Most teams learn about timezone bugs the hard way: a monitor pages at an odd hour, the on-call finds a green job history, and someone closes the ticket as a false alarm. A week later the same page returns. Or it does not return, and a partner file is late instead. Daylight saving transitions and mixed timezone schedules produce exactly that shape of incident. The job behavior looks intermittent. The monitoring looks flaky. The root cause is usually a schedule expression that does not mean what the author thought it meant.
These bugs recur across stacks. Kubernetes CronJobs, cloud schedulers, Windows Task Scheduler, and home-grown runners all hit the same edges. The symptoms land in the alerting channel first, so they get blamed on the monitor. Fixing the monitor without fixing the schedule leaves you with a quieter dashboard and the same missed work.
What the schedule string actually promises
A cron expression is not a wall-clock promise by itself. It is an expression evaluated in a timezone, by a particular scheduler implementation, against a particular definition of this minute. Teams write 0 6 * * * and say 6 a.m. The unanswered question is 6 a.m. where, and what happens when that local day has 23 or 25 hours.
If the scheduler is set to UTC and the business expectation is America/New_York, the job drifts relative to local operations twice a year and every time someone mentally translates morning into the wrong offset. If the scheduler is set to a named timezone, you still need to know whether the implementation skips, runs twice, or picks one side during the fall-back hour. Those behaviors differ by product. Assuming they match the last system you used is how you get a page that should not have fired.
I have seen a nightly settlement export configured as 0 2 * * * in UTC because someone wanted after midnight Eastern. That mapping is wrong for most of the year, and it is differently wrong after a DST change. During Eastern Daylight Time, 02:00 UTC is 22:00 the previous evening local. The export ran before the day's late postings finished. The job succeeded. The file was incomplete. The completeness monitor looked like it was crying wolf until someone compared the file contents to the ledger close time.
Spring forward, fall back
In the spring, many local timezones skip an hour. In America/New_York, 02:00 becomes 03:00. A job scheduled for 02:30 local has no wall-clock instant to land on that morning. Some schedulers skip the run. Some shift it. Some queue it into the next valid minute. If your monitoring expects a heartbeat every calendar day inside a fixed UTC window, a skipped local run looks like an absence alert. If your monitoring is also expressed in local time and the scheduler shifted the run, you may see a late heartbeat instead.
The false-alarm story writes itself. On-call opens the scheduler UI, sees that yesterday ran and tomorrow is scheduled, and assumes the monitor double-counted. The more careful check is whether today's expected local run exists in the history at all. On spring-forward mornings, not in history can be the correct scheduler behavior and still be the wrong business outcome. If the job must run once per local business day, a skip is a miss. Your alert is doing its job. The schedule definition is the bug.
A useful habit: for every job whose cadence is described in local business language (before the New York open, after the London close), store the timezone next to the cron expression in the same config, and document the DST policy in one sentence. Skip if the local minute does not exist is a policy. Run once in the next valid minute is a different policy. Always schedule in UTC and convert deliberately is a third. Pick one and test it twice a year on purpose.
In the fall, clocks repeat an hour. A job scheduled for 01:15 local can have two valid instants. Some systems run it twice. Some run it once on the first pass. Some run it once on the second. If the job is not idempotent, a double run is a data incident. If the job is idempotent but your heartbeat monitor expects exactly one check-in per local day, a double run can look like a duplicate or a late second pulse depending on how you key the event.
I watched a cleanup job delete temporary payment artifacts based on files older than two hours. It was scheduled hourly. On the fall-back night it ran an extra time in the repeated hour and deleted a file that a downstream retry still needed. The cleanup job's own logs looked normal. The incident showed up as a missing artifact alert in a different service. The first hypothesis was a flaky object-store integration. The timeline only made sense after someone plotted runs against local time and saw two executions stamped into the repeated hour.
If you must schedule inside the ambiguous hour, make the job idempotent and key heartbeats by a logical run id, not only by wall clock. Better, move the schedule out of the ambiguous hour. 03:15 local is boring. Boring is good for batch work.
UTC windows that disagree with local SLOs
A common monitoring mistake is to express the alert window in UTC while the SLO is explained to the business in local time. Example: the partner file must land by 7 a.m. Eastern becomes a monitor that expects a heartbeat before 12:00 UTC. That conversion is correct only for one offset. When Eastern moves between UTC-5 and UTC-4, the UTC deadline is wrong for half the year. Either you page too early and train people to ignore the alert, or you page too late and discover the miss after the partner has already called.
The fix is not clever math in the alert text. The fix is to evaluate the deadline in the same timezone the SLO uses. If the business speaks Eastern, the monitor should use America/New_York. If the job itself must stay on UTC for portability, keep the job on UTC and convert the expectation at the monitoring layer with a named zone, not with a hard-coded offset. Hard-coded offsets are fossilized bugs. They are correct on the day they are written and wrong after the next transition, the next region expansion, or the next person who copies the rule to a second job in London.
Operators often compare two jobs and conclude the monitoring SaaS is inconsistent because one alerted and the other did not. Check the runners first. One job may be a Kubernetes CronJob with a timezone set. The other may be a legacy crontab on a host whose localtime still says UTC. One cloud scheduler may be using the account's default region zone. Another may have an explicit zone override in Terraform that drifted from the console.
During incident review, capture these fields every time: schedule expression, timezone setting, scheduler product, DST behavior for that product, monitoring window, monitoring timezone, and the business deadline in plain language. When those fields disagree, the alert is usually the only component telling the truth. Also watch for weekday rules around holidays that are local by nature. A job that skips US federal holidays but runs on a UTC calendar can fire on a local holiday morning and miss the observed day the business cares about. That is adjacent to DST, and it produces the same false-alarm argument: it ran when it was supposed to according to the cron. According to whose calendar.
Do not wait for the next transition to discover your policy. Keep a fixture environment where you can freeze or simulate clock transitions, or at least a checklist you run before the second Sunday in March and the first Sunday in November in the US, plus the equivalent dates for any other zone you serve. For each critical job, answer four questions in writing. What timezone evaluates the schedule. What happens if the local time does not exist. What happens if the local time occurs twice. What timezone evaluates the alert window. If any answer is "I am not sure," the next DST weekend can produce a page that someone will close as noise.
When an alert does fire around a transition, resist the reflex to widen the window until the page stops. Widening hides both the scheduler skip and the real outage next month. Confirm whether the run happened, whether it happened once, and whether the artifact matches the business deadline. Then change the schedule, the timezone setting, or the job's idempotency. Change the monitor only when the expectation was wrong.
Timezone bugs feel like monitoring bugs because monitors are the first place absence shows up. Absence is still the right signal. The work is to attach that signal to a schedule definition the whole team can explain on a bad Monday morning, including the morning the clocks moved.