The jobs that wake people up are rarely the ones that hurt the most. A failing payment settle that pages every night gets attention, ownership, and a runbook. A weekly reconciliation that has exited zero for eighteen months gets none of those things. It sits in a scheduler, green every cycle, and the team treats the green as proof. That is the risk. A job that never fails teaches the organization that silence means health, and silence is exactly what a broken scheduled path produces.
I have watched this pattern in payments ops more times than I can count. The flashy failure is the API that returns 500s. The expensive failure is the batch that stopped writing the file the finance team uses to close the day, and that nobody noticed until the close was late. The batch had not thrown. The scheduler had not marked a miss. The dashboard that tracks failed jobs in the last 24 hours showed an empty panel. The work simply did not happen, and the monitoring had no concept of work that does not happen.
Why never-failing jobs go unwatched
Operators allocate attention based on pain. A noisy job earns retries, alerts, and a named owner. A quiet job earns a comment in the deploy notes and then falls out of the weekly review. Over a year or two, the quiet job accumulates the worst combination of properties: high business consequence, low operational familiarity, and monitoring that only asks whether the last exit code was zero.
That monitoring question is the wrong one for most batch work. Exit code zero means the process terminated without an unhandled exception. It does not mean the expected rows moved, the expected file landed, or the expected side effect reached a downstream system. Plenty of production scripts exit zero after processing zero records. Plenty more exit zero after writing an empty artifact because an upstream query returned an empty set. The job succeeded. The business did not.
The second reason these jobs stay invisible is ownership drift. The engineer who wrote the nightly settlement export left. The schedule is still correct. The credentials still work, usually. The runbook, if it ever existed, lives in a wiki page that nobody has opened since the last audit. When something finally does go wrong, the on-call person has to reverse-engineer intent from the script and the destination path. That delay is part of the cost. The longer a job goes without an incident, the fewer people can explain what good looks like for it.
Failure modes that look like success
A job that never fails is often a job that fails in ways that do not raise an exception.
The schedule fires, the container starts, the main query runs against a view that was altered last Tuesday, and the result set is empty. The script logs that it processed zero records at info level and exits. Nothing pages. Three days later, a partner asks why their report is missing updates.
The schedule fires, the job authenticates with a service account that still has login rights but lost write permission on one table during a least-privilege cleanup. Reads succeed. Writes no-op or skip. Depending on the ORM and the error handling, the process may still exit cleanly. The heartbeat, if it fires at the start of the job, already went out. The monitoring service is satisfied.
The schedule fires in a region that was drained for maintenance. The orchestrator rescheduled the CronJob into a namespace with a stale ConfigMap. The job runs against a staging database that accepts the writes. Production stays unchanged. From the scheduler's point of view, the job ran on time and completed. From the business's point of view, production is stale.
None of these look like a crash. They look like a healthy job that has always been healthy. That is why the never-fails property is not a compliment. It is a warning that your observability may be measuring the wrong thing.
When I inventory scheduled work for a team, I do not sort by error rate. Error rate is useful for noisy services. For cron and batch, I sort by a different set of questions. What happens if this job misses one cycle. What happens if it misses a week. Who notices first: an internal operator, a finance close process, a customer, or a regulator. How long would it take a stranger on the on-call rotation to decide whether a given run was good. Is there an independent signal that the work occurred, or do we only have the job's own exit status.
The jobs that score badly on those questions are almost always the quiet ones. Nightly ledger snapshots. Weekly compliance exports. Monthly retention sweeps. Partner file drops that land before a market opens. They run infrequently enough that a miss does not produce a spike on a shared dashboard, and they matter enough that a miss becomes an incident with a long timeline.
A concrete example from payments: a daily ACH return file ingest that had a perfect success rate for months. The ingest job checked for a file, and when the file was missing it logged and exited zero, because an empty day is normal on holidays. Then the upstream bank changed the filename pattern. Every weekday looked like a holiday to the ingest. The job succeeded. The return items sat unprocessed. The first signal was a dispute volume anomaly two days later. The postmortem action item was not to add retries. It was to stop treating absence of work as success without a bound.
Monitoring that treats absence as the signal
The corrective posture is simple to state and easy to under-implement. You need a signal that the work happened, on a schedule you can defend, independent enough that the job cannot grade its own homework.
A completion heartbeat is the minimum. The job sends it only after the meaningful work finishes, with a small payload: records processed, bytes written, file name, destination checksum if you have one. A monitoring service expects that heartbeat inside a window. No heartbeat means page, even if the scheduler history is green.
For anything customer-facing or financially material, add an outcome check that does not share the job's code path. If the job is supposed to drop a dated returns file into a bucket by 06:30 local, a separate check at 07:00 asserts that today's object exists and is non-empty. If the job is supposed to insert settlement rows, a separate query counts today's rows and compares them to a floor that you set from history, not from hope.
Send the heartbeat at the end, not the start. Include failure heartbeats with a status field so a known bad run is distinct from silence. Silence is the expensive state. A known failure is annoying and usually recoverable the same morning. Also write down what good means in one place the on-call can find: expected window, expected artifact, expected volume band, owner, escalation. The document can be short. The point is that the next person does not have to reconstruct intent from git blame at 2 a.m.
The resistance is usually not technical. Heartbeats are cheap. Outcome checks are a few SQL statements or an object-store HEAD. The resistance is cultural. A job with no pages feels finished. Spending a sprint on monitors for jobs that have never woken anyone feels like inventing work. I treat that feeling as the reason to do the work. The page you install for a quiet job is insurance against a class of incident that arrives with no warning and with a timeline that starts days before anyone was looking.
If you need a starting rule that fits a normal engineering team, use this. Every production scheduled job gets an end-of-run heartbeat with a result payload. Every job whose miss would affect money, customers, or a compliance deadline also gets an independent outcome check. Both feed the same on-call path as other production alerts, with a severity that matches the business impact, not the historical page rate. Then pick the ten quietest jobs by age-without-incident and put them at the front of the queue. Age-without-incident is a poor proxy for safety in batch systems. It is often a proxy for how long the job has been outrunning your ability to notice.
A job that never fails can be well engineered. It can also be a job whose failure mode produces no exception, no log line anyone reads, and no dashboard panel that turns red. Until you instrument for absence and for outcome, you cannot tell which kind you are running. The green history is not the answer. It is the reason the question stayed unasked.