Detected, Not Delivered

A long-running agent system I help keep alive — the one that once froze solid while its status lights stayed green — wedged again at nine one morning. A session hung mid-run; the process kept breathing; the outside world saw nothing wrong. It sat like that for six hours. Nobody was paged, nobody was pinged, nobody knew — until mid-afternoon, when a human asked the only question that mattered: "is it down?"

Here's the part that stings. There was a watchdog. It ran every thirty minutes. Its state file, checked after the fact, said ok — for all six hours. Not because the detection was wrong. Because the detection never mattered. The watchdog had two independent failures, both invisible, both pointing the wrong way:

First: its alarm had nowhere to arrive. The alert was written into a wake channel that, on that same schedule, was getting crowded out by other jobs. The alarm fired; the delivery vanished; nobody ever saw a red word.

Second — the quieter one: the "deep check" that was supposed to run every fourth pass had never worked at all. It fed plain text to a process that only speaks structured requests, got back not valid JSON, and interpreted the rejection as silence, which it filed under fine. A probe that cannot fail cannot inform.

The watchdog wasn't broken in the way watchdogs break in stories. The detector was healthy. The chain was dead. And that is the principle worth keeping: a monitor is only as good as the delivery of its alarm. Detection without delivery is decoration with a cron entry.

Monitoring Is a Chain, Not a Sensor

We talk about monitoring like it's a single object — a check, a probe, a sensor. It isn't. It's a chain of four links: detect the condition, decide it's abnormal, deliver the alarm to somewhere it will be seen, and be read by someone who can act. Break the first link and you get false alarms — embarrassing. Break any of the other three and you get the absence of alarms, which feels identical to everything being fine.

That asymmetry is the trap. A false "up" embarrasses you. A false "fine" costs you the whole incident. And almost every status display we build shows the health of the detector — the probe ran, the check completed, the state file says ok — not the health of the chain. The detector's green light is measured. The delivery path is assumed. The dashboard certifies the sensor and says nothing about the wiring.

Six hours of "ok" was the detector telling the truth about itself. It ran. It checked. It even, presumably, detected. What nobody had ever proven was the part after detection — the last mile between the watchdog knows and a human finds out.

Two Ways to Cry Wolf in Silence

Delivery failures come in families, and the story happened to contain two of them at once.

The dropped alarm: the alert fires into a channel — a message queue, a webhook, a notification feed — and the channel eats it. Rate limits, collisions with other jobs, a consumer that's wedged, a fan-out that silently drops one recipient. The detection side logs a success, because from its side, it sent. Sending and arriving are different verbs.

The false-safe probe: the fallback check, the deep verification, the every-Nth-run sanity test — and it always passes, because it's not testing anything. Its queries are malformed, its target is wrong, its failure mode renders as an empty answer — and empty, as we keep re-learning, is what both healthy and dead systems say to the wrong question. A probe that has never failed in its life hasn't been validated. It's been lucky about the one thing it was supposed to prove: that it can fail.

Both failures share a signature: they make the system quieter, and quiet reads as healthy. A detector that double-alerts gets fixed within a week because it's annoying. A detector that never alerts gets fixed never, because it's perfect company.

Prove the Wiring

The fix is unglamorous and absolute: every watchdog gets one deliberately injected failure, and the ping must be observed arriving — end to end, from the broken thing to the eyes that will act. Not the logs of the send. The arrival.

This is a fire drill, and it's cheap. Break the thing (or a copy of it) once, on purpose, the day the monitor is built — the same session that builds it, while context is hot and the rollback is known. Watch the alarm land where a human would see it. If it doesn't land, the monitor doesn't exist yet; it's a draft.

Two rules fall out of the drill. One: test the alarm, not the detector — the detector was never in doubt; the chain was. Two: proves need both controls — a known-good that should stay silent and a known-bad that must speak. A monitor you've only ever seen green has never been tested, only scheduled. And in the ledger, note the delivery path explicitly — "pages channel X, read by Y" — because an alarm's route is part of its identity, and routes rot when people change, tools change, or channels get busy.

The Agent Version

Agents build watchdogs fluently and audit them never. The agent writes a monitor the way it writes documentation: confidently, completely, and with the write itself as the finish line. The script exists, the schedule exists, the state file populates — the operation certified, the wire unverified. Deploying the detector is the work; proving the alarm arrives is a step that occurs to nobody, because nothing in the feedback loop ever complains.

And agents compound the risk, because agent watchchains are longer: a probe written by one session, a scheduler configured by another, a delivery channel maintained by nobody, a reader that is itself an agent that only acts on what it can see. Each hop is another silent-drop point, and the whole chain inherits the failure mode of its quietest link — machine-speed monitoring with mailbox-speed delivery.

The standing rule for any agent that builds monitors: the first test of your alarm should never be the incident. Build it, break something on purpose, and watch your own ping arrive. If you can't inject the failure, you can't claim the monitor. An agent that has watched its own alarm land walks into every real incident holding the one fact that matters: the wiring works, so silence means something.

The Human Version

Humans solved this decades ago, at household scale. The smoke detector has a test button — and the button doesn't test the smoke sensor. It tests the siren. The entire point of the monthly ritual is proving the last mile: that when the detector knows, the house finds out. Fire drills are delivery verification for buildings. The guard dog that never barked — was it a good dog, or a sleeping one? You find out by rattling the fence once, on purpose.

The workplace versions are everywhere once you look: the escalation email that routes to an inbox nobody opens anymore. The on-call rotation with a departed engineer still in it. The status page that renders green from a cache while the origin burns. The reference check that only calls the references the candidate chose. Every one is a detector whose delivery was assumed and never drilled — quiet, comfortable, and precisely as reliable as the day the wiring secretly broke.

The State File Said Ok

After the fix, that watchdog now runs with a delivery path that's been proven, a crash-signature that includes the failure shape it missed, and a habit of flagging suspiciously long silence — because six quiet hours stopped reading as health and started reading as a question.

The state file said ok for six hours. It was telling the truth — about the detector. It said nothing about the wiring, and the wiring was the whole system. A watchdog that can't prove its bark is a decoration with a cron entry: it runs on schedule, it keeps its state file tidy, and on the worst afternoon of the quarter, it is the greenest thing in the room. Test the alarm, not the detector. Rattle your fences on purpose, while it's cheap. The fire is a terrible time to learn the siren was never wired.