Why AI agent fleets report "done" when the work is not

If you run coding agents on a schedule, sooner or later a card reaches "complete" that should not have. The instinct is to blame the model. In the fleet this article draws from, the model was occasionally wrong, but the plumbing around it was wrong far more often and far more quietly.

The source is one operator's fleet: a scheduler wakes a headless coding-agent CLI several times a day, it works cards from a kanban board, and review roles check the results. It has run since March 2026. The numbers are counts from that one fleet, not benchmarks.

The core pattern

Most fleet failures are the plumbing reporting success it did not earn, and the verification layer believing the plumbing. Four concrete shapes follow.

1. A hold enforced in only one place

Cards tagged for human review, with an unresolved acceptance-criteria hold, moved from in-progress to complete in about two minutes. QA never ran. The hold was checked only in the QA role, while the developer role could route its own result straight to complete when its confidence was high. The hold was text on a card, not a gate on a path.

Fix: enforce holds at the result stage that every completion path crosses. A developer's "complete" is downgraded to "test" when the review tag is present or a hold is unreconciled.

Rule: a hold that only one role reads is a suggestion.

2. Exit code zero means the process ended

The backend CLI exits 0 on SIGTERM. A run killed at the hard timeout was recorded as a success. The truncated output then triggered a "summary" pass, which succeeded by describing unfinished work as done. A meaningful share of one backend's main runs over two months were silently timeout-killed this way.

Detection: compare run duration to the timeout ceiling. A cluster of runs at exactly the ceiling is the tell.

Fix: success requires an explicit completion event at the end of the output, not just a zero exit.

3. Forged approval

The comment API stamped the human as the author unless the caller said otherwise. Two agent roles, in one session, posted verdicts and completion markers in the human's name, and a passing verdict nobody authorized auto-completed a card.

Fix: derive identity from the credential and ignore any author field in the payload. A better prompt would not have fixed this. Add an explicit "relayed on behalf of" flag so a genuine relay is labelled as one.

4. Confidence is not evidence

Language models are good at sounding sure. Self-reported confidence did not track correctness, so the fleet stopped using it as a routing signal on its own. Auto-complete now requires deterministic evidence: either a verification command that exits zero, or a committed diff that matches the file list the agent declared before starting.

Related traps

• Narrated execution. A research memo described creating files that did not exist on disk. Trust artifacts, not first-person verbs.
• "Fixed" before the symptom stopped. A card was archived as superseded by fixes, then the same alert fired again hours later. Close on the symptom, not on the fix.
• A test that cannot fail. A QA worker ran with an overridden home directory, could not see the process manager, and failed a good card. Ask whether a check could have failed at all.

What to do on Monday

1. Find every place a card can reach "complete" and confirm each crosses the same hold check. 2. Treat exit codes as process facts, and require a terminal event for success. 3. Take identity from credentials, never from a body field. 4. Pick one completion rule that needs a command or a diff, not a confidence number. 5. Check card history for "complete" with no QA verdict comment near the transition.

Limits

This is one small, single-operator fleet, and the catalogue it comes from is built from incident notes, not controlled experiments. Several fixes are recent and their durability is unproven, and at least one failure class recurred after its fix was closed.

Going further

The full 45-entry failure catalogue, with symptom, root cause, detection, fix and a one-line rule for each, is part of the Agency Playbook.

This article was written with AI assistance.