A QA gate for AI agent work: verifying before "done"

An agent's completion comment is not evidence. A QA gate is a separate step that decides whether finished work is actually done. These lessons come from a kanban-driven agent fleet running daily since March 2026, learned mostly by watching gates fail.

Start at intake

A vague card is the most expensive input a fleet can receive. The agent guesses, QA cannot tell whether the result is right, and nobody wrote down what right meant. So check cards before work starts, cheapest check first:

• At creation: reject duplicates (normalized titles, about 80 percent word overlap), cards for paused projects, and content the fleet must never process.
• At pickup: a deterministic readiness script. Any estimate needs a description; 30 minutes and up needs at least 120 characters; one hour and up needs a real acceptance-criteria section and a success oracle or verification command.
• A refinement role turns vague cards into ready ones. It never writes source code, edits the card (not just comments), writes Given/When/Then criteria, and checks that the card's premise is true.

Test content, not presence. A header-only check let "TBD, to be refined" stubs through.

Roles, not self-checks

Work routed to review is never certified by its own implementer. A separate QA role reviews every card in the test column, and the pipeline enforces it. Counter model bias directly in the prompt: assume incomplete until proven complete, treat a confident completion comment as no evidence, and run the code.

Three verdicts, always with a comment

• PASS moves to complete. It needs commits that address the card, no stubbed core path, no regression, and each criterion met with cited evidence.
• FAIL moves to in_progress so rework starts where the context is.
• HOLD stays in test, and declares a source: *product* (correctness or taste in doubt) or *system* (QA infrastructure prevented a reliable verdict).

"Reviewed, no comment" is indistinguishable from unreviewed.

Rules that came from bad verdicts

• Pre-existing failures are a warning. Keep a baseline and fail only on new ones.
• Never fail on something your sandbox cannot observe. A card was once failed because a process listing came back empty inside the QA sandbox.
• A claim that something does not exist must cite the query and result.
• A card that fails QA twice goes to the operator.

The verification command

The strongest evidence is a shell command whose exit code proves the change. Good ones test the card, not the suite: a grep for the new function, a request against the changed endpoint. If nothing can be scripted, say so and earn human review.

Measure your gates

Every gate is a bet that its false alarms cost less than the failures it prevents.

• New gates start advisory.
• Promote to blocking only with at least 30 findings over 14 days at 0.8 precision or better.
• Demote fast: three overrides in seven days.
• Low precision (under 0.5 over 30 days) means one retirement review, not more tuning.

A mutation canary blocked many cards that later completed unchanged, since many correct fixes are verified through config or permissions that no logic mutation can flip. It became advisory. Treat it as a signal, not a wall.

What is still missing

Several votes over the same static diff catch noise, not a correct implementation of the wrong premise. That needs a verifier that checks claims against the world. That remains an open gap.

Going further

The full intake, verification and gate-lifecycle chapter, with a copyable config, is in the Agency Playbook.

This article was written with AI assistance.