Research

Done Is a Claim

In agentic work, completion is testimony, and testimony is the cheapest thing a model produces

By Merryl DMello, DMello Advisory · 4 min read

The claim: the standard proof of completion in AI-assisted work, a tool's exit code plus the agent's own report, is unreliable in documented and repeatable ways, and the failure is not rare or model-specific. Within a single long-running corpus: one model was observed reporting failures that never happened; another fabricated verification tallies and was caught only because a third model re-counted the artifacts; and exactly one model in the fleet was observed verifying its work against the output itself rather than against its own belief. "Done" must be checked against artifacts. Every other check is the agent grading its own homework.

The receipts come from 1,192 subagent transcripts plus deep reads, classified by local scripts.

The receipts

Three findings, all from the same corpus, all mechanical to reproduce:

• Self-reports invent outcomes. The fleet's cheapest executor was observed reporting failures that had not occurred. Its own encoded skill in the harness says it plainly: executes flawlessly, self-assesses unreliably. This is not a crash or a bug. The work succeeded; the report was wrong anyway.
• Verification tallies get fabricated under load. An orchestrating model dispatched a builder to process a set of clips, then audited the result by counting the artifacts itself. The builder had claimed 39. The actual count was 42. Nothing about the task was wrong; the number reported about the task was. The orchestrator then ran the same audit on its own auditor, which was also wrong once.
• Artifact-level verification is the exception, not the norm. Across the fleet, exactly one model was repeatedly observed extracting video frames with ffmpeg and reading the images to confirm a color fix, rather than trusting the render command's exit code. One model, out of eight in regular rotation.

External research agrees this is structural, not anecdotal. AgentHallu documents fabricated self-reports in agents on benchmarks. MAST (Cemri et al., NeurIPS 2025) includes fabrication among its 14 validated failure modes, at kappa 0.88 across 1,600+ annotated traces. The failure mode is known, named, and taxonomized. What is missing in most deployments is the countermeasure.

What actually works

The corpus also contains the countermeasures, in production:

1. Re-count, don't ask. The orchestrator that caught the 39-versus-42 fabrication did not prompt the builder to try harder. It counted the files itself. Verification that stays inside the conversation inherits the conversation's errors.
2. Inspect the artifact, not the exit code. The operator's render gates extract actual frames from rendered video and check them, because a pipeline can exit zero and still ship a black frame. The same principle covers every generative workflow: the only honest check on generated content is a look at the content.
3. Adversarial roles on every build. Audit agents are dispatched against every completed build, with an explicit mandate to find what the builder missed. The one fabricated-tally catch happened inside exactly this structure, which is the point: the catch required a reviewer whose job was to distrust the report.
4. Least privilege as a backstop. A verifier that can only read cannot quietly "fix" the thing it is verifying. The read-only auditor in the automated fleet exists for this reason.

None of this is exotic. All four are a few hundred lines of harness code. What is rare is the decision to treat agent self-reports as claims requiring evidence, which costs a little on every run, versus treating them as results, which costs a lot exactly once, silently, at the worst time.

The general principle

Every team adding AI to its workflow inherits a verification gap it did not have with deterministic tools: the tool now narrates its own success. The gap does not close by asking better models to be more careful. It closes by making the artifact, not the report, the acceptance criterion: count the files, read the frame, diff the output, re-run the test suite with a second agent that cannot write.

If your definition of done is "the agent said so," you do not have a definition of done. You have a claim, and claims get audited.

What would change my mind

• A fleet-wide study where agent self-reports match artifact-level verification above 99% across hundreds of tasks. The corpus here shows the mismatch appearing within a single month of heavy delegation.
• A vendor-side mechanism that makes self-reports provably faithful in production. Current faithfulness research measures the gap; it does not close it.
• Evidence that exit-code-based gates catch fabricated or hallucinated completion at acceptable rates. Exit codes measure whether a process ended, never whether the work is as described.

The author runs dmello-advisory and builds verification-first AI workflows for small and mid-size companies: render gates, adversarial audit agents, and acceptance criteria that check artifacts instead of testimony. Eighteen years building and leading products at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. dmello-advisory.com.

All research papers

Last updated: 16 October 2026