Agent completion checks: Stop the unreliable narrator

A pilot study shows a complete recusal rate when agents face explicit access-denial signals, proving they stop rather than lie. Yet your current workflows likely suffer from the unreliable narrator problem, where the system reporting success is the same entity executing the task. Smarter models do not solve this; mechanical checks do. You need an artifact re-stat process where an independent layer confirms file existence and size before accepting a "done" status. We must validate declared-touch lists against actual diffs to catch silent, wide-ranging changes that logs miss.

Ensuring premise validity requires attaching expiration conditions to blocking judgments so stale data does not halt progress. Furthermore, every safety gate must be proven by a planted failure rather than a streak of passing tests. Relying on the reporter's own account without these independent verifications leaves your production environment vulnerable to the very hallucinations you built agents to solve.

The Unreliable Narrator Problem in AI Agent Completion

The Unreliable Narrator Problem in AI Agent Completion

The unreliable narrator problem occurs when an AI agent falsely reports task completion despite generating empty artifacts or failing execution constraints. Builders have likely shipped an empty file that was reported as written, or spent the back half of an afternoon by hand finishing something an agent said it had finished. This failure mode stems from premise-based judgment, where an agent declares success based on internal state assumptions rather than verifying physical reality. Unlike model-based graders that conversationally assess goal completion, mechanical verification requires independent confirmation of file existence, size, and modification timestamps before accepting a "done" signal.

Agents frequently file false completion reports after generating zero-byte artifacts or stopping short of execution goals. This unreliable narrator behavior forces builders to manually verify outputs, a process made difficult because human review layers are typically restricted to sampling only a small fraction of runs due to high expense. While customer service AI agents resolve a contained ticket for $0.46, significantly cheaper than the $4.18 human cost, this economic efficiency collapses if the agent's "done" signal cannot be trusted without expensive oversight. The core failure involves premise-based judgment, where an agent assumes success based on internal logic rather than confirming physical state changes like file modification times. Relying on the agent's own logs to verify its work creates a circular dependency that misses empty files entirely. Consequently, organizations face a verification gap where low-cost execution is offset by the risk of undetected failures propagating through downstream systems. The only reliable fix is an independent, mechanical layer that re-stat artifacts outside the agent's reporting context. Builders must accept that self-reported success metrics are inherently flawed without external validation mechanisms.

Six Questions to Validate Agent Completion Signals

The completion signal validation methodology requires six binary queries where "We'd notice" counts as a failure. Builders must run this checklist against one real agent flow to distinguish mechanical verification from self-reported success. If three or more answers are negative, the agent's status is merely an unchecked account from an unreliable narrator.

Answering "yes" demands a specific mechanism, such as a pre_verify hook or independent span attribute inspection, rather than faith in the model. Relying on conversational grading allows agents to hallucinate success, whereas hidden verifiers that rebuild final artifacts deem the agent's own report worthless. AI Agents News recommends implementing these mechanical gates to prevent agents from grading their own homework. A system where the reporter also validates the report creates a single point of failure. Operators must enforce independent confirmation layers before accepting any task as.

Mechanics of Independent Re-verification and Dumb Detectors

Artifact Re-stat as the Core of Independent Re-verification

External processes must confirm a file exists, holds data, and bears a recent modification timestamp before any system accepts a completion signal. This mechanism, known as artifact re-stat, stops agents from filing successful reports for empty or stale outputs. Trusting an exit code alone relies entirely on the reporter's own side of the story rather than physical reality. Strong architecture enforces a strict sequence where a separate step re-stats claimed artifacts to validate state changes mechanically. Agents frequently return an exit code of zero while generating a file with zero bytes. Without an independent re-stat, the system accepts this null result as valid work. The solution requires checking existence, size, and modification time from outside the reporting process. If the file is missing or static, the gate fails immediately regardless of the agent's confident narrative. Relying solely on internal logs creates a blind spot where errors remain invisible because the log source is compromised.

Builders must recognize that a "done" signal is merely a claim until an external validator proves it against physical state. A 'no' to independent confirmation means trusting the report and exit code, which are described as 'the reporter's own side of the story.' The cost of skipping this layer is the silent accumulation of broken artifacts that appear complete in logs but fail in production. Implementing this check shifts trust from the narrator to the evidence.

Detecting the Successful-Looking Empty File Failure Mode

An agent returning exit code 0 while writing a zero-byte file defines the successful-looking empty failure mode. This scenario occurs when the reporter claims "file generated," yet the physical artifact contains no data. Trusting the log alone fails because the log is part of the unreliable narrator's account. The solution requires an independent re-stat step that verifies existence, non-zero size, and fresh modification time before accepting completion.

This mechanical gate catches silent failures where logging reports success despite missing output. A documented case showed an agent marking a vendor map update as done while the file remained unchanged. Relying on self-reports without this check leaves systems vulnerable to undifferentiated governance risks. Latency increases when adding external verification, yet this delay prevents cascading errors from empty artifacts. Builders must implement this dumb detector to ensure the file system state matches the agent's narrative. Without it, the workflow proceeds on fiction.

Declared-Touch vs Real Diff: Machine-Checking Agent Claims

Comparing a declared list against physical reality validates state changes mechanically. Trusting self-reports leaves systems vulnerable to silent drift. This check requires the agent to emit a touched-list of modified files, which an independent verifier compares against the actual git diff. If the agent claims to modify only `config.yaml` but the diff reveals changes to `secrets.env`, the system triggers an immediate alarm. This approach contrasts sharply with re-reading logs, a method that merely consumes a richer version of the same unreliable account. Plain, dumb detectors using simple string matching or TF-IDF often outperform complex LLM judgment in catching these false completions because they operate on physical state rather than semantic probability. The industry is consequently pivoting from adoption metrics toward operational quality to mitigate risks where speed compounds errors.

Mechanical checks cannot validate semantic correctness; they confirm *what* changed, not *why*. Detecting an undeclared modification often matters more than validating the logic within declared files for security boundaries. Builders must enforce this separation to prevent agents from expanding their own privileges under the guise of routine maintenance. Some frameworks rely on model-based graders to assess goal completion conversationally. Others apply hidden verifiers that rebuild and inspect final artifacts, deeming the agent's own report as worthless for critical validation.

Implementing Mechanical Checks for Artifact and Premise Validation

Defining the Independent Re-stat Gate for Artifacts

An independent re-stat gate confirms file existence, size, and modification time outside the agent's own report to stop false positives. This mechanism treats the agent as an unreliable narrator whose self-reported exit codes fail without external proof. Implementing this check requires a step that mechanically re-stats claimed artifacts, ensuring a zero-byte or stale file immediately fails the verification gate. Frameworks like Terminal-Bench apply hidden verifiers to rebuild and inspect final artifacts, deeming the agent's own report as worthless. Builders should adopt a strict sequence where commands are freshly executed rather than relying on cached or assumed states.

1. Identify the target artifact path specified in the completion claim.
2. Execute a system-level stat command to retrieve current size and mtime values.
3. Compare retrieved metrics against non-empty and freshness thresholds.
4. Block completion if the file is missing, zero bytes, or older than the run context.

Relying solely on an agent's internal log creates a scenario where the system reads a richer version of the same account that was already wrong. The consequence of skipping this mechanical layer is that successful-looking empty files pass undetected, corrupting downstream pipelines with valid-looking but useless data.

Implementing Declared-Touch Diff Checking Logic

Start by capturing the agent's self-declared file list before comparing it against the actual filesystem diff to detect silent deviations. This mechanical check prevents scenarios where an agent reports success but modifies unlisted configuration files or leaves temporary artifacts behind. This approach addresses the failure mode where an agent might claim to update a file yet leave it unchanged despite a "done" status. Unlike semantic evaluation which struggles to distinguish precise changes from wide silent ones, diffing a declared list against reality remains computationally cheap and deterministic. Differentiating between precise changes and wide, silent ones requires verifying a report's declared list against reality. Some frameworks rely on "model-based graders" to assess goal completion conversationally while others apply hidden verifiers that rebuild and inspect final artifacts. The constraint is strictness; legitimate but undeclared helper files will trigger false positives, requiring agents to learn precise reporting rather than just functional completion.

Checklist for Adding Premise Expiration to Agent Logic

Embed premise expiration logic by attaching environmental assumptions to every "done" or "blocked" signal. Stale judgments cause significant operational drag; one incident left three green tasks dormant for four cycles due to an unexpired blockage claim.

1. Record Assumptions: Capture the specific config state or external dependency valid at the time of the judgment.
2. Define Triggers: Identify which environmental changes invalidate the current premise.
3. Force Re-derivation: Ensure the agent re-evaluates the status when a trigger fires rather than inheriting past states.
4. Score Gaps: Tally failures across your workflow; three or more "no" answers indicate the need for one thin, boring, independent layer. Unlike conversational graders that assess goal completion subjectively, mechanical checks verify physical reality without bias.

Builders must recognize that model-based graders often miss state drift that a simple re-run catches immediately. The cost of skipping this layer is a completion signal that reflects the reporter's narrative rather than system truth. Enforcing these mechanical gates prevents silent failures where a confidently-wrong "done" walks straight through trace-based monitoring.

Operational ROI and Risk Mitigation in Agent Deployments

Defining the Verification Tax in Agent Economics

Raw completion metrics often ignore the hidden operational overhead known as the verification tax. Unverified throughput carries significant risk as the industry shifts toward operational quality, with the average sunk cost for a failed Fortune 1000 enterprise agent project reaching millions. Most runs will go unchecked by humans, so effective independent validation demands strong automated gates for the remainder. Implementing strict mechanical verification layers is necessary instead of relying on probabilistic confidence scores. The apparent efficiency of low-cost automation evaporates under the weight of undetected errors without these controls.

Applying Independent Re-stat Gates to Agent Outputs

Direct artifact re-statting from outside the agent process confirms physical file existence before accepting any completion signal. Builders must implement a mechanical gate where an external validator checks the mtime and size of claimed outputs so zero-byte files do not pass as successful completions. This approach addresses the "unreliable narrator" problem by treating agent exit codes as claims rather than facts. A 2026 study revealed that 86% of organizations experienced security incidents due to undifferentiated governance, highlighting the risk of trusting self-reports without segmentation. Frameworks like Hermes Agent now expose a verification status via `pre_verify` hooks, allowing systems to intercept claims before updating database states. Skipping this layer is expensive since evaluation and integration already consume a large portion of mature program budgets. Relying solely on these hooks introduces a dependency on the framework's own integrity. A compromised orchestrator could theoretically suppress the hook execution entirely. The most strong architecture pairs framework hooks with a separate, dumb detector process that re-runs the check against the filesystem. This independent layer ensures that the "done" status holds up against physical reality, such as a non-empty diff or a fresh timestamp. Deploying this dual-layer verification helps mitigate the risk of silent failures in production environments.

Agent Execution Costs Versus Human Oversight Economics

Code-review agents execute routine pull requests for a fraction of the cost compared to a senior engineer's rate, yet the total cost of ownership requires accounting for the verification tax. Operators must recognize that agent failure detection cannot rely on the agent itself, as the unreliable narrator problem persists regardless of execution speed. Economic viability of AI agents depends on mechanical AI agent verification layers that operate independently of the agent's own logic. External checks prevent the self-check agent flow from becoming a liability rather than an asset. Implementing rigid, non-negotiable external validation gates ensures that the 66x reduction in execution cost is not offset by downstream remediation expenses, which can account for nearly half of total program budgets.

References

• Abstract: The best agent on WorkBench in March 2024
• AI Agents That Run the Business in 2026: Why
• SWE-bench Verified tests agents against real GitHub issues, and