Agent runtime rules: boost scores by 14 points

Qwen3.5-35B-A3B performance jumped from 23.8% to 38.1% on Terminal-Bench-2.0 simply by automating its runtime constraints. Manual prompt tuning is replaced by Self-Harness, a framework that freezes model weights to iteratively optimize the agent's surrounding architecture. Instead of guessing which prompt tweak might fix a flaky tool call, this approach forces the agent to mine its own execution traces for specific failure patterns and propose minimal, validated edits to its own ruleset.

The lever is the execution trace. Raw logs expose recurring collapse modes such as infinite retry loops or context window saturation, and weakness mining turns each pattern into a small, targeted edit instead of a broad, destabilizing rewrite. Regression testing against held-out tasks then decides whether that edit ships, while loop breakers cap the failures that burn the most tokens. Reliability ends up carried by failure-recovery logic and verification rules, not by the weights. This is not about improved models; it is about building a self-correcting infrastructure that makes mediocre models perform reliably and great models perform consistently.

The Role of Execution Traces in Modern Agent Reliability

Defining the AI Agent Harness Beyond Model Weights

System prompts, tool definitions, memory structures, and runtime policies form the AI agent harness that governs behavior independent of model weights. Claude Opus 4.8 achieves 88.6% on SWE-bench Verified, yet an agent's behaviour depends equally on this surrounding configuration. Raw execution traces provide the primary evidence source for improvement. Summarizing these traces destroys critical signal needed for optimization, causing accuracy to drop from 50% to 34.9% according to recent findings. This data gap forces a shift from manual tuning to automated loops that analyze failures directly. Agentic Harness Engineering demonstrates that systems mining uncompressed traces outperform baselines like Codex by identifying repeated systemic failures rather than isolated errors. The approach requires machine-checkable outcomes to function. Without objective pass/fail signals, the feedback loop cannot validate harness proposals. Swapping models without adjusting the harness yields diminishing returns because the architecture determines if potential is reached in production. Over-engineering rules can stifle model capability while under-specifying leads to flaky execution.

Real-World Impact of Harness Engineering on SWE-bench Scores

Codex reaches a 77.3% score on Terminal-Bench 2.0 by wrapping the model in a strict execution harness. Raw performance metrics often mislead operators because the delta between a base model and a harnessed agent defines production viability. Systems like SWE-agent and OpenHands demonstrate that Agentic Harness Engineering outperforms human-designed configurations by mining execution traces for systemic failures. The mechanism relies on replacing compressed failure summaries with uncompressed trajectory data to preserve signal fidelity.

Inside Weakness Mining and Automated Harness Optimization

Weakness Mining Mechanics Using Uncompressed Execution Traces

Running task batches with verifiable outcomes generates the raw data required to categorize failed traces without retraining model weights. This Weakness mining stage executes a set of challenges, capturing every tool call and error message in an uncompressed log. Operators must preserve this full fidelity because summarizing execution history destroys the specific signal needed for self-improvement, causing accuracy to collapse significantly compared to raw data retention. The process identifies recurring failure patterns, such as infinite tool loops or incorrect retry logic, which appear consistently across the test batch. Unlike external search processes used by Meta-Harness, this method operationalizes improvement as a bounded edit process driven strictly by internal evidence.

An assigned proposer role converts each categorized failure pattern into a small, targeted edit tied to that specific mechanism. Qwen3.5 initial performance sits at 23.8% before optimization, a baseline reflecting unaddressed execution errors. The mechanism operates as a bounded edit process, distinct from the external search strategies used by Meta-Harness. Instead of scanning broad codebases, the system generates diverse, minimal modifications derived strictly from identified weaknesses in the Self-Harness Architecture. This approach directly addresses the token snowball effect, where minor recursive errors compound into expensive, multi-step failures. Operators must recognize that automated harness optimization creates a tension between edit minimality and coverage breadth. Small edits reduce regression risk but may miss complex, multi-variable failure modes requiring broader intervention. The implication for production networks is clear: reliance on raw trace fidelity remains non-negotiable for valid proposals. Unlike summarized data, uncompressed logs preserve the precise command sequences needed to craft proven runtime policies.

The cost of this precision is the requirement for verifiable task outcomes during the mining phase. Without objective pass/fail signals, the proposer lacks the feedback loop necessary to validate changes. This constraint limits immediate application to domains with deterministic evaluation metrics. AI Agents News highlights that such systematic loops replace manual hunches with evidence-based configuration updates. Validating a harness change means comparing exact command sequences, which is precisely what a summary obscures.

1. Execute task batches to capture full execution traces.
2. Categorize failures using raw logs rather than abstracts.
3. Generate edits only from high-fidelity pattern matches.

Ignoring this constraint forces the system to regress toward random guessing. For more on validating these loops, consult resources from AI Agents News.

Implementing Regression Testing and Loop Breakers for Agents

Defining Loop Breakers and Regression Testing in Self-Harness

Hard tool-call limits stop infinite retry cycles by terminating execution after 50 attempts. This mechanism swaps vague instructions for deterministic runtime policies that kill specific failure patterns. Operators place these guards inside the agent's policy file to interrupt recursive errors before resource consumption spikes.

1. Define a maximum tool-call threshold in the runtime configuration.
2. Implement a rule prohibiting exact duplicate commands in the retry logic.
3. Validate proposed edits against held-out tasks to prevent performance regression.

The Self-Harness Architecture uses this validation stage to accept changes only when they fix failing cases without hurting success on other tasks. Testing across models like MiniMax M2.5 and GLM-5 shows that freezing model weights while optimizing the harness yields relative performance gains of up to 60%. AI Agents News highlights that this empirical debugging process transforms ambiguous failures into testable edits.

Fixing MiniMax M2.5 Timeout Errors with Targeted Policy Edits

MiniMax M2.5 previously explored dataset configs until timeout, shipping zero artifacts due to missing runtime limits.

1. Identify infinite exploration patterns in the raw execution logs before any compression occurs.
2. Insert a loop breaker policy that forces redirection after a fixed number of tool calls.
3. Mandate early generation of required artifacts to satisfy downstream verification steps immediately.

Specific edits replace generic prompt extensions with legible, deterministic rules tied to the failure mode. Operators must note that Proposal Validation rejects changes unless they improve failing cases without regressing held-out tasks. This gate prevents the harness from over-correcting based on noise in a single trace batch. Unlike manual tuning, the system auto-optimized versions beat hand-engineered entries by using a proposer role to read failed traces. AI Agents News highlights that such infrastructure determines whether feedback loops fit within nightly deployment windows.

Validating Agent Stability on Terminal-Bench-2.0 Artifacts

Terminal-Bench-2.0 execution requires artifact management checks before any loop breaker activates.

1. Run baseline tasks to capture raw failure traces covering command use and verification.
2. Mine weaknesses from logs to identify specific retry loops or missing artifact generation.
3. Propose minimal harness edits that target the identified failure mode without generic prompt padding.
4. Validate changes against held-out datasets to confirm improvement without degrading prior capabilities.

The proposal validation stage rejects edits that improve one metric while breaking another, ensuring net stability gains. High CI/CD integration viability depends on this strict, evidence-based gatekeeping. Blindly increasing prompt length fails where targeted policy edits succeed. Skipping regression testing leads to unverified behavior in production artifacts. AI Agents News recommends freezing model weights to isolate harness efficacy during these tests.

Why a Model Swap Restarts the Loop

Replacing a Qwen3-32B backbone with GPT-4o-mini causes Effectiveness under Token Budget scores to plummet from a strong majority to a small fraction without automatic adaptation. Cheaper models display distinct failure modes that static prompts miss entirely, so a swap that looks like a configuration change behaves like a rewrite. Self-Harness answers it as a bounded edit process: Weakness Mining spots error patterns in raw traces, Harness Proposal creates minimal code edits, and Proposal Validation accepts them only if held-out tasks improve without regression. Unlike Meta-Harness, which searches broadly over source code, this method refines policy iteratively, which turns a model upgrade from an engineering project into a configuration update.

Evaluation speed decides whether that promise survives a deadline. Parallelizing workloads across 50 sandboxes cuts validation time from days to roughly one hour, and slower feedback stalls the loop until adaptation after a swap becomes impossible. The cost of maintaining human-engineered harnesses scales linearly with model churn rates, while automation ties harness proposals directly to observed execution data.

Strategic ROI: Self-Harness Automation Versus Manual Tuning Costs

Manual harness rewrites consume engineering weeks while automated loops validate edits in hours against objective pass/fail gates. Economic divergence stems from token consumption patterns where unoptimized agents generate expensive failure cascades during context management. Static configurations force operators to choose between over-provisioning resources or accepting frequent service degradation.

The formalization of harness engineering shifts focus from prompt text to the code infrastructure governing tool calls and memory. This architectural pivot allows smaller models to achieve production readiness without the latency penalties of larger alternatives. Rapid iteration creates tension with stability. Continuous automated changes risk introducing subtle regressions if the validation dataset lacks sufficient diversity. Teams must maintain rigorous held-out test sets to prevent the system from optimizing for specific benchmark quirks rather than general reliability. Organizations lacking machine-checkable success criteria cannot adopt this methodology regardless of potential efficiency gains. AI Agents News recommends implementing strict acceptance gates before deploying any self-modifying agent logic to production environments.