Agent runtime rules: boost scores by 14 points

Qwen3.5-35B-A3B performance jumped from 23.8% to 38.1% on Terminal-Bench-2.0 simply by automating its runtime constraints. Replaced by Self-Use, a framework that freezes model weights to iteratively optimize the agent's surrounding architecture. Instead of guessing which prompt tweak might fix a flaky tool call, this approach forces the agent to mine its own execution traces for specific failure patterns and propose minimal, validated edits to its own ruleset.

You will learn how execution traces serve as the primary data source for identifying recurring collapse modes, such as infinite retry loops or context window saturation, without requiring external supervision. The article details the mechanics of weakness mining, where the system categorizes failed tasks to generate targeted use proposals rather than broad, destabilizing changes. Finally, we examine the critical role of regression testing and loop breakers in the validation phase, ensuring that new runtime policies improve held-out performance without degrading existing capabilities. By shifting focus from the model to the failure-recovery logic and verification rules that govern it, teams can achieve the stability of a seasoned engineer's oversight without the manual labor. This is not about improved models; it is about building a self-correcting infrastructure that makes mediocre models perform reliably and great models perform consistently.

The Role of Execution Traces in Modern Agent Reliability

Defining the AI Agent Use Beyond Model Weights

System prompts, tool definitions, memory structures, and runtime policies form the AI agent use that governs behavior independent of model weights. Claude Opus 4.8 achieves 88.6% on SWE-bench Verified, yet an agent's behaviour depends equally on this surrounding configuration. Raw execution traces provide the primary evidence source for improvement. Summarizing these traces destroys critical signal needed for optimization, causing accuracy to drop from 50% to a significant share according to recent findings. This data gap forces a shift from manual tuning to automated loops that analyze failures directly. Agentic Use Engineering demonstrates that systems mining uncompressed traces outperform baselines like Codex by identifying repeated systemic failures rather than isolated errors. The approach requires machine-checkable outcomes to function. Without objective pass/fail signals, the feedback loop cannot validate use proposals. Swapping models without adjusting the use yields diminishing returns because the architecture determines if potential is reached in production. Over-engineering rules can stifle model capability while under-specifying leads to flaky execution.

Real-World Impact of Use Engineering on SWE-bench Scores

Codex reaches a dominant score on Terminal-Bench 2.0 by wrapping the model in a strict execution use. Raw performance metrics often mislead operators because the delta between a base model and a harnessed agent defines production viability. Systems like SWE-agent and OpenHands demonstrate that Agentic Use Engineering outperforms human-designed configurations by mining execution traces for systemic failures. The mechanism relies on replacing compressed failure summaries with uncompressed trajectory data to preserve signal fidelity. Summarizing trace data destroys optimization signals, causing accuracy to collapse from 50.0% to 34.9% during the refinement cycle. However, this approach demands machine-verifiable success criteria for every task in the evaluation set.

Inside Weakness Mining and Automated Use Optimization

Weakness Mining Mechanics Using Uncompressed Execution Traces

Running task batches with verifiable outcomes generates the raw data required to categorize failed traces without retraining model weights. This Weakness mining stage executes a set set of challenges, capturing every tool call and error message in an uncompressed log. Operators must preserve this full fidelity because summarizing execution history destroys the specific signal needed for self-improvement, causing accuracy to collapse significantly compared to raw data retention. The process identifies recurring failure patterns, such as infinite tool loops or incorrect retry logic, which appear consistently across the test batch. Unlike external search processes used by Meta-Use, this method operationalizes improvement as a bounded edit process driven strictly by internal evidence.

A assigned proposer role converts each categorized failure pattern into a small, targeted edit tied to that specific mechanism. Qwen3.5 initial performance sits at 23.8% before optimization, a baseline reflecting unaddressed execution errors. The mechanism operates as a bounded edit process, distinct from the external search strategies used by Meta-Use. Instead of scanning broad codebases, the system generates diverse, minimal modifications derived strictly from identified weaknesses in the Self-Use Architecture. This approach directly addresses the token snowball effect, where minor recursive errors compound into expensive, multi-step failures. Operators must recognize that automated use optimization creates a tension between edit minimality and coverage breadth. Small edits reduce regression risk but may miss complex, multi-variable failure modes requiring broader. The implication for production networks is clear: reliance on raw trace fidelity remains non-negotiable for valid proposals. Unlike summarized data, uncompressed logs preserve the precise command sequences needed to craft proven runtime policies.

The cost of this precision is the requirement for verifiable task outcomes during the mining phase. Without objective pass/fail signals, the proposer lacks the feedback loop necessary to validate changes. This constraint limits immediate application to domains with deterministic evaluation metrics. AI Agents News highlights that such systematic loops replace manual hunches with evidence-based configuration updates. This collapse occurs because summarization algorithms discard the low-level signal required for proven Weakness mining While compression reduces token costs, it simultaneously destroys the fidelity needed for the proposer role to generate valid use edits. The mechanism demands uncompressed trajectory data to distinguish between similar-looking but distinct failure modes. Validating use changes requires comparing exact command sequences, which summaries obscure. Attempts to guide validation using compressed logs fail because the failure signal is lost in translation.

1. Execute task batches to capture full execution traces.
2. Categorize failures using raw logs rather than abstracts.
3. Generate edits only from high-fidelity pattern matches. Ignoring this constraint forces the system to regress toward random guessing. For more on validating these loops, consult resources from AI Agents.

Implementing Regression Testing and Loop Breakers for Agents

Defining Loop Breakers and Regression Testing in Self-Use

Hard tool-call limits stop infinite retry cycles by terminating execution after 50 attempts. This mechanism swaps vague instructions for deterministic runtime policies that kill specific failure patterns. Operators place these guards inside the agent's policy file to interrupt recursive errors before resource consumption spikes.

1. Define a maximum tool-call threshold in the runtime configuration.

1. Implement a rule prohibiting exact duplicate commands in the retry logic.
2. Validate proposed edits against held-out tasks to prevent performance regression.

The Self-Use Architecture uses this validation stage to accept changes only when they fix failing cases without hurting success on other tasks. Testing across models like MiniMax M2.5 and GLM-5 shows that freezing model weights while optimizing the use yields significant relative performance gains. A constraint exists: the approach demands machine-checkable outcomes. Tasks lacking objective pass/fail signals cannot feed the necessary feedback loop. Without verifiable metrics, the system cannot tell a beneficial edit from a harmful one. Self-Use Evaluation on Terminal-Bench-2.0 The framework assumes access to uncompressed execution traces because summarized data often lacks the fidelity required for precise weakness mining. AI Agents News highlights that this empirical debugging process transforms ambiguous failures into testable edits.

Fixing MiniMax M2.5 Timeout Errors with Targeted Policy Edits

MiniMax M2.5 previously explored dataset configs until timeout, shipping zero artifacts due to missing runtime limits.

1. Identify infinite exploration patterns in the raw execution logs before any compression occurs.

1. Insert a loop breaker policy that forces redirection after a fixed number of tool calls.
2. Mandate early generation of required artifacts to satisfy downstream verification steps immediately.

The Self-Use Evaluation on Terminal-Bench-2.0 Specific edits repla generic prompt extensions with legible, deterministic rules tied to the failure mode. Operators must note that Proposal Validation rejects changes unless they improve failing cases without regressing held-out tasks. This gate prevents the use from over-correcting based on noise in a single trace batch. Unlike manual tuning, the system auto-optimized versions beat hand-engineered entries by using a proposer role to read failed traces. A drawback is strict: this loop requires machine-checkable outcomes to function as an objective acceptance gate. Without verifiable pass/fail signals, the regression testing phase cannot distinguish a fix from a hallucination. AI Agents News highlights that such infrastructure determines whether feedback loops fit within nightly deployment windows.

Validating Agent Stability on Terminal-Bench-2.0 Artifacts

Terminal-Bench-2.0 execution requires artifact management checks before any loop breaker activates.

1. Run baseline tasks to capture raw failure traces covering command use and verification.

1. Mine weaknesses from logs to identify specific retry loops or missing artifact generation.

1. Propose minimal use edits that target the identified failure mode without generic prompt padding.
2. Validate changes against held-out datasets to confirm improvement without degrading prior capabilities.

The proposal validation stage rejects edits that improve one metric while breaking another, ensuring net stability gains. Operators must avoid summarizing traces prematurely, as this destroys the signal needed for proven weakness mining High CI/CD integration viability depends on this strict, evidence-based gatekeeping. Blindly increasing prompt length fails where targeted policy edits succeed. Skipping regression testing leads to unverified behavior in production artifacts. AI Agents News recommends freezing model weights to isolate use efficacy during these tests.

Replacing a Qwen3-32B backbone with GPT-4o-mini causes Effectiveness under Token Budget scores to plummet from a strong majority to a small fraction without automatic adaptation. Cheaper models display distinct failure modes that static prompts miss entirely.org/abs/2606.09498) treats improvement as a bounded edit process instead of a full rewrite. A three-stage loop executes automatically. Weakness Mining spots specific error patterns in raw traces. Use Proposal creates minimal code edits. Proposal Validation accepts changes only if they boost held-out tasks without regression.

This cycle turns model upgrades from expensive engineering projects into routine configuration updates. Unlike Meta-Use, which searches broadly over source code, this method focuses strictly on iterative policy refinement. Evaluation speed acts as the primary limitation. Parallelizing workloads across 50 sandboxes cuts validation time from days to roughly one hour. Slow feedback stalls the loop and makes real-time adaptation impossible. Operators need infrastructure capable of rapid, machine-checkable verification to sustain the cycle. The outcome is a model-agnostic tuning definition where the use evolves empirically alongside the underlying weights.

Teams deploy this loop when swapping model backbones because static prompts fail to adapt to new failure modes automatically. The mechanism mines raw execution traces to find systemic errors, then proposes minimal code edits for validation. Compressing these traces into summaries destroys the signal needed for proven self-improvement. Teams must retain full log fidelity rather than relying on aggregated reports. Infrastructure must support high-volume trace storage to enable automatic correction. The feedback loop breaks without this capacity, forcing a return to manual tuning.

Ambiguous failures where an agent appears broken become testable edits with an objective accept/reject gate. The process converts vague observations into specific runtime policies that prevent recurrence of identical errors.mindstudio.ai/blog/use-engineering-the-discipline-6- notably outperforms methods using summarized data by preserving the context required for fixes. This approach demands machine-checkable success criteria for every task in the evaluation set. Tasks lacking binary pass/fail signals cannot participate in the automated validation stage. Network teams must build rigorous benchmarks before expecting autonomous improvement.

The cost of maintaining human-engineered harnesses scales linearly with model churn rates. Automation breaks this link by tying use proposals directly to observed execution data.

Strategic ROI: Self-Use Automation Versus Manual Tuning Costs

Manual use rewrites consume engineering weeks while automated loops validate edits in hours against objective pass/fail gates. Economic divergence stems from token consumption patterns where unoptimized agents generate expensive failure cascades during context management. Static configurations force operators to choose between over-provisioning resources or accepting frequent service degradation.

The formalization of use engineering shifts focus from prompt text to the code infrastructure governing tool calls and memory. This architectural pivot allows smaller models to achieve production readiness without the latency penalties of larger alternatives. Rapid iteration creates tension with stability. Continuous automated changes risk introducing subtle regressions if the validation dataset lacks sufficient diversity. Teams must maintain rigorous held-out test sets to prevent the system from optimizing for specific benchmark quirks rather than general reliability. Organizations lacking machine-checkable success criteria cannot adopt this methodology regardless of potential efficiency gains. AI Agents News recommends implementing strict acceptance gates before deploying any self-modifying agent logic to production environments.