Coding agent scores: why 83.4% hides real costs

Codex paired with GPT-5.5 reached an 83.4% score on Terminal-Bench v2 as of June 2026. Raw model intelligence means nothing without the right execution harness to channel it. The Artificial Analysis Coding Agent Index exposes this reality by aggregating results from DeepSWE, Terminal-Bench v2, and SWE-Atlas-QnA into a single composite metric.

The index, now at version v1.1, hides more than its headline number shows. Prompt cache hit rates alone move effective costs by up to 23% on identical task flows, which is wider than the gap between the top two configurations.

Fixed models like Claude Opus 4.7 perform differently when swapped between harnesses such as Cursor CLI and Claude Code. The agent framework often dictates success more than the underlying language model. Understanding these variables is critical for engineering teams selecting tools for complex software tasks.

What the Coding Agent Index v1.1 Actually Measures

Artificial Analysis Coding Agent Index v1.1 Composition

The Artificial Analysis Coding Agent Index v1.1 derives its composite score by averaging pass@1 rates across three distinct benchmark suites. This metric aggregates performance on 113 software engineering tasks from DeepSWE, created by Datacurve, and 84 agentic terminal challenges from Terminal-Bench v2, developed by the Laude Institute. The index also incorporates SWE-Atlas-QnA, which tests technical question-answering capabilities. Each component measures pass@1, representing the probability that a single agent execution successfully resolves a task without error.

Recent evaluations show top configurations like Codex paired with GPT-5.5 achieving 83.4% on terminal tasks, while Claude Code with Fable 5 reaches 83.1%. These figures illustrate that marginal gains in the composite index often mask significant variance in specific domains. A high aggregate score does not guarantee proficiency in repository understanding if the agent excels primarily at command-line navigation. Builders must examine the underlying task counts and domain specificity rather than relying solely on the headline average. The index serves as a filtering mechanism, yet distinct operational requirements demand scrutiny of the individual benchmark results that comprise the final value.

Interpreting Agent Wall Time and Cost per Task Metrics

Time per Task quantifies the mean agent wall time required to complete a single benchmark iteration, where lower duration indicates superior orchestration efficiency. This metric captures the latency of multi-step workflows, distinguishing between rapid tool execution and delays caused by excessive reasoning loops or context switching. Operators must analyze this duration alongside success rates, as quicker agents that fail binary checks offer no operational value compared to slower, reliable alternatives.

Cost per Task measures the mean API expenditure in USD for each completed unit, serving as a primary indicator of economic viability for production deployments. Recent market analysis highlights $/task as a standard unit for comparing economic efficiency alongside raw accuracy scores. This financial metric forces a choice between model capability and token consumption, particularly when rubric-scored partial credit allows for cheaper, incomplete solutions versus expensive, fully correct outputs.

The benchmark suite mixes effectively binary outcomes with rubric-scored partial-credit outcomes, complicating direct cost comparisons across different task types. Binary tasks demand full correctness, often inflating costs for agents that struggle with complex implementation steps. In contrast, rubric-based evaluations may reward partial progress, allowing agents to achieve lower Cost per Task figures by optimizing for incremental credit rather than full resolution. Builders must weight these metrics against their specific service level agreements, prioritizing either strict completion or cost-effective partial automation. The gap between the top two harness pairs says that model architecture differences now matter less than agent orchestration strategies for terminal workflows. What survives as a distinction is token efficiency and execution stability, not marginal accuracy: a high score guarantees neither lower latency nor reduced API spend during complex, multi-step terminal interactions.

Inside Coding Agent Mechanics: Token Efficiency and Execution Dynamics

Defining Input, Output, and Cached Token Types

Separating Input Tokens from Cached Input Tokens exposes the real shape of API bills. Input Tokens cover non-cached data sent to the model, including prompts, instructions, and tool context that skip provider caches. Cached Input Tokens are reused prompt segments billed at lower rates when telemetry confirms retrieval from provider-side storage instead of fresh processing. Output Tokens make up the visible response the model generates while executing a task. These distinctions matter because prompt cache hit rates swing widely depending on provider routing, which shifts effective cost structures. Some providers send repeated requests to different backend replicas.

Provider Routing Risks to Prompt Cache State

Inconsistent backend replica affinity forces repeated prompt re-processing, dragging down cache hit rates and swelling Input Tokens. When a provider spreads sequential agent requests across disjoint server clusters lacking shared state, the system fails to retrieve Cached Input Tokens despite identical task flows. This routing behavior causes the model to treat reused context as fresh data, wiping out provider-side caching discounts. Local IDE environments often let parallel agents manage state internally, but cloud-hosted paths frequently lack the sticky sessions needed for consistent cache retrieval. The operational result is a measurable gap between theoretical and actual API costs for high-volume deployments. Builders chasing token efficiency must check whether their chosen setup keeps cache affinity across the full execution window. Without explicit affinity controls, reported cost per task metrics may underestimate spend by counting cache misses as standard input processing. AI Agents News recommends validating replica routing policies before scaling agentic workflows to production.

Comparing Model-Harness Pairs Across Diverse Benchmark Environments

Defining Model-Harness Pairs in Agent Benchmarks

Fixing the underlying model at Claude Opus 4.7 separates orchestration efficiency from raw reasoning power. This control exposes how agent logic diverges even when base intelligence stays static. Evaluations typically contrast distinct harnesses like Claude Code, Cursor CLI, and Opencode to measure these structural gaps. Top contenders in rigorous benchmarks often differ by less than a fraction of a percent, suggesting architecture now outweighs marginal model upgrades. The table below contrasts how these pairs orchestrate work and manage context.

A higher pass@1 score does not guarantee lower API costs. Aggressive retry logic may boost accuracy while consuming excessive tokens, inflating the cost per task. Selecting a model-harness pair requires balancing precision against economic constraints rather than chasing single-metric supremacy. Providers route requests differently, altering effective pricing regardless of the chosen harness.

Applying SWE-bench Verified and Terminal-Bench v2 to Agent Evaluation

SWE-bench Verified validates agent performance against real-world GitHub issues rather than synthetic prompts. This dataset requires the coding agent to navigate existing repository structures and resolve specific bugs without human intervention. Leading systems now successfully address over 80% of these complex problems, marking a shift from simple code completion to autonomous engineering. High accuracy on repository tasks does not guarantee proficiency in system interaction. Terminal-Bench v2 assesses command-line reasoning and environment manipulation, competencies that standard code generation often overlooks.

Optimizing for one benchmark may degrade performance in another due to divergent prompting strategies. A configuration tuned for deep file navigation might fail strict terminal timeout constraints. Selecting an agent requires matching the evaluation metric to the specific operational workload rather than relying on a single composite index. Deploying candidates against both suites identifies whether a workflow demands repository context or shell access.

Cursor vs Claude Code: IDE-First Agents and Subscription Costs

Engineers choosing between Cursor and Claude Code must weigh integrated development environment depth against raw orchestration flexibility. Cursor operates as an IDE-first AI coding agent optimized for parallel agent execution, allowing simultaneous task handling within the editor context. This architecture contrasts with CLI-centric harnesses that manage system interactions externally. Individual IDE-first plans are priced at $16/month, so fixed costs simplify budgeting compared to variable API spend, though scaling requires enterprise tiers. Users prioritizing deep editor integration and concurrent workflow management will find the parallel agents capability decisive for productivity. Teams requiring custom sandbox environments or specific system-level controls may prefer the modular nature of CLI-based alternatives. AI Agents News recommends evaluating task concurrency needs before locking into fixed monthly plans.

Optimizing API Costs and Token Usage for High-Value Agent Deployments

Deconstructing Pay-Per-Token API Cost Components

The pay-per-token metric isolates raw API expenditure by summing distinct billing elements rather than averaging a flat rate, specifically intended to show API costs rather than consumer plan pricing or the full operational cost of deploying the system in production. This cost model explicitly separates standard input pricing from discounted cached-input pricing, while adding output pricing to the final total. Such granularity reveals that prompt cache hit rates can change effective costs by up to 23%, even when benchmark task flows remain identical across runs. Engineers reduce API costs in agent workflows by targeting output verbosity. Maximizing token efficiency while maintaining high pass@1 performance on complex tasks creates a specific tension. Builders evaluate cost per task alongside execution speed to select the most economically viable model-harness pairs for their specific engineering constraints.

What the Published Cost Numbers Assume About Caching

Published cost figures leave out the tricks that would raise reuse: Artificial Analysis adds no custom relay headers and no affinity controls, so the numbers describe what a typical setup observes rather than an optimized best case. Anyone quoting those figures for a budget should state which of the two they are budgeting against.

Infrastructure and Engineering Costs Excluded from API Metrics

The reported pay-per-token figures exclude infrastructure, engineering, and supervision expenses necessary for production stability. Benchmark data highlights $/task as a standard economic unit, yet this metric captures only the raw API expenditure rather than the total cost of ownership for deployed systems. Subscription plans often mask these variables, providing fixed access that decouples usage from immediate marginal cost but obscures true operational efficiency. Selecting an agent based solely on API costs risks underestimating the engineering burden required to maintain reliability. True value assessment requires balancing these external factors against the narrow cost vs performance trade-offs visible in benchmark charts. Evaluations in 2026 compare agents across parameters including context window utilization, memory retention capabilities, security posture, and task handling complexity, moving beyond simple code generation accuracy.

References

• Updated June 18, 2026 (post-Fable-5). Coding agents scored on
• As organisations ramp up the use of AI coding
• 18 Best AI Coding Agents in 2026 — Agentic.ai