Study finds harness choice barely moves agentic coding scores

The Harness Effect Is Smaller Than Vendors Suggest

A new paper published on arXiv challenges the prevailing assumption that the specific agent framework, or "harness," used to drive a large language model is the primary determinant of coding performance. The study, titled "Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite," presents paired same-model contrasts to measure whether vendor-native pairings actually outperform neutral or third-party harnesses. The authors argue that while practitioners often assume the vendor-tuned pairing solves more tasks, the data suggests the model itself is the dominant variable, with the harness contributing a statistically negligible average advantage.

The research team conducted 792 graded runs out of 800 planned, using a private, contamination-controlled suite of 256 repository and post-cutoff contest tasks. By isolating the model from the control flow, prompts, and tools, the study provides a rare clean look at the "harness effect" in agentic coding. The core finding is that neither of the two primary contrasts resolved an average advantage for either harness, suggesting that the choice of SDK or agent framework is less critical than the underlying model's capabilities.

Methodology and Key Metrics

The study focused on two main paired contrasts. The first compared the claude-agent-sdk against deepagents using the claude-opus-4-8 model. The second compared the openai-codex SDK against deepagents using the gpt-5.5 model. Additionally, gemini-3.5-flash and deepseek-v3.2 were included as side cells for broader context. All runs were graded by an isolated oracle to ensure consistency and prevent contamination from public benchmarks.

For the Claude Opus 4.8 contrast, the native harness trailed by an average of 1.25 percentage points compared to the neutral harness, with a task-bootstrap 95% confidence interval of [-10.0, +7.5]. This wide interval indicates that the difference is not statistically significant. Similarly, for the GPT-5.5 contrast, the native harness led by 1.25 percentage points (55.6% vs 54.4%), but the confidence interval of [-4.4, +6.9] again fails to confirm a meaningful advantage. These results suggest that the marginal benefit of using a vendor-specific harness is, on average, close to zero.

However, the average masks significant stratification. The authors note that the Opus 4.8 average combines opposite strata: the native harness trailed by 9.0 percentage points on 61 repository tasks but led by 23.7 percentage points on 19 contest tasks. The label-permutation p-value of 0.003 indicates this split is unlikely to be random, though the authors caution that the partition was chosen after seeing the data and requires a designed replication to confirm. This suggests that harness performance may be highly task-dependent, with neutral harnesses potentially excelling at structured repository work while native harnesses may have an edge in competitive programming scenarios.

Cost and Completion Trade-offs

Beyond accuracy, the study examines the economic and operational efficiency of different harnesses. When re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 compared to the native harness. For GPT-5.5, the neutral harness cost 1.2 times as much. These are observed-usage estimates, meaning they reflect actual token consumption rather than theoretical costs. The higher cost of the neutral harness suggests that vendor-native integrations may be more token-efficient, possibly due to optimized prompt structures or tool calling conventions.

Completion rates also showed divergence from correctness. The authors found that 22 of 81 runs that were cancelled at the wall-clock ceiling had already produced a passing patch. This indicates that many agentic coding systems are capable of solving tasks but may exceed time limits due to inefficient exploration or verification loops. This finding is critical for developers deploying these systems in production, where time-to-solution is often as important as the final success rate.

Implications for Developer Workflows

For developers building agentic coding tools, the study suggests that investing heavily in a custom harness may not yield significant performance gains if the underlying model is strong. Instead, the focus should shift to optimizing the model selection and managing cost and latency. The data implies that the "harness effect" is not a universal multiplier but a task-specific variable. Teams should benchmark their specific use cases rather than relying on general assumptions about vendor-native pairings.

The cost implications are also significant. If a neutral harness costs 1.3 to 1.6 times more per solved task, organizations with high-volume coding tasks should carefully evaluate the total cost of ownership. The 22 runs that produced passing patches but were cancelled due to time limits highlight the need for better timeout management and early termination strategies. Developers should consider implementing heuristic checks to stop agents once a valid patch is generated, rather than allowing them to continue exploring until the wall-clock limit is reached.

What to Watch

The study releases the orchestrator, grading oracle, reanalysis code, and derived aggregates, but the tasks remain private. This limits the ability of other researchers to replicate the exact findings, though the code release allows for methodological scrutiny. The authors acknowledge that the partition of tasks into repository and contest categories was chosen post-hoc, and a designed replication is needed to confirm the stratification effects. Future work should focus on pre-registered task partitions and larger sample sizes to narrow the confidence intervals.

Additionally, the cost analysis is based on frozen list prices and observed usage, which may not reflect real-world billing complexities, such as volume discounts or dynamic pricing. The 58 runs on the Anthropic account that left no usage record introduce uncertainty into the cost ratios, with the Opus ratio potentially moving between 0.7 and 2.3 depending on how that spend is allocated. This unresolved billing ordering means that the cost advantage of native harnesses is not definitively established.

Finally, the study's reliance on post-cutoff contest tasks and private repository tasks limits generalizability to other types of coding tasks, such as legacy code maintenance or multi-language projects. Developers should treat these findings as a strong signal for the specific tasks tested, but not as a universal rule for all agentic coding applications.