Model Tier or Harness? What Decides Agentic Coding Quality

When a coding agent produces a bad diff, the standard fix is one config line: point it at a stronger model. The benchmark boards that publish model and harness as separate columns complicate that reflex. Among models of comparable capability, switching the harness moves the score at least as much as switching the model, and the model’s advantage reasserts itself only as tasks lengthen. My working default is therefore the reverse of that reflex: a mid-tier model inside a harness with a closed verification loop and a strict context budget, escalating tier by task horizon, not by task importance.

By harness I mean the software layer around the model: context construction, the tool interface, the verification loop, orchestration. The decision sits with whoever pays for both levers: the model subscription on one side, and the engineering time that harness work costs on the other.

Where the Tier Argument Is Right

“Buy the stronger model” deserves its strongest form before anything else, because on one class of work it is simply correct.

SWE-bench Pro exists because harnesses saturated the easier benchmark. The paper’s current version (arXiv 2509.16941, v2, November 2025) evaluates widely used coding models under a unified scaffold, and every one stays below 45% Pass@1. Claude Sonnet 4.5 leads the public set at 43.6%; GPT-OSS 120B trails at 16.2%. That is a 27-point spread with the harness held fixed, and more than half of the benchmark unsolved at every tier. The hard tasks are model-bound. One appendix comparison is worth carrying forward, though: GPT-5 scores 23.3% under a 50-turn, $2 budget cap at medium reasoning effort, against 41.8% uncapped at high. Even on a model-bound board, the run budget shifts the score.

METR’s time-horizon work is the cleanest capability argument. Time Horizon 1.1 (January 2026) estimates Claude Opus 4.5’s 50% time horizon, the human-task length the model completes half the time, at 320 minutes, with a wide 95% confidence interval of 170 to 729 minutes. It also estimates a post-2023 doubling time of 130.8 days against a longer-run trend of 196.5 days. The ability to hold a multi-hour task together is a property of the model, and it is improving on a curve no amount of tooling bends.

The harness has a ceiling of its own, too. A verification loop can reject a wrong answer; it cannot invent the correct architectural decision the model never proposed. Therefore, on tasks whose horizon exceeds what your current model can hold together, no verification loop rescues the run. The rest of the evidence is about why most delegated work does not sit in that regime.

When the Scaffold Was the Whole Story

Two 2024 results pull in opposite directions, and both were state of the art within weeks of each other. SWE-agent (NeurIPS 2024) argued that “LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use.” Its purpose-built agent-computer interface reached 12.5% pass@1 on the original SWE-bench and 87.7% on HumanEvalFix, results the paper describes as “far exceeding the previous state-of-the-art achieved with non-interactive LMs.” The models were the same; the interface multiplied the score.

Agentless (arXiv 2407.01489) went the opposite way and deleted the agent loop entirely. A fixed three-phase pipeline (localize, repair, validate) solved 32.00% of SWE-bench Lite issues, 96 fixes, at $0.70 per issue, beating every open-source agent of the period on both score and cost. The two figures come from different benchmark tracks and are not directly comparable, but they point at the same lesson. In that era, the software around the model moved the number more than the model did, and two opposite harness designs each beat the field.

What the Models Absorbed

The elaborate scaffolds then stopped paying as much, because model post-training absorbed them. mini-swe-agent, from the SWE-bench team, ships an agent class of “just some 100 lines of python” and states that it “scores >74% on the SWE-bench verified benchmark” (the README does not attribute the figure to a specific model). The same README names the shift plainly: “as LMs have become more capable, a lot of this is not needed at all to build a useful agent.”

One caveat travels with the 100-line claim. The 100 lines are the control loop; Simon Willison puts the mini-swe-agent codebase around it at roughly 9,000 lines of Python. The loop commoditised; the system around it did not.

The official SWE-bench leaderboard now runs a standardised bash-only track on mini-swe-agent precisely so that the model is the only variable. In the February 2026 refresh, Claude Opus 4.5 (high reasoning) led at 76.8%, with Gemini 3 Flash and MiniMax M2.5 at 75.8%, Opus 4.6 at 75.6%, and GPT-5.2 at 72.8%. Willison states the design intent: “This benchmark uses the same system prompt for every model, which is important for a fair comparison but does mean that the quality of the different harnesses or optimized prompts is not being measured here.” Note the shape of that list, not the standings, which are a snapshot of that refresh: five frontier models within four points of each other once the harness is held fixed.

A source-code taxonomy of thirteen open-source coding agent scaffolds (arXiv 2604.03515) confirms the shift structurally. Agent designs converge “where external constraints dominate (tool capability categories, edit formats, execution isolation)” and still diverge in context compaction, state management, and multi-model routing, with eleven of the thirteen composing multiple control primitives. What remains divergent, and therefore still worth engineering effort, is context management.

The Harness Gap Is Now a Context Gap

Terminal-Bench 2.1 is the sharpest current exhibit because it publishes agent and model as separate columns, so either one can be held fixed. From the board as of mid-August 2026:

Model	Harness	Score
Fable 5	Claude Code	83.8%
Fable 5	Terminus 2	80.4%
GPT-5.5	Codex	83.1%
GPT-5.5	Terminus 2	78.0%
Gemini 3 Pro	Terminus 2	73.9%
Gemini 3 Pro	Gemini CLI	65.8%

Read it twice. Holding the harness fixed at Terminus 2, the spread across the three models is 6.5 points (73.9% to 80.4%). Holding the model fixed at Gemini 3 Pro, the spread across its two harnesses is 8.1 points (65.8% to 73.9%). On this board, for this task class, the harness column is worth at least as many points as the model column.

The board also contains a crossover worth stating precisely. GPT-5.5 under Codex (83.1%) outscores Fable 5 under Terminus 2 (80.4%), even though Fable 5 leads when both run under the shared harness. That is a crossover between near-peers, and it is the strongest published shape of the effect. I found no public result showing a strict tier inversion, meaning a genuinely lower-tier model in a strong harness beating a frontier model in a weak one head to head. What the data supports is a claim about magnitude within the frontier band: on bounded task classes, harness choice moves the score at least as much as a tier upgrade does.

The academic version of that claim is scoped the same way, and the scope is the point. “Stop Comparing LLM Agents Without Disclosing the Harness” (arXiv 2605.23950) formalises the Binding Constraint Thesis: “for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness … is often a stronger determinant of agent performance than the model it wraps.” In its 3x3 factorial study (GPT-5.4, Kimi K2.6, and GLM-5.1, each run under three harness configurations on a 100-task stratified SWE-bench Verified subset), harness-attributable variance was 7.80 times model-attributable variance (18.48 against 2.37, in squared percentage points), and model rankings reversed across harnesses. Read the scope condition before quoting the ratio: comparable frontier capability. Within that band, the harness explains more of the score variance than the model choice does. The paper offers no support for the stronger claim that a cheap model in a good harness beats a frontier model in a bad one.

What harness work means in practice

Anthropic’s engineering guidance gives the concrete work list. Context engineering is defined there as “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference,” with the objective of finding “the smallest set of high-signal tokens that maximize the likelihood of your desired outcome.” The mechanism this defends against is measured: Chroma’s 2025 Context Rot study across 18 models found that “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows.” An advertised window degrades long before its limit is reached.

In practice the moves are:

• Compaction. Summarise a near-full window and reinitialise with the summary, preserving architectural decisions, unresolved bugs, and implementation state.
• Structured note-taking. Persist state outside the window, in a plan file or a NOTES.md, and re-inject it on demand instead of carrying it through every turn.
• Sub-agent exploration. A subagent burns tens of thousands of tokens reading the codebase and returns a condensed summary of 1,000 to 2,000 tokens; the lead agent’s window never sees the exploration.
• Progressive tool disclosure. Anthropic’s code-execution-with-MCP write-up reports one workflow dropping from 150,000 tokens to 2,000 tokens, a 98.7% reduction, by letting the model read tool definitions from a filesystem on demand instead of loading every definition up front.
• A check the agent can run. Tests, a build exit code, a linter, a screenshot diff. The Claude Code guidance is blunt about why this one is mandatory: “Claude stops when the work looks done. Without a check it can run, ‘looks done’ is the only signal available, and you become the verification loop.” Its escalation path runs from a prompt instruction, to a goal condition re-evaluated every turn, to a Stop hook that blocks the turn from ending (Claude Code overrides the hook after eight consecutive blocks), to an adversarial review subagent that “has a fresh model try to refute the result, so the agent doing the work isn’t the one grading it.” The names are Claude Code’s; the pattern, a machine-checkable completion gate, ports to any harness that exposes hooks or scriptable gates.

The first four moves manage the token budget; the fifth supplies the completion signal the model lacks. Anthropic’s own summary ties the list together: “most best practices are based on one constraint: Claude’s context window fills up fast, and performance degrades as it fills.”

The edit-format variable

How a model is asked to emit a change interacts with how well it changes code, which is why Aider’s polyglot leaderboard (225 Exercism problems across six languages) tracks edit format as a first-class column next to score and cost. The board’s last update is dated 20 November 2025, so treat these rows as a snapshot of that moment: gpt-5 (high) at 88.0% using diff format for $29.08; o3-pro (high) at 84.9$146.32; and an architect pairing of o3 (high) planning with gpt-4.1 editing at 78.2% for $17.55. The third row is the pattern in miniature. A cheaper configuration landed within seven points of o3-pro at roughly an eighth of the price, because the harness split the work into roles each model could do well.

The Cost Side of the Same Decision

Tier has a list price, and harness work changes what you effectively pay. Claude API list prices as of mid-August 2026, per million tokens:

Model	Input	Output
Claude Fable 5	$10	$50
Claude Opus 5	$5	$25
Claude Sonnet 5	$2	$10
Claude Haiku 4.5	$1	$5

Sonnet 5 to Opus 5 is a 2.5x multiplier; Haiku 4.5 to Opus 5 is 5x. Two pricing mechanics let harness work compete directly with those ratios: cache reads bill at 0.1x base input, and the Batch API discounts input and output by 50%. A tier drop from Opus 5 to Sonnet 5 cuts 60% everywhere. A cached prefix cuts 90% on every input token it covers, and batching cuts 50% on everything that does not need an interactive response. A harness that stabilises its prompt prefix and routes non-interactive work through batches moves effective cost more than most tier decisions do.

One accounting trap: Anthropic’s pricing page notes that Claude 4.7 and later models use a newer tokenizer that “produces approximately 30% more tokens for the same text.” Per-token prices across model generations are therefore misleading on their own. Cost per merged PR is the unit that survives tokenizer changes, price changes, and model swaps; the ROI part of the AI tools series covers how to build that number.

A Decision Rule for the Next Unit of Spend

The rule, rooted at the default, with every branch naming its override trigger:

The same signals in table form, for cases the flowchart does not map cleanly:

Signal	Points to harness	Points to tier
Failure mode	Right idea, wrong execution; forgets an earlier instruction	Wrong approach chosen at step one
Task horizon	Minutes to about an hour of human work	Multi-hour, multi-subsystem
Verifiability	A test or build settles it	Correctness is a judgement call
Context pressure	Compaction fires repeatedly mid-task	Window barely half full at failure
Repeatability	Same shape of task, many instances	One-off, novel design work
Marginal spend	Cache hit rate below 50%, no batch usage	Caching and batching already tuned

Treat the one-hour and four-hour splits as rough anchors. METR’s live time-horizons page (updated May 2026) estimates the frontier 50% horizon at roughly twelve hours (Claude Opus 4.6, 719 minutes), still with wide error bars and a note that measurements above sixteen hours strain its task suite. A bounded task under an hour sits far below that ceiling; that is exactly the region where the harness decides the outcome.

What Each Lever Costs

Harness investment, the default, bills in engineering time and in a maintenance surface: hooks, subagent definitions, instruction files, and skills all rot as the codebase moves, and someone has to own the agent configuration the way someone owns CI. It also fails in a specific way. Anthropic’s guidance lists “the over-specified CLAUDE.md” as a named failure pattern and warns that “bloated CLAUDE.md files cause Claude to ignore your actual instructions.” A second failure mode is subtler: a verification loop that gates on a weak check produces confident garbage faster than no loop at all.

Tier escalation costs 2.5x to 5x on tokens at list price plus latency on interactive work, and close to zero operational overhead, which is exactly why it is everyone’s first move. Its failure mode is silent. On bounded tasks it buys nothing measurable, and because it hides the real constraint, a team can spend a quarter before discovering the harness was the bottleneck.

Per-task routing, the default plus an escalation trigger, is where the decision rule above eventually lands you. The cost is a routing policy that has to be maintained and re-measured after every model release. The scaffold taxonomy lists multi-model routing among the components where designs still diverge, which is a polite way of saying nobody has converged on doing it well.

Reasoning effort deserves its own line because it looks like a free tier upgrade. The Holistic Agent Leaderboard, across 21,730 rollouts on nine models and nine benchmarks, reports “higher reasoning effort reducing accuracy in the majority of runs.” The knob turns the wrong way more often than assumed; A/B test it on your own task classes before treating high effort as the safe default.

Common Pitfalls

• Quoting agent scores without the agent column. A benchmark row that names only the model is an incomplete comparison. When quoting a number, quote model, harness, and version together, the way Terminal-Bench and the bash-only SWE-bench track force you to.
• Treating the vendor’s scaffold number as your number. Vendor-reported scores come from vendor-tuned harnesses. Use the standardised track as the baseline and read the vendor delta as an upper bound on what harness tuning could buy.
• Letting the window fill because the window is large. Context rot sets in before the limit. Clear state between unrelated tasks and delegate exploration to subagents so the reading never lands in the main window.
• Accepting “looks done.” A plausible diff that never ran is the canonical agent failure. Delegate nothing without a check the agent can execute, and escalate the check as autonomy grows: prompt, then goal condition, then a blocking hook.
• Correcting the same mistake a third time in one session. After two failed corrections the context is polluted with dead approaches, and every further turn conditions on them. Reset and rewrite the opening prompt with what you learned.
• Loading every MCP tool definition up front. Tool schemas are context. Prefer progressive disclosure, and drop servers whose function a CLI already covers; Anthropic’s guidance calls CLI tools “the most context-efficient way to interact with external services.”
• Benchmarking the agent on your easiest task class. Saturated benchmarks flatter every configuration; SWE-bench Pro exists because the easier boards stopped discriminating. Hold out a hard set that resembles the work you delegate.
• Actioning every finding from an adversarial reviewer. A reviewer prompted to find gaps will find gaps. Anthropic warns that chasing them all “leads to over-engineering: extra abstraction layers, defensive code, and tests for cases that can’t happen.”

What to Measure Before Moving Budget

The rule above is built from public benchmarks; whether it holds on your backlog is an empirical question. These are the numbers I plan to collect on real workloads before moving budget on the strength of it, and the same set works as a standing dashboard for a team:

• First-pass verification rate: the share of tasks where the agent’s own check passed with no human turn. The single best harness metric.
• Human interventions per completed task, trended per task class: corrections, redirects, rewinds.
• Cost per merged PR: the one cost unit that survives a tokenizer or price change.
• Context utilisation at task end, and compactions per task: the leading indicator. A rising compaction count predicts a quality drop before the drop shows in output.
• Escalation rate: tasks that needed a tier bump to finish. Near zero on a frontier default means overpaying; high on short tasks means the harness is the constraint.
• Rework rate: agent-authored PRs reverted or reopened within 14 days.
• Cache hit ratio and batch share of non-interactive tokens: the two spend levers that never touch tier.
• Median wall-clock to green CI, per task class.

Two sprints of those numbers answer the flowchart’s questions with data.

Where the Rule Holds

The default, a mid-tier model with a closed verification loop and a deliberate context budget, covers the bounded, machine-checkable work that fills most sprints: a failing test to green, typed refactors, migrations across many similar files. It stops holding where METR’s curve says it stops: multi-hour horizons, novel design work, and failures that pick the wrong approach in the first step. There the model sets the ceiling, and tier is the spend that pays. Until your own numbers exist, treat this as a calibrated starting point; the measurement plan above is the next step, and it is still ahead of me.

References

• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (arXiv) - The 2024 paper that made the agent-computer interface a first-class variable, with the SWE-bench and HumanEvalFix results cited above.
• Agentless: Demystifying LLM-based Software Engineering Agents (arXiv) - The three-phase localize, repair, validate pipeline that beat contemporary open-source agents on score and cost without an agent loop.
• mini-swe-agent (GitHub) - The roughly 100-line control loop the standardised SWE-bench bash-only track runs on, with the README statements quoted above.
• SWE-bench Verified leaderboard - The official board, including the bash-only track that holds the harness fixed so models can be compared cleanly.
• The SWE-bench February 2026 update (Simon Willison) - The write-up behind the February 2026 bash-only figures, the 9,000-line harness description, and the note on what a shared system prompt excludes.
• Terminal-Bench 2.1 leaderboard - The board that publishes agent and model as separate columns; source of the six-row table above.
• Stop Comparing LLM Agents Without Disclosing the Harness (arXiv) - The position paper behind the Binding Constraint Thesis, its 3x3 factorial study, and the harness disclosure protocol.
• Holistic Agent Leaderboard (arXiv) - The 21,730-rollout evaluation behind the finding that higher reasoning effort reduced accuracy in most runs.
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv) - The contamination-resistant long-horizon benchmark; in the v2 results, every evaluated model stays below 45% Pass@1 under a unified scaffold.
• Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures (arXiv) - The thirteen-scaffold analysis showing where agent designs converged and where context management still diverges.
• Effective context engineering for AI agents (Anthropic) - The definitions of context engineering, compaction, structured notes, and sub-agent architectures cited above.
• Best practices for Claude Code (Anthropic) - The verification-loop escalation path, the context-management guidance, and the named failure patterns quoted above.
• Code execution with MCP (Anthropic) - Progressive tool disclosure, including the 150,000-to-2,000-token worked example.
• Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma) - The 18-model study showing reliability degrading as input grows, well before the advertised window limit.
• Time Horizon 1.1 (METR) - The time-horizon estimates and doubling times that define the regime where model tier sets the ceiling.
• Task-Completion Time Horizons (METR) - The continuously updated horizon page behind the twelve-hour frontier estimate in the decision-rule section.
• Aider LLM leaderboards - The polyglot board that tracks edit format alongside score and cost; source of the November 2025 snapshot.
• Claude API pricing - List prices, cache and batch multipliers, and the tokenizer note that breaks cross-generation per-token comparisons.