Research

Stop Buying One Model for Everything

The corpus says model choice is procurement, not preference

By Merryl DMello, DMello Advisory · 4 min read

The claim: "which model should we get" is the wrong question, and the way it is usually answered, by benchmark rankings and vibe checks on chat quality, predicts almost nothing about what matters in daily work. One long-running corpus, eight models in regular rotation, same user, same projects, same harness, shows that the right question is organizational: which model, in which role, at which tier. The data to answer it is already in your own usage logs, and almost nobody reads them.

What the fleet actually shows

Across 1,192 delegated tasks, per-model behavior as a worker (errors per task, output tokens per task):

Three procurement-relevant findings fall out of this table, and none of them are visible on a benchmark leaderboard.

First: the most reliable worker is not the newest model. An older-generation model (opus-4-8) is the fleet's most dependable implementer: lowest error rate of any model in any role, reads the spec and the tests exhaustively, runs the test suite itself before claiming success. The newest model of the same family, meanwhile, is a manager by default: it spawned 814 subagents across 31 sessions, roughly 26 per session, while two other models spawned zero. Assigning the manager to implementation work is paying director rates for clerking, and assigning a worker-model to orchestration produces a worker without a plan.

Second: a high error rate can be the right buy. The cheapest model in the table carries a 3.25 error rate, about eight times the best. It ran in production for a month anyway, profitably, on high-volume low-stakes scheduled work: 80 reel summarizations, 60 classification runs, 18 more of another summarization job. The boundary was strict and never crossed: volume work to the cheap model, judgment work (a daily portfolio analyst, a truth-drift auditor) to a model that had never produced a fabrication incident. Errors that a wrapper catches are cheap. Errors in judgment are not. Buying reliability where you need volume is the expensive mistake.

Third: token appetite varies four-fold at similar quality. Between 4.7K and 20.7K output tokens per task across models that all complete their work, the cost of the same task differs by a factor of four before you have written a single line of optimization. A team routing everything to one model pays that spread on every task, invisibly, forever.

Why the question stays wrong

The tooling encourages the mistake. Model routers on the market (NotDiamond, Martian, RouteLLM) route on benchmarks and aggregate signals; none of them learn from your own transcripts, your own correction rates, your own error-per-task counts. The evidence that would answer the procurement question correctly accumulates in every team's logs daily, and the default is to delete it in 30 days and re-ask the preference question instead.

The operator in this corpus eventually wrote the org chart by hand: the strongest model directs, cheaper models implement, the cheapest runs scheduled volume. Every line of it came from measured behavior, not a launch announcement. Any team can produce the same chart from its own logs in an afternoon of scripting, which is cheaper than a year of over-provisioned subscriptions.

What would change my mind

• A benchmark that predicts delegation rate, correction response, or errors-per-delegated-task. Nothing published today does; that is the finding of the fingerprint paper in this series.
• A single-model deployment that matches a tiered fleet's cost and quality on mixed workloads, sustained for months. Every measured comparison I know of varies the model and the task together, which is the error this paper argues against.
• Evidence that model-router services using aggregate signals outperform an org chart learned from the buyer's own history. The routers exist; the comparison does not.

The author runs dmello-advisory and designs model fleets for small and mid-size companies: the right model in the right tier, measured against your own logs instead of a leaderboard. Eighteen years building and leading products at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. dmello-advisory.com.

All research papers

Last updated: 16 October 2026