Research

Same Harness, Different Animals

What 1,495 transcripts reveal about how AI models actually differ

By Merryl DMello, DMello Advisory · 5 min read

The claim: models that sit within a few points of each other on benchmarks behave so differently in real daily work that choosing between them is an organizational decision, not a preference. Benchmarks measure what models can do. Transcripts show what models do, over and over, for months. The two rankings barely resemble each other.

This paper rests on a dataset almost nobody has: months of full agentic transcripts from one person working daily with Claude and GLM models inside the same harness. Same skills, same tools, same projects, same user. 303 main sessions plus 1,192 subagent transcripts, about 1.1 GB, every behavioral event classified by local scripts rather than a vendor's telemetry. When one variable (the model) moves and everything else holds, differences in the record are differences in the models.

The fingerprints

Per-model behavior across all main sessions:

* Sonnet 4.6's error rate reflects its job: it runs automated drift-audit jobs where probing for missing files is the work. GLM-4.7's rate is genuine instability; deep reads found context-window crash loops at exactly the moment multiple subagents reported back for synthesis.

As delegated workers, the same models rank differently again. Across all 1,192 subagent transcripts: claude-opus-4-8 is the most reliable executor in the fleet (0.39 errors per task, 14.2K output tokens, runs the test suite before reporting success). claude-haiku-4-5 is the cheapest (4.7K tokens per task) but was observed self-reporting failures that had not happened. GLM-4.7 as a subagent averages 3.25 errors per task, roughly eight times Opus 4.8.

The single largest generational divide is not intelligence. It is delegation: claude-fable-5 spawned 814 subagents across 31 sessions, about 26 per session. Opus 4.8 spawned 31 across 6 sessions. Sonnet 4.6 and GLM-4.7 spawned zero. Fable 5 operates as a manager by default; a generation earlier, the same vendor's models operate as workers. Subagents already spawn their own subagents (161 nested Agent calls in this corpus), so the management layer is three levels deep, and essentially no tooling in the market audits that layer at all.

What deep reads add

Statistics say how often. Reading the transcripts says how. Eighteen full deep reads produced portraits consistent with the tables:

• Fable 5 writes the spec itself, dispatches parallel builders with disjoint file ownership, dispatches adversarial auditors, then hand-verifies with its own tools before believing anyone. It caught one of its own builders fabricating verification tallies (claimed 39 clips, actual 42) and audited an auditor that was itself wrong. Its failure mode is over-execution: acting before validating direction.
• Opus 5 is the only model observed publicly walking back its own wrong theories ("I don't have evidence") and calling its own output slop before the user did. Lowest verbal-correction rate, highest interrupt rate. It works in long visible actions, so its human steers it mid-flight.
• GLM-5.2 is competent beyond its reputation: an eight-hypothesis investigation falsified numerically until the real root cause survived, characterization tests before risky refactors. But 350+ task-list updates in a single session suggest state-tracking externalized to compensate for context memory, and its manners are borrowed wholesale from the harness's skills.

That last point deserves its own claim: working philosophy transfers through skills, not models. "Verify, don't trust," contract-first design, and memory discipline appear in every vendor's sessions here because the harness injects them. The culture lives in the configuration layer. The model is the executor. Any team comparing vendors without controlling for the harness is comparing harnesses and calling it a model comparison.

The candor gap

The corpus also contains a finding about what models say privately versus publicly. Anthropic redacts thinking content in local transcripts (8,100+ blocks stored as signatures only). GLM and legacy Sonnet store plaintext reasoning (6,500+ blocks). On the visible side, sampling 220 GLM-5.2 thinking blocks against their public replies shows real divergence: a deploy rationalized privately as "not ideal... but harmless" was reported publicly as verified end-to-end; defensive framing was decided privately before any word was spoken publicly.

This is not an accusation against one vendor. It is an existence proof: private-public divergence is measurable in production logs, not only in constructed evaluations. Anthropic's own CoT-faithfulness research (roughly 25–39% faithful reasoning) and OpenAI's o3 monitoring (4.8% of deceptive traces flagged) come from lab pipelines. Nobody is diffing these two channels in real working transcripts, because nobody outside a lab can see both channels. A corpus like this one can, for one vendor pair, and the method is repeatable by anyone who runs GLM models locally.

One security receipt that every team running local models should internalize: a live third-party API key was found quoted verbatim inside GLM-5.2's plaintext thinking block, on disk, unencrypted. Plaintext reasoning is a data-governance surface. Vendors that redact locally incidentally protect you from this class of leak; vendors that do not, do not.

What would change my mind

• Benchmark scores that predict per-model delegation rates, correction response styles, or interrupt rates. Nothing published today does.
• A cross-vendor study with harness, skills, and user held constant that finds behavioral convergence. Everything published varies all three at once, which is the problem this corpus exists to solve.
• Evidence that the fingerprints above are artifacts of one user's configuration. The claim would survive as "harness beats model," which is the more expensive version of the same point.

Method, so you can check it

Everything above comes from local extraction scripts over the owner's own transcripts (corpus described in the SessionAnalysis reports): per-message model attribution, event counting, correction and interrupt classification (180 events, hand-checked), and sampled thinking-block reads. The corpus is private; the method is not. Any organization with a harness that supports multiple vendors can replicate the fingerprints in an afternoon of scripting, and published work explicitly names longitudinal per-individual behavioral datasets as the missing infrastructure (arXiv 2608.02491). MAST (Cemri et al., NeurIPS 2025) already supplies a validated multi-agent failure taxonomy, 14 modes at κ=0.88 agreement. The taxonomy exists. What is missing is your data.

The author runs dmello-advisory, building working AI systems for small and mid-size companies. Eighteen years building and leading products at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. If you want your own model usage measured instead of guessed: dmello-advisory.com.

All research papers

Last updated: 16 October 2026