Research

Your Rules May Be Dead Weight

A personal AI harness with 100+ behavioral rules and zero measurements

By Merryl DMello, DMello Advisory · 4 min read

The claim: the context-file stack that serious AI users curate (CLAUDE.md, skills, rule files, memory) is deployed almost entirely on faith, the one controlled test of the premise that exists says it may not help at all, and most of it cannot even be checked because nobody instruments it. Until rules are measured, a rule library is a hypothesis library.

The evidence comes from three directions: a controlled 2026 ablation study, one fully mapped personal rule stack, and a correction record that shows rules failing in ways the rules were supposed to prevent.

The ablation nobody argues with

"Do Context Files Help Coding Agents?" (2026) ran a 288-run controlled ablation on context files like CLAUDE.md and found no measurable improvement in correctness. This is one study, on coding agents, with the usual scope limits. It is also the only controlled evidence in the field, and it points against the entire premise of the prompt-stack industry: curated context files, applied to real tasks, did not measurably outperform their absence.

The reaction from practitioners is always the same: "my rules obviously help." Maybe. Cursor-rules taxonomies and vendor evals lean both ways. The point is not that the study settles it. The point is that after two years of context-file practice, one 288-run study is nearly the whole evidence base, and it is negative. The default posture should be "unvalidated," not "obviously."

One rule stack, mapped

The corpus behind this paper contains a fully mapped personal instruction stack:

• a global CLAUDE.md used as a four-line router
• an imported process plugin: about 14 skills, 40 to 50 behavioral rules
• roughly 55 personal skills, of which about 8 do real behavioral work
• 40+ dated incident memory files across 25 projects, each recording one corrected behavior

More than 100 rules shape every session. Not one of them has a measured effect. The stack's owner is unusual only in having the incident files that make the gap visible: every rule carries provenance, and no rule carries an outcome.

Two receipts from the same corpus sharpen this. First, an imported process doctrine cost 45 minutes of ceremony before the first task of a project started; the fix was a home-grown override memory file. The rule was not just dead weight, it was negative weight, and it took a human's irritation to discover that. Second, the same correction types recur across projects even though rules exist against them: wrong assumptions about external state repeated across three projects by the same model family; premature handoffs three times inside one project by another. A rule that the same model violates three times in a month is not load-bearing. It is decoration.

There is one counter-example worth taking seriously, and it proves the rule: one skill library in this corpus treats skills as hypotheses. Mutate, test, score, promote or discard, with statistical promotion and human veto (430 tests). It is the only part of the stack where adoption is earned by measurement. The rest of the ecosystem, commercial and personal, works by vibes.

The experiment that would settle it

The corpus has something the ablation study lacks: natural experiments. Sessions exist with and without a given skill loaded, for the same user, same projects, same models, months of both conditions. The measurement is not hard:

1. Pick the rules claimed to matter most (verification gates, audit-before-complete, delegation patterns).
2. Define the behavioral delta each rule is supposed to cause (false "done" claims, rework rate, correction rate per project).
3. Compare matched sessions with and without the rule, dated cohorts, mechanical metrics where possible.
4. Report the effect, including zero effects. A rule with no measurable delta gets deleted, not defended.

This costs a few days of scripting over transcripts any heavy user already has. Nobody does it, because the industry sells rule curation (prompt libraries, skill marketplaces, rule packs) and measuring would threaten the product. The honest version of a skill marketplace answers "why should I install this" with an effect size. No marketplace on earth currently does.

What would change my mind

• A controlled replication of the 288-run ablation that finds large positive effects for personal context files. Then the interesting question becomes which rules, and my measurement argument stands anyway.
• Any published example of a personal rule stack with per-rule effect measurements. If one exists, it is well hidden.
• Evidence that recurring-rule-violations in this corpus were caused by rules failing to load rather than failing to bind. Distinguishable in transcripts (the rule text appears; the behavior still violates it). The deep reads found the latter.

The uncomfortable summary

Teams now spend real money curating instruction stacks whose effects are unmeasured, defending them with anecdotes, against a small controlled literature that says the effect may be zero or negative. The stack in this paper is over 100 rules strong and its owner could not tell you which 10 actually matter. Neither can anyone else about theirs. That is the claim, and it is aimed at the author's own stack first.

The author runs dmello-advisory and instruments AI workflows for small and mid-size companies: measured systems instead of vibes, working software instead of slideware. Eighteen years at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. Start here: dmello-advisory.com.

All research papers

Last updated: 16 October 2026