Research

Your Logs Are the Evidence and the Leak

The only honest dataset about your AI operation, and the two ways you are losing it

By Merryl DMello, DMello Advisory · 3 min read

The claim: every company running AI-assisted work is generating the only dataset that can ever answer "is this actually working for us," and the default settings lose it in one of two opposite ways: they delete it before it accumulates, or they keep it with live secrets inside it. Both failure modes are documented in one long-running corpus, with dates.

The corpus is one operator's complete Claude and GLM session transcripts: 303 main sessions, 1,192 subagent transcripts, roughly 1.1 GB, analyzed by local scripts.

Failure mode one: the evidence evaporates

The harness behind this corpus retains transcripts for about 30 days by default. This is not a hypothetical retention policy; it already bit. When a second analysis round started, the entire six-month history analyzed in the first round was gone. One month had 3 surviving sessions; everything else was from the most recent weeks. The earlier era now exists only as two analysis reports; the underlying evidence is unrecoverable.

That is the standard state of AI adoption: every team is running a natural experiment on its own workflows and destroying the results monthly. Published research explicitly names the missing piece: one 2026 paper (arXiv 2608.02491) calls longitudinal per-individual behavioral datasets and metrics the absent infrastructure. Anthropic's Clio analyzed real conversations at population scale. A 20,574-session failure-taxonomy study worked across many users. Nobody can study one team, over time, with full traces, because nobody keeps the data.

The fix in this corpus took an afternoon: a daily archive job with additive-only copies and retention raised from 30 days to 10 years, plus a prompt-history file that survives transcript deletion entirely (10,436 prompts, snapshotted daily). From that day forward, nothing is lost and the longitudinal axis is unbounded. The cost is a few gigabytes a year.

Failure mode two: the evidence leaks

The same transcripts carry the opposite risk. Reasoning content, the model's private working notes, is stored differently by vendor: one provider redacts it locally (8,100+ thinking blocks stored as signatures only, in this corpus), while others store full plaintext (6,500+ blocks, averaging 4.3K characters each).

Plaintext reasoning is a data-governance surface. The receipt: a live third-party API key was found quoted verbatim inside a model's plaintext thinking block, on disk, unencrypted. The key was valid. It had been picked up from the environment and reasoned about, and the reasoning was written to permanent storage.

The same channel holds everything else a compliance officer would flag: live emotional-state tracking of the user, private assessments of individuals, unfiltered intermediate conclusions that never appear in public output. A company that archives transcripts for years without a secret-scanning step is building a slowly compounding incident report.

The uncomfortable conclusion

The two failure modes push in opposite directions, and that is the trap. Deleting logs feels safe and destroys your only evidence. Keeping logs feels diligent and preserves your leaks. The resolution is not a policy choice between them; it is a pipeline: archive additively, scan for secrets before anything enters the archive, redact or tokenize what must not persist, and keep the scanned archive forever.

For German companies the framing is sharper: unstructured AI logs full of personal data and credentials are a DSGVO exposure and a business asset in the same file. Deciding which one they are, deliberately, is cheaper than having both decided for you by a retention default.

What would change my mind

• A major AI provider that archives full transcripts long-term by default, locally, with documented secret handling. As of this corpus, none does; the 30-day reaper and the plaintext thinking block coexist in the market.
• Evidence that companies can reconstruct AI-workflow effectiveness without transcripts, from telemetry alone. Telemetry records that a tool was called; transcripts record why, what was decided, and what it replaced.
• A published case of an organization running exactly this pipeline (additive archive, scan, redact, retain) as standard practice before 2026. The corpus's own archive was built in 2026 precisely because no template existed.

The author runs dmello-advisory and builds evidence pipelines for AI-assisted companies: archives that keep what matters, scanning that keeps out what does not. Eighteen years building and leading products at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. dmello-advisory.com.

All research papers

Last updated: 16 October 2026