Prove the surface before you trust it

The rule that does the most work: a negative result is admissible only if the same capture surface first showed a known-good signal. Every run plants a control token that must come back. If the control does not come back, the surface is blind — and a blind surface’s negatives are discarded, not reported with a caveat. A caveat is how a bad reading gets into a summary table anyway.

“The same surface” is the load-bearing phrase. A control that proves some other channel was working proves nothing about the channel the finding came through, and it is very easy to write that test by accident and feel rigorous.

The transcript is not ground truth for what the model saw

The obvious instrument is the session transcript: run the thing, grep the log, see whether the content showed up. It lies, and it lies in the direction that produces findings. Content injected as conversational turns is genuinely visible there. Content delivered through the system prompt is not serialised per turn — so grepping the transcript for it matches only the model’s own echo of it, and an echo is absent whenever the model simply had no reason to repeat itself. Read naively, that absence looks exactly like “the content was never delivered.”

The uncomfortable consequence, which I would rather state than hide: for some questions the model’s own self-report in its output is the only admissible evidence available. That is a weak instrument. It is still better than a strong-looking instrument that is measuring the wrong thing.

It also forces a distinction between two kinds of probe that look interchangeable. Asking “did it repeat this token?” cannot separate read it and had no reason to say it from never received it — two different states of the world, one observation. No amount of extra sampling resolves that, because the ambiguity is in the design rather than in the noise. The fix is to make the planted sentinel an instruction that only a model which actually read the content could satisfy, so that compliance is evidence and silence is not required to mean anything.

And the scored token is always a nonce, never a real word, with a standing rule that the harness’s own artifacts — the driving prompt, the runbook, a docstring — must never contain the token being scored. Otherwise the grep eventually matches the harness quoting itself and reports it as behaviour. Measurement apparatus contaminating its own measurement is an ordinary laboratory failure; text-based evaluation just makes it a one-line mistake.

A fixture where only one thing is allowed to vary

The routing question is: given a set of tools, does the model call the right one? To make the answer mean anything, every difference between tools except the one under test has to be removed. So the generated tools all share one identical trivial schema, and each is named positionally rather than descriptively — which leaves the description as the only channel carrying signal. You can check that the naming actually achieves that, and check the seam the fixture is injected through, without any of my code:

Removing the confound is the easy half. The half that decides whether a measured collapse is real is asserting the fixture’s validity in both directions before the run — so that a “collapse” can never turn out to have been a lexical accident:

Each of those refuses to generate rather than warning, because a fixture that is invalid in this way still runs fine and still produces a number. Alongside them sits a control tool that is always present and is invoked by name rather than by description match, so that a cell where nothing routes can still demonstrate the plumbing was connected — the positive-control rule from earlier, applied one level down.

One honest note about that first reproduction, because the distinction matters more than it looks. The generator also carries an assertion that tool names must be neutral — and under the positional scheme that assertion can never fire, exactly as the command above shows. Neutrality is a property of the construction, not something the guard established; the guard is a tripwire for whoever changes the naming scheme later. “A check proved this” and “the construction makes this true and a check watches for drift” are different claims, and only the second one is true here.

Instrument failure and model failure are different outcomes

If a harness bug can present as a model failure, the whole measurement is unfalsifiable — every improvement to the harness looks like the model getting better. So the scorer separates “I failed to observe” from “I observed a failure” as an explicit output bucket rather than a footnote:

That last row is the one I would defend hardest. The temptation with an exclusion bucket is to let it grow, because every exclusion makes the numbers cleaner — and an empty answer looks like a null result. It is not one. The run completed, the model answered, the answer was nothing. Exclusion criteria have to be written against what the instrument did, never against how unhelpful the result was.

The automatable mode is blind to the failure that matters most

The cheap way to run any of this is headless — one non-interactive invocation per trial, scriptable, parallel, free of a terminal. Most of the axes run that way and produce real failures. But in that mode a capability resolves only when it is named explicitly, which means the model never gets to decide whether to reach for it. So the question “does this fire on its own?” is not merely hard to answer headlessly. It is unaskable — the mode has removed the behaviour under test.

Which is why the harness has a third layer it would rather not have: a driver that runs real interactive sessions through a pseudo-terminal, so that auto-firing can be observed at all. It costs real sessions, so it cannot run freely the way the structural probes do. Accepting that cost was the correct call and it was not a comfortable one.

The transferable warning is aimed at anyone else’s CI. Pressure that shortens a capability’s description can stop it firing on its own, while every explicit-invocation test keeps passing. A green headless suite is consistent with the feature being silently dead in normal use. The general shape is worth looking for in any evaluation you own: when the mode you can afford to automate is structurally incapable of exhibiting the failure you most fear, coverage figures from that mode are measuring your budget, not your system.

Grading, and grading the grader

Scoring open-ended output eventually wants a model in the loop, which is circular: a non-deterministic judge rating a non-deterministic actor. The structure that makes it tolerable is to keep the deterministic graders primary and treat the model judge as the noisy instrument it is — never the sole vote on a question, and itself subject to measurement.

Two details carry most of the weight. The judge’s reply is parsed structurally — scanning for brace-balanced spans with string and escape awareness — rather than by pattern match, because a nested object can otherwise shadow its parent and invert a verdict while producing perfectly well-formed output; that is the failure mode you never notice, since nothing looks broken. And the judge is calibrated against human labels rather than assumed: agreement between the judge and a hand-labelled sample is computed as its own artifact, and the reliability figure the aggregation depends on is replaced with a measured one instead of a guessed one.

The reason I trust that layer, though, is the thing its own documentation says against it: with a single grader per question the reliability estimate is trivially perfect and the weighting is inert, and that is written down in the code that implements the weighting rather than in a backlog somewhere. A measurement layer that documents where its headline mechanism does nothing is one I will believe about the cases where it does.

Written to be quoted carefully

The harness’s documentation ends with a section headed read before quoting results, and it exists because I expected to be the person who over-quoted them. It says the measurements are single-needle retrieval rather than synthesis over ambiguous real queries; that per-cell sampling is small; that the fixtures are deliberately clean, so they demonstrate mechanisms rather than any real corpus’s ambiguity, and should be replicated against a slice of your own material before anyone draws a hard conclusion; and that every finding is tied to the model and runtime versions it was produced against.

That last one is why no finding appears on this page. Results from a harness like this have a shelf life measured in releases, and a portfolio page has no refresh path — which is exactly how a number that was true when written becomes a false claim nobody edited. The instrument discipline is the durable part; the readings are perishable.