The green test that measured nothing

Any engineer distrusts a red build. The dangerous artefact is the green one, because green carries an implicit claim: this was measured, and it held. That claim has two failure modes and only one is famous. A test can be wrong about the system. A test can also be right about nothing, pointed at an artefact nobody ships, a file nobody loads, a promise nobody runs. It passes forever, and it protects nothing.

We spent a season collecting specimens of the second kind, in our own repositories, which is where such a collection should begin. Five of them, with the code, and the habit that would have caught each one earlier.

Specimen one: the suite that verified a ghost

Migrating a site between two renderers, our suite read the published output directory and checked hundreds of properties against it:

SWA = ROOT / "swa"                    # the directory the deploy uploads

def test_every_route_declares_its_canonical():
    for page in SWA.rglob("index.html"):
        assert '<link rel="canonical"' in page.read_text()

Green for weeks. Four hundred and twenty routes verified. The problem: swa/ was still being written by the old renderer. The new one, the entire point of the project, wrote to .build/astro/, and no test ever opened that folder. We were verifying the artefact we were replacing.

The cure was not more assertions. It was one question asked of the whole suite, which bytes does this actually open?, answered by making the suite compose the publishable artefact first, and adding a test whose only job is to fail if the published directory is not that composition:

def test_the_published_artefact_is_the_composition():
    # `data-page-owner` is written by the new renderer and nothing else.
    # If no published page carries it, swa/ is still the old site.
    owned = [p for p in SWA.rglob("*.html") if "data-page-owner" in p.read_text()]
    assert owned, "the published artefact is the old renderer; the composition never shipped"

Boring, structural, and the highest-value test in the repository, because it validates the one relationship every other test silently assumes.

Specimen two: the gate that read text, not behaviour

Our evidence collector carries a hard promise: read-only. It is enforced by parsing every PowerShell file and walking the syntax tree for mutating verbs:

def test_no_powershell_file_has_a_write_path():
    for f in COLLECTOR.rglob("*.ps1"):
        ast = parse_powershell(f.read_text())
        verbs = [c.command_name for c in ast.find_all(CommandAst)]
        assert not [v for v in verbs if v.split("-")[0] in MUTATING], f

A good gate. Also a gate over text. One release taught us the difference: a nested Import-Module -Force unloaded a shared helper from the caller's session, and the collector could not run at all, on no tenant and in no mode:

# The defect. -Force on a nested import re-imports the module fresh,
# evicting the already-loaded copy from the caller's session.
Import-Module (Join-Path $PSScriptRoot 'Evidence.psm1') -Force   # Initialize-Evidence now gone upstream

Every text gate stayed green, because the text was fine. The program the text described was broken. The fix was a second guard that loads the modules exactly as the collector loads them and asks the live session what survived:

Describe 'the modules load in the collector''s own order' {
    It 'keeps Initialize-Evidence reachable after the siblings import' {
        Import-Module ./modules/Evidence.psm1
        Import-Module ./modules/Sharing.psm1        # must not evict the above
        Get-Command Initialize-Evidence -ErrorAction Stop | Should -Not -BeNullOrEmpty
    }
}

The lesson generalises past PowerShell: for every property you enforce statically, ask what happens if the code passes the check and still fails to do the thing. If nothing would notice, your gate proves analysability, not behaviour, and it should say so out loud rather than borrow the authority of the stronger claim.

Specimen three: the published table that never ran

Our engine documents how it reasons over incomplete evidence: a table of operators and bounds, which side can decide, which side never can:

operator      lower bound proves    what it cannot prove
>  (min)      pass                  fail  (needs an upper bound)
not-contains  fail on known part    pass  (needs the whole set)
==            nothing               decides only when single and fully observed

It was in the architecture document, cited, polished, and, as a coverage report revealed, barely executed by any test. True the day it was written, unguarded ever after; any refactor could invert a row in silence and the document would keep promising the old behaviour with a straight face. So the table became executable, one case per cell, generated from the same source the document renders:

@pytest.mark.parametrize("operator,bound,expected", CASES_FROM_THE_TABLE)
def test_each_cell_of_the_published_table(operator, bound, expected):
    assert decide(operator, bound) is expected

def test_a_bound_of_none_decides_nothing():
    assert decide(">", bound=None) is Outcome.UNKNOWN

A published guarantee either runs in CI or it is a hope with typography.

Specimen four: the fixture that flattered the test

The quietest. A test evaluates a rule and expects unknown; it passes:

def test_denied_sharing_run_is_all_unknown():
    doc = load("coverage-denied-sharing.json")
    assert doc.counts.unknown == 13        # ← where does 13 come from?

The fixture was generated when a selection bug made the engine run every rule instead of the profile's two. The 13 was copied from that broken output. It was not a specification; it was a fossil of a defect, kept alive by a fixture directory nothing refreshed. We caught it only when the fixtures were regenerated from the current engine and the fossil disagreed with reality: the profile selects two rules, so the honest count was three, not thirteen.

The habit that catches it sooner is one comment:

    # Three, because `--profile sharing` selects three rules since SPO-SHARE-005.
    # It read thirteen for as long as a bare profile name failed open to every
    # rule on disk, and this fixture was generated then.
    assert doc.counts.unknown == 3

A pinned number you cannot source is a number you copied from whatever the system did the day you wrote the test, and the system that day may have been wrong.

Specimen five: the tests that proved the rules and never the wiring

The largest, and the closest to home. Sixty-five tests, all green, covered an API rule by rule: validation, rate limits, the moderation ceiling, the shape of a token. Not one of them opened the socket a request opens. Every test exercised a pure helper; none drove the HTTP handler that assembles those helpers into an answer. The rules were proven. The wiring between them was assumed.

Behind that green sat a mechanism that could not work in production. A newsletter edition records who already received it, so that a retry sends only to those it missed. The marker goes to a table named newssent:

// The whole idempotency of a send rests on this row, and nothing creates the
// table it lives in.
await marker().upsertEntity(
  { partitionKey: editionId, rowKey: key(subscriber.email) },
  "Replace",
);

Production had no newssent table. The first send would mail the edition, fail to write the marker into a table that did not exist, and report the recipient as a failure. Worse, the guard that reads the marker treats a missing table as "not sent yet", so every retry mails the whole list again. The idempotency was dead on arrival, and every test stayed green, because the in-memory double each test used answered createTable with a shrug and upsert with success. A fake cannot fail to exist.

What measured it was not another assertion. It was the same handler, driven against a real Table Storage running locally, with the persisted row read back:

// Real handler, real Table Storage, real OData. Only a table that can truly be
// absent exposes the missing create; only a real query proves the filter.
const res = await handler("newsletter-send")(admin(edition), ctx());
const written = await readAll(sent);       // read the state back, do not trust the return
assert.equal(written.length, res.sent);    // "sent" must imply "recorded"

The fix was one line the other stores already had, a defensive createTable before the loop. The defect is not the point. The point is that a suite can prove every rule a system owns and never prove that the system runs, and the gap stays invisible until a test opens the same socket, and the same storage, that production will.

One guarantee is worth naming here, because naming it is half the work. Delivery of a confirmed message to the inbox, and of a newsletter edition, is at-least-once: a crash between the send and its record repeats the send on retry, it never drops it. That is a choice. A duplicate is a nuisance; a lost message is a client. The test does not assert "exactly once", which would be a lie; it asserts that after a crash at each point the work is still recoverable, and nothing is silently skipped.

Green is a claim, not a state

Every passing test asserts a relationship between three things: the code, the artefact under test, and the world the artefact will meet. Each specimen broke a different leg. The suite measured the wrong artefact. The gate measured text instead of behaviour. The document promised what nothing executed. The fixture froze a broken world and named it expected. The last suite proved every rule and never ran them together.

The cure is one question in five disguises, and it is worth asking of your greenest suite today: if this stopped being true, would anything here go red? Take your most reassuring dashboard, your steadiest CI badge, and trace one green light down to the bytes it opens. If the trail ends at an artefact nobody ships, a promise nobody runs, or a number with no source, the light is not measuring the system.

It is decorating it.

Tags#testing#engineering#ci#havecode