30-second answer

The engine ships 67 evidence fixtures. A registry outside the evidence documents, fixture-registry.json, classifies every one of them: 65 are synthetic (constructed by hand, every URL, GUID, date and name fabricated), and exactly 2 are sanitized-observation (read from a real tenant, then stripped of anything that could identify it). Only those two carry may_be_presented_as_tenant_observation: true. A result computed from a synthetic fixture is a true result about constructed evidence, and the classification is what keeps those two sentences apart.

What this proves

  • Which inputs were ever real. site-class-observed-20260810.json came from a real site read on 2026-08-10, with the tenant, URL and group GUID replaced. tenant-sharing-observed-20260808.json came from a sandbox run on 2026-08-08. Everything else is a construction that preserves shape, collection-state combinations and the product's own error text, and fabricates the rest.
  • That the engine's arithmetic is honest either way. A rule evaluated against list-over-limit.json produces the same fail it would produce against a real list with the same numbers. The result is true; the evidence was never observed anywhere.

What it does not prove

  • Anything about any tenant. 63 of the 64 SharePoint fixtures use the host contoso.sharepoint.com. A digest, a timestamp and a plausible URL make a construction look like a reading; the registry is the only thing that says it is not one.
  • That real evidence entered the repository. It never does: the engine's .gitignore refuses evidence/ and *.tenant.json, and the 63 real evidence documents from the first tenant run never entered git.

PowerShell

PowerShell
# Which shipped fixtures were ever near a real tenant?
$reg = Get-Content .\src\m365_governance\data\fixture-registry.json |
    ConvertFrom-Json
$reg.fixtures |
    Where-Object { $_.origin -ne 'synthetic' } |
    ForEach-Object { '{0}: {1}' -f $_.path, $_.origin }

Example output

Text
fixtures/sharepoint/site-class-observed-20260810.json: sanitized-observation
fixtures/sharepoint/tenant-sharing-observed-20260808.json: sanitized-observation

Explanation

The obvious design would put the classification inside each evidence document, next to acquisition. The engine refuses that on purpose: acquisition answers how real evidence reached an Assessment (collected or imported), and adding synthetic to that enum would let a production Assessment validate a construction. Two questions live in two places: what the document is for the engine, and what the file is for the repository. A test refuses the enum change by name, and a second test refuses any fixture that claims an API the collector has never used.

The distinction earned its place in public: the engine's example result was once displayed with a collection timestamp that read as a real tenant observation at a specific hour. The number was right, the provenance was a construction, and the difference between those two things is what an evidence-first product sells.

Production considerations

  • If you build your own fixtures for rule testing, register their origin somewhere the evidence schema cannot reach. The moment a constructed document can validate as collected evidence, every report you produce inherits the doubt.
  • Sanitising is replacement, not deletion: keep the shape, the real collection-state combinations and the error text, and fabricate every identifier. A fixture must be able to reproduce a defect without being able to identify anybody.
  • SPO-LIST-001: the rule whose fixture, list-over-limit.json, is the engine's public worked example, and a construction

References

Found something wrong? Suggest a correction. The article source is not public; the engine it cites is.