There is a category of failure that no error log will ever show you, because nothing fails. The API responds. The value parses. The report renders. And the answer is wrong in the specific way that ends careers: it is wrong while looking authoritative.

We hit it three times in one season, building an evidence engine for Microsoft 365 governance. Each time the mechanism was different, each time it is reproducible, and each time the fix was structural rather than a patch. This is the long version, with the code.

Shape one: the default that reads as a finding

SharingCapability is the property that tells you the most permissive external sharing a SharePoint site allows. Its four values, in order of exposure:

Text
Disabled                          no external sharing
ExistingExternalUserSharingOnly   guests already in the directory
ExternalUserSharingOnly           new and existing guests, sign-in required
ExternalUserAndGuestSharing       Anyone links: no sign-in, no identity

The obvious way to inventory a tenant is to enumerate every site once and read the property off each:

PowerShell
# The tempting one-liner. It is also wrong, and Microsoft says so.
Get-SPOSite -Limit All | Select-Object Url, SharingCapability

Here is the note Microsoft puts on the Get-SPOSite reference, quoted verbatim:

If the Limit or Filter parameters are provided then the following site collection properties will not be populated and may contain a default value: AllowDownloadingNonWebViewableFiles, AllowEditing, [...] DefaultLinkPermission, DefaultSharingLinkType, [...] SensitivityLabel, [...] SharingCapability, SharingDomainRestrictionMode.

Twenty-three properties. The moment you pass -Limit or -Filter, none of them is read; each returns its type default. SharingCapability is a .NET enum, its default is the zero member, and the zero member is Disabled.

Read that chain again slowly. The bulk enumeration, the first thing every inventory script reaches for, returns the most locked-down value in the vocabulary for every property it did not populate. Not null. Not an error. The safest possible word, on a site it never actually looked at.

The correct read is per site, by identity:

PowerShell
# One site, actually populated. Note the admin endpoint: sharing capability
# is a tenant property about a site, not a property of the site object.
Connect-PnPOnline -Url https://contoso-admin.sharepoint.com -Interactive -ClientId $appId
(Get-PnPTenantSite -Identity https://contoso.sharepoint.com/sites/finance).SharingCapability

The two disagree, and they disagree silently. We reproduced it against a real tenant and one site in five differed: enumeration said Disabled, the direct read said ExternalUserAndGuestSharing. A site handing out Anyone links, reported as sealed, by a correct API doing exactly what its documentation says.

A dashboard built on the first snippet shows a wall of green. Every tile is a working API call. Every tile can be wrong.

The engineering answer is not "always remember to read per site". Humans do not reliably remember. The answer is that a value read through a path known not to populate it must be recorded as not evidence, structurally, so that nothing downstream can consume it:

PowerShell
# From the collector. The enumerated value is never trusted; it is only
# used to obtain the list of URLs to read properly.
$urls = Get-PnPTenantSite | Select-Object -ExpandProperty Url        # the list, not the facts
foreach ($u in $urls) {
    $site = Get-PnPTenantSite -Identity $u                            # the populated read
    New-ScalarFact -Value ([string]$site.SharingCapability) -RawField 'SharingCapability'
}

The enumeration is demoted to what it is honestly good for (producing a list of addresses) and the fact is only ever taken from the read that Microsoft documents as complete.

Shape two: the provenance nobody checked

The second one we found in our own repository, which is the honest place to find it.

Every evidence document we ship carries a provenance block: when it was collected, by what, through which API. Twenty-seven fixtures declared this:

JSON
{
  "provenance": {
    "collected_at": "2026-08-05T14:02:11Z",
    "collector": "spo-collector",
    "source_api": "Microsoft Graph v1.0"
  }
}

The collector has never used Graph. It reads through PnP.PowerShell and CSOM; its tenant paths go through the SharePoint Admin API. Microsoft Graph v1.0 was copied forward from an assumption in week one and never questioned, because every check we had looked like this:

Python
# What the schema enforced: source_api is a string. It was a beautiful string.
assert isinstance(doc["provenance"]["source_api"], str)

The field the entire product exists to guarantee, how do you know this?, was false in twenty-seven documents, and every gate was green, because the gate checked the shape of the claim and never its truth.

The fix ties the claim to reality. There is now a gate that reads which collection paths the collector actually declares, and refuses any source_api that is not one of them:

Python
def test_no_fixture_claims_an_api_the_collector_never_uses():
    # The paths the collector really declares, read from its own source.
    paths = set()
    for f in [ORCHESTRATOR, *MODULES.glob("*.psm1")]:
        paths |= set(re.findall(r"-SourceApi\s+'([^']+)'", f.read_text()))
    # Every published source_api must be one the collector can produce.
    bad = [p.name for p in FIXTURES.rglob("*.json")
           if (api := load(p)["provenance"].get("source_api")) and api not in paths]
    assert not bad, f"fixtures claim an API the collector never uses: {bad}"

The gate's own first draft was wrong: it read only the modules and missed the orchestrator, so it accused correct SharePoint-Admin evidence of lying. That error is preserved in the commit history on purpose. A checker that was wrong once can be wrong again, and remembering exactly how is the cheapest insurance there is.

Shape three: the timestamp that dressed a construction

The subtlest. Our public example result showed a real rule failing against a document library, stamped Collected: 2026-08-05T14:02:11Z. The rule logic was right. The number was right. The library never existed: the evidence was a fixture, built by hand to exercise a code path, wearing a collection timestamp because the renderer printed one for every result.

Nothing on that page was false sentence by sentence. Assembled, it asserted something false: that this had been observed in a tenant. The gap between a true result about a construction and an observation of a tenant is the whole distance between a demo and a finding, and the page had erased it.

The repair is a registry that lives outside the evidence documents and classifies every fixture by origin:

JSON
{
  "path": "fixtures/sharepoint/list-over-limit.json",
  "origin": "synthetic",
  "may_be_presented_as_tenant_observation": false
}

And a test that refuses the evidence schema ever learning the word that would collapse the distinction:

Python
def test_the_evidence_schema_knows_nothing_about_fixtures():
    # `acquisition` says how REAL evidence arrived: collected | imported.
    # Teaching it `synthetic` would let a production Assessment validate a
    # construction. The classification lives in the registry, not the schema.
    assert "synthetic" not in json.dumps(load(EVIDENCE_SCHEMA))

Two questions, two homes: what the evidence is lives in the schema; what a file is for lives in the registry. The day a constructed document can validate as collected evidence, the difference is gone for everyone.

Why the confident wrong answer is the worse one

Three shapes, one root. In each case a system preferred producing an answer to admitting the limits of what it knew. The enum had a default, so it answered. The string validated, so it passed. The timestamp existed, so it read as observed.

The discipline that catches all three is uncomfortable, because it makes reports look worse before they look honest: unknown is a valid result, and it is the honest one whenever the evidence was not actually read. The engine has six outcomes, and two of them describe the machinery rather than the tenant.

Text
pass  fail  not-applicable  unknown  invalid-evidence  error

unknown means the evidence was not there to read: collect again, with the access you were missing. It is resolved by a fixed order that no engine is free to reorder: a required path in a missing, not-supported or permission-denied state forces unknown before any pass or fail can be reached. A site whose SharingCapability came from an enumeration is not compliant and not violating. It is unread, and the report must say so.

Now weigh the two failures. Missing evidence tells you where to look: "16 sites unreadable under this identity" is a named list and a next step. The confident wrong answer tells you to stop looking. It spends your attention budget on the sites that least need it, while the site reporting Disabled quietly hands out anonymous links to anyone who asks.

If you keep one habit from this: for every value on your dashboards, trace the reading path and ask what it returns when it cannot populate the field. If the answer is anything other than a visible refusal, you do not have a monitoring gap. You have a confidence machine, and it is pointed at the things you are least worried about.

Tags#governance#security#sharepoint#evidence

Comments