A missing .NET trace does not establish that the operation never ran

An incident review searches for an operation, finds no trace and concludes that the operation did not happen. That conclusion contains an unstated premise: every relevant execution would necessarily have produced a retrievable trace.

Before accepting it, establish that premise. Was the operation instrumented? Was the relevant source being listened to? Was recording selected? Did export succeed? Does the query reach the right environment and retention window?

This is not an argument against tracing. It is a way to keep a powerful diagnostic tool from being asked to prove something its observation path cannot establish.

Separate work from its observation

.NET exposes ActivitySource and Activity for distributed tracing instrumentation. Creating an activity can return null when there is no interested listener. The application must still perform its work; optional tracing cannot become a prerequisite for business execution. Microsoft's instrumentation walkthrough

Here is a small console program using only the runtime APIs. It needs no exporter, Azure subscription, credentials or NuGet instrumentation package. Its two operations are synthetic; no customer transaction is being executed.

using System.Diagnostics;

using var source = new ActivitySource("Ph7x.LocalTraceExample");
int executions = 0;
int observations = 0;

void Execute()
{
    using var activity = source.StartActivity("example.operation");
    executions++;
    activity?.SetTag("example.sequence", executions);
}

Execute(); // Business work runs without a listener.

using var listener = new ActivityListener
{
    ShouldListenTo = candidate => candidate.Name == source.Name,
    Sample = (ref ActivityCreationOptions<ActivityContext> options) =>
        ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = activity => observations++
};
ActivitySource.AddActivityListener(listener);
Execute();

Console.WriteLine($"Executions: {executions}; observed activities: {observations}");

Run it in a console project with a currently supported .NET SDK, replacing Program.cs with the code above. The local verification for this article used .NET SDK 10.0.402. The observed output was:

Executions: 2; observed activities: 1

The counter establishes that both calls reached the increment. The listener establishes that one completed activity reached that callback. There is no exporter here, so the result establishes nothing about ingestion or persistence in a telemetry service.

This boundary makes the example useful: it disproves the general claim “no trace means no execution” without requiring an explanation of any particular production incident.

Sampling changes what a search represents

OpenTelemetry distinguishes head sampling, decided early in a trace, from tail sampling, which can consider more of a trace before deciding. Sampling reduces the trace data retained or processed; its configuration is part of the explanation of what a query can retrieve. OpenTelemetry sampling concepts

A retained trace can show a concrete execution path. An empty search does not automatically establish the absence of a path. Nor does a list of retained traces automatically enumerate every request. Those are stronger statements requiring a stronger observation contract.

For an incident, our recommendation is to write down the applicable sampling policy alongside the query, rather than discovering it only after someone has used a count as a total. Preserve the environment, service identity, time window and filters as well. A query without its scope is difficult to reproduce and easy to overinterpret.

Follow the observation through each boundary

Treat the telemetry path as a sequence of independently testable stages:

The table is a troubleshooting method, not a guarantee that every backend exposes every stage. Where a stage is not observable with the current authority, record that limitation. Do not fill the gap with a guessed success state.

A practical first check is a controlled local operation with a known identifier that carries no customer data. First observe it before the exporter; then look for it at the destination. Change one boundary at a time. If it disappears, the difference narrows the investigation. It does not by itself identify which component is at fault.

Avoid repairing a diagnostic gap with sensitive data

The temptation is to attach the entire request or customer identity to every span. That can make one query easier while creating a different operational problem.

Prefer a non-sensitive correlation value and attributes that answer a defined diagnostic question. Review retention, access and cardinality before increasing what is emitted. This is an engineering recommendation; the acceptable fields and handling rules belong to the application's established data policy.

Similarly, do not alter production sampling during an investigation merely to make a chart look fuller. A controlled experiment should name the change, its duration, its operational cost and the observation it is meant to establish.

Write the conclusion at the strength of the evidence

“We found no matching stored trace in this query” is reproducible. “The operation never ran” requires establishing that the observation mechanism would have captured it and that the search could have found it.

Use the stronger statement only when those additional premises have been proved. Otherwise retain the narrower one and name the next measurement. That discipline is also why licence assignment and usage alone cannot prove that a licence is safe to remove: a missing observation and an absent dependency are different facts.

Tags#dotnet#azure#observability