A readiness probe that checks the database

An ASP.NET Core API runs as four instances behind a load balancer. Its health endpoint connects to the API's SQL database, because a health check that ignores the database feels like one that checks nothing. The database becomes unreachable for twenty minutes, and all four instances fail their probes together: they are asking one database the same question. On Kubernetes, with the same endpoint serving as the liveness probe, every container is restarted, then restarted again after growing delays. The API could have served cached reads and asked other callers to retry. For twenty minutes and then some, it served nothing.

Each part worked as designed. A probe answers one question for one consumer, and a probe that checks a dependency every instance shares turns that dependency's failure into every instance's failure at once. The consumer decides what follows. Kubernetes takes every Pod out of its Services, or restarts every container, and a degraded service becomes an outage. App Service declines to remove every instance and, an hour later, starts replacing them one at a time, which no new instance can fix. A probe that checks nothing fails the other way: it proves that the process answers, and nothing about whether a request will succeed.

This analysis covers ASP.NET Core applications on Azure App Service or Kubernetes. Microsoft's documentation and the Kubernetes documentation were checked on 7 November 2026. No tenant or subscription was accessed.

Who acts on a probe

Kubernetes defines each probe by the action that follows it (Liveness, Readiness, and Startup Probes). "Liveness probes determine when to restart a container." "Readiness probes determine when a container is ready to accept traffic", and a Pod that fails one is removed from its Services while its container keeps running. A startup probe holds both back until it succeeds.

Sources for the table: Liveness, Readiness, and Startup Probes and Monitor App Service instances by using Health check.

Health checks in ASP.NET Core says it before any code: "The monitoring system dictates what types of health checks to create and how to configure their endpoints." Our reading: a check asks a consumer to act, and belongs on a probe only if that action helps.

What App Service does with an unhealthy instance

App Service pings one path you choose, on every instance, once a minute; there is no default (App Service Health check, above). An instance that does not answer "between 200 and 299 (inclusive) after 10 requests" is removed from the load balancer, and WEBSITE_HEALTHCHECK_MAXPINGFAILURES lowers the count to as few as 2. An instance that "remains unhealthy for one hour" is "replaced with a new one". On scale-out, App Service pings the same path "to ensure new instances are ready".

Two rules apply when many instances fail together (App Service Health check, above). Removal is guarded. By default "no more than half of the app's active instances are excluded from the load balancer at a time", and when every instance is unhealthy, none is removed, because that "would effectively cause an outage for your application. However, the instance replacement still occurs." Replacement is rationed: one instance per hour and three per day per App Service plan, under a "nonconfigurable limit" per scale unit that resets every 12 hours.

Microsoft's advice for the path is direct (App Service Health check, above): "if your application depends on a database and a messaging system, the Health check endpoint should connect to those components". It should return a 500-level code when it cannot connect, and 200 "only when the app is fully warmed up". Because of the guard, that advice is safer here than elsewhere: an instance that alone cannot reach the database leaves rotation, and an outage that fails every instance removes none. Our reading of what remains: an outage longer than an hour replaces instances that were never faulty, from a daily allowance of three that a broken instance may need later, and a failure that reaches some instances before others can still, by default, remove up to half the capacity.

When every instance fails at once

The Kubernetes pages we read describe no such guard. Every Pod that fails readiness leaves its Services, and the only fallback in EndpointSlices is for Pods being deleted: proxies "may route traffic to endpoints that are both serving and terminating if all available endpoints are terminating". A Pod that is not ready does not qualify, because serving "maps to the Pod's Ready condition". When a shared database fails every Pod's probe, the Service has no ready endpoint left.

Kubernetes offers the dependency check as an option: the readiness probe "additionally checks that each required back-end service is available. This helps you avoid directing traffic to Pods that can only respond with error messages" (Kubernetes probes, above). Microsoft's AKS guidance takes the database case: an unavailable dependency "causes the readiness probe to fail for all the pods in your service", and the removal "creates cascading failures upstream"; it prefers "retry handling within your service" (Microservices architecture on AKS). The Health Endpoint Monitoring pattern calls the result "a cascading failure where the monitoring mechanism reduces availability".

Our reading of the disagreement: the consumers differ. Removing a Pod helps when another Pod can do better, and when the database has failed for all of them, none can. A probe cannot tell its own instance's fault from a shared one; only a consumer that sees every instance can, and App Service's guard is that consumer treating a failure everywhere as shared. On Kubernetes, the decision falls to whoever writes the probe.

Liveness is worse, because its action is a restart. A liveness probe must "truly indicate unrecoverable application failure, for example a deadlock" (Kubernetes probes, above), and "only helps if restarting the pod is likely to restore it to a healthy state" (AKS microservices, above). With the defaults, our arithmetic says a liveness probe on the database restarts every container within about thirty seconds. Failing liveness probes can put a container into CrashLoopBackOff, where restarts back off "(10s, 20s, 40s, …)" up to five minutes (Pod Lifecycle). Our reading: a container may still be waiting when the database returns, so the application's outage can outlast the database's, and the containers come back with empty connection pools at once: the "synchronized login bursts" that Microsoft's SqlClient pooling guidance says to avoid.

Kubernetes also calls it a common pattern to give liveness "the same low-cost HTTP endpoint as for readiness probes, but with a higher failureThreshold" (Kubernetes probes, above). Our reading: that is safe only while the endpoint checks nothing a restart cannot repair.

What a database check proves, and what an empty one proves

ASP.NET Core's default is one rule: "The app is considered healthy if it can respond at the health endpoint URL." Microsoft's liveness endpoint filters every check out, so it "reports a Healthy status for all calls" (ASP.NET Core health checks, above). That suits liveness, and proves nothing about the requests an instance will receive; as an App Service path, it would pass a new instance before it is warm.

A database check proves less than it seems. AddDbContextCheck calls EF Core's CanConnectAsync by default (AddDbContextCheck), which connects "in the normal way" and notes that "being able to connect to the database does not mean that it is up-to-date with regard to schema creation, etc." (CanConnectAsync). SqlClient pools connections by default. "The pool doesn't send a validation query before every checkout because that round trip removes much of the pooling benefit" (SqlClient pooling, above). Our reading: the instance's own pool may answer the check. Microsoft still prefers a connection to a query, which "runs the risk of overloading the database" (ASP.NET Core health checks, above).

There is one failure only a database check catches. Pools "aren't shared across application instances, containers, or hosts", and when every pooled connection is busy, an open waits for Connect Timeout, 15 seconds by default, then fails (SqlClient pooling, above). A pool exhausted by a leak fails one instance's check and no other's, and removal helps. The timeouts also disagree: Kubernetes gives a probe 1 second by default (Kubernetes probes, above). Our reading: a database that is slow rather than down can fail every Pod's probe while requests with longer timeouts still succeed.

Keeping the questions apart in ASP.NET Core

ASP.NET Core separates checks with tags and serves each probe from its own endpoint (ASP.NET Core health checks, above). The registration below is illustrative and was not run. It follows Microsoft's two-endpoint example and adds the EF Core check:

builder.Services.AddHostedService<StartupBackgroundService>();
builder.Services.AddSingleton<StartupHealthCheck>();

builder.Services.AddHealthChecks()
    // Instance-local: has this instance finished starting?
    .AddCheck<StartupHealthCheck>("startup", tags: new[] { "ready" })
    // Shared: every instance asks the same database.
    .AddDbContextCheck<AppDbContext>("database",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "ready" });

var app = builder.Build();

// Liveness: runs no checks.
app.MapHealthChecks("/healthz/live", new HealthCheckOptions
{
    Predicate = _ => false
});

// Readiness: the checks tagged "ready".
app.MapHealthChecks("/healthz/ready", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("ready")
});

AppDbContext is registered as usual; StartupHealthCheck and StartupBackgroundService are Microsoft's example classes. An endpoint's status is "the worst individual result" of its checks, so unfinished startup makes readiness Unhealthy, a 503 by default, while Degraded "returns HTTP 200, so a load balancer or orchestrator that only checks the status code still treats the app as available" (ASP.NET Core health checks, above). failureStatus sets what the database check reports when it fails (AddDbContextCheck, above), so a failed database shows in the report and moves no instance.

That is a choice with a cost: an instance whose own pool is exhausted also reports Degraded and stays in rotation. With the default failure status, the same line removes that instance and, on Kubernetes, every other one when the database fails.

What we would check where

Two questions decide whether a check belongs on a probe. Does the action fix it? A restart can clear a deadlock or a leak, but restarting a process "doesn't repair a failed shared dependency" (Health Endpoint Monitoring, above). Does the alternative share it? Removal helps only if the consumer can send the request somewhere that does not depend on the failed thing. A database shared by every instance fails both tests at the level of instances, and can pass the second at a larger one: a router choosing between regions that have their own databases has a better destination. The same pattern makes failover depend on "redundant application capacity across the failure boundary".

App Service's one path does the work of three probes: it gates new instances, removes failing ones and replaces those that stay unhealthy. Its guard answers the second question for you, so we would follow Microsoft and connect to the database, accepting that an outage longer than an hour spends replacements. We would decide that in advance, because "Health check configuration changes restart your app" (App Service Health check, above).

What we recommend, in order

These are our recommendations, not Microsoft's.

1. Name the consumer of every health endpoint and its action on failure: restart, removal, replacement or an alert. An endpoint with two consumers answers to the more destructive one.
2. Keep dependencies off liveness and startup. Microsoft's pattern agrees for liveness, so that "a dependency outage doesn't trigger unnecessary application restarts" (Health Endpoint Monitoring, above). Startup work that needs the database retries inside the application and reports not ready, as Microsoft's example does with a configuration download (ASP.NET Core health checks, above).
3. On readiness, check what only this instance can get wrong, and settle each shared dependency, removal or Degraded, with the two questions above.
4. Prefer a connection to a query, bounded below the probe's own timeout: 1 second by default on Kubernetes, a minute on App Service.
5. Watch shared dependencies with a consumer that only reports: a health check publisher, an availability test, the dependency's own metrics.
6. Let the application degrade, not the platform. Fail the requests that need the database quickly, and serve the rest. On App Service, Microsoft notes, a shared outage "can therefore require application-level degradation or recovery rather than instance removal" (Health Endpoint Monitoring, above).
7. Test the outage. Take the database away in a test environment for over an hour; count removals, restarts and replacements, and time the recovery.

What this does not settle

• What App Service does with a new instance that never passes. Microsoft documents the ping "to ensure new instances are ready", not the outcome when one never is.
• The size of the per-scale-unit replacement limit. Microsoft documents its existence and its 12-hour reset, not its value.
• Whether CanConnectAsync reaches the server when the pool holds an idle connection. The EF Core and SqlClient pages we read do not say.
• What a check reports when its registration's timeout expires. The HealthCheckRegistration.Timeout reference says only that it is "the timeout used for the test".
• What a client receives when a Kubernetes Service has no ready endpoint. It depends on the proxy or ingress in front, which the probe pages do not describe.

Where this fits

• A retry that sends a POST twice: the AKS guidance answers a flaky dependency with retries inside the service, not with the probe (AKS microservices, above). That article covers which retries are safe.
• The green test that measured nothing collects signals that were green because they pointed at nothing. A liveness endpoint that runs no checks is one by design; used for readiness, it is one by accident.
• A missing .NET trace does not establish that the operation never ran separates what ran from what was recorded. Health check requests "won't show in the web server logs" (App Service Health check, above), so a quiet log establishes nothing about the probe.

Sources, checked on 7 November 2026

• Azure App Service: Monitor App Service instances by using Health check
• ASP.NET Core and .NET: Health checks in ASP.NET Core, AddDbContextCheck, HealthCheckRegistration.Timeout, DatabaseFacade.CanConnectAsync, SQL Server connection pooling with Microsoft.Data.SqlClient
• Azure Architecture Center: Health Endpoint Monitoring pattern, Microservices architecture on Azure Kubernetes Service
• Kubernetes: Liveness, Readiness, and Startup Probes, Configure Liveness, Readiness and Startup Probes, Pod Lifecycle, EndpointSlices

Tags#azure#dotnet#observability#engineering