Readiness, draining and already accepted work

28 min guided lesson · Read, predict, then run

External .NET10 SDK10.0.403, real Kestrel and HTTP driver. Experiments are limited to owned loopback processes. SQLite10.0.14 is a local-file classroom ledger, not a network production database.

Read or download the complete SDK project, then follow its setup on your computer. The separate C# companion practice below runs in SharpRune’s .NET 9 console environment.

Readiness and accepted work answer different questions

Readiness says whether a replica should receive new traffic. Liveness asks a different question about keeping the process running. Draining additionally needs a policy for work the application already accepted. The testbed exposes /ready, /live and a Development-only POST /lab/drain so those responsibilities can be observed separately. Drain changes an admission flag under a shared admission lock; it does not kill Kestrel, cancel every accepted item, or notify a real load balancer in this desktop experiment. An already submitted WorkItem remains owned by its lane until it completes or its own lifetime is canceled.

Follow the actual admission and shutdown code

Handle creates an owned item and asks LabRuntime.Admit for one atomic admission decision. Admit checks Accepting and calls the channel’s TrySubmit under the same admissionSync lock that Drain uses to clear the flag. No await occurs inside that lock. Admission therefore has an explicit order relative to drain: already accepted work remains owned, while an admission after the transition is rejected with503; a full accepting lane returns429 instead. /ready reads the flag and /live remains independent. Separately, WorkLane.StopAsync completes the writer and awaits workers, canceling their stop lifetime if the shutdown budget expires.

public void Drain() { lock (admissionSync) Volatile.Write(ref accepting, 0); }
    public AdmissionDecision Admit(WorkLane lane, WorkItem item) {
        lock (admissionSync) {
            if (!Accepting) return AdmissionDecision.Draining;
            return lane.TrySubmit(item) ? AdmissionDecision.Accepted : AdmissionDecision.Full;
        }
    }

Trace ownership without pretending to orchestrate Pods

The portable program enqueues one integer while accepting is true, switches the flag to false, refuses a second conceptual admission, and dequeues the item it already owns. Its exact lines are accepted, not ready, rejected and completed:1. Nothing in that program runs a probe, stops a process or updates endpoint discovery. It illustrates why stopping new admission need not discard existing ownership. The external project adds HTTP responses, cancellable worker lifetimes and a hosted-service shutdown path, but those mechanisms still require separate observations instead of being inferred from the portable labels.

Practice one: drain after proving an item is accepted

Start a fresh owned Development server, close the main gate, and begin one /work request. Wait until /stats shows active work before POST /lab/drain. Now check /ready for 503, /live for 200, and a newly started /work for 503. Hint: the initial request must remain connected and must not use a short budgetMs value; otherwise client or budget cancellation changes the test. Open the main gate and await the original response. This sequence demonstrates controlled admission after drain together with the opportunity for previously accepted work to finish. After the drain response, also start eight bounded parallel admission attempts and verify all receive503 before releasing the original item.

Practice two: separate cancellation from successful draining

Use a different fresh fixture, close the main gate and submit /work?budgetMs=50. Wait for its 504 response and inspect canceled after the worker observes the token. Open the gate and finish cleanup. Hint: a request budget expiring is not a readiness transition, so /ready can still be 200 throughout this exercise. Compare this canceled operation with the successful previously accepted operation from the first exercise. The distinction prevents a counter showing no active work from being mistaken for proof that every admitted request completed successfully before a drain.

Do not infer routing convergence from one readiness response

Kubernetes readiness probes influence service endpoints, while liveness probes can trigger restart decisions. The supplied application returns deliberately simple endpoint responses rather than registering the ASP.NET Core health-check middleware. The Kubernetes manifest uses probes, but that manifest has not been deployed here. There is also no preStop hook automatically calling the Development-only drain route; Production does not map that route. Application admission, service discovery convergence, connection reuse and the host's shutdown budget therefore remain distinct concerns. A real deployment must test their combined timing and account for requests already in transit.

Record outcomes across the lifecycle boundary

Useful evidence includes the active-item snapshot before drain, the three endpoint status checks, the original response after release, and final completed or canceled deltas. The admission flag cannot be restored through an undrain endpoint in this fixture; restart the owned server for an independent scenario. Unexpected process failure may bypass graceful shutdown entirely, so this experiment establishes no durable delivery guarantee. A local SQLite receipt also cannot preserve arbitrary in-memory queued work after a crash. Write conclusions around the controlled sequence actually observed, including cancellation and timeout outcomes rather than counting all disappeared work as success.

External .NET10 SDK10.0.403, real Kestrel and HTTP driver. Experiments are limited to owned loopback processes. SQLite10.0.14 is a local-file classroom ledger, not a network production database.