Circuit breaking and controlled recovery

28 min guided lesson · Read, predict, then run

External .NET10 SDK10.0.403, real Kestrel and HTTP driver. Experiments are limited to owned loopback processes. SQLite10.0.14 is a local-file classroom ledger, not a network production database.

Read or download the complete SDK project, then follow its setup on your computer. The separate C# companion practice below runs in SharpRune’s .NET 9 console environment.

A breaker limits repeated dependency attempts

A circuit breaker answers whether a dependency call should be attempted now. A retry answers whether to make another attempt for one logical operation. Keeping those decisions separate makes the experiment interpretable: an open response should avoid calling the protected operation, rather than silently retrying it. The real endpoint here is /guarded, backed by one ClassroomCircuit object in one Kestrel process. Its controlled dependency outcome is a Boolean in that process. HTTP transport is real, but there is no remote dependency behind this particular teaching endpoint and no production resilience package is being simulated.

Read the locked state transition

Invoke takes one lock, reads the current UTC time, and returns called=false with status 503 while the open deadline remains in the future. Otherwise it increments calls and evaluates the configured outcome synchronously. A failure increments failures; the second consecutive failure opens a 250-millisecond window. A failure after an expired window opens another window immediately. A success resets failures, openUntil and probe, returning closed with called=true. The entire outcome occurs inside the lock, so this is deliberately small synchronous teaching code. Its probe flag is not evidence of an asynchronous single-flight recovery implementation.

public sealed class ClassroomCircuit {
    private readonly object sync = new(); private int failures, calls; private bool probe; private DateTimeOffset openUntil = DateTimeOffset.MinValue;
    public int DependencyCalls { get { lock (sync) return calls; } }
    public IResult Invoke(bool failing) {
        lock (sync) {
            DateTimeOffset now = DateTimeOffset.UtcNow;
            if (openUntil != DateTimeOffset.MinValue && now < openUntil) return Results.Json(new { state = "open", called = false }, statusCode: 503);
            if (probe) return Results.Json(new { state = "probe-busy", called = false }, statusCode: 503);
            bool halfOpen = openUntil != DateTimeOffset.MinValue; if (halfOpen) probe = true;
            calls++;
            // The dependency is an explicitly controlled in-process classroom outcome, not a real remote service.
            if (failing) {
                failures++; if (halfOpen || failures >= 2) openUntil = now.AddMilliseconds(250); probe = false;
                return Results.Json(new { state = openUntil == DateTimeOffset.MinValue ? "closed" : "open", called = true }, statusCode: 503);
            }
            failures = 0; openUntil = DateTimeOffset.MinValue; probe = false; return Results.Ok(new { state = "closed", called = true });
        }
    }
}

Trace the deterministic companion separately

The portable program prints closed, open, probe and closed by manipulating integer state and fixed arithmetic times. It sets openUntil to ten, compares five and eleven against that value, and finally clears the failure state. Those lines explain a policy progression without clocks, HTTP or parallel callers. They do not measure the real 250-millisecond window. In particular, printing probe in this companion is a label chosen by the model; it does not execute ClassroomCircuit.Invoke or discover a half-open circuit elsewhere. Use its exact output to check your reasoning before interpreting the external responses.

Practice one: verify avoided calls

Start a fresh owned Development server and record dependencyCalls from /stats. Make two /guarded calls while dependencyFailing is true, then make a third promptly inside the open window. Inspect both status and called, then compare the counter with its baseline. Hint: the first two failures have called=true, whereas a short-circuited response has called=false. If your manual interaction outlasts 250 milliseconds, another dependency attempt is allowed and the counter can increase; repeat with a bounded scripted sequence rather than assuming the timing. Keep the experiment within the existing request limits.

Practice two: distinguish reopening from recovery

In a separate fresh run, open the circuit, keep the dependency failing, and make one bounded call after the window has expired. Predict called=true and another open response. Then POST /lab/dependency/false, wait for the current window to expire, and make another bounded call. Hint: changing the development flag does not bypass an already open deadline; the next permitted successful attempt performs the reset. Record the counter increments and the final closed response. This exercise tests the recovery transition, independently of the earlier test that open calls avoided work.

Choose failure and ownership boundaries explicitly

The circuit state is process-local. Starting another replica creates another failure counter and another deadline, so one open replica does not prove that all replicas stopped calling a dependency. This sample also has no moving throughput window, exception classification, cancellation handling for remote calls, or durable state. DateTimeOffset is the classroom clock choice; elapsed-time policies in a real library deserve their own clock and scheduling analysis. Do not copy the lock around a slow network operation. Protecting a meaningful dependency requires a suitable asynchronous resilience design and tests for concurrent recovery behavior.

Report state evidence rather than a speed guarantee

A useful result records the fresh-process baseline, the three immediate responses, the dependencyCalls change, and a later recovery outcome. These establish the implemented state transitions and avoided calls. A status 503 alone is insufficient because both an executed failure and an open rejection return that status. The local experiment does not establish remote-service recovery, cluster coordination, throughput improvement or a universal retry policy. Keep breaker outcomes separate from queue saturation and admission counters, which belong to other endpoints. Restart only your owned process when resetting the fixture, and record that reset with the observation.

External .NET10 SDK10.0.403, real Kestrel and HTTP driver. Experiments are limited to owned loopback processes. SQLite10.0.14 is a local-file classroom ledger, not a network production database.