Circuit Breaker in Go: Stop One Failure From Becoming Everyone's Problem

Meta: "Circuit breaker pattern in Go production: how to stop cascading failures before they take your whole system down. Real implementation, real tradeoffs. (~155 chars)"

I

Nobody planned for that. Nobody's timeout config said "if orders-service gets slow, please also kill the recommendation engine." But that's exactly what happened, because in distributed systems, latency is contagious.

The circuit breaker pattern is how you stop it. It's not complicated in theory. The execution is where most teams stumble.

What a Circuit Breaker Actually Does

The name comes from your fuse box at home. When a circuit draws too much current, the breaker trips and cuts the flow protecting everything downstream from the surge. Once the problem is fixed, you reset it and current flows again.

A circuit breaker in software works the same way. It wraps a call to an external dependency a service, a database, a third-party API. It watches how that call behaves. When failures cross a threshold, it "trips" and instead of continuing to hammer the failing dependency, it immediately returns an error to callers without even trying. Give the dependency a moment to breathe. Then, carefully, let a small amount of traffic through to check if it's recovered.

Three states:

Closed is normal operation. Requests flow through. The breaker counts failures in the background.

Open means the breaker has tripped. Requests fail immediately no attempt made. This is the state that saves you from cascade failure, because it stops the timeout queue from building up.

Half-open is the probe state. After a configurable wait period, the breaker lets one or a few requests through. If they succeed, it resets to Closed. If they fail, it flips back to Open and waits again.

What makes this powerful isn't the error handling you have that already. It's the speed. A closed breaker fails your caller in microseconds instead of holding a connection open for 30 seconds waiting for a timeout that was never going to succeed.

Building It in Go Without a Framework

There are libraries that do this Sony's gobreaker is the one most Go teams reach for first, and it's solid. But I want to show the core implementation before we talk about libraries, because if you don't understand what's inside, you'll misconfigure it and it'll either never trip or never recover.

Untitled
Copy
1type State int
2
3const (
4    StateClosed   State = iota
5    StateOpen
6    StateHalfOpen
7)
8
9type CircuitBreaker struct {
10    mu              sync.Mutex
11    state           State
12    failureCount    int
13    successCount    int
14    lastFailureTime time.Time
15
16    // Configuration
17    failureThreshold int
18    successThreshold int
19    timeout          time.Duration
20}
21
22func NewCircuitBreaker(failureThreshold, successThreshold int, timeout time.Duration) *CircuitBreaker {
23    return &CircuitBreaker{
24        state:            StateClosed,
25        failureThreshold: failureThreshold,
26        successThreshold: successThreshold,
27        timeout:          timeout,
28    }
29}
30
31func (cb *CircuitBreaker) Call(fn func() error) error {
32    cb.mu.Lock()
33
34    switch cb.state {
35    case StateOpen:
36        // Check if timeout has elapsed try half-open
37        if time.Since(cb.lastFailureTime) > cb.timeout {
38            cb.state = StateHalfOpen
39            cb.successCount = 0
40        } else {
41            cb.mu.Unlock()
42            return errors.New("circuit open: dependency unavailable")
43        }
44    }
45
46    cb.mu.Unlock()
47
48    // Execute the function
49    err := fn()
50
51    cb.mu.Lock()
52    defer cb.mu.Unlock()
53
54    if err != nil {
55        cb.failureCount++
56        cb.lastFailureTime = time.Now()
57
58        if cb.state == StateHalfOpen || cb.failureCount >= cb.failureThreshold {
59            cb.state = StateOpen
60            cb.failureCount = 0
61        }
62        return err
63    }
64
65    // Success path
66    if cb.state == StateHalfOpen {
67        cb.successCount++
68        if cb.successCount >= cb.successThreshold {
69            cb.state = StateClosed
70            cb.failureCount = 0
71        }
72    } else {
73        cb.failureCount = 0
74    }
75
76    return nil
77}

This is minimal but functional. A few things worth noting. The mutex is held as briefly as possible just long enough to read and update state, not during the actual call. The half-open success threshold (successThreshold) matters: if you reset to Closed on a single success, you'll flap badly when a service is recovering unevenly. Wait for two or three consecutive successes before trusting it.

Configuring It for Real Traffic

Here's where theory meets production and things get uncomfortable. The question everyone asks is: what numbers do I use?

There's no universal answer, but there's a sensible frame. Your failure threshold should reflect what's actually abnormal not what sounds low. If your payment gateway normally has a 0.5% error rate, setting a threshold of 5 failures out of 100 requests means you'll trip on noise constantly. You want a threshold that signals "something is genuinely wrong", not "we're having a mildly bad minute."

For a service like a credit-check API in an Indonesian fintech context something your system calls maybe 200 times a minute during peak a reasonable starting config might be:

Untitled
Copy
1breaker := NewCircuitBreaker(
2    10,                // failureThreshold: 10 consecutive failures
3    3,                 // successThreshold: 3 consecutive successes to reset
4    30 * time.Second,  // timeout: wait 30s before probing
5)

But that config assumes the failure mode is binary the service either works or it doesn't. In practice, most degradation is latency-based. The service is responding, just slowly. A pure error-counting breaker won't trip on that at all. Your callers will keep getting through, they'll just take 10 seconds each.

This is why most production implementations count both errors and timeouts:

Untitled
Copy
1func callWithTimeout(ctx context.Context, breaker *CircuitBreaker, fn func() error) error {
2    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
3    defer cancel()
4
5    return breaker.Call(func() error {
6        done := make(chan error, 1)
7        go func() { done <- fn() }()
8
9        select {
10        case err := <-done:
11            return err
12        case <-ctx.Done():
13            return fmt.Errorf("timeout: %w", ctx.Err())
14        }
15    })
16}

Now a slow service trips the breaker just like a failing one. That's the behavior you actually want.

The Part Most People Get Wrong

Most engineers put one circuit breaker on the wrong thing.

The instinct is to wrap your HTTP client and call it done. One breaker, one dependency. That works if all your calls to that dependency are equivalent. But they rarely are.

Consider your notification service. You call it for two reasons: transactional alerts (payment received high priority, user is waiting) and marketing pings (promo notification low priority, can be dropped). If you wrap both in the same breaker and the notification service gets slow, you trip the breaker on marketing traffic and now your transactional alerts are being dropped too. Wrong outcome.

Separate breakers per criticality, not per service:

Untitled
Copy
1type NotificationClient struct {
2    transactionalBreaker *CircuitBreaker
3    marketingBreaker     *CircuitBreaker
4    client               *http.Client
5}
6
7func (n *NotificationClient) SendTransactional(ctx context.Context, msg Message) error {
8    return n.transactionalBreaker.Call(func() error {
9        return n.send(ctx, msg, "/transactional")
10    })
11}
12
13func (n *NotificationClient) SendMarketing(ctx context.Context, msg Message) error {
14    return n.marketingBreaker.Call(func() error {
15        return n.send(ctx, msg, "/marketing")
16    })
17}

You can tune them independently. Your marketing breaker might have a much lower threshold and a much shorter recovery window you're happy to shed that load aggressively. Your transactional breaker might have a higher threshold and more conservative half-open behavior, because false trips on payment confirmations have a real cost.

This pairs naturally with building reliable AI features in production

AI inference calls are exactly the kind of high-latency, variable-reliability dependency that needs a circuit breaker tuned to latency, not just error rate.

What Gojek Does at 3AM

Gojek's engineering team has been public about their approach to resilience architecture since around 2019. Their production Go services use circuit breakers as a standard component in any service-to-service call not something you add after an incident, but something you configure before you ship.

The context that made this necessary is familiar to anyone who's operated during Indonesian peak hours. On New Year's Eve and Harbolnas, Gojek sees traffic spikes that would read as an attack on any other day. During those spikes, some services slow down. Without circuit breakers, those slow services would become a drain on everything upstream connection pools exhausted, thread pools saturated, queues backing up.

With breakers in place, a slowing service trips its callers' breakers quickly. Callers fail fast, free their connections, and either serve degraded results from cache or return a graceful error. The slowing service, now relieved of the additional inbound load, has a chance to recover. The circuit breaker's half-open state then acts as a controlled ramp traffic comes back gradually rather than all at once the moment the service recovers, which is exactly when you don't want to slam it with full load again.

Their observability setup monitors breaker state transitions as a signal. A breaker flipping to Open is an alert. A breaker flipping Open repeatedly is a high-priority page. That metric alone breaker trips per service per minute gives their on-call engineers a faster signal than most error rate dashboards, because it catches latency degradation before errors start accumulating.

For teams using the saga pattern for distributed transactions which every fintech service should the circuit breaker sits inside each saga step's execute function. If the step's dependency is tripped, the step fails immediately, the saga compensates, and you never build up a queue of stuck in-flight transactions waiting on a dead service.

FAQ

Q: Is circuit breaker the same as a retry with backoff? A: No they solve different problems and work best together. Retry with backoff handles transient failures: a momentary network blip, a brief service hiccup. Circuit breaker handles sustained degradation: a service that's going to be slow or down for more than a few seconds. If you only have retries, a degraded service will cause your callers to retry repeatedly, amplifying load. If you only have a circuit breaker, you won't recover from the transient errors that retries handle cleanly. Use both.

Q: What should the fallback return when the circuit is open? A: Depends on the dependency. If it's a recommendation engine, return a cached or default response the user gets something. If it's a payment processor, fail explicitly with a clear error no silent degradation on money. The worst thing you can do is return a success response when you haven't actually processed anything. Design your fallback before you need it.

Q: Should I use a library or build my own in Go? A: Use Sony's gobreaker for most production cases. It handles the concurrency correctly, has a ReadyToTrip hook for custom threshold logic, and is battle-tested. Build your own only if you need behavior the library doesn't support like separate thresholds for timeouts vs errors, or per-endpoint granularity within the same client.

Q: How do I test a circuit breaker properly? A: Write integration tests that inject failures: stub the downstream dependency to return errors until the threshold, verify the breaker trips, wait the timeout, verify it transitions to half-open, inject a success, verify it resets. Don't just test the happy path. The breaker's value is entirely in its failure behavior, and that behavior is what most teams skip testing.

Q: What metrics should I expose from my circuit breakers? A: At minimum: current state per breaker (closed/open/half-open), failure count in the current window, total transitions to Open. Ideally also: success rate in half-open state, time spent in Open state per service. These metrics give your on-call team a much earlier signal than generic error rate dashboards, because they surface latency degradation before your error rate climbs.

Distributed systems fail. That's not pessimism that's the operating condition. The engineers who sleep through incidents aren't the ones who've eliminated failure modes; they're the ones who've made sure failures don't travel.

A circuit breaker is one of the cheapest ways to contain that travel. SpectreDev runs this pattern in every service-to-service call we build for clients, because it's the thing we've learned the hard way you want before the incident, not after.

Internal links used:

• building reliable AI features in production

AI inference = high-latency dependency needing breaker

• saga pattern for distributed transactions circuit breaker inside saga steps

External links used:

• None

Word count: ~1,980