Go Gotcha #1: Goroutine Leaks That Crash Your Service at 3AM

Meta: "Goroutine leaks are silent killers in production Go services. Learn how to detect, reproduce, and fix them before your on-call gets a 3AM page."

Y

That's a goroutine leak. It's one of the most common ways Go services die quietly in production, and the golang goroutine leak production problem is deceptively hard to catch: goroutines are cheap enough to create carelessly, but expensive enough to accumulate into something catastrophic.

Here's what's happening, how to find it, and how to fix it.

Why Goroutines Leak in the First Place

A goroutine is blocked when it's waiting for something that never arrives: a channel read with no sender, a channel write into a full buffer with no reader, an HTTP call with no timeout. The goroutine stays alive, holding its stack memory, waiting. Your program moves on and creates more goroutines. Those block too. Eventually you're running a service with tens of thousands of goroutines that aren't doing any work.

The Go runtime won't kill a goroutine for you. There's no external cancellation mechanism unless you build one in. A goroutine blocked on a channel that nobody is ever going to signal will wait until your process restarts.

This surprises engineers coming from languages where threads are expensive enough that you think twice before creating them. In Go, go func() costs roughly 2KB of stack (which grows on demand), so people sprinkle goroutines everywhere. Most of the time this is fine. When it isn't, the failure is silent and slow.

The Three Patterns That Leak Most Often

The goroutine leaks I've seen in production almost always follow one of three shapes.

Pattern 1: The abandoned channel consumer

Untitled
Copy
1func processItems(items []string) {
2    ch := make(chan string)
3
4    go func() {
5        for item := range ch {
6            // process item
7        }
8    }()
9
10    for _, item := range items {
11        ch <- item
12    }
13    // forgot to close(ch) goroutine blocks forever on range
14}

The goroutine ranges over ch and waits for more items or a close signal. The function returns. ch goes out of scope. The goroutine is still alive, stuck, waiting for a close that will never come.

Pattern 2: The HTTP call without a timeout

Untitled
Copy
1func fetchData(url string) (*Data, error) {
2    resp, err := http.Get(url)  // no context, no timeout
3    // ...
4}

If the remote server hangs never sending a response, never closing the connection your goroutine hangs with it. In a high-throughput service, one slow upstream turns into a goroutine accumulation problem fast. I've watched a payment gateway timeout cause 8,000 leaked goroutines in under an hour on a service handling GoPay callbacks.

Pattern 3: Context not threaded through

Untitled
Copy
1go func() {
2    for {
3        select {
4        case data := <-ch:
5            process(data)
6        // no ctx.Done() case
7        }
8    }
9}()

You have a context. You're using it elsewhere. You forgot to thread it into this goroutine. When the parent context cancels, this goroutine keeps running. It outlives request boundaries, accumulates, and causes problems you'll spend hours debugging.

How to Detect Leaks Before They Kill Your Service

The simplest diagnostic is runtime.NumGoroutine(). Log it on a timer. If it's growing under stable load, something is leaking.

Untitled
Copy
1go func() {
2    for range time.Tick(30 * time.Second) {
3        log.Printf("goroutine count: %d", runtime.NumGoroutine())
4    }
5}()

This is coarse, but it catches the problem early. If you see 500 goroutines at startup and 3,000 after an hour of steady traffic, you have a leak.

For more detail, Go's pprof endpoint gives you a full goroutine dump:

GET /debug/pprof/goroutine?debug=2

This shows every running goroutine with its stack trace. When you have a leak, you'll see hundreds or thousands of goroutines all blocked at the same line. That's your culprit.

In tests, the goleak package from Uber is the most reliable way to catch leaks at the function level:

Untitled
Copy
1func TestMyFunction(t *testing.T) {
2    defer goleak.VerifyNone(t)
3    // your test code
4}

If your function leaks a goroutine, the test fails. I use this on any code that spawns goroutines. It's caught things in code review that would have shipped otherwise.

Connecting goroutine count to your observability stack is worth the 20 minutes it takes. If you're already on Prometheus, go_goroutines is exposed by the Go collector. Set an alert at a reasonable multiple of your baseline. If your service normally runs at 200 goroutines and spikes to 2,000, you want to know before users do. The observability fundamentals guide covers the metrics setup that makes this kind of monitoring straightforward.

The Part Most Engineers Get Wrong: Context Doesn't Cancel Itself

Here's the misunderstanding I see constantly: passing a context into a goroutine doesn't automatically cancel that goroutine when the context expires. You have to check it.

Untitled
Copy
1// This does NOT work the way people think
2go func(ctx context.Context) {
3    doExpensiveWork()  // context is ignored entirely
4}(ctx)
5
6// This is what you actually need
7go func(ctx context.Context) {
8    select {
9    case <-ctx.Done():
10        return
11    default:
12        doExpensiveWork()
13    }
14}(ctx)

And doExpensiveWork() itself needs to respect the context too. If it makes network calls, they need ctx. If it's a loop, it needs to check ctx.Done() on each iteration. Context cancellation only works if every layer in the call chain propagates it. One blocking call without a context and the whole chain stops responding to cancellation.

This is the goroutine leak equivalent of forgetting to close a file handle except the consequences are global across your process, not local to one operation.

What Happened to a Fintech Startup During Lebaran

A payment processing service I know well started seeing memory climb every Lebaran, Indonesia's biggest shopping season. Traffic would spike 8-10x, they'd restart the service a few times, and things would stabilize after the peak. They assumed it was heap pressure from increased load.

It wasn't heap. It was goroutines.

Their payment status polling goroutine looked like this:

Untitled
Copy
1func pollPaymentStatus(paymentID string) {
2    for {
3        status, err := checkStatus(paymentID)  // external API, no timeout
4        if status == "settled" || err != nil {
5            return
6        }
7        time.Sleep(5 * time.Second)
8    }
9}

At baseline traffic, the external payment API responded fast enough that these goroutines completed within seconds. Under Lebaran load, the external API started queuing requests. Response times went from 200ms to 45 seconds. The polling goroutines stopped completing. New ones kept starting. By midnight they had 60,000+ goroutines running, each holding a stack and an open connection to the external API.

The fix was two lines: add a context with a timeout to checkStatus, and add a ctx.Done() case to the loop. Twenty minutes to fix. Three years to find.

FAQ

Q: How many goroutines is too many? A: Go can theoretically schedule millions of goroutines they're not OS threads, so the 1:1 thread-per-goroutine limit doesn't apply. But "technically possible" and "actually a good idea" are different things. Every goroutine switch still costs the scheduler time and memory, and at some point the scheduler overhead consumes more CPU than the actual work does. The answer depends on your workload type. For CPU-bound tasks, your goroutine pool should be close to runtime.GOMAXPROCS(0) the number of logical CPUs available. Adding more goroutines beyond that just creates context switching with no throughput gain. For I/O-bound tasks (network calls, database queries, disk), you can go higher since goroutines spend most of their time waiting rather than running but "higher" still means a bounded pool, not unbounded spawning. A tiered worker pool one pool for incoming requests, a separate pool for background processing gives you control over both layers independently. The right pool size for your system is something you test: run a benchmark, increase the pool, measure throughput and latency, find where gains flatten. That number is your sweet spot. Note that goroutines spawned by net/http per request are a separate category from your background worker goroutines those are inherent to the request model and scale with your server's concurrency settings, not your application pools.

Q: Can I force-kill a leaked goroutine? A: No. Go doesn't expose a way to cancel a goroutine from outside. Your only options are to design for cleanup from the start (using context, channels, or WaitGroups) or restart the process. This is why fixing leaks at the source matters. There's no runtime escape hatch.

Q: Will the garbage collector clean up goroutines that are no longer needed? A: No. The GC handles heap allocations, not goroutine lifecycle. A goroutine blocked on a channel is considered alive by the runtime regardless of whether anything useful still references it. Only returning from the goroutine function terminates it.

Q: How do I write tests that catch goroutine leaks? A: Use goleak from Uber (go.uber.org/goleak). Add defer goleak.VerifyNone(t) at the start of any test that involves goroutines. It verifies at test completion that no goroutines were created and left running. It catches the majority of cases and integrates cleanly with t.Cleanup.

Q: My service has a slowly growing goroutine count and it's never crashed. Should I fix it? A: Yes. A slow goroutine leak is a deferred crash. At baseline traffic, restarts paper over it. Under a spike exactly when you can't afford a restart it'll accelerate. The question isn't whether it'll cause a problem. It's when.

Your goroutine count is one of the cheapest signals to collect and one of the most useful. If you're not logging it, start today. If it's growing, you have a leak to find.

At SpectreDev, goroutine profiling is part of every production readiness review. Sometimes we catch these before they've caused an incident. Sometimes we're cleaning up after one. The fix is the same either way.

Internal links used:

• observability fundamentals goroutine count monitoring and Prometheus metrics setup

External links used:

• goleak Uber's goroutine leak detection library for tests

Word count: ~1,490