Go Gotcha #2: sync.Mutex Is Killing Your Throughput

"Lock contention is invisible until load exposes it. Here's how sync.Mutex serializes Go throughput and when sync.RWMutex is the fix you already have."

T

It's the mutex.

Specifically, it's a sync.Mutex protecting a shared cache, and every read and write is being serialized through a single lock. The golang mutex performance throughput problem isn't that the mutex is broken it's that you chose the wrong kind, and you won't see the cost until concurrency is high enough to make goroutines queue up waiting for it.

What sync.Mutex Actually Does

sync.Mutex is a mutual exclusion lock. When one goroutine calls Lock(), every other goroutine calling Lock() on the same mutex blocks until the first calls Unlock(). One at a time. Serialized. No exceptions.

For writes, this is exactly what you want. Two goroutines modifying a map concurrently without synchronization causes a data race your program panics, or worse, silently corrupts data.

The problem is when you protect reads the same way:

Untitled
Copy
1type Cache struct {
2    mu    sync.Mutex
3    items map[string]string
4}
5
6func (c *Cache) Get(key string) (string, bool) {
7    c.mu.Lock()
8    defer c.mu.Unlock()
9    v, ok := c.items[key]
10    return v, ok
11}
12
13func (c *Cache) Set(key, value string) {
14    c.mu.Lock()
15    defer c.mu.Unlock()
16    c.items[key] = value
17}

Reads don't modify shared state. Two goroutines can safely read the same map at the same time. But sync.Mutex doesn't know that. It treats every Lock() call identically: one goroutine in, everyone else waits. In a read-heavy system a cache, a config store, a lookup table this serializes operations that could be running in parallel.

sync.RWMutex: The Fix You Already Have

Go's standard library ships sync.RWMutex, which distinguishes readers from writers.

Multiple goroutines can hold a read lock simultaneously. A write lock is exclusive when a writer holds it, no readers enter; when readers are active, writers wait. This is exactly the pattern a read-heavy cache needs.

Untitled
Copy
1type Cache struct {
2    mu    sync.RWMutex
3    items map[string]string
4}
5
6func (c *Cache) Get(key string) (string, bool) {
7    c.mu.RLock()          // multiple goroutines can hold this simultaneously
8    defer c.mu.RUnlock()
9    v, ok := c.items[key]
10    return v, ok
11}
12
13func (c *Cache) Set(key, value string) {
14    c.mu.Lock()           // exclusive readers block until this is done
15    defer c.mu.Unlock()
16    c.items[key] = value
17}

If your workload is 95% reads and 5% writes common for caches and config lookups sync.RWMutex can multiply throughput under contention without changing your architecture. Two-line change. Measurable improvement.

The tradeoff: RWMutex has higher overhead than Mutex when there's no contention. For write-heavy workloads or low-traffic paths, Mutex can actually be faster. Profile before assuming.

The Deeper Problem: Holding Locks Across I/O

Switching to RWMutex won't help if you're holding a lock across a network call or a database query. This turns a contention problem into a complete bottleneck.

Untitled
Copy
1func (c *Cache) GetOrFetch(key string) (string, error) {
2    c.mu.Lock()
3    defer c.mu.Unlock()
4
5    if v, ok := c.items[key]; ok {
6        return v, nil
7    }
8
9    // BAD: making a network call while holding the mutex
10    v, err := fetchFromDB(key)
11    if err != nil {
12        return "", err
13    }
14    c.items[key] = v
15    return v, nil
16}

While fetchFromDB is waiting on a network response maybe 50ms, maybe 500ms every other goroutine calling GetOrFetch is blocked. Your mutex has become a global pause. Under load, one slow database query causes a cascade: all goroutines block, the request queue grows, timeouts start firing.

The fix is to do I/O outside the lock, then acquire the lock only to write the result:

Untitled
Copy
1func (c *Cache) GetOrFetch(key string) (string, error) {
2    c.mu.RLock()
3    if v, ok := c.items[key]; ok {
4        c.mu.RUnlock()
5        return v, nil
6    }
7    c.mu.RUnlock()
8
9    // I/O happens without holding any lock
10    v, err := fetchFromDB(key)
11    if err != nil {
12        return "", err
13    }
14
15    c.mu.Lock()
16    c.items[key] = v
17    c.mu.Unlock()
18    return v, nil
19}

Yes, two goroutines might both miss the cache and both fetch from the database for the same key. That's acceptable it's a cache, not a transaction. Serializing every request through a lock while doing network I/O is much worse.

The Part Most Engineers Miss: Contention Is Invisible

Lock contention doesn't show up in the obvious metrics. CPU stays low because goroutines aren't doing CPU work while blocked they're waiting. Memory is fine. Error rates are zero. Everything looks healthy until you notice latency is three times higher than it should be and throughput has flatlined.

Go's pprof mutex profile is the right tool:

Untitled
Copy
1go tool pprof http://localhost:6060/debug/pprof/mutex

This shows which mutexes are most contended and how long goroutines are waiting on each one. Enable it first:

Untitled
Copy
1runtime.SetMutexProfileFraction(1)

The output points you directly at the lock and the goroutines competing for it. The benchmark to run when you're unsure whether you have a contention problem:

Untitled
Copy
1func BenchmarkCacheGet(b *testing.B) {
2    c := NewCache()
3    b.RunParallel(func(pb *testing.PB) {
4        for pb.Next() {
5            c.Get("key")
6        }
7    })
8}

b.RunParallel creates real contention. If your throughput doesn't scale with GOMAXPROCS, you're looking at lock contention.

What Happened at a Marketplace During Harbolnas

An e-commerce platform handling product catalog lookups used a global in-memory cache protected by sync.Mutex. At normal traffic, response time was around 12ms. During Harbolnas (Indonesia's National Online Shopping Day), traffic spiked 15x and response time climbed to 340ms a 28x degradation for a 15x traffic increase.

The cache hit rate was 94%. Almost nothing was hitting the database. The bottleneck was the mutex: 94% of requests were reads, all serialized through a write lock, all blocking each other.

Switching to sync.RWMutex took one engineer about 45 minutes, including running the benchmark to verify the fix. Under the same load simulation, response time dropped to 18ms. Still slightly above baseline due to cache invalidation overhead during writes, but well within acceptable range.

The expensive part wasn't the fix. It was the three hours of profiling to understand why a 94% cache hit rate still produced 340ms latency.

FAQ

Q: When should I use sync.Mutex vs sync.RWMutex? A: Use sync.Mutex when writes are frequent relative to reads, or when your critical section is very short and the coordination overhead of RWMutex isn't worth it. Use sync.RWMutex when reads dominate and concurrent reads are safe caches, config lookups, read-heavy data structures. If you're not sure, benchmark with b.RunParallel under realistic read/write ratios. The numbers will tell you.

Q: Can I use a channel instead of a mutex? A: Yes, and sometimes that's the right call particularly for ownership transfer or fan-out patterns. But for protecting shared state with many readers, a channel forces serialization through a goroutine, which is often slower than RWMutex. Choose based on what reads more clearly for the access pattern you're protecting. Mutexes for shared state, channels for communication.

Q: How do I find which mutex is causing contention? A: Enable the mutex profile with runtime.SetMutexProfileFraction(1), then hit /debug/pprof/mutex with go tool pprof. It shows cumulative wait time per mutex and the stack traces of goroutines waiting on each one. This is far more useful than CPU or memory profiles when your problem is blocking, not computation.

Q: My benchmark shows sync.RWMutex is slower than sync.Mutex. Why? A: RWMutex has higher overhead when there's no real contention the reader-writer coordination costs more than a simple mutex in the uncontested case. Run your benchmark with enough goroutines to create actual contention. If RWMutex is still slower under load, your workload is probably write-heavy, and the simpler serialization of Mutex wins.

Q: Is there anything faster than sync.RWMutex for high-read workloads? A: For specific patterns, yes. sync.Map is optimized for cases where each key is written once and read many times, or where goroutines operate on completely disjoint key sets. For counters and simple numeric values, sync/atomic operations are lock-free and significantly faster. Both come with constraints: sync.Map doesn't work well for write-heavy or general-purpose maps, and atomic ops only cover primitive types. Use sync.RWMutex as your default; reach for the alternatives only when profiling confirms you need them.

Lock contention is a performance problem that hides until load forces it into the open. The fix is almost always straightforward once you've diagnosed it correctly: wrong mutex type, I/O held inside a lock, or a design that serializes what could run in parallel.

If your system needs to handle traffic spikes without falling over, getting the synchronization right early is cheaper than profiling it at 2AM on Harbolnas.