gRPC in Go: The Patterns Your Microservices Actually Need

Meta: "gRPC in Go works out of the box. Production microservices need more. Here are the patterns that make it reliable at scale and the mistakes that don't show up until too late." (178 chars)

M

Getting gRPC running in Go takes an afternoon. Writing a .proto file, generating the stubs, wiring up a server and client the tooling is solid and the official docs are good. What the docs don't cover is what happens when your service is handling ten thousand concurrent streams, a downstream dependency goes down, your client is retrying without backoff, and your connection pool is exhausted.

This post is about the patterns that matter in production. Not the hello world. The stuff that determines whether your microservices hold under load or cascade into each other.

Why gRPC Over REST for Internal Services

REST works. If your team knows it well and your services are coarse-grained, REST is fine. But when you're building internal service-to-service communication at any meaningful scale, the tradeoffs shift.

gRPC uses HTTP/2 by default, which means multiplexed streams over a single connection rather than one request per connection. Under high call volume dozens of internal services talking to each other on every user-facing request that difference in connection overhead compounds fast. Gojek's internal architecture has talked publicly about the pressure their service mesh was under at peak traffic. The cost of connection establishment at that scale isn't trivial.

The other advantage: protobuf. Binary serialization, smaller payloads, schema-enforced contracts. When your order service calls your inventory service five hundred thousand times a day, the difference between a 2KB JSON payload and a 400-byte protobuf payload adds up. The protobuf vs JSON comparison

covers the benchmarks in detail the short version is the gap widens significantly under concurrent load, not just in raw message size.

The tradeoff: gRPC is more operationally complex. Load balancers need to understand HTTP/2. Debugging is harder you can't curl a gRPC endpoint directly. Your observability stack needs gRPC-aware tooling. Go into it with eyes open, not because it's fashionable.

The Basic Setup Worth Getting Right

Before the patterns: the baseline setup that most tutorials skip over.

Proto organization. Keep your .proto files in a separate repository or a dedicated directory at the root of your monorepo. Generate stubs into a shared package. Don't let each service generate its own version of the same proto you'll have drift within weeks.

/proto
  /order/v1/order.proto
  /inventory/v1/inventory.proto
/gen
  /go/order/v1/
  /go/inventory/v1/

Version your protos explicitly. order/v1 is a contract. When you need a breaking change, create order/v2 and migrate consumers incrementally. Never edit a published proto in place.

Server setup with graceful shutdown. This is the one most teams get wrong initially. The default gRPC server in Go doesn't handle SIGTERM gracefully:

Untitled
Copy
1server := grpc.NewServer()
2pb.RegisterOrderServiceServer(server, &OrderService{})
3
4// Wire shutdown before serving
5go func() {
6    sigChan := make(chan os.Signal, 1)
7    signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
8    <-sigChan
9    
10    // GracefulStop waits for in-flight RPCs to complete
11    server.GracefulStop()
12}()
13
14if err := server.Serve(listener); err != nil {
15    log.Fatalf("failed to serve: %v", err)
16}

GracefulStop() stops accepting new connections and waits for in-flight RPCs to finish. Stop() terminates immediately. Use GracefulStop() with a timeout in any Kubernetes-deployed service it pairs directly with the graceful shutdown patterns covered in the Go Gotcha #6 post

.

Interceptors: Where Cross-Cutting Logic Lives

Interceptors are gRPC's middleware layer. Every team eventually builds them. Build them right from the start and you won't rewrite them later.

Two types: unary (single request/response) and streaming. They compose you can chain multiple interceptors using grpc.ChainUnaryInterceptor.

Logging interceptor. Every RPC should log its duration and outcome. Not optional in production.

Untitled
Copy
1func loggingInterceptor(
2    ctx context.Context,
3    req interface{},
4    info *grpc.UnaryServerInfo,
5    handler grpc.UnaryHandler,
6) (interface{}, error) {
7    start := time.Now()
8    resp, err := handler(ctx, req)
9    
10    logger.Info("rpc",
11        zap.String("method", info.FullMethod),
12        zap.Duration("duration", time.Since(start)),
13        zap.Error(err),
14    )
15    
16    return resp, err
17}

Auth interceptor. Extract and validate tokens before the handler sees the request. Fail fast before any business logic runs.

Untitled
Copy
1func authInterceptor(
2    ctx context.Context,
3    req interface{},
4    info *grpc.UnaryServerInfo,
5    handler grpc.UnaryHandler,
6) (interface{}, error) {
7    md, ok := metadata.FromIncomingContext(ctx)
8    if !ok {
9        return nil, status.Error(codes.Unauthenticated, "missing metadata")
10    }
11    
12    token := md.Get("authorization")
13    if len(token) == 0 {
14        return nil, status.Error(codes.Unauthenticated, "missing token")
15    }
16    
17    if err := validateToken(token[0]); err != nil {
18        return nil, status.Error(codes.Unauthenticated, "invalid token")
19    }
20    
21    return handler(ctx, req)
22}

Recovery interceptor. Panics in a handler will crash the goroutine and close the connection. A recovery interceptor converts panics to codes.Internal errors and keeps the server running. Build this one; don't rely on not panicking.

Untitled
Copy
1func recoveryInterceptor(
2    ctx context.Context,
3    req interface{},
4    info *grpc.UnaryServerInfo,
5    handler grpc.UnaryHandler,
6) (resp interface{}, err error) {
7    defer func() {
8        if r := recover(); r != nil {
9            logger.Error("panic in rpc handler",
10                zap.String("method", info.FullMethod),
11                zap.Any("panic", r),
12                zap.ByteString("stack", debug.Stack()),
13            )
14            err = status.Error(codes.Internal, "internal server error")
15        }
16    }()
17    
18    return handler(ctx, req)
19}

Chain them in the right order: recovery wraps everything, then auth, then logging:

Untitled
Copy
1server := grpc.NewServer(
2    grpc.ChainUnaryInterceptor(
3        recoveryInterceptor,
4        authInterceptor,
5        loggingInterceptor,
6    ),
7)

Client-Side: Connection Management and Retry

This is where most gRPC issues in production originate. Not in the server in how clients manage connections and handle failure.

Use a single ClientConn per target service. grpc.Dial is expensive. HTTP/2 multiplexes streams over one connection creating a new ClientConn per request throws away that advantage entirely. Create the connection at startup, inject it as a dependency, reuse it everywhere.

Untitled
Copy
1// At startup
2conn, err := grpc.Dial(
3    orderServiceAddr,
4    grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
5    grpc.WithDefaultCallOptions(grpc.WaitForReady(true)),
6)
7if err != nil {
8    log.Fatalf("failed to connect: %v", err)
9}
10defer conn.Close()
11
12orderClient := pb.NewOrderServiceClient(conn)

WaitForReady vs fail-fast. By default, gRPC in Go uses fail-fast: if the connection isn't ready when you make a call, it returns immediately with an error. WaitForReady(true) changes this to block until the connection is ready or the context deadline expires. For internal service-to-service calls where you control both ends, WaitForReady is usually right it handles transient connection drops without forcing your calling code to implement retry logic for connectivity issues.

Retry policy via service config. Don't implement retry logic manually in your application code. gRPC has a built-in retry policy configured via service config:

Untitled
Copy
1serviceConfig := `{
2    "methodConfig": [{
3        "name": [{"service": "order.v1.OrderService"}],
4        "retryPolicy": {
5            "maxAttempts": 3,
6            "initialBackoff": "0.1s",
7            "maxBackoff": "1s",
8            "backoffMultiplier": 2,
9            "retryableStatusCodes": ["UNAVAILABLE", "RESOURCE_EXHAUSTED"]
10        }
11    }]
12}`
13
14conn, err := grpc.Dial(
15    addr,
16    grpc.WithDefaultServiceConfig(serviceConfig),
17)

Only retry on UNAVAILABLE and RESOURCE_EXHAUSTED. Never retry on INVALID_ARGUMENT, NOT_FOUND, or ALREADY_EXISTS those aren't transient failures. And never retry without backoff. A fleet of services simultaneously retrying with no backoff on a struggling downstream is how cascading failures start. The circuit breaker pattern exists for exactly this scenario the Circuit Breaker post

covers the Go implementation.

Context and Deadline Propagation

Every gRPC call should carry a deadline. Without one, a slow downstream can hold a goroutine indefinitely. Under load, that means goroutines accumulate until memory is exhausted.

Set deadlines at the entry point of your request the HTTP handler or the top-level RPC and propagate the context through every downstream call.

Untitled
Copy
1func (s *OrderService) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.CreateOrderResponse, error) {
2    // Incoming context already has a deadline from the client
3    // Use it for all downstream calls
4    
5    inventory, err := s.inventoryClient.CheckStock(ctx, &inventorypb.CheckStockRequest{
6        ItemId: req.ItemId,
7    })
8    if err != nil {
9        if status.Code(err) == codes.DeadlineExceeded {
10            return nil, status.Error(codes.DeadlineExceeded, "inventory check timed out")
11        }
12        return nil, err
13    }
14    
15    // ...
16}

Two things to get right here. First: don't create a new context with a longer deadline in a downstream call than you have in the current context. If the caller gave you 500ms and you call downstream with a 2-second deadline, you'll hold resources long after the caller has given up and moved on. The deadline should shrink as it propagates, not grow.

Second: check error codes explicitly. codes.DeadlineExceeded and codes.Canceled are not the same thing. Canceled means the caller gave up don't retry. DeadlineExceeded means time ran out may be worth retrying if you have budget left. This is the Go context cancellation propagation problem in a gRPC context, covered directly in the Go Gotcha #8 post

.

The Part Most Teams Miss: Status Codes Matter

gRPC has a defined set of status codes. They're not just conventions clients use them to decide whether to retry, how to handle the error, what to log.

Most teams use three codes for everything: OK, Internal, and NotFound. That's not enough.

Using the right code means your retry policy can distinguish retryable from non-retryable errors automatically. It means your clients log the right thing. It means your SRE team can write alerts that distinguish "clients are sending bad requests" from "downstream is unavailable" without reading log messages.

Return Internal only when you genuinely don't know what went wrong. Everything else should have a more specific code.

Real-World Example: Payment Retry Saga at a Fintech Startup

A fintech startup in Jakarta was processing payments through an internal gRPC service. The payment service called a third-party payment gateway over HTTP, wrapped in a gRPC call to their internal billing service.

Their initial setup: no retry policy, no deadline propagation, default gRPC connection management. When Traveloka-style peak traffic hit during a promotional campaign, their payment gateway started returning 503s intermittently. Their billing service had no circuit breaker. Clients retried manually, with no backoff, in a tight loop. Connection count to the billing service spiked. The billing service's goroutine count climbed. Memory pressure triggered GC pauses. Latency spiked. More clients timed out and retried. Classic cascade.

The recovery involved three changes, deployed over two days:

First: deadline propagation. Every call into the billing service now carried the deadline from the original HTTP request. When the gateway was slow, the billing service stopped holding goroutines past the point where the caller was still listening.

Second: retry policy via service config, with exponential backoff. Clients stopped hammering a struggling service. Peak retry load dropped by 80% under the same gateway failure scenario.

Third: a circuit breaker around the gateway call inside the billing service. When the gateway failure rate crossed a threshold, the circuit opened, and billing returned codes.Unavailable immediately rather than waiting for the gateway to time out.

Their payment success rate during the same gateway degradation went from 34% to 91%. The remaining 9% were requests that genuinely arrived during full gateway unavailability, handled with a queued retry mechanism rather than a synchronous call.

FAQ

Q: Should I use gRPC for all service-to-service communication or only some? A: Use it for high-frequency, latency-sensitive internal calls where the schema contract matters. Keep REST for external-facing APIs your clients don't want to deal with protobuf. For async communication between services, a message queue is often more appropriate than either. gRPC is synchronous; it's not the right tool for fire-and-forget or fan-out patterns.

Q: How do I handle gRPC behind a Kubernetes load balancer? A: Standard Kubernetes Service load balancing (kube-proxy) operates at the TCP layer and doesn't understand HTTP/2 streams. All traffic from one pod to your gRPC service goes through a single connection, which defeats the load balancing entirely. Use client-side load balancing with headless Services and a gRPC name resolver, or route gRPC through an Envoy sidecar (service mesh). This is the most common production surprise with gRPC on Kubernetes.

Q: Is gRPC-Gateway worth using for exposing gRPC services as REST? A: For teams that need both a gRPC interface for internal services and a REST interface for external clients, yes it's significantly less work than maintaining two separate implementations. The tradeoff is an extra layer in the request path and a more complex proto setup with HTTP annotations. Worth it if you're in that situation; unnecessary overhead if you're not.

Q: How do I test gRPC handlers without spinning up a real server? A: Use bufconn from google.golang.org/grpc/test/bufconn. It creates an in-memory listener that behaves like a real network connection without the port binding overhead. Your test creates a server on the bufconn listener, creates a client connecting to it, and runs assertions against real RPC calls. No mocking the gRPC layer, no brittle stub behavior.

Q: When does gRPC streaming actually make sense? A: Server streaming makes sense when results arrive incrementally a search endpoint that streams results as they're found, a log tail, a file download chunked into frames. Client streaming makes sense for bulk data ingestion where you're sending many records. Bidirectional streaming is for real-time protocols where both sides push updates. For most CRUD-style internal service calls, unary RPC is simpler and easier to reason about. Reach for streaming when your use case actually needs it, not because the feature exists.

The patterns here aren't optional at scale. Deadline propagation, correct status codes, and sane retry policy are the difference between a microservices setup that holds under pressure and one that turns a gateway blip into a multi-service outage.

If you're building a new service, wire these in from the start. Retrofitting them into a running system under pressure is the harder lesson.

Internal links used:

• protobuf vs JSON comparison

payload size context

• Circuit Breaker post

cascade failure prevention

• Go Gotcha #8 post

context propagation mechanics

• Go graceful shutdown

GracefulStop pattern

External links used:

• None required

Word count: ~2,510