I

The money didn't disappear. It's caught between three microservices in an inconsistent state a textbook distributed transaction failure. The debit committed. The credit didn't. The system has no automated path out.

By morning, there are 300 quote-tweets.

The saga pattern was designed for exactly this. Not to prevent failures you can't prevent them in distributed systems but to make partial failures recoverable without a human reading transaction logs at 2 AM.

Why Two-Phase Commit Breaks at Fintech Scale

The classical solution to distributed transactions is two-phase commit. A coordinator polls every participant: "Ready to commit?" Everyone confirms. Coordinator issues the final commit. If anyone hesitates, everyone rolls back. Clean in theory.

Here's the problem. Two-phase commit works by holding database locks across all participants while the coordinator waits for consensus. At GoPay's transaction volume millions of transfers per day this means every payment holds locks across your debit service, credit service, ledger, and notification service simultaneously. Locks mean serialization. Serialization under concurrent load means throughput collapse.

Then there's the coordinator failure case. When the coordinator crashes after telling some participants to commit but before reaching others, those participants sit frozen holding locks, waiting for an instruction that never arrives. Resolving this requires a human examining transaction logs and manually deciding what committed and what didn't.

Picture that scenario landing during Harbolnas, when traffic spikes 15x overnight.

Two-phase commit makes a tradeoff that was acceptable in the monolith era: availability and throughput in exchange for atomicity. Fintech at Indonesian scale can't take that trade. You need a different model.

What the Saga Pattern Actually Does

A saga replaces the single distributed transaction with a chain of local transactions. Each step commits immediately to its own database and either publishes an event or triggers the next step directly. No global coordinator. No cross-service locks. Just a sequence of fast, local commits.

The mechanism that makes this safe: every step has a corresponding compensation action. If step three fails, the saga doesn't attempt a global rollback steps one and two already committed and can't be atomically undone. Instead, it executes compensations in reverse: undo step two, then undo step one. Each compensation is a deliberate business action that reverses the effect of its corresponding forward step.

For a wallet transfer, the saga looks like this:

1. Debit service deducts Rp 500,000 from sender → publishes FundsDebited
2. Credit service adds Rp 500,000 to recipient → publishes FundsCredited
3. Notification service sends confirmation SMS → publishes NotificationSent

Step 2 fails. The saga triggers the compensation for step 1: refund the deduction to the sender. The user's money comes back. The recipient never had it to begin with. No inconsistent state. No 300 quote-tweets.

No locks spanning services. No coordinator that can strand participants mid-commit. Failure has a defined response.

Choreography vs Orchestration Pick One

Two coordination models exist for sagas. The difference between them shapes how maintainable your codebase is six months after you ship it.

Choreography: each service reacts to events. The debit service publishes FundsDebited. The credit service subscribes to that event and starts work when it arrives. No central brain just a chain of reactive services.

This sounds elegant. It is, until you have eight saga steps with three branching failure paths. Now the full business logic for a single transaction is distributed across eight services. Debugging a failed saga means tracing events across Kafka partitions, correlating logs from four different services, reconstructing a sequence that no single place describes. New engineers can't read the system they have to feel their way through it.

Orchestration: a dedicated saga orchestrator calls each participant in sequence, waits for results, and manages compensations when things go wrong. The entire transaction flow lives in one place. A new engineer can read it and understand the full business logic in ten minutes.

The orchestrator sounds like the two-phase commit coordinator central, potentially dangerous. The difference is fundamental: the orchestrator doesn't hold database locks. It tracks state and makes API calls. It can crash, restart, and recover from its last saved checkpoint without corrupting anything.

For most Indonesian fintech teams, orchestration is the right call. Choreography's elegance is real but fragile distributed business logic is the thing that kills you when a new payment edge case surfaces at 3 AM. Build the orchestrator. Keep the flow readable.

The Part Most Teams Get Wrong: Compensation Isn't Undo

Almost every first saga implementation gets the happy path right and the compensation wrong.

The mistake is treating compensation as the arithmetic inverse of the forward step. The debit took Rp 500,000 away, so compensation adds Rp 500,000 back. Feels correct.

Here's what actually happened. The debit ran at 11:48:03. The compensation triggered at 11:48:51. In those 48 seconds, the user might have received a cashback reward applied to that balance. Another concurrent transaction might be mid-flight. A naive "add the original amount back" might produce a balance that doesn't reconcile with your ledger and you won't catch it until month-end.

Compensation transactions need to be semantic inverses, not arithmetic ones. They express a business intent "this saga failed, restore the sender's pre-debit state as of this transaction reference" not a raw delta. They also need to be idempotent. The orchestrator will retry on timeout. The compensation endpoint must detect a duplicate call and return the same result without processing twice.

In Go, every saga endpoint forward and compensation starts with an idempotency check:

Untitled
Copy
1func (s *DebitService) Compensate(
2    ctx context.Context,
3    req CompensationRequest,
4) (*CompensationResponse, error) {
5    // Check idempotency key before doing anything
6    if existing, err := s.store.GetByIdempotencyKey(ctx, req.IdempotencyKey); err == nil {
7        return existing, nil // Already processed return cached result
8    }
9
10    // Fetch current account state for a semantic (not arithmetic) compensation
11    account, err := s.store.GetAccount(ctx, req.AccountID)
12    if err != nil {
13        return nil, fmt.Errorf("fetch account: %w", err)
14    }
15
16    entry := LedgerEntry{
17        AccountID:      req.AccountID,
18        Amount:         req.OriginalDebitAmount,
19        Direction:      Credit,
20        SagaID:         req.SagaID,
21        IdempotencyKey: req.IdempotencyKey,
22        Reason:         "saga-compensation",
23        Timestamp:      time.Now().UTC(),
24    }
25
26    if err := s.store.ApplyLedgerEntry(ctx, account.Version, entry); err != nil {
27        return nil, fmt.Errorf("apply compensation: %w", err)
28    }
29
30    result := &CompensationResponse{
31        Status:    "compensated",
32        SagaID:    req.SagaID,
33        Timestamp: entry.Timestamp,
34    }
35
36    _ = s.store.StoreByIdempotencyKey(ctx, req.IdempotencyKey, result)
37    return result, nil
38}

The account.Version in ApplyLedgerEntry is optimistic locking the compensation only applies if the account hasn't been concurrently modified in a way that conflicts. This is the detail that prevents the balance inconsistency nobody finds until auditors arrive.

A Payment Saga in Go: Top-Up Flow

This is a composite of patterns common across Indonesian e-wallet platforms. Not any single company's internal architecture, but the decisions these teams actually make.

A bank top-up spans four services:

1. Bank connector: initiates the transfer and gets a bank reference number
2. Pending transaction service: creates the record the user sees immediately ("Processing...")
3. Balance service: credits the wallet on bank confirmation
4. Audit service: writes an immutable entry to the transaction log

The orchestrator persists saga state after each step. If the process crashes, it restarts and resumes from the last saved checkpoint no re-processing committed steps:

Untitled
Copy
1func (o *TopUpOrchestrator) Execute(ctx context.Context, saga *TopUpSaga) error {
2    // Step 1: Initiate bank transfer
3    if saga.Step < 1 {
4        ref, err := o.bankConnector.Initiate(ctx, saga.BankRequest)
5        if err != nil {
6            return fmt.Errorf("bank initiation: %w", err)
7        }
8        saga.BankRef = ref
9        saga.Step = 1
10        if err := o.persist(ctx, saga); err != nil {
11            return err
12        }
13    }
14
15    // Step 2: Create pending transaction record
16    if saga.Step < 2 {
17        if err := o.pendingTx.Create(ctx, saga); err != nil {
18            _ = o.bankConnector.Cancel(ctx, saga.BankRef, saga.ID)
19            saga.Status = "compensated"
20            _ = o.persist(ctx, saga)
21            return fmt.Errorf("pending record: %w", err)
22        }
23        saga.Step = 2
24        _ = o.persist(ctx, saga)
25    }
26
27    // Step 3: Credit balance (after bank confirmation)
28    if saga.Step < 3 {
29        if err := o.balance.Credit(ctx, saga); err != nil {
30            _ = o.pendingTx.MarkFailed(ctx, saga)
31            _ = o.bankConnector.Cancel(ctx, saga.BankRef, saga.ID)
32            saga.Status = "compensated"
33            _ = o.persist(ctx, saga)
34            return fmt.Errorf("balance credit: %w", err)
35        }
36        saga.Step = 3
37        _ = o.persist(ctx, saga)
38    }
39
40    // Step 4: Audit log (non-compensatable retry until success)
41    if saga.Step < 4 {
42        backoff := 100 * time.Millisecond
43        for {
44            if err := o.audit.Write(ctx, saga); err != nil {
45                time.Sleep(backoff)
46                backoff = min(backoff*2, 5*time.Second)
47                continue
48            }
49            break
50        }
51        saga.Step = 4
52        saga.Status = "completed"
53        _ = o.persist(ctx, saga)
54    }
55
56    return nil
57}

Two things worth calling out.

Every section guards with saga.Step < N. The orchestrator is reentrant by design. Restart it after a crash and it skips already-committed steps, resumes at the right point, and never double-processes. This is state machine persistence the capability that makes orchestration viable in production where processes can die at any moment.

Step 4 retries with exponential backoff indefinitely. The audit log is append-only. You don't compensate an audit entry you write a compensating entry if the saga failed. An infinite retry loop on a non-compensatable, idempotent write is correct here. The write eventually succeeds or the process stays alive until it does.

The [→ Read: API Design for High-Throughput Systems] post covers the idempotency and versioning patterns that every endpoint in this saga depends on. Those aren't optional extras they're the contract sagas require to function.

FAQ

Q: Is the saga pattern only for payment and fintech systems? A: No. The saga pattern applies anywhere a multi-step business process spans multiple services or databases. E-commerce order fulfillment reserve inventory, charge card, trigger shipping uses the same model. So does ride-hailing dispatch and logistics tracking. Fintech just makes failures most visible because money inconsistencies are noticed immediately.

Q: What's the right message queue for choreography-based sagas in Go? A: For high-volume Indonesian payment flows, Kafka is the standard choice it has the durability guarantees and replay capability that payment systems need. RabbitMQ works for lower-volume systems. The broker matters less than your consumer design: every consumer must be idempotent and handle at-least-once delivery. Build that first, then choose the broker.

Q: What do you do when a compensation itself fails? A: Retry with exponential backoff and make the compensation idempotent so retries are safe. If a compensation genuinely can't complete automatically a bank reversal the bank's API rejects it escalates to a human resolution queue. Design this queue before you need it, because you will need it. The question is whether you have a process for it or an ad-hoc crisis when it happens.

Q: How does this relate to CQRS? A: They solve different problems at different layers. [→ Read: CQRS: The Pattern CTOs Misapply Most] CQRS separates reads from writes within a single service. Sagas coordinate writes across multiple services. In practice, mature fintech architectures use both: CQRS within individual services for read performance and event sourcing, sagas across services for distributed transaction integrity. They compose well.

Q: What's a minimum viable saga for a Seed-stage startup that can't build a framework yet? A: A database table with columns for saga_id, current_step, status, payload, and updated_at. A recovery job that runs at startup and resumes any saga where status is in_progress and updated_at is older than your timeout. Idempotency keys on every service endpoint. That's roughly 300 lines of Go and covers 90% of what a framework gives you. Add abstraction when the complexity demands it, not before.

There's no version of distributed transactions that's free of tradeoffs. The saga pattern's cost is upfront complexity designing compensation logic before you need it, building idempotent endpoints, maintaining orchestrator state. The reward is that when something breaks at 11:48 PM, the system recovers on its own. No 300 quote-tweets. No data team audit the next morning.

If your payment flow spans multiple services and compensation logic is still on the backlog, that item is more urgent than it looks. SpectreDev has worked through this with fintech teams in Indonesia and Australia the failure modes in production are more varied than staging suggests.