How do you design a high-concurrency system in Go?

Long Answer

High concurrency in Go emerges from a few disciplined ideas: isolate work in goroutines, express coordination with channels, and constrain throughput with backpressure. The goal is sustained p95 latency, bounded memory, and predictable shutdown. Below is a blueprint that balances throughput and safety.

1) Concurrency model and contracts
Goroutines are lightweight threads, but uncontrolled fan-out causes memory spikes and tail latency. Establish explicit contracts: who produces, who consumes, when to stop. Use context.Context as the first parameter for all long-lived operations. A function that spawns goroutines is responsible for their lifecycle.

2) Structured concurrency
Adopt errgroup to fan-out work and join on completion or first error. This collects errors, propagates cancellation, and prevents leaked goroutines. Combine with WithCancel or WithTimeout so slow branches do not hold the group hostage. Prefer returning values through local variables guarded by closure scope or via typed result channels that you close exactly once.

3) Backpressure and worker pools
Bound concurrency with pools. A worker pool is a fixed number of goroutines pulling tasks from a buffered channel. The buffer sets queue depth; the pool size sets parallelism. If producers exceed capacity, they block or fail fast, preventing unbounded memory growth. Separate pools per dependency (database, cache, third-party API) to avoid head-of-line blocking across unrelated resources. For uneven work, use a single shared task queue; for dissimilar latency classes, use multiple queues.

4) Pipelines and cancellation
Model multi-stage flows as pipelines: source → transform → sink, each stage a goroutine reading from an input channel and writing to an output channel. Every for range ch must also select on ctx.Done() to exit promptly. Close output channels only by the sender, and do so exactly once when the sender is finished. Document ownership to avoid double closes or sends on closed channels.

5) Avoiding deadlocks
Deadlocks arise from circular waits, unbuffered channels with mismatched send/receive, and forgetting to drain. Rules: never hold a lock when sending on a channel whose receiver might take the same lock; never send to a channel you also read from while holding a lock; avoid synchronous bidirectional handshakes. If two goroutines must exchange data, prefer a single direction plus an ack channel or merge through a coordinator.

6) Race-free data sharing
Prefer immutable messages. When mutation is required, confine state behind a goroutine (actor model) and interact via messages. If you must share memory, choose sync.Mutex for composite invariants and sync/atomic for counters and flags. Keep critical sections small, avoid locking around I/O, and never hold a lock across select. Document lock ordering to prevent cycles.

7) Timeouts, retries, and idempotency
Wrap external calls with context deadlines and circuit breakers. Retries must be idempotent or guarded with de-duplication keys. Expose concurrency limits per dependency with semaphores (buffered channels or semaphore.Weighted), not global goroutine counts. Measure and tune limits based on saturation and p95 latency.

8) Error handling and observability
Return rich errors with %w wrapping. Surface metrics for inflight tasks, queue depth, worker utilization, p50/p95/p99 latency, error rates, and backoff counts. Log with correlation IDs from context. Add health checks that report queue length and breaker state. Create a debug endpoint that dumps goroutine stacks when thresholds are exceeded.

9) Resource safety and shutdown
On shutdown, stop accepting new work, cancel contexts, close ingress channels, drain work, and wait with a bounded grace period. Release connections and flush buffers. Use sync.WaitGroup only for local goroutines you create; prefer errgroup for lifecycles tied to context.

10) Testing and verification
Run go test -race always. Add stress tests that introduce timeouts, slow consumers, and bursty producers. Fuzz message payloads and reorder deliveries. Use -run TestLeak style tests with goroutine snapshots to detect leaks. In canaries, alert on goroutine growth and queue saturation.

11) Monolith or services
Inside a service, use pools and pipelines. Across services, prefer idempotent RPCs or message queues to smooth bursts. Concurrency within a process complements, rather than replaces, horizontal scale.

With these patterns—structured concurrency, bounded pools, context-aware pipelines, careful sharing, and strong observability—you can push Go to very high concurrency while keeping correctness and latency under control.

‍

Table

‍

Common Mistakes

Creating a goroutine per request without bounds and relying on GC to save you. Using unbuffered channels between slow producer and slow consumer, causing incidental deadlocks. Closing channels from multiple places or sending on closed channels. Holding a Mutex while performing I/O or while sending on a channel. Sharing mutable maps or slices across goroutines without protection and hoping the race detector will catch every case. Using WaitGroup without cancellation, leaking workers on failures. Retrying non-idempotent operations, duplicating writes. Global concurrency limits that starve critical paths. Ignoring queue depth and goroutine counts in monitoring. Treating context.Context as optional, leading to stuck calls during shutdown.

‍

Sample Answers

Junior:
I would use a worker pool with a buffered job channel to bound concurrency. Each worker reads jobs in a for select loop that also listens to ctx.Done(). I would validate with go test -race and add a WaitGroup to wait for workers on shutdown.

Mid:
I prefer errgroup.WithContext for fan-out. For external calls I use semaphores to cap concurrency and deadlines for timeouts. I design pipelines with clear ownership of channel closes and add metrics for queue depth and p95 latency. Shared state is wrapped in a small struct with a Mutex and narrow critical sections.

Senior:
I split work by dependency, each with its own pool and breaker. I use actors for hot state, message passing for coordination, and immutable payloads. I prevent deadlocks by banning send/recv while holding locks and by documenting lock order. I gate deploys on race-free tests, leak checks, and SLOs for p95 and saturation.

‍

Evaluation Criteria

Look for structured concurrency (errgroup, context), explicit backpressure (bounded pools, semaphores), and clear pipeline ownership (who closes channels, when). Candidates should prevent deadlocks by avoiding sends under locks, by documenting lock order, and by testing with the race detector. Strong answers separate per-dependency limits, use timeouts and idempotent retries, and expose metrics for queue depth, goroutines, p95, and breaker trips. Red flags: unbounded goroutines, casual channel closes from multiple owners, no context, locking around I/O, or relying solely on WaitGroup without cancellation. Bonus: actor model for hot state, atomic for counters, circuit breakers, and leak tests with goroutine snapshots.

‍

Preparation Tips

Build a sample service that fetches URLs with a bounded pool and per-host semaphores. Add errgroup to fan-out, context timeouts, and retries with jitter. Instrument queue depth, inflight workers, goroutine counts, p95 latency, and error rates. Create a three-stage pipeline (fetch → parse → store) with channels and ensure each stage halts on ctx.Done(). Add a circuit breaker around storage. Write tests under -race and add a leak test that samples runtime.NumGoroutine. Fuzz the parser. Load test with bursts to validate backpressure and confirm memory does not grow without bound. Finally, practice a graceful shutdown: stop intake, close channels, drain pools, and verify that no goroutines remain.

‍

Real-world Context

A crawler began as one goroutine per URL and crashed under spikes. Introducing a bounded pool per host and a global semaphore stabilized memory, and p95 latency flattened. A payments system used errgroup with deadlines to orchestrate risk checks; failures cancelled downstream steps and eliminated leaks. A streaming pipeline deadlocked when a stage sent while holding a lock; refactoring to message passing and documented lock order fixed it. A cache service replaced shared mutable maps with an actor goroutine that owned state; races vanished and throughput grew. After adding metrics for queue depth and goroutine counts, on-call could spot saturation early and bump pool sizes safely. Race detector and leak tests caught a subtle double close in a rare error path before release.

‍

Key Takeaways

• Use structured concurrency with errgroup and context.
  
• Bound throughput with worker pools, buffers, and semaphores.
  
• Prefer message passing and immutable data; lock minimally when needed.
  
• Prevent deadlocks by avoiding sends under locks and by clear ownership.
  
• Prove safety with the race detector, leak tests, and saturation metrics.
  
  

Practice Exercise

Scenario:
You must design a Go service that ingests tasks from HTTP, enriches them with two external APIs, and stores results. Traffic is bursty. Requirements: stable memory, p95 ≤ 200 ms under 2k RPS bursts, graceful shutdown, and no goroutine leaks.

Tasks:

1. Ingress: Accept tasks and enqueue into a bounded buffered channel; reject with 429 when full. Attach context with a per-request deadline.
   
2. Pools: Create three pools: fetchPool for API A, verifyPool for API B, and storePool for persistence. Size pools based on backend limits; guard each with a semaphore.
   
3. Fan-out: Use errgroup.WithContext to call both APIs in parallel per task; propagate cancellation on first error.
   
4. Pipelines: Build stages: parse → enrich → verify → store. Each stage reads from input, writes to output, and selects on ctx.Done(). Only the producer closes its output channel.
   
5. Safety: For shared counters use atomic, for shared maps use an actor goroutine. Never hold a Mutex while sending on channels or during I/O.
   
6. Resilience: Add per-dependency timeouts, retries with jitter, and a circuit breaker around each API. Ensure writes are idempotent with request keys.
   
7. Observability: Expose metrics: inflight workers, queue depth, goroutine counts, p50/p95 latency, breaker state. Add a /debug/pprof and a goroutine dump on threshold.
   
8. Shutdown: Stop intake, cancel contexts, close ingress channel, drain pipelines, and wait with a 15 s grace.
   
9. Tests: Run go test -race, a leak test, and a stress test that bursts producers and injects slow API responses.
   
   

Deliverable:
A short design doc, metrics dashboards, and passing tests proving bounded memory, clean shutdown, and p95 ≤ 200 ms under burst load.