How do you implement error handling, retries, and monitoring?

Long Answer

In heterogeneous environments—APIs, queues, files, third parties—integration workflows fail in many ways: timeouts, throttles, schema drift, duplicates, and partial success. Building reliability and maintainability starts with clear failure semantics, idempotent steps, bounded retries, and first-class observability so issues are detected and fixed fast.

1) Classify failures and handle locally first
Treat errors as transient (timeouts, 429/5xx, network blips), persistent (bad request, schema mismatch), and systemic (outage, dependency down). Handle what you can at the edge: validate and normalize inputs; reject impossible payloads early with actionable error codes. For persistent errors, skip retries and route to a review path. For transient errors, retry with discipline.

2) Idempotency and deduplication
Retries are safe only if actions are idempotent. Assign deterministic operation IDs; store processed IDs with TTL to prevent duplicates. For side effects (payments, emails, shipments), use an outbox pattern: write the event and the mutation in one transaction, then publish from the outbox to a queue; consumers mark message handling with a dedupe key. When calling external APIs, send an Idempotency-Key and verify server behavior.

3) Timeouts, budgets, and backoff
Every outbound call needs a timeout shorter than the user or job budget; propagate a per-workflow deadline to avoid runaway retries. Use exponential backoff with jitter (for example, base × 2^n ± random) and a max retry count or max elapsed time. Combine with circuit breakers: open on consecutive failures or high latency; half-open to probe; close on recovery. Prefer hedged requests for tail latency only when side effects are safe.

4) Retries by layer (sync vs async)
For synchronous APIs, retry idempotent GET/PUT with small backoff; never retry unsafe POSTs without an idempotency key. For asynchronous steps, prefer queue-driven workers with retry schedules (for example, 30s, 2m, 10m, 1h) and dead-letter queues (DLQs) after N attempts. Include failure metadata (error type, stack, payload digest, correlation ID) so DLQ messages are easily triaged and replayed.

5) Sagas and compensation
Distributed transactions need sagas: a sequence of steps with compensating actions. Model each step as “execute/undo”; persist saga state and events. On failure, run compensations in reverse (cancel reservation, refund payment, release slot). Emit saga progress and outcomes to a stream for auditability.

6) Contract stability and schema governance
Most “errors” are contract drift. Use schema registries and versioned contracts with compatibility checks (backward/forward). Reject unknown critical fields; tolerate benign additions. Add feature flags to roll out new fields gradually. Provide contract tests against partner sandboxes.

7) Observability and monitoring
Treat monitoring as part of the contract. Emit structured logs (JSON) with trace_id, span_id, correlation_id, partner name, attempt number, backoff, and outcome. Capture metrics following RED/USE: request rate, error rate, duration; queue depth, age, DLQ size. Define SLOs (success rate, p95 latency) and alert on error-budget burn rather than single spikes. Use distributed tracing across services and third-party boundaries to see where time and failures occur.

8) Runbooks and replayability
Every alert must map to a runbook: hypothesis, quick checks, safe mitigations (pause consumer, open breaker, increase backoff, flip feature flag), and replay instructions. Make workflows replayable: store the minimal payload and context; ensure reprocessing is safe (idempotent) and bounded (do not storm dependencies). Provide an operator console to requeue DLQ items, edit minor payload issues, or override with compensations.

9) Security and compliance in failure paths
Failures are not an excuse to leak data. Redact PII in logs by policy; encrypt DLQ storage; enforce least privilege on replay tools. Retain failure artifacts only as long as policy allows; apply hashing for identifiers. For partners, separate sandbox and production credentials; prevent cross-environment replay.

10) Maintenance, tests, and cost control
Write failure-first tests: simulate timeouts, throttles, malformed payloads, duplicate deliveries. Use chaos and load tests on queues to ensure backpressure and breakers behave. Track cost metrics for logs, traces, and retries; apply sampling, cardinality limits, and log levels to keep spend predictable without losing signal.

Designing error handling, disciplined retries, and actionable monitoring yields workflows that degrade gracefully, self-heal when possible, and surface clear signals to operators—turning integration from an opaque tangle into a maintainable, auditable system.

‍

Table

‍

Common Mistakes

Endless retries without timeouts or budgets, amplifying outages. Missing idempotency, causing duplicate charges or emails. Retrying client errors (400s) instead of fixing payloads. Ignoring partner rate limits and creating retry storms. No circuit breaker, so threads pile up during dependency failure. Dumping everything into logs without structure or redaction, making triage slow and risky. Skipping DLQs, so poison messages block queues. Lacking saga compensations, leaving half-completed business flows. No correlation IDs, so you cannot follow a request across hops. Alerting on noisy host metrics rather than SLO impact, burning out on-call.

‍

Sample Answers (Junior / Mid / Senior)

Junior:
“I validate inputs early and classify errors. For transient failures I retry with exponential backoff and timeouts; for bad requests I stop and log. I add IDs so operations are idempotent and send failed messages to a dead-letter queue for review.”

Mid:
“I implement circuit breakers and per-workflow deadlines. All producers use the outbox pattern; consumers dedupe by message key. We expose RED metrics and traces with correlation IDs. DLQs include payload digest and stack so operators can replay safely. Multi-step flows are sagas with compensations.”

Senior:
“I design failure budgets and SLO-based alerts, not raw CPU. Contracts are versioned with compatibility tests. Retries have jitter and caps; breakers protect dependencies. A runbook catalog covers pauses, backoff tuning, and replay. Security applies to failure paths: redaction, encryption, and least-privilege consoles. Chaos tests validate backpressure and recovery.”

‍

Evaluation Criteria

Strong answers show: error classification; idempotency and dedupe; retries with exponential backoff + jitter, timeouts, and circuit breakers; DLQs with replay; saga compensations; structured logs with correlation IDs; RED/USE metrics and distributed tracing; SLO-based alerting and clear runbooks. They mention schema governance and the outbox pattern for exactly-once intent. Red flags: infinite retries, missing idempotency, no DLQ, retrying 400s, no breakers, unstructured logs, or alerts tied to hosts rather than user impact. Bonus: cost controls (sampling), security of failure data, and chaos validation.

‍

Preparation Tips

Build a demo pipeline: HTTP ingress → normalize → enqueue → worker → partner API. Add idempotency keys and an outbox table on writes. Implement retry middleware with exponential backoff + jitter, per-step timeouts, and a circuit breaker. Configure a DLQ with a replay tool that shows payload, attempt count, and last error. Emit JSON logs with trace_id and RED metrics; wire distributed tracing across components. Define SLOs and burn-rate alerts; write runbooks for partner outage, schema error, and duplicate delivery. Add tests that simulate 429 throttles, 500 storms, malformed payloads, and partial saga failure, verifying compensations and replay.

‍

Real-world Context

A marketplace suffered duplicate charges during partner flaps. Adding idempotency keys, outbox publishing, and consumer dedupe eliminated repeats despite retries. A logistics integrator faced queue jams from poison messages; introducing DLQs with replay and payload linting cut manual unblocks from hours to minutes. A fintech stabilized third-party calls with circuit breakers and jittered backoff; error budget alerts replaced CPU pages, halving false alarms and MTTR. A subscription platform modeled sign-up + billing as a saga; compensations (refund, revoke access) turned partial failures into predictable outcomes and reduced support tickets.

‍

Key Takeaways

• Classify errors; handle persistent ones without retry and transient ones with discipline.
  
• Make operations idempotent; use outbox and consumer dedupe to enable safe retries.
  
• Apply timeouts, exponential backoff with jitter, and circuit breakers.
  
• Use DLQs and replay tools for poison messages and auditability.
  
• Orchestrate multi-step flows with sagas and compensations, and instrument everything with metrics, logs, and traces.
  
  

Practice Exercise

Scenario:
You own an order-to-shipment integration spanning a storefront, payment processor, inventory service, and a 3PL API. Incidents include duplicate shipments, stuck queues, and hard-to-trace partner outages.

Tasks:

1. Add idempotency: assign an operation ID at checkout; persist a processed-key store; implement an outbox to publish “order_paid” atomically with the payment write.
   
2. Introduce retry middleware with per-step timeouts and exponential backoff + jitter; cap attempts by elapsed time. Respect partner 429s.
   
3. Protect dependencies with a circuit breaker; expose breaker state in metrics and a safe toggle.
   
4. Model the flow as a saga: reserve inventory → charge card → create 3PL shipment; define compensations (release stock, refund, cancel label) and persist saga state.
   
5. Configure DLQs for workers after N failures; build a replay console that shows payload digest, attempts, last error, and compensation status.
   
6. Implement observability: structured logs with correlation IDs across all hops, RED metrics, queue depth, DLQ size, and distributed tracing to the 3PL boundary.
   
7. Create SLOs (success rate, p95 end-to-end latency) and burn-rate alerts; write runbooks for “3PL outage,” “schema mismatch,” and “duplicate event.”
   
   

Deliverable:
A working prototype or design doc proving disciplined error handling, safe retries, actionable monitoring, and maintainable integration workflows with clear recovery paths.