How do you monitor, log, and troubleshoot web services?

Long Answer

Production-grade web services monitoring is a system, not a tool. It links signals, logs, traces, SLOs, and disciplined ops so issues are spotted early, diagnosed quickly, and fixed safely. My playbook: golden-signal telemetry, structured logs, distributed tracing, budget-based alerts, and a repeatable troubleshoot-and-rollback loop.

1) Golden signals & health
Track latency, traffic, errors, saturation—plus dependency health. Expose RED (Rate, Errors, Duration) per endpoint and tenant. Publish SLIs that mirror real journeys (e.g., “/checkout p95 < 300 ms; errors < 0.5%”). Combine app metrics with infra (CPU, GC, DB pools, queue depth) so alerts reflect user pain, not noisy host graphs.

2) Structured logs for forensics
Emit JSON logs with ts, level, service, version, env, trace_id, request_id, tenant, route, status, duration, and redacted params. INFO for lifecycle, WARN for handled anomalies, ERROR for failures with stack. Add fields (error.kind, retryable, feature_flag) for surgical search. Enforce redaction; tier retention to control cost.

3) Tracing & correlation
Adopt OpenTelemetry; propagate W3C trace context so every request carries a correlation ID across services and queues. Add span attributes for DB statements (sanitized), cache hits/misses, external calls, and flags. Tail-sample on errors/latency.

4) SLOs & alerting
Define user-centric SLOs and compute error budgets. Alert on budget burn (e.g., “2% in 1h”) with multi-window rules to avoid pager noise yet page instantly for real incidents. Link alerts to runbooks with deploys, flags, dashboards, and a first-hour checklist.

5) Troubleshooting flow
When paged: open the service dashboard; pivot to logs by trace_id; follow the trace to the failing span; verify infra (CPU, memory, DB, queues, dependencies); mitigate safely (rollback, disable flag, raise pool caps, shed load); record the timeline.

6) Change awareness & rollback
Most incidents are change-triggered. Annotate graphs with deploys and traffic shifts. Use canary/progressive delivery and auto-rollback on guardrail breaches.

7) Dependency & edge visibility
Instrument DBs, caches, queues, and third-party APIs. Track pool exhaustion, timeouts, retry rates. At the edge, watch CDN hit ratio, origin errors, TLS failures. Run regional synthetics to prove end-to-end paths.

8) Reliability hygiene
Run blameless postmortems; convert fixes into backlog: tests, circuit breakers, bulkheads, idempotency, backpressure. Load-test critical flows; rehearse game days.

9) Cost & data quality
Control observability sprawl via cardinality budgets, log sampling, and retention policies. Keep data trustworthy: schemas in version control, label linting, and checks that new services ship with metrics, logs, and traces.

Together, this web services monitoring stack surfaces the right symptoms, provides forensics to explain them, and gives operators safe levers to recover fast—turning outages into short, teachable moments instead of brand-damaging events.

‍

Table

‍

Common Mistakes

Paging on noisy host metrics instead of user-visible SLIs, so teams chase ghosts. Unstructured logs without correlation IDs, making incidents a needle-in-haystack search. Sampling traces only by fixed rate, missing the rare bad path that hurts users. One-alert-per-symptom spam that causes pager fatigue; no runbooks to guide responders. Ignoring change as the root cause: no deploy annotations, no canaries, no fast rollback. Relying on ad-hoc ssh + grep rather than searchable, redacted JSON logs. Zero visibility into dependencies—DB pools, queues, third-party APIs—so fixes poke in the dark. Letting observability costs explode via high-cardinality labels and infinite retention. Skipping postmortems, so the same failure repeats. Finally, treating web services monitoring as a tool install, not a practice owned by engineering and SRE together. Teams also forget to redact PII, leaking secrets; or they trust client metrics alone, missing edge/CDN & regional outages that users feel.

‍

Sample Answers (Junior / Mid / Senior)

Junior:
“I’d start with basic web services monitoring: collect latency, error rate, and traffic per endpoint, and pipe JSON logs with request_id to a searchable store. I’d add simple SLOs and alerts with runbooks so I know the first checks to run.”

Mid:
“I design dashboards around SLIs and golden signals, add structured logs with correlation IDs, and enable OpenTelemetry tracing across services. Alerts are budget-burn with multi-window rules to avoid noise. My first-hour playbook is dashboards → logs → traces → rollback or flag flip.”

Senior:
“I make web services monitoring a product: user-centric SLOs, tail-sampled traces on anomalies, semantic logs, and change-aware alerts wired to auto-rollback. We instrument dependencies (DB, cache, queues, third-party APIs), run synthetics from key regions, and practice blameless postmortems that feed backlog work. Cost/cardinality guardrails keep the data useful and affordable. I also require deploy annotations and feature flags on risky paths so incidents can be contained in minutes without guessing or hot-patching.”

‍

Evaluation Criteria

Strong answers frame web services monitoring as a layered practice: golden signals + SLIs/SLOs; structured JSON logs with correlation IDs; OpenTelemetry tracing across hops; and alerting on error-budget burn, not single spikes. Candidates should describe a repeatable troubleshooting flow (dashboards → logs → traces → infra checks → mitigation) and show change awareness (deploy annotations, canary/progressive delivery, fast rollback). Look for dependency observability (DB pools, caches, queues, third-party APIs), edge visibility (CDN hit ratio, origin errors), and synthetics. Security and cost matter: PII redaction, access controls, cardinality budgets, and retention tiers. Evidence of learning loops—blameless postmortems, fixes turned into backlog, and game-day drills—signals maturity. Weak answers: tool name-dropping without SLIs, paging on CPU, or no plan to correlate logs/traces. Also welcome: rollback automation and flag-gated releases.

‍

Preparation Tips

Build a small service and practice web services monitoring end-to-end. Instrument RED metrics per route; export SLIs (p95 latency, error rate) and set SLOs. Emit JSON logs with request_id/trace_id and semantic fields; push to a search store. Add OpenTelemetry tracing and verify a trace stitches across HTTP + queue hops. Create burn-rate alerts (multi-window) tied to runbooks that list the first-hour checks. Simulate incidents: bad deploy, database saturation, third-party timeouts. Follow the playbook: dashboard → logs → traces → safe rollback/flag flip; capture timelines. Add deploy annotations, canary, and auto-rollback in CI/CD. Instrument dependencies (DB, cache, queue) and edge (CDN). Create synthetic probes from two regions and a ‘quiet hours’ schedule. Finally, run a blameless postmortem and convert fixes into tests, circuit breakers, and budgets; prune high-cardinality labels and set retention tiers. For interviews, rehearse a 60–90s story with before/after: pager noise down, MTTR down, and which signals proved the fix. Bring a trace view and an SLO burn chart to anchor it.

‍

Real-world Context

A payments API saw p95 jump from 220 ms to 1.6 s after a Friday deploy. Because web services monitoring was wired to change, the dashboard showed a deploy marker at the surge. A burn-rate alert paged SRE; the first-hour runbook led them to logs filtered by trace_id. Tracing highlighted a new ORM path causing N+1 queries and DB pool exhaustion. Mitigation: flag-off the feature and auto-rollback. p95 fell in minutes; customers saw only brief degradation.

Elsewhere, a media service had intermittent 5xxs that didn’t reproduce. Tail-sampled traces on errors revealed a 3rd-party thumbnail API timing out for EU users. Edge metrics showed low CDN hit ratio from a single region. A synthetic probe confirmed the path. The team added a circuit breaker, cached default thumbnails, and moved traffic via canary to a healthier POP. Incidents dropped to near zero, and costs fell after trimming high-cardinality labels and long log retention. Both cases show how signals + logs + traces + change awareness deliver fast, reliable fixes instead of long firefights.

‍

Key Takeaways

• Treat web services monitoring as a product: SLIs/SLOs + golden signals.
  
• Use structured JSON logs and OpenTelemetry traces with correlation IDs.
  
• Alert on error-budget burn; attach runbooks and deploy context.
  
• Keep rollback/flags ready; instrument dependencies and edge.
  
• Control cost/cardinality; learn via postmortems and drills.
  
  

Practice Exercise

Scenario: You own a checkout service. Users report sporadic timeouts; error budget is burning fast during peaks.

Tasks:

1. Instrument web services monitoring: export RED metrics per route and dependency (DB, cache, payment API). Publish SLIs (p95 latency, error rate) and set SLOs.
   
2. Logging: emit JSON logs with request_id/trace_id, tenant, route, status, duration, error.kind. Enable structured queries for “time>1s AND route=/charge”.
   
3. Tracing: enable OpenTelemetry across HTTP + queue hops. Add spans for DB calls and payment API with retry/timeout tags; verify correlation from edge to DB.
   
4. Alerting: create a 2-window burn-rate alert (1h + 6h) tied to a runbook. The runbook lists first-hour checks, rollback steps, and flag names.
   
5. Drill: deploy a known slow query; watch burn-rate page you. Follow the playbook: dashboard → logs → traces → rollback/flag flip. Capture the timeline.
   
6. Prevention: add a slow-query budget per endpoint, DB connection caps, and a circuit breaker for the payment API. Trim high-cardinality labels; set log retention.
   
7. Evidence: attach screenshots—SLO chart, trace of the slow span. Summarize MTTR and ticket volume before/after.
   
8. Synthetics: add regional probes for /healthz and checkout; alert on SLO breaches.
   
9. Change safety: add deploy annotations and canary; auto-rollback on error spikes.
   
10. Report: measure cost after pruning labels and moving logs to warm storage.
    
    

Deliverable: a 90-second walkthrough proving detection was fast, diagnosis precise, and recovery safe—with the policies you’ll keep to prevent recurrence.