How to architect chatbot web apps for speed & reliability?

Designing a chatbot-driven web app that feels instant yet stays reliable under stress starts with an architecture that assumes the LLM can be slow, costly, or unavailable. Combine predictable routing, layered caching, strict concurrency control, and graceful degradation so users always see progress and never hit a dead end.

1) Entry, routing, and streaming UX‍

Use WebSockets or Server-Sent Events for token streaming behind an edge gateway. Classify requests early (intent, safety, tools) with a tiny model or rules so cacheable/toolable paths skip big models. Emit a quick typing signal and stream partial tokens ~300 ms to anchor perceived speed.

2) Layered caching‍

• Prompt+context cache: normalize, hash, and cache with TTL/semantic keys for FAQs.
• Tool cache: memoize deterministic results with freshness windows.
• Embedding cache: store vectors by content hash.
• Embedding cache: store vectors by content hash.
• Namespaces + encryption prevent cross-tenant leakage.

3) Concurrency and rate limits‍

Use multi-axis limits (user, session, IP/ASN, key) and per-conversation budgets (e.g., ≤3 active generations, ≤30k tokens/min). Queue excess and offer a “quick answer” fallback. Global circuit breakers shed load when error rate or p95 latency breach thresholds.

4) Timeouts, retries, hedging‍

Set hard timeouts on LLM and tools (8–12 s). Retry once with jitter for transient faults. For idempotent reads, hedge to a second region after a short delay; cancel the loser. Apply an overall deadline to cap total downstream work.

5) Model routing and fallbacks‍

Pick the smallest model that meets task quality; escalate only on low confidence. Fallback ladder: primary → smaller model → RAG summary → rules/template reply. If a provider degrades, fail over to a secondary via a compatibility layer that standardizes prompts and safety.

6) Retrieval and grounding‍

RAG improves accuracy and cost. Build tenant-scoped indexes; chunk and embed with ACL metadata. Gate retrieval by intent; limit top-k; compress passages before generation. Invalidate caches on document updates.

7) State, windows, and cost‍

Maintain short conversation windows; summarize older turns into rolling memory with citations. Enforce per-turn token budgets; warn and auto-trim long inputs. Track spend by tenant and surface quotas to the UI.

8) Observability and quality‍

Emit RED metrics per route/model/tenant. Log redacted prompts and model metadata (id, tokens, latency). Sample outputs for offline evals (grounding, toxicity). Add canaries to catch silent failures and trigger rollbacks.

9) Security and privacy

‍Sign requests; enforce quotas server-side. Encrypt caches; avoid storing raw chats unless allowed. Minimize vendor payloads; mask secrets; pin regions for regulated tenants.

10) Validation plan‍

Set SLOs: p95 TTFT < 500 ms (hit), < 2.5 s (cold); p95 TLT < 8 s; errors < 1%. Load-test with 10× spikes and injected upstream timeouts; measure TTFT, completion rate, and cost per conversation.

Common Mistakes

Treating the LLM as always-on and skipping caching, so identical prompts re-pay full latency and cost. Using a single global rate limit, letting per-user token churn crash upstreams. Retrying without backoff or jitter, creating retry storms under partial outages. Letting conversations bloat so each turn drags a massive context window. Failing to stream partial tokens, leaving a blank screen. No fallback ladder; when the API blips, chats just die. Cross-tenant caches or raw PII in logs. Zero observability—no TTFT/canaries—so incidents linger. Relying on one region with no failover. Skipping realistic load tests; mobile networks and packet loss go untested. Overusing the biggest model by default; re-embedding unchanged content; letting slow tools block the main path; versionless prompts and safety settings, making rollbacks guesswork.

Sample Answers (Junior / Mid / Senior)

Junior:‍

“I’d stream responses with SSE, cache common answers, and limit requests per user. If the LLM is slow, I’d show partial text and retry once. I’d add a smaller fallback model so the chat never stops.”

Mid:

“I’d add layered caches (prompt, tool, embeddings) and per-chat budgets. Concurrency is capped; overflow queues inform users. LLM/timeouts are enforced; retries use jitter. I’d route to smaller models first, escalate on low confidence, and use RAG for accuracy. Metrics would track TTFT and error rates to spot regressions.”

Senior:‍

“Edge gateway + streaming; early intent routing to skip expensive calls. Multi-axis rate limits and circuit breakers protect upstreams. Fallback ladder: primary → small model → RAG summary → rules. Hedged requests across regions and secondary providers handle incidents. Strict privacy (encrypted caches, minimal payloads). Observability with RED metrics, canaries, and cost per chat. SLOs define success and drive automated rollbacks.”

Evaluation Criteria

Interviewers look for a coherent system design that treats LLMs as unreliable at times and mitigates accordingly:

• Streaming UX with sub-second time-to-first-token.
  
• Layered caching (prompt, tools, embeddings) with privacy controls.
  
• Multi-axis rate limits and per-chat token/concurrency budgets.
  
• Timeouts, single retry with jitter, and request-level deadlines; hedging.
  
• Model routing that prefers smaller models and escalates on low confidence.
  
• RAG grounding with tenant-scoped indexes and ACLs.
  
• Fallback ladder and cross-region/provider failover.
  
• Observability: RED metrics, structured logs, canaries, cost tracking.
  
• Security: signed requests, encrypted caches, minimal vendor payloads.
  
• State discipline and rolling summaries to control context.
  
• Clear validation plan with target SLOs (TTFT, TLT, error rate) and canary checks.
• Governance: quotas per tenant, cost controls, versioned prompts/safety. Answers that only say “add a cache” or “add more servers” score low; those that quantify SLOs and validate via load tests score high.

Preparation Tips

Build a small chat app with SSE streaming. Add a prompt hash cache (Redis), tool memoization, and an embedding cache. Implement token-bucket limits per user and API key; add per-chat concurrency caps and a queue with UI notices. Add timeouts and a single retry with jitter; set an overall deadline. Route by confidence: try a small model, escalate when needed. Wire a simple RAG index (chunk, embed, top-k) and cache retrievals. Add a fallback ladder: small model → RAG summary → rules/template. Introduce hedged reads across two regions and a secondary provider. Instrument TTFT, TLT, tokens, and cost per chat; add canaries and dashboards by route/tenant/model. Load-test: mix cache hits/misses and spike 10×; run chaos drills (timeouts, 500s, region loss) to verify breakers and failover. Control cost with rolling summaries. Document a rollback plan and a prompt/version change log to bisect regressions fast.

Real-world Context

A conversational support portal saw p95 latency swing wildly during launches. Introducing streaming SSE plus a prompt cache cut perceived wait to <400 ms for common queries. A retailer added token-bucket limits and per-chat concurrency caps; upstream errors dropped 70% during flash sales. A SaaS team swapped “always GPT-XL” for a router that tries a small model then escalates; costs fell 45% with no quality loss on benchmarks. Another org added RAG with tenant-scoped indexes and cached retrievals; grounding reduced follow-up questions by 30%. During a provider outage, hedged requests to a second region and a secondary vendor kept TLT within SLOs. Across cases, the wins came from the same trio: layered caching, rigorous limits, and a clear fallback ladder—plus observability that proved the gains.

Key Takeaways

• Stream early; target sub-second time-to-first-token.
  
• Cache prompts, tools, and embeddings; encrypt and namespace caches.
  
• Enforce multi-axis limits, timeouts, and a single jittered retry.
  
• Route to the smallest model first; add RAG and a fallback ladder.
  
• Measure TTFT/TLT/cost; hedge and fail over when providers wobble.

Practice Exercise

Scenario: You’re launching a chatbot for a global retailer. Traffic is spiky (promo drops cause 10× bursts), mobile users dominate, and leadership demands sub-2s perceived response even during incidents.

Tasks:

1. Implement SSE streaming so users see a typing indicator and first tokens in <400 ms. Add a prompt hash cache (Redis) and memoize deterministic tool calls with freshness windows.
   
2. Add token-bucket limits per user and API key, plus per-chat concurrency caps. Queue overflow and surface a “Quick answer” option that uses a smaller model or a cached summary.
   
3. Configure timeouts (LLM/tools 10 s) and a single retry with jitter; add a request-level deadline. Enable hedged reads to a second region and prepare a secondary provider with a prompt/safety compatibility layer.
   
4. Add RAG: tenant-scoped index, top-k retrieval, and passage compression; cache retrieval results. Control context by summarizing old turns with citations.
   
5. Instrument TTFT, TLT, token counts, and cost per chat. Add canaries that run every minute. Build dashboards by tenant and model.
   

Deliverable: Run a load test that spikes 10× with 20% upstream timeouts. Provide before/after charts for TTFT/TLT and a brief rollback plan describing which toggles (limits, hedging, fallback ladder) you would change first during an incident.

Stretch: Add privacy checks: encrypt caches at rest, redact logs, and verify no PII leaves the region. Document SLOs (TTFT, TLT, error rate) and criteria for success, then present a 60–90s verbal walkthrough of design choices, trade-offs, and measured gains.