How would you architect a multi-tenant Python web service?

Long Answer

Designing a large Python web service architecture for multi-tenant, event-driven workloads means aligning domain boundaries with tenancy, selecting the right execution model (ASGI versus WSGI), and moving slow or spiky work to background jobs (Celery or RQ) so request latency stays flat.

1) Domain boundaries and control versus data planes
Split the system into a control plane (authentication, organizations, billing, provisioning, feature flags) and data planes that hold tenant data and business logic. Organize code by bounded contexts, not layers: accounts, catalog, billing, analytics. Each context exports commands, queries, and events; internal models do not leak. In Django, this maps to apps per domain; in FastAPI, to routers and packages per domain. Public contracts (OpenAPI) are small and versioned, while internal types remain private.

2) Multi-tenancy models and isolation
Select a tenancy model per risk and scale: shared schema with tenant_id, separate schemas per tenant, or separate databases for high compliance. Enforce isolation in three layers: identity (JWT claims or session-to-tenant binding), application (automatic tenant scoping in ORM managers or repository methods), and storage (row-level predicates or schema routing). For Django, use a request middleware that resolves tenant from subdomain or header and sets the current tenant context; for FastAPI, use dependencies to resolve and inject tenant context into handlers and repositories. Never trust client-sent tenant fields without cross-checking the authenticated claim.

3) API shape, read models, and caching
Keep writes canonical and event-driven. Model aggregates that emit domain events (OrderPlaced, InvoicePaid) to a durable log or outbox table. For read-heavy endpoints, build read models (denormalized projections) and cache them per tenant with strict keys, time-to-live, and invalidation on relevant events. In Django, pair the ORM with custom managers and select-related prefetching; in FastAPI, prefer repository abstractions so the web layer stays free of ad-hoc queries.

4) ASGI versus WSGI and concurrency
Prefer ASGI servers (Uvicorn, Hypercorn) for concurrent I/O, WebSockets, and Server-Sent Events. Framework choice follows need: FastAPI excels at typed, async APIs; Django now supports ASGI for views, channels, and async ORM reads, though many third-party packages remain sync. Use WSGI only for legacy admin flows or libraries that block. Where sync is unavoidable, isolate with thread executors in Django or anyio.to_thread in FastAPI, but measure carefully to avoid starving the event loop. Keep handlers small, await only non-blocking calls, and stream large payloads to prevent head-of-line blocking.

5) Background jobs and the outbox pattern
Side effects belong off the critical path. Use Celery (prefetch limits, acks-late, dead-letter queues, rate limits) or RQ (simple Redis-backed) for idempotent tasks. Persist domain events in an outbox within the same transaction as the write, then publish to the broker in a reliable worker. Tag every message with tenant_id, correlation IDs, and schema version. Shape queues per domain and tier; apply per-tenant concurrency caps to avoid noisy neighbors. For heavy compute, run dedicated worker pools with autoscaling and backpressure.

6) Event-driven workflows and saga coordination
For multi-step operations (for example, order → payment → fulfillment), implement sagas. Begin with choreography (services react to events); move to orchestration when compensations or visibility matter. All handlers must be idempotent, time-bound, and retry-safe. Store processed message keys to deduplicate. Publish state changes as immutable facts; rebuild projections by replay when needed.

7) Data integrity, migrations, and versioning
Normalize your write models, then project for reads. Migrations follow expand → migrate → contract. Use online schema changes and backfills via workers. Version event schemas and API contracts; require consumer contract tests in CI so producers cannot ship breaking changes. For files and exports, stream to object storage and reference via signed URLs, not inline blobs.

8) Observability and tenant-aware operations
Instrumentation is tenant scoped. Emit structured logs with tenant, route, correlation ID, and domain. Track RED metrics (rate, errors, duration), worker queue depth, retry rates, and event lag per tenant. Define SLOs (for example, P95 < 300 ms on read endpoints, outbox-to-publish lag < 10 seconds) and alert on budget burn. Provide admin tooling to pause a tenant (throttle jobs, block writes) without affecting others.

9) Security, quotas, and platform guardrails
Apply least privilege across services. Rotate secrets, pin dependencies, and use a Web Application Firewall and rate limiter at the edge. Enforce per-tenant limits: request rate, concurrent jobs, storage, and expensive queries. Return clear 429 or 403 errors with machine-readable reasons. For customer-managed keys, isolate encryption contexts per tenant and audit all privileged access.

10) Delivery, environments, and testing
Keep a monorepo or coordinated repos with a golden CI pipeline: static checks, type checks (mypy), tests (unit, integration with ephemeral Postgres/Redis, contract tests), and smoke end-to-end. Use seedable tenants and fixture builders. Spin preview environments that include the broker and workers. Run chaos profiles that drop broker messages, slow the database, and spike event rates to verify backpressure, retries, and invariants.

This design yields a multi-tenant Python web service that keeps latency predictable, scales with events and workers, and evolves safely through typed contracts, explicit isolation, and disciplined ASGI/WSGI boundaries.

‍

Table

‍

Common Mistakes

• Treating multi-tenancy as only a database column and forgetting identity and policy checks.
  
• Mixing blocking I/O or ORMs inside async paths, stalling the ASGI loop.
  
• Skipping the outbox and issuing dual writes to the database and broker, causing drift.
  
• One global Celery queue where one tenant or task type starves others.
  
• Overusing synchronous webhooks in request paths instead of publishing events and returning quickly.
  
• Building wide, unversioned contracts; clients break on minor changes.
  
• Caching without tenant-aware keys and cache-control, leaking data between tenants.
  
• Big-bang migrations without expand → migrate → contract and backfill, leading to downtime.
  
• No tenant-scoped metrics; noisy neighbors remain invisible until an outage.
  
  

Sample Answers (Junior / Mid / Senior)

Junior:
“I would resolve the tenant from subdomain or header and verify it against a token claim. I would keep handlers small and async, and move slow work to Celery. I would cache read models per tenant and use idempotency for tasks. I would document small, versioned endpoints.”

Mid:
“My Python web service architecture uses bounded contexts, ASGI for APIs and WebSockets, and WSGI only for legacy admin. Tenancy is enforced at identity and the ORM level. Writes emit events into an outbox that a publisher worker sends to the broker. Celery queues are per domain with per-tenant rate limits and dead-letter queues. Read models and tenant-aware caches keep P95 low.”

Senior:
“I separate control and data planes, define contracts and events per domain, and enforce multi-tenancy in identity, app, and storage layers. Async paths run on ASGI; blocking libraries are isolated or moved to worker services. The outbox guarantees exactly-once effects; consumers are idempotent and versioned. Per-tenant quotas, SLOs, and observability guide scaling and fairness. Migrations follow expand → migrate → contract with replayable projections.”

‍

Evaluation Criteria

A strong response defines a multi-tenant Python web service architecture with: bounded contexts; tenant resolution and enforcement across identity, application, and database layers; ASGI for concurrent I/O and WSGI only where legacy demands; and background jobs via Celery or RQ with outbox-driven events and idempotent consumers. It should include read models, tenant-aware caching, quotas and rate limits, and tenant-scoped observability and SLOs. It should describe migrations and versioned contracts. Red flags: dual writes without outbox, blocking code in async handlers, global unpartitioned queues, cache keys without tenant, and untyped or unversioned APIs.

‍

Preparation Tips

• Build two domains (orders, billing) as Django apps or FastAPI routers with OpenAPI.
  
• Implement tenant resolution (subdomain or header) and cross-check against token claims; enforce scoping in repositories or ORM managers.
  
• Stand up ASGI (Uvicorn) for APIs and WebSockets; keep a small WSGI admin if needed.
  
• Add an outbox table and a publisher worker; emit OrderPlaced and InvoicePaid.
  
• Wire Celery with domain queues, dead-letter queues, retries with jitter, and per-tenant concurrency caps.
  
• Create a read model for “order summary,” cache per tenant with strict keys, and invalidate on events.
  
• Set tenant-scoped dashboards for RED metrics, queue lag, and errors; define two SLOs with alerts.
  
• Practice an expand → migrate → contract: add a field, backfill via workers, switch reads, and remove legacy code.
  
  

Real-world Context

A marketplace migrated to ASGI for its APIs while keeping a legacy WSGI admin. Blocking payment calls were isolated; tail latency dropped. Introducing an outbox stopped occasional order duplication during retries. Celery queues per domain with per-tenant caps prevented a single importer from starving everyone. Read models reduced API P95 by thirty percent, and tenant-scoped caches eliminated cross-tenant leaks. When adding tax fields, the team used expand → migrate → contract with backfill workers and saw zero downtime. Tenant dashboards revealed a noisy neighbor pattern early; quotas and alerts kept the platform fair. The Python web service architecture scaled cleanly and stayed trustworthy.

‍

Key Takeaways

• Separate control and data planes; draw domain boundaries and version contracts.
  
• Enforce multi-tenancy at identity, application, and storage layers.
  
• Prefer ASGI for concurrent I/O; isolate or retire WSGI and blocking code.
  
• Use Celery or RQ with an outbox and idempotent consumers for event-driven workloads.
  
• Build read models and tenant-aware caching; set quotas, SLOs, and tenant-scoped observability.
  
  

Practice Exercise

Scenario:
You are building a multi-tenant Python web service for analytics with ingestion, processing, and dashboards. Tenants ingest bursts of events, expect near real-time charts, and require strict isolation. The stack is Django or FastAPI, ASGI, Postgres, Redis, and Celery.

Tasks:

1. Choose tenant resolution (subdomain or header) and implement cross-check against token claims. Add middleware or dependencies that set a request-scoped tenant context, and enforce scoping in ORM managers or repositories.
   
2. Define domains (ingestion, processing, dashboards, billing). For each, list commands, queries, and events. Publish events to an outbox in the same transaction as writes.
   
3. Stand up ASGI for ingestion and dashboards, with streaming endpoints for charts; isolate any blocking libraries using executors. Keep any legacy admin as WSGI only if required.
   
4. Configure Celery queues per domain with per-tenant concurrency caps, dead-letter queues, and retry policies with jitter. Add idempotent consumers and deduplication keys.
   
5. Build a read model for “tenant time-series summary” with rolling windows; cache results with tenant-aware keys and invalidate on new events.
   
6. Implement quotas and rate limits per tenant (requests per minute, events per minute, storage). Return 429 or 403 with machine-readable reasons and Retry-After.
   
7. Add tenant-scoped observability: RED metrics, outbox lag, queue depth, consumer retry rates, and dashboard P95. Create alerts on error budget burn and noisy neighbor detection.
   
8. Demonstrate an expand → migrate → contract change: add a “channel” field to events. Backfill via workers, flip reads to use the new field, and remove legacy code.
   
9. Provide a runbook: how to pause a tenant safely, how to replay events, and how to recover from dead-letter queues.
   
   

Deliverable:
A concise architecture note, configuration snippets, and a validation plan proving the Python web service architecture supports multi-tenant, event-driven workloads with correct ASGI/WSGI boundaries and robust Celery/RQ background jobs.