Scenario / Data pipelines

Backpressure & Ingestion Pipelines: IoT-Scale Writes

An IoT telemetry platform gets a firmware rollout. Ingestion is cheap until streams, analytics, and storage all back up.

Run this scenario

Briefing

Ingestion pipelines & backpressure

An ingestion pipeline accepts high-volume events, buffers them durably, and lets downstream consumers process them at a sustainable pace.

When producers outrun consumers, data piles up. Without backpressure controls, that backlog spills into user-facing latency or data loss.

When producers outrun consumers, the system needs a shock absorber. That is what queues and event streams are for.

  • Use queues or streams as shock absorbers between producers and consumers.
  • Add workers or analytics sinks when backlog grows faster than it drains.
  • Protect live request paths from telemetry and batch-processing pressure.

Contract

Uptime

99.75%

P95 latency

180ms

Budget

$820/mo

Traffic shape

Rollout-driven ingestion spike with downstream pressure. Baseline 120 users; peak around 3,300 users over 60 hours.

Available components

Server

HTTP request handler Every web app needs at least one server. More servers let you handle more simultaneous requests before latency starts climbing.

Postgres

Primary data store Without a database, your app has no memory. Most dynamic requests eventually depend on it.

Stream

Durable event stream Streams are shock absorbers for high-volume ingestion and fan-out systems where one producer feeds many downstream consumers.

Analytics

Async metrics sink Analytics traffic can be high-volume and noisy. Keeping it async prevents dashboards and event collection from hurting core product latency.

Queue

Async job buffer Moving background work out of the request path keeps the app responsive even when extra processing is needed.

Worker

Background job processor Separating background work keeps checkout, page loads, and other user actions from competing with batch processing.

Redis

In-memory cache layer Popular pages, profiles, and product data often get requested again and again. Serving those from memory is much faster and cheaper.

Replica

Read-only DB copy Many applications read far more often than they write. Replicas let you spread those reads across more machines.

LB

Load balancer If you run more than one server, something needs to decide where each request goes. That is the load balancer.

Rate limiter

Request throttle During abuse events, legitimate traffic competes with junk traffic for server capacity. Filtering noisy traffic at the edge protects the rest of the stack.

Common mistakes

  • Streams and workers must scale together to control backlog.
  • Retries need backoff, priority, and enough worker capacity to avoid self-inflicted load.
  • Telemetry should be isolated behind async sinks before it grows noisy.
  • Count writes per user action. Batch, queue, or sample non-critical writes before they dominate the primary path.

Interview adjacency

  • Design an IoT ingestion pipeline
  • Design event streaming
  • Explain backpressure