Backpressure is the control layer that keeps a real-time messaging system usable when producers generate work faster than brokers or consumers can process it. This playbook explains what to measure, how to set starting thresholds, which broker and client controls to review, and how to turn the findings into a repeatable monitoring and incident routine.
Overview
In a healthy messaging system, production and consumption remain close enough that queues, partitions, WebSocket buffers, and worker pools do not grow without limit. Backpressure appears when that balance breaks. A consumer may slow because of a database dependency, a broker may approach a quota, or a burst of WebSocket messages may exceed what connected clients can receive.
Flow control is the collection of mechanisms used to respond. It can include limiting producer rates, restricting in-flight work, pausing consumption, reducing batch sizes, applying connection-level limits, or shedding work that is optional. The objective is not to eliminate every queue. Queues absorb normal variation. The objective is to keep backlog bounded and recovery predictable.
Backpressure should be designed across three points:
- Producers: limit request rates, bound retries, apply timeouts, and decide whether callers should receive a retryable error or wait.
- Brokers: monitor queue depth, partition or subject pressure, storage, network capacity, and service quotas.
- Consumers: cap concurrency, control fetch or delivery batches, acknowledge only completed work, and expose processing latency.
The right settings depend on message size, ordering requirements, delivery guarantees, workload variance, and downstream capacity. Treat all numeric values below as starting points for load testing, not universal limits.
What to track
Use a dashboard that connects symptoms to causes. A single lag number rarely explains whether the issue is a slow consumer, an overloaded broker, or an unusually large producer burst.
| Signal | What it indicates | Useful response |
|---|---|---|
| Consumer lag or age of oldest message | Work is arriving faster than it is being completed | Inspect processing time, consumer health, partition assignment, and downstream dependencies |
| Queue depth or unacknowledged messages | Pending work is accumulating | Reduce intake, increase safe consumer capacity, or route failures to a dead-letter path |
| Publish and delivery rate | The current workload and its direction | Compare producer rate with completed-message rate over the same interval |
| Processing latency and error rate | Whether consumers are slow or failing | Separate transient retries from permanent failures before adding workers |
| In-flight messages and worker utilization | How much work is actively consuming resources | Bound concurrency so a downstream outage does not exhaust memory or connections |
| WebSocket send-buffer size and disconnects | Clients cannot receive messages as quickly as they are produced | Apply per-client limits, coalesce updates, or disconnect persistently slow clients |
| Broker storage, network, and quota utilization | The platform may become the bottleneck | Compare usage with the service's documented limits and capacity plan |
Broker-specific controls should be reviewed alongside these signals. For Kafka consumers, examine max.poll.records, poll intervals, fetch sizing, partition assignment, and the relationship between polling and message processing. A large poll batch can improve throughput but increase memory use and delay individual acknowledgments. For RabbitMQ, prefetch limits the number of unacknowledged deliveries per consumer or channel, depending on the client configuration. A lower value can improve fairness and bound memory; a higher value may improve throughput for short, uniform tasks.
NATS deployments should use the flow-control and heartbeat features supported by the selected client and subscription mode, then verify behavior under a slow subscriber. WebSocket applications need an explicit send-buffer policy because a broker may be healthy while individual clients fall behind. Do not let buffers grow indefinitely: set a maximum, define what can be dropped or coalesced, and record the decision.
A small Prometheus-style starting point might look like this:
max by (consumer_group) (messaging_consumer_lag_messages)
rate(messaging_published_messages_total[5m])
rate(messaging_processed_messages_total[5m])
max by (service) (messaging_oldest_message_age_seconds)
rate(messaging_processing_failures_total[5m])Metric names vary by broker, exporter, and application. Keep the query structure, but map it to the names your platform actually exposes. Alert on sustained conditions rather than a single sample. For example, a lag alert may require both a minimum lag and a positive lag growth rate for several minutes.
Cadence and checkpoints
Review backpressure controls at least monthly for fast-changing systems and quarterly for stable workloads. A useful review should include more than dashboard inspection.
- Compare demand and capacity: record peak publish rate, completed-message rate, largest backlog, and time required to recover after a burst.
- Check configuration drift: compare producer limits, consumer batch sizes, prefetch, retry policies, WebSocket buffer limits, and autoscaling settings with the approved service configuration.
- Run a focused load test: test normal traffic, a short burst, a slow dependency, and a failed consumer. Measure whether backlog remains bounded and whether recovery causes a second overload.
- Review failure handling: confirm that retries have limits, poison messages do not block useful work, and dead-letter queues are monitored and replayable.
- Validate quotas and dependencies: check broker, cloud, database, network, and connection limits before raising concurrency.
- Record decisions: preserve the chosen thresholds, their rationale, test date, owner, and next review date.
A reusable configuration record can be kept beside each service:
service: order-events-consumer
workload: asynchronous order updates
peak_publish_rate: measured during load test
max_consumer_concurrency: tested value
batch_or_prefetch_limit: tested value
max_acceptable_lag_seconds: service objective
backlog_alert_duration: sustained interval
retry_limit: bounded value
dead_letter_destination: approved destination
slow_client_policy: coalesce, sample, or disconnect
last_tested: YYYY-MM-DD
owner: team-name
next_review: YYYY-MM-DDThresholds should reflect business impact. A notification stream may tolerate coalescing and a short delay, while a workflow that drives financial or operational state may require bounded age and durable recovery. If message ordering matters, increasing parallelism may create a correctness problem even when it improves throughput.
How to interpret changes
Lag rises while publish rate is steady: investigate consumer processing latency, errors, rebalances, unavailable dependencies, and partition or queue distribution. Adding consumers is useful only when there is assignable work and the downstream systems can handle the added load.
Lag rises with a publish-rate spike: determine whether the burst is expected. If it is, use producer rate limits, admission control, batching, or capacity that has been tested for the peak. If it is not, inspect retries, duplicate producers, webhook storms, and replay jobs.
Queue depth rises but broker metrics look normal: the bottleneck is often a consumer or its dependency. Check database latency, connection pools, CPU, garbage collection, external API response time, and message-size distribution.
WebSocket buffers grow for only a subset of clients: treat it as a client-specific flow-control problem. Apply per-connection limits and consider sending the latest state instead of every intermediate update. Disconnecting a persistently slow client can be safer than allowing one connection to consume unbounded memory.
Errors and retries rise together: do not respond by raising retry concurrency automatically. A retry storm can intensify the original outage. Use exponential backoff with jitter, cap attempts, isolate failing destinations, and preserve enough observability to distinguish transient from permanent errors.
When replaying backlog, separate replay capacity from live traffic where possible. A controlled replay with a bounded rate is safer than releasing all stored messages at once. For recovery design, see the Event Replay Guide and the message delivery guarantees guide.
When to revisit
Revisit this playbook on a monthly or quarterly schedule, and immediately after a material change. Triggers include a new broker or client-library release, a migration between Kafka, RabbitMQ, NATS, or another messaging platform, changed managed-service quotas, a new event schema, increased message size, traffic seasonality, consumer rebalancing, or a production incident.
Also review the design when a team adds a new downstream dependency, changes acknowledgment behavior, enables a replay workflow, or introduces a new WebSocket client type. These changes can alter the safe relationship between intake, concurrency, memory, and recovery time even when average traffic is unchanged.
For the next review, copy the configuration record, export the previous four weeks of lag and backlog data, and compare peak demand with completed work. Then verify one controlled overload scenario. If the system cannot state who slows producers, how consumers stop accepting work, what happens to slow clients, and how operators recover safely, the flow-control design is not finished.
Use the wider capacity and observability checklists to complete the review: message broker capacity planning, Kafka observability, and WebSocket scalability. The practical goal is a messaging system that degrades deliberately, keeps backlog bounded, and gives operators enough warning to act before overload becomes data loss or a prolonged outage.
