Message broker capacity planning becomes much easier when traffic, payload size, retention, replication, and consumer growth are converted into repeatable calculations. This guide provides a practical sizing method, transparent assumptions, and worked examples for estimating throughput, storage, partitions, and processing capacity before launch or during a capacity review.
Overview
A broker can appear healthy while one resource is approaching its limit. Producers may have sufficient bandwidth, but storage may fill faster than expected. Consumers may be connected, but lag can grow during peak traffic. A design may handle average throughput while failing when a traffic burst, retry storm, or new consumer group arrives.
Capacity planning should therefore examine four separate questions:
- Throughput: How many messages per second must the system accept and deliver?
- Storage: How much data must be retained, including replicas and operational headroom?
- Parallelism: How many partitions, workers, or consumers are needed to process the workload?
- Growth and failure: What happens when traffic increases or a broker, consumer, or network path is unavailable?
The calculations below apply broadly to event streaming platforms, message queues, and pub/sub systems. The exact limits depend on the broker, client configuration, message distribution, persistence settings, acknowledgment behavior, and hardware. Treat the formulas as a planning model, then validate them with a representative load test.
For system-shape decisions, compare the requirements with the patterns described in this catalog of event-driven architecture patterns. Capacity planning is most useful after the delivery model, ordering requirements, and retention policy are clear.
How to estimate
Start with a worksheet containing the following values:
- Average messages per second
- Peak-to-average traffic factor
- Average and maximum payload size
- Number of consumer groups or queue workers
- Retention period
- Replication factor
- Protocol and metadata overhead
- Target utilization and failure headroom
1. Calculate peak message rate
Peak messages per second = average messages per second × peak factor
Use an observed peak when telemetry exists. If it does not, document an explicit planning factor rather than quietly sizing from the average. A factor of 3, for example, means the design must handle a peak three times the average; it is an assumption, not a universal recommendation.
2. Convert messages into byte throughput
Payload throughput = peak messages per second × average payload bytes
Apply a separate overhead factor for headers, serialization, protocol framing, indexes, and other broker-specific costs:
Estimated ingress bytes per second = payload throughput × overhead factor
Use the largest realistic payload for a safety check. Averages are appropriate for storage estimates, but large messages can determine network, memory, batching, and latency behavior.
3. Estimate retained storage
Logical retained storage = ingress bytes per second × retention seconds
Then account for replication:
Broker storage = logical retained storage × replication factor
Finally, add headroom for segment files, compaction behavior, rebalancing, recovery, temporary duplication, and uneven partition distribution. A team may choose a planning buffer such as 20% to 30%, but the appropriate value should reflect the broker and failure objectives rather than being treated as a fixed rule.
4. Size partitions or workers from measured capacity
For a partitioned event stream:
Partitions required = ceiling(peak workload ÷ safe throughput per partition)
“Safe throughput per partition” must come from a load test using the intended payload size, acknowledgment mode, replication, compression, and consumer behavior. A partition count also affects ordering and consumer parallelism, so adding partitions solely to increase throughput can have application consequences.
For a worker-based queue, estimate concurrency using processing time:
Required in-flight work ≈ peak messages per second × average processing time in seconds
Add capacity for slow jobs, retries, and the utilization target. Queue depth is not capacity by itself; it is a signal that arrival rate currently exceeds processing rate or that workers are unavailable.
Inputs and assumptions
Good sizing depends less on mathematical precision than on making assumptions visible.
Traffic shape
Separate sustained traffic from bursts. Record the duration of a peak, not only its highest one-second value. A short burst may be absorbed by batching or local buffers, while a sustained peak requires the broker and consumers to keep up continuously.
Payload and serialization
Measure serialized message size, including headers where relevant. JSON, Avro, Protobuf, and compressed payloads produce different byte rates. If a schema change can increase event size, include that scenario in the worksheet. A schema registry and versioning process can help make those changes predictable, but it does not remove the need to test their capacity impact.
Retention and replication
Retention should match recovery, replay, audit, and consumer-outage requirements. Do not confuse logical retention with physical disk capacity: replication multiplies stored data, and operational headroom is required for healthy recovery. Compacted topics, tombstones, tiered storage, and queue deletion policies need separate treatment because their storage behavior differs from simple time-based retention.
Consumer growth
Each independent consumer group may read the stream separately. Estimate the resulting broker read traffic and downstream processing load. In a partitioned system, adding consumers beyond the available partitions may not increase active parallelism for a group. In a queue, concurrency is constrained by processing time, downstream limits, ordering, and retry behavior.
Monitor the assumptions with metrics such as producer rate, broker disk usage, request latency, under-replicated data, consumer lag, queue depth, retry rate, and dead-letter volume. The Kafka observability checklist provides a useful framework for connecting these signals to capacity decisions.
Worked examples
Example 1: Retained event stream
Assume an application produces 400 messages per second on average, with a peak factor of 3. The average serialized payload is 2 KB, the planning overhead factor is 1.2, retention is three days, and the replication factor is 3.
- Peak rate: 400 × 3 = 1,200 messages per second
- Payload throughput: 1,200 × 2 KB = 2.4 MB per second
- Estimated ingress: 2.4 × 1.2 = 2.88 MB per second
- Logical three-day retention: 2.88 × 259,200 seconds = approximately 746 GB
- With three replicas: approximately 2.24 TB before additional headroom
If a load test demonstrates that one partition safely handles 150 messages per second for this workload, the initial throughput calculation requires at least eight partitions: 1,200 ÷ 150 = 8. The final design may use more if consumer parallelism, future growth, or recovery time justifies it. The test should also verify replication traffic, disk performance, and lag during a broker failure.
Example 2: Asynchronous job queue
Assume a webhook integration receives 50 jobs per second on average and 200 jobs per second at peak. Each job takes an average of three seconds to process. At the peak, the queue needs approximately 600 active processing slots: 200 × 3 = 600.
If the operating target is 70% utilization, divide by 0.7 to obtain about 857 slots, then round up to an operationally manageable worker count. This is only a starting point. Test the effect of retries, downstream rate limits, long-running outliers, and worker restarts. For webhook workloads, a durable queue and a dead-letter process are often as important as raw throughput; see the webhook-to-queue integration guide for the reliability workflow.
When to recalculate
Revisit the worksheet during quarterly capacity reviews, before a major launch, and whenever traffic or message shape changes. Recalculate immediately when any of these conditions occurs:
- Average or peak message rate changes materially
- Payloads become larger or a serialization format changes
- Retention, replication, or acknowledgment settings change
- A new consumer group, region, or data pipeline is added
- Consumer lag, queue depth, disk usage, or request latency trends upward
- A broker, instance type, client library, or managed-service configuration changes
- Recovery objectives require faster replication or replay
Keep the model next to the operational dashboard and record the date, measured inputs, benchmark configuration, and open assumptions. Then validate the result with a load test that includes peak traffic, realistic payloads, retries, and at least one failure scenario. Capacity planning is not a one-time broker selection exercise; it is a repeatable reliability practice that turns growth into an observable and manageable change.
