Message broker capacity planning is easier when traffic, storage, consumer growth, and pricing inputs are separated into measurable assumptions. This worksheet-style guide shows how to estimate broker requirements before production, compare queue and streaming workloads, and build a cost model that can be updated after load tests or changes in traffic.
Overview
A broker sizing exercise should answer four practical questions: how much data must the system accept, how quickly must it deliver that data, how long must messages remain available, and what happens when traffic or consumers grow? The answers apply whether you are evaluating a managed Kafka service, a traditional message queue, a pub/sub system, or another event streaming platform.
Capacity is not the same as peak throughput. A design can appear adequate on average and still fail during traffic bursts, consumer slowdowns, replays, deployments, or retry storms. A useful estimate therefore combines normal traffic with peak traffic, operational overhead, and deliberate headroom.
Use the following worksheet as a planning model rather than a promise of exact performance. Broker quotas, instance limits, partition behavior, storage classes, network charges, and managed Kafka pricing vary by provider and region. Replace the illustrative assumptions with values from your chosen platform's current documentation and pricing calculator.
The basic sizing model
Start with these relationships:
- Ingress rate: messages per second multiplied by average message size.
- Peak ingress: normal ingress multiplied by the expected peak factor.
- Retention storage: ingress bytes per second multiplied by retention seconds.
- Required capacity: peak workload multiplied by a headroom factor, then checked against broker, partition, connection, and storage limits.
Keep payload size separate from protocol overhead. Headers, keys, indexes, replication, compression behavior, and metadata can all affect the amount of data written or transferred. If you do not yet have measurements, use a conservative planning allowance and label it as an assumption.
How to estimate
1. Define the traffic envelope
Record both average and peak values for each message-producing workload. If several applications share a broker, calculate them separately before adding them together.
| Input | Worksheet question |
|---|---|
| Message rate | How many messages are produced per second during normal and peak periods? |
| Message size | What are the average, high-percentile, and maximum payload sizes? |
| Peak factor | How much higher is launch, campaign, batch, or incident traffic than normal traffic? |
| Retention | How many hours or days must messages remain available for consumers and replay? |
| Replication | How many copies are required by the platform or reliability design? |
For example, suppose an application produces 2,000 messages per second at an average size of 4 KB. The logical ingress is approximately 8 MB per second. If a launch is expected to produce three times normal traffic, plan around 24 MB per second before adding overhead and headroom. This is a workload assumption, not a benchmark for a particular broker.
2. Convert throughput into storage
Retention storage is calculated from the bytes written over the retention window. At 24 MB per second, one hour of logical data is approximately 86.4 GB before replication, compression differences, indexes, and provider-specific overhead. A 24-hour retention window would therefore represent roughly 2.07 TB of logical data under the same sustained peak assumption.
Decide whether retention is based on average traffic or a sustained peak. Average-based storage estimates may be reasonable for stable workloads, but they can understate capacity when bursts are long, consumers fall behind, or replay requirements require a complete history. If the platform stores replicated copies, include the replication factor in the physical storage estimate. If compression is important, model a measured compression ratio rather than assuming one.
3. Size partitions and consumer concurrency
In partitioned event streams, partitions are both an ordering boundary and a unit of parallel consumption. Estimate the number of partitions from the throughput required per partition and the consumer concurrency required by each workload:
Candidate partitions = the larger of: peak ingress divided by tested per-partition write capacity, or required consumer concurrency divided by useful consumers per partition.
The result is only a starting point. Partition capacity depends on message size, batching, acknowledgments, compression, replication, disk performance, and consumer behavior. A topic that needs many consumers may require more partitions even when its write rate is modest. Conversely, adding partitions does not automatically make a poorly designed consumer faster.
For queue-based systems, use an equivalent model based on worker concurrency, processing time, acknowledgment rate, and backlog tolerance. A queue may not expose partitions in the same way, but it still has limits for connections, in-flight messages, delivery rate, storage, and consumer work.
4. Estimate connections and network transfer
Count producers, consumers, administrative clients, connectors, stream processors, and failover instances. Use the maximum number of simultaneous connections rather than the daily average. Include rolling deployments and autoscaling behavior, because a short-lived connection surge can trigger limits even when message throughput is unchanged.
For network planning, separate ingress, broker replication traffic, egress to consumers, cross-zone or cross-region transfer, and connector traffic. A single message may be read by several consumer groups, so consumer egress can be much larger than producer ingress. Calculate each path independently before applying any provider-specific billing assumptions.
5. Add operational headroom
Headroom covers traffic uncertainty, uneven partition distribution, maintenance activity, consumer recovery, replay, and temporary retries. Choose the allowance according to the confidence of your inputs and the consequence of saturation. A forecast based on measured production traffic can use a different allowance from a forecast based only on product expectations.
Do not hide headroom inside an unexplained multiplier. Record it as a named assumption, such as “peak factor,” “consumer recovery allowance,” or “storage reserve.” This makes future reviews more useful and shows which assumption changed.
Inputs and assumptions
Build the worksheet with one row per topic, queue, or workload. The following fields are enough for an initial comparison between message queue solutions and event streaming platforms:
- Workload name: payments, notifications, audit events, webhooks, or another bounded use case.
- Producers and consumers: application count, consumer groups, worker count, and expected autoscaling range.
- Average and maximum message size: include headers and encoding where possible.
- Normal and peak message rate: record the duration of the peak, not just its multiplier.
- Processing time: average and slow-case time per message or batch.
- Delivery requirement: at-most-once, at-least-once, or an application-level idempotent design.
- Retention and replay: how long data must be available and whether historical reprocessing is expected.
- Replication and availability: required copies, failure domains, and recovery expectations.
- Connection model: long-lived connections, connection pooling, or short-lived producers.
- Cost inputs: broker capacity, storage, requests, data transfer, connectors, observability, and support as applicable.
Separate fixed and variable costs. Fixed costs can include provisioned broker capacity or minimum service tiers. Variable costs may follow storage, requests, data transfer, or processing volume. A lower per-message rate can still produce a higher total bill if the architecture creates multiple consumer copies, long retention, cross-region transfer, or extensive replay traffic.
Also document what the estimate excludes. Common exclusions include application servers, databases, WebSocket gateways, schema management, logging, alerting, disaster recovery environments, and engineering time. A broker estimate is more useful when it is clearly bounded than when it appears comprehensive but omits major supporting components.
Worked examples
Example A: transactional queue
A webhook service receives 300 messages per second normally and 900 messages per second during a short peak. Each message averages 12 KB. Workers take 80 milliseconds on average, with occasional slower processing. The planning model should calculate peak ingress, the number of concurrent workers needed to clear that rate, the maximum acceptable backlog, and the storage required if messages remain available for retries for several hours.
The key risk is not only broker write capacity. If downstream processing slows, the queue grows. Size worker concurrency and backlog storage together, then define an alert threshold before the queue reaches its maximum. For failed deliveries, include retry traffic and decide whether a dead-letter queue needs separate storage and monitoring. See the dead-letter queue design guide for retry and recovery considerations.
Example B: retained event stream
An analytics system receives 2,000 events per second at 4 KB each, with a three-times launch peak. It has four independent consumer groups: analytics, fraud checks, search indexing, and an archive process. The logical write rate is based on the producer stream, but network and read capacity must account for four consumers. Retention storage is calculated from the retained producer data, while consumer lag planning must account for the slowest group and the time available for recovery.
In this case, adding partitions should be evaluated against both producer throughput and consumer parallelism. Test with representative message sizes and batches, then observe partition skew and consumer lag. The Kafka and streaming observability runbook and backpressure playbook provide useful operational follow-through.
Example C: cost comparison
To compare two broker options, hold the workload constant and create one column for each provider or deployment model. Enter the same peak ingress, retention period, consumer count, replication assumption, connection count, and transfer paths. Then add the service's current capacity, storage, request, and transfer inputs without treating one provider's pricing unit as equivalent to another's.
The result should show at least three scenarios: expected traffic, launch peak, and growth forecast. Include a sensitivity check for message size and consumer count. These two variables often change the cost model substantially because they affect storage, network transfer, and processing volume.
When to recalculate
Revisit the worksheet whenever a material input changes, not only when the broker is close to a limit. Recalculate before a major launch, after a load test, when message schemas grow, when a new consumer group is added, or when retention and replay requirements change. Also review the model after a migration, region change, availability redesign, or change in delivery guarantees.
Pricing and quotas should be checked whenever the managed service, tier, region, or contract changes. Replace old capacity limits and pricing inputs with current provider documentation rather than carrying forward assumptions from an earlier design. Benchmarks should also be refreshed when the client libraries, serialization format, compression settings, instance type, or workload mix changes.
Use production measurements to close the loop. Track ingress and egress bytes, message rate, message size distribution, partition skew, consumer lag, processing latency, retries, dead-letter volume, connection counts, storage growth, and broker saturation. Set alerts for the symptoms that matter to users, not only infrastructure utilization. For replay and recovery planning, the event replay guide can help connect capacity assumptions with operational procedures.
Finally, turn the worksheet into a reviewable decision record. Date each version, list the assumptions, note the test environment, record the provider and region, and identify which values are measured versus estimated. That discipline makes capacity planning reusable: the same model can support a launch review, a broker comparison, an annual budget exercise, or a post-incident correction without starting from zero.