Kafka and other event-streaming systems can appear healthy while messages quietly accumulate, producers slow down, or a single partition limits the whole workload. This guide provides a practical observability model: the metrics to track, alerting principles, review checkpoints, and an incident runbook for turning signals into action without treating one dashboard number as the complete story.
Overview
Effective streaming observability connects four questions: Are events being accepted? Are they stored safely? Are consumers processing them fast enough? Can operators explain a failure and recover without creating duplicates or data loss?
A useful monitoring setup combines metrics, logs, traces, and business-level checks. Broker metrics may show that requests are succeeding, while a consumer group is still missing its processing objective. Conversely, consumer lag may rise briefly during a predictable traffic burst and resolve without intervention. The goal is therefore not to eliminate every fluctuation. It is to identify changes that threaten delivery, freshness, durability, capacity, or recovery.
Metric names vary by Kafka version, client library, exporter, and managed service. Treat the names below as a monitoring model rather than a promise that every deployment exposes identical labels. Before creating alerts, confirm the metric definition, unit, aggregation, and sampling interval in your platform documentation.
What to track
Organize dashboards by the path an event takes through the system. This makes it easier to distinguish an input problem from a storage problem or a processing problem.
| Area | Signals to monitor | What a change may indicate | First action |
|---|---|---|---|
| Producers | Record send rate, request latency, error rate, retries, timeouts, and batch or compression behavior | Client saturation, broker rejection, network trouble, oversized requests, or an unavailable partition leader | Check producer logs and error types, then compare client latency with broker request latency |
| Broker and requests | Request latency, request failures, under-replicated partitions, offline partitions, controller or coordination health, and network or disk pressure | Resource contention, replication risk, leadership changes, or a broker that cannot keep up | Check whether the issue affects one broker, one topic, or the whole cluster |
| Partitions | Partition count, leader distribution, bytes and records per partition, replica health, and hot-partition concentration | Uneven keys, an overloaded leader, insufficient parallelism, or a topic layout that no longer matches traffic | Compare the busiest partition with the topic average and review recent key or traffic changes |
| Consumers | Consumer lag, lag growth rate, records consumed, poll or fetch latency, processing latency, rebalance count, and commit failures | Slow handlers, blocked downstream services, too few consumers, failed commits, or unstable group membership | Identify the affected group and determine whether lag is growing, stable, or recovering |
| Storage and retention | Log-disk usage, retention activity, segment growth, fetch or read errors, and available capacity | Unexpected traffic, retention misconfiguration, slow cleanup, or insufficient storage headroom | Estimate time to capacity and verify retention settings before deleting or expanding anything |
| Schemas and contracts | Serialization failures, unknown schema or version errors, rejected event counts, and schema compatibility checks | Producer-consumer version drift, malformed events, or an incompatible deployment | Find the first failing producer and compare its deployed schema with the consumer expectation |
| Business outcomes | Event freshness, completed workflows, notification delivery, failed jobs, and end-to-end processing time | A technically available pipeline is not producing the intended result | Trace one representative event from production to the final action |
Consumer lag deserves more than one number
Consumer lag is the difference between the latest available position and the position a consumer group has committed. Track it by group, topic, and partition. A total across all partitions can hide one severely delayed partition, while a raw record count can be misleading when event volume varies widely.
Pair lag with lag age or event freshness when possible. A lag of a few thousand records may be harmless for a low-volume topic but serious for a high-volume workflow. Also record the rate of change: rising lag suggests the consumer is processing more slowly than events arrive; falling lag usually indicates recovery. Set alerts on sustained conditions rather than every short-lived increase.
Use symptoms and causes together
Dashboards should place related signals together. For example, consumer lag beside processing latency, downstream error rate, rebalance count, and records-consumed rate is more useful than a lag panel by itself. For operational detail, the Kafka observability checklist can serve as a companion review list.
Cadence and checkpoints
Observability improves when it is reviewed at several time scales. On-call monitoring should focus on active symptoms, while periodic reviews look for drift that has not yet become an incident.
- Per shift or on call: Check active alerts, consumer groups with rising lag, failed requests, under-replicated partitions, storage headroom, and unresolved dead-letter or retry traffic.
- Weekly: Review the highest-lag topics, recurring rebalances, noisy alerts, producer retry patterns, and end-to-end freshness. Confirm that dashboards still cover the groups and topics that matter.
- Monthly: Compare peak throughput, partition skew, consumer utilization, broker resource use, retention growth, and alert volume with the previous period. Remove alerts that have no clear owner or response.
- Quarterly or after major changes: Revisit capacity assumptions, recovery procedures, access controls, schema compatibility, replay plans, and failure-test results. Revalidate thresholds after topology, client, broker, or managed-platform changes.
Keep a small change log beside the monitoring configuration. Record topic creation, partition changes, consumer deployments, retention updates, schema changes, and major traffic events. Without this context, a dashboard comparison can mistake a deliberate architecture change for an unexplained regression.
Alert thresholds should be tied to service objectives and workload behavior. As a starting point, alert on a consumer group whose lag is continuously increasing for a defined window, on partitions that remain under-replicated, on failed requests that exceed the normal baseline, and on storage that may reach its operational limit within the recovery or expansion window. Use warning and critical levels, and test each alert against a documented response.
How to interpret changes
When an alert fires, avoid changing multiple variables at once. First establish scope: Is the problem limited to one consumer group, topic, partition, broker, region, or deployment? Then establish timing: Did it begin after a release, traffic increase, configuration change, infrastructure event, or downstream outage?
- Confirm the symptom. Check the alert window, dashboard data, and a second signal. A missing metric can look like zero activity.
- Separate input from processing. Compare producer rate with consumer rate. If production increased but consumer throughput did not, investigate capacity, partition parallelism, or handler latency. If production itself stopped, start with the producer and broker path.
- Check partition concentration. A single hot partition points toward key distribution or leader placement; evenly rising lag across partitions points more toward consumer capacity or a shared dependency.
- Inspect dependencies. Review database latency, API errors, thread pools, connection pools, and rate limits used by the consumer. Many streaming incidents are downstream bottlenecks expressed as lag.
- Protect the system before recovering. Apply backpressure, pause noncritical consumers, reduce expensive work, or route poison messages according to the approved procedure. Do not blindly increase concurrency if the dependency is already overloaded.
- Recover and verify. Watch lag age, processing rate, error rate, and business completion—not only whether the alert clears.
For messages that repeatedly fail, use a controlled retry and dead-letter process rather than allowing one poison event to block useful work indefinitely. See dead-letter queue design and recovery patterns for operational considerations. If replay is necessary, document the offset range, deduplication strategy, affected consumers, and validation checks before starting; the event replay guide provides a useful planning reference.
After the incident, record the leading indicator that should have detected it earlier, whether the alert was actionable, and which dashboard context was missing. This turns monitoring from a collection of charts into an improving control system.
When to revisit
Revisit this monitoring plan monthly for routine tuning and quarterly for a broader reliability review. Update it sooner after a production incident, a significant traffic change, a new consumer group, a partition or retention change, a broker or client upgrade, a schema migration, or a move between managed and self-managed infrastructure.
Use the following short checklist during each review:
- Do alerts measure a sustained condition and have a named owner?
- Can an operator distinguish producer, broker, partition, consumer, schema, and downstream failures?
- Are lag thresholds based on freshness or recovery objectives rather than an arbitrary record count?
- Are logs and traces correlated with topic, partition, offset, consumer group, and deployment where appropriate?
- Can the team explain how to pause, retry, dead-letter, replay, and verify recovery safely?
- Have capacity, retention, permissions, encryption, and schema controls changed since the last review?
Finally, test the runbook with a small, approved failure exercise or load test. A monitoring design is ready for production when it detects a meaningful failure, points responders toward a likely cause, and supports a measured recovery. Recheck the dashboards after every major architecture change, and keep the thresholds and metric definitions close to the systems they describe.