Apache Kafka explained

What Kafka is (and is not)

Kafka is best understood as an append-only distributed log, not a traditional queue that deletes messages after delivery. Records stay on disk for a configurable retention period (or forever with compaction). Multiple independent consumer groups can each read the same topic from different offsets — analytics, billing, and search indexing all consume the same orders stream without competing for messages.

That persistence model makes Kafka ideal for event streaming: high-volume telemetry, audit trails, change-data capture from databases, and the backbone of event-driven microservices. It is a poor fit for task queues where each job should be processed exactly once and then disappear — RabbitMQ or SQS often win there.

Core vocabulary

• Broker — a Kafka server that stores topic partitions and serves produce/fetch requests.
• Topic — a named category of records (like a table or channel).
• Partition — an ordered, immutable sequence of records within a topic; the unit of parallelism.
• Offset — a monotonically increasing ID for each record within a partition; consumers track their position by offset.
• Cluster — multiple brokers coordinated by Apache ZooKeeper (older deployments) or KRaft (Kafka 3.x+ native quorum).

Topics, partitions, and ordering

When you create topic orders with six partitions, Kafka spreads incoming records across six independent logs. Ordering is guaranteed only within a single partition. If all events for order_id=48291 must arrive in sequence, the producer should set the message key to that order ID — Kafka hashes the key to pick a partition, so every event for the same key lands in the same partition and preserves order.

More partitions mean higher write and read parallelism, but also more file handles, memory overhead, and rebalancing cost when consumer groups change. A common starting point is one partition per expected consumer instance in the busiest group, rounded up to a power of two. Revisit after measuring lag — adding partitions later is possible but reordering keys across old and new partitions does not happen automatically.

Record format is typically key + value + timestamp + headers. Values are often Avro, Protobuf, or JSON serialized with a schema registry so consumers can evolve safely across deploys.

Producers: acks, batching, and idempotence

A Kafka producer sends batches of records to the leader broker for each target partition. The acks setting controls durability:

• acks=0 — fire-and-forget; fastest, no guarantee the broker received anything.
• acks=1 — leader acknowledged write; risk of loss if leader dies before replication.
• acks=all (or -1) — wait until all in-sync replicas confirm; production default for critical events.

Producers batch records to amortize network round trips. Tune linger.ms and batch.size for throughput vs latency. Enable idempotent producers (enable.idempotence=true) so retries after transient errors do not create duplicate records — Kafka assigns a producer ID and sequence number per partition.

Compression (lz4, zstd, snappy) on the wire and on disk reduces bandwidth dramatically for JSON-heavy payloads at the cost of CPU.

Consumers and consumer groups

A standalone consumer subscribes to topics and polls for records, committing offsets manually or automatically. In production, consumers join a consumer group identified by group.id. Kafka assigns each partition to at most one consumer in the group — if you run six consumers against a six-partition topic, each consumer owns one partition. Add a seventh consumer and it sits idle.

When consumers join, leave, or crash, the group rebalances: partitions are redistributed. Rebalances pause consumption briefly — frequent churn (aggressive autoscaling, flaky pods) causes "rebalance storms." Mitigations include cooperative-sticky assignors (incremental rebalance), static membership with group.instance.id, and keeping processing time under max.poll.interval.ms.

Offset commit strategy matters. Auto-commit is simple but can acknowledge messages before your handler finishes — a crash loses data. Manual commit after successful processing is safer; pair with idempotent handlers because at-least-once delivery still duplicates on retry.

Replication, ISR, and durability

Each partition has one leader broker that handles reads and writes, and zero or more followers that replicate the log. The in-sync replica set (ISR) lists followers caught up within replica.lag.time.max.ms. With min.insync.replicas=2 and acks=all, Kafka refuses writes if only one broker is available — trading availability for durability during partial outages.

replication.factor is typically 3 in production clusters spanning availability zones. Under-replicated partitions (followers falling behind) are a leading indicator of disk pressure, network issues, or broker overload — monitor them in metrics, not only consumer lag.

Retention, compaction, and disk planning

By default Kafka deletes segments older than retention.ms (often seven days). High-volume topics need explicit capacity planning: bytes in = partition count × replication factor × retention window.

Log compaction keeps only the latest record per key — ideal for changelog topics like user-profiles or Kafka Streams state stores. Compacted topics still respect min.compaction.lag.ms so consumers can read historical updates before old segments merge.

Tombstones (records with null value) delete keys during compaction. Misconfigured compaction on high-cardinality keys can balloon disk usage before the cleaner runs.

Delivery semantics

Kafka provides at-least-once by default: producers may retry, consumers may reprocess after crash before offset commit. Achieve effectively exactly-once within Kafka using idempotent producers + transactional writes (transactional.id) for consume-transform-produce pipelines — common in stream processing with Kafka Streams or Flink's Kafka connector.

Exactly-once to external systems (PostgreSQL, email APIs) still requires idempotent side effects or outbox patterns — Kafka cannot magically deduplicate your database writes. The transactional outbox pattern pairs well: write business state and an outbox row in one DB transaction, then a separate connector publishes to Kafka.

Worked example: order event pipeline

An e-commerce checkout service publishes to topic orders (12 partitions, key = order_id, acks=all, idempotent producer). Three downstream consumer groups run independently:

1. fulfillment-service (6 instances) — reserves inventory; commits offset only after warehouse API returns 200.
2. analytics-etl (3 instances) — batches events to a data warehouse every 30 seconds; tolerates minutes of lag.
3. fraud-scorer (4 instances) — uses tumbling windows to flag velocity anomalies in real time.

Partition count (12) exceeds the largest group's instances (6), leaving headroom for Black Friday scaling. Replication factor 3 across three AZs survives single-broker loss. Retention 14 days lets analytics replay after a bug fix without re-emitting from the monolith.

Kafka vs RabbitMQ vs SQS decision table

Common pitfalls

• Too few partitions — consumer group cannot scale past partition count; hot partitions when key distribution is skewed.
• Rebalance storms — slow handlers exceed max.poll.interval.ms; consumers get kicked, partitions shuffle, lag spikes.
• Auto-commit before processing — silent message loss on crash between poll and handler completion.
• Ignoring consumer lag — lag is the queue depth metric; alert on p95 lag per group/topic, not only broker uptime.
• Unbounded retention on verbose topics — debug topics with full payloads fill disks; use shorter retention or separate clusters.
• Schema breakage — deploy consumers before producers when removing required fields; use a schema registry with compatibility rules.
• Treating Kafka like a database — it is a log, not a query engine; compacted topics help but do not replace OLTP stores.

Practitioner checklist

• Choose partition count from expected peak throughput and consumer parallelism.
• Set message keys to preserve ordering for entities that need it.
• Use acks=all, min.insync.replicas=2, replication factor 3 in production.
• Enable idempotent producers; add transactions only when consume-transform-produce needs it.
• Commit offsets after successful side effects; design handlers to be idempotent.
• Monitor consumer lag, under-replicated partitions, and disk usage per broker.
• Pick retention or compaction per topic based on replay vs latest-state needs.
• Register schemas and enforce backward/forward compatibility before multi-team topics.
• Load-test rebalance behavior before autoscaling consumer pods aggressively.
• Document which consumer groups own which topics — orphan consumers are a common incident source.

Key takeaways

• Kafka is a distributed, durable commit log — not a delete-on-read task queue.
• Partitions provide parallelism; keys route related events to the same partition for ordering.
• Consumer groups divide partitions among instances; each group reads independently at its own offset.
• Replication and ISR with acks=all protect against broker loss at the cost of write availability during outages.
• Pair Kafka with idempotent consumers and clear retention policy; use stream processors when you need windows, joins, and aggregations.

Related reading

• Message queues explained — broker comparison, delivery guarantees, and when async beats HTTP
• Stream processing explained — windows, watermarks, and Kafka Streams / Flink
• Event-driven architecture explained — events vs commands and system design patterns
• Change data capture (CDC) explained — streaming database changes into Kafka