What is a distributed message queue?

A durable, append-only log that sits between producers and consumers so work can be handed off asynchronously. It is split into partitions for scale and replicated across machines so an acknowledged message survives a broker failure.

How does partitioning affect message ordering?

Ordering is a per-partition guarantee, not a global one. A producer hashes a key to a partition (partition = hash(key) mod N), so all messages for one key stay in order in one partition — but messages in different partitions have no defined order relative to each other.

What are in-sync replicas (ISR) and when is a message committed?

Each partition has a leader and follower replicas; the ISR is the set currently caught up. With acks=all a write is not committed until every in-sync replica has it, and consumers only ever see committed messages — so a single broker loss cannot lose acknowledged data.

Is exactly-once delivery possible?

Not over the wire. The honest answer is at-least-once delivery plus idempotent processing: deliver possibly twice, but make applying a message twice have the same effect as once (dedupe on an ID, or upsert). Kafka’s "exactly-once" is this pattern via idempotent producers and transactions.

How is Kafka different from RabbitMQ and SQS?

Same three decisions traded differently. Kafka is a retained, replayable partitioned log (pull, offsets). RabbitMQ deletes on acknowledgement and routes flexibly. SQS standard is managed and best-effort/at-least-once; SQS FIFO adds strict ordering and exactly-once processing at a per-partition throughput cap.

Design a Distributed Message Queue

From one producer and one consumer to partitions, replication, and back-pressure — the high-level end of the spectrum, drawn and animated.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Distributed Message Queue — built to be understood, not skimmed.

Questions

What is a distributed message queue?: A durable, append-only log that sits between producers and consumers so work can be handed off asynchronously. It is split into partitions for scale and replicated across machines so an acknowledged message survives a broker failure.
How does partitioning affect message ordering?: Ordering is a per-partition guarantee, not a global one. A producer hashes a key to a partition (partition = hash(key) mod N), so all messages for one key stay in order in one partition — but messages in different partitions have no defined order relative to each other.
What are in-sync replicas (ISR) and when is a message committed?: Each partition has a leader and follower replicas; the ISR is the set currently caught up. With acks=all a write is not committed until every in-sync replica has it, and consumers only ever see committed messages — so a single broker loss cannot lose acknowledged data.
Is exactly-once delivery possible?: Not over the wire. The honest answer is at-least-once delivery plus idempotent processing: deliver possibly twice, but make applying a message twice have the same effect as once (dedupe on an ID, or upsert). Kafka’s "exactly-once" is this pattern via idempotent producers and transactions.
How is Kafka different from RabbitMQ and SQS?: Same three decisions traded differently. Kafka is a retained, replayable partitioned log (pull, offsets). RabbitMQ deletes on acknowledgement and routes flexibly. SQS standard is managed and best-effort/at-least-once; SQS FIFO adds strict ordering and exactly-once processing at a per-partition throughput cap.

Related explainers

Loading a Safetensors Checkpoint on a Multihost Cluster, Line by Line