What is distributed consensus and how do systems elect a leader or coordinate state?
Take a moment.
Say your answer out loud or write it down first — retrieval is what makes it stick. Reveal when you're ready.
Short answer
Distributed consensus gets a group of nodes to agree on a value or state despite failures. Protocols: Paxos (theoretical foundation, hard to implement), Raft (understandable consensus — leader election + log replication, used by etcd and Consul), and Zab (ZooKeeper). Use them for leader election, configuration stores, and coordination — not for request-path data. ZooKeeper, etcd, and Consul are the off-the-shelf implementations most teams use.
Detailed explanation
Many distributed problems reduce to consensus: agreeing on who the leader is, what the current configuration is, or whether a transaction committed. Paxos (Lamport) proves consensus is achievable but is notoriously hard to implement correctly. Raft (Ongaro, 2014) achieves the same result with understandability as a goal: it splits the problem into leader election (a candidate requests votes; first to get a majority wins a term), log replication (leader appends entries, followers acknowledge, committed once a majority replicate), and safety (only the leader with the latest term serves clients). Used by etcd, Consul, TiKV, and CockroachDB. Zab powers ZooKeeper's coordination. Practical guidance: do not implement consensus yourself — use etcd (Kubernetes' backing store), Consul (service discovery + KV), or ZooKeeper (Kafka, Hadoop). They expose a strongly consistent KV lock, leader election primitives, and watch/stream APIs. Trade-offs: consensus groups need an odd number of nodes (3 or 5) to tolerate 1 or 2 failures; writes go through the leader so latency is bounded by the slowest follower; and partitions without a majority halt writes (CP in CAP terms). Use consensus for metadata and coordination, not for high-throughput request data.
Example
etcd (Raft-based) in Kubernetes:
- 5-node etcd cluster.
- Leader handles all writes; followers replicate the log.
- If the leader fails, remaining nodes elect a new leader in ~1 second.
- kube-apiserver reads/writes cluster state to etcd.
Leader election pattern:
1. Services attempt to create an ephemeral key '/leader'.
2. First creator becomes leader; others watch the key.
3. If the leader dies, its ephemeral key expires; next watcher takes over.Interview tip
Name Paxos as the theory and Raft as the practical, understandable alternative. Then name one real system (etcd, Consul, or ZooKeeper) and its use case. Saying 'I would not implement this myself' is the correct senior-engineering answer.
Common mistakes
- Saying you would implement consensus from scratch.
- Not knowing Raft's leader election plus log replication split.
- Forgetting that consensus groups need an odd node count for majority.