Distributed systems consistency explained

What consistency actually means

In distributed systems literature, consistency usually means: if I write value X, what will the next reader see? That sounds simple until you have three replicas in different availability zones and a client that retries a failed write to a second replica while the first is still catching up.

Do not confuse this with consistency in the CAP theorem sense (linearizability — all nodes agree on a single global order of operations) with casual uses like "the UI feels consistent." Engineers also talk about:

• Replication lag — how far behind a read replica trails the primary.
• Cache coherence — whether a CDN or Redis layer serves data older than the database.
• Ordering — whether event B always appears after event A for all consumers.

Each layer — database, cache, queue, search index — can have a different consistency profile. A system is only as strong as its weakest link in the read path.

ACID on one node vs distributed reality

Relational databases give you Atomicity, Consistency, Isolation, Durability inside a transaction boundary on one server. Commit succeeds or rolls back; concurrent transactions are isolated; committed data survives a crash.

Replicate that database for availability and the story fractures:

• Synchronous replication — the primary waits for a replica to acknowledge before committing. Stronger consistency, higher write latency, and if the replica is unreachable you may block writes entirely.
• Asynchronous replication — the primary commits immediately and ships changes later. Lower latency, but a reader hitting the replica may see data that is seconds (or minutes) old — or miss a write that existed only on the primary before it failed.

Distributed systems rarely give you single-node ACID globally without paying for it in latency, availability, or operational complexity. That trade-off is what CAP and the consistency-model menu below help you navigate.

The CAP theorem — and what people get wrong

Eric Brewer's CAP theorem states that a distributed data store cannot simultaneously provide all three of:

• Consistency (C) — every read receives the most recent write or an error (linearizability).
• Availability (A) — every request receives a non-error response, without guarantee that it reflects the latest write.
• Partition tolerance (P) — the system continues despite arbitrary message loss or delay between nodes (network partitions).

In real networks, partitions happen — cables cut, AZs fail, packets drop. So P is not optional; during a partition you choose between C and A. A CP system (like etcd or ZooKeeper during a split) may reject writes to preserve a single truth. An AP system (like Dynamo-style stores) keeps accepting writes on both sides of a partition and reconciles conflicts later.

Common misconceptions

• "Pick two forever." CAP describes behavior during a partition, not your steady-state design. Most production systems are mostly available and mostly consistent, degrading only when the network misbehaves.
• "NoSQL = eventual consistency." CockroachDB, Spanner, and FoundationDB offer strong distributed consistency with SQL interfaces. MongoDB defaults changed over the years — read the docs for your deployment mode.
• "Strong consistency is always better." Social like counts, analytics dashboards, and search indexes tolerate seconds of lag. Payment ledgers and inventory reservation do not.

Consistency models on the spectrum

Between "every read is instantly global" and "readers may never agree," engineers use named models:

Strong consistency (linearizability)

Once a write completes, all subsequent reads return that value (or a later one). Feels like a single copy of the data. Achieved via synchronous replication, consensus protocols (Raft, Paxos), or routing all reads through the leader. Cost: latency and reduced write throughput under load.

Eventual consistency

If writes stop, all replicas eventually converge to the same state. No bound on how long "eventually" takes. Common in DNS, Cassandra, and cache-aside patterns where TTL expiry is the reconciliation mechanism. Works when stale reads are acceptable or self-correcting (a view counter off by five is fine; a bank balance off by five is not).

Causal consistency

Operations that are causally related are seen in the same order by all nodes. If Alice posts a comment and Bob replies, everyone sees Alice's post before Bob's reply — but unrelated posts may appear in different orders on different replicas. Stronger than eventual, weaker than linearizable, and often enough for collaborative apps and comment threads.

Read-your-writes

A session guarantee: after you write, you always read your own update, even if other users see older data briefly. Implemented by sticky sessions to the primary, client-side version tokens, or routing writes and subsequent reads to the same replica. Essential for "I clicked Save — why doesn't my profile show the new bio?"

Monotonic reads

A client never sees time go backward — if you read version 5, your next read will not return version 3. Prevents jarring UI flicker when load balancers hit different replicas with different lag.

BASE: the pragmatic counterpart to ACID

Large-scale web systems often follow BASE instead of ACID at the global level:

• Basically Available — the system responds even under partial failure.
• Soft state — state may change without input because of background replication and reconciliation.
• Eventual consistency — replicas converge over time.

This is not an excuse for lost payments. It means you decompose the problem: keep the money ledger strongly consistent (small, critical path) while denormalized read models, notification fan-out, and search indexes catch up asynchronously through event-driven pipelines. The pattern is sometimes called CQRS — command side strong, query side eventual.

Quorums, leaders, and conflict resolution

Many distributed databases use quorum reads and writes: with N replicas, require W nodes to acknowledge a write and R nodes to participate in a read. If R + W > N, a read overlaps a write quorum and should see fresh data (simplified — clock skew and concurrent writes complicate this).

Leader-based replication sends all writes to one primary; followers tail a log. Reads from followers are eventually consistent unless you use "read from primary" or "sync replica" modes. Simpler mental model, but the leader is a hotspot and failover must promote a new leader without splitting brain.

When two partitions both accept writes, you get conflicts. Resolution strategies include:

• Last-write-wins (LWW) — timestamp or version vector picks a winner. Simple; can silently drop legitimate updates.
• Application merge — shopping carts union items; CRDTs merge counters and sets mathematically without coordination.
• Reject and retry — return a conflict error so the client merges consciously (optimistic locking with version columns).

Consistency in caches and async pipelines

A Redis cache in front of PostgreSQL introduces a second consistency boundary. Cache-aside: read cache, on miss load DB and populate; on write, update DB then delete cache key. The delete-then-reread window can still serve stale data if another request repopulates from a lagging replica. Mitigations:

• Short TTLs on hot keys so staleness self-heals.
• Write-through or write-behind with careful invalidation pub/sub.
• Version stamps in cache values — reject cached entries older than the client's known version.

Message queues add another axis. At-least-once delivery means consumers must be idempotent — processing the same event twice should not double-charge. Exactly-once end-to-end is a myth across heterogeneous systems; you approximate it with idempotent consumers plus deduplication tables or transactional outbox patterns.

Blockchains as an extreme consistency design

Public blockchains like Solana push consistency to an extreme: thousands of validators must agree on an ordered log of transactions. Finality is not instant — clients choose between processed, confirmed, and finalized commitment levels with different safety/latency trade-offs (see our Solana consensus guide). The lesson generalizes: stronger guarantees cost time and coordination. A dice game waiting for finalized confirmation trades speed for safety; a like button does not need that overhead.

Choosing the right model — a decision framework

Before picking technologies, classify each data path:

1. Can a stale read cause money loss, safety issues, or legal liability? If yes — strong consistency on the write path, synchronous commit or consensus, read from primary or quorum.
2. Is the operation idempotent and self-healing? View counts, CDN assets, recommendation scores — eventual consistency with TTL or background reconciliation is fine.
3. Does the user who wrote need to see it immediately? Enforce read-your-writes even if global consistency is eventual.
4. Do you need ordering across related events? Use partition keys in Kafka, single-shard queues, or causal ordering — not "fire and forget" to random workers.
5. What happens during a partition? Document whether you prefer failed writes (CP) or divergent state that merges later (AP). Test failover, not just happy path.

Document these choices per entity — "UserProfile: read-your-writes, 5s max staleness on followers" — so the next engineer does not bolt a cache on the payment table without understanding the implications.

Key takeaways

• Consistency is a spectrum, not a single setting — strong, eventual, causal, and session guarantees address different failure modes.
• CAP applies during partitions — you trade perfect consistency against always answering requests when the network splits.
• Replicate in layers — database, cache, queue, and search each need explicit consistency rules; the weakest link defines user-visible behavior.
• Match guarantees to business risk — ledgers strong, dashboards eventual, user edits read-your-writes.
• Design for retries — at-least-once delivery and partial failures make idempotency and version-based conflict detection non-optional.
• Test partition and failover — chaos experiments reveal whether your "strong" path actually blocks writes or silently forks state.

Related reading

• SQL fundamentals explained — ACID transactions on a single database
• Message queues explained — delivery guarantees and ordering
• Event-driven architecture explained — async decoupling and idempotent consumers
• Application caching explained — cache-aside and invalidation patterns