Part 6: Many Burrows, One Colony

One burrow, one chinchilla: simple. One mountain, 10,000 chinchillas across 500 burrows, separated by ridges and rivers: that’s distributed systems. The fundamental challenge is COORDINATION across distance.

6.1 The Impossible Triangle

The problem: Three chinchillas on three different ledges. A seed update happens on ledge 1. Can all three chinchillas see it immediately? Can all three chinchillas always be available? What if a rockslide cuts off ledge 3?

The answer: You CAN’T have all three. This is the most fundamental law of distributed systems.

The real name: CAP Theorem

Choose 2 of 3:

• Consistency: Every chinchilla sees the same data at the same time
• Availability: Every chinchilla gets a response (even if it might be stale)
• Partition tolerance: The system works even when communication between ledges is broken

Why you can’t have all 3: If ledge 3 is cut off (partition), you have two choices:

1. Block requests to ledge 3 until it reconnects -> Consistent, but NOT Available
2. Let ledge 3 serve potentially stale data -> Available, but NOT Consistent

You MUST handle partitions (they WILL happen in any real network), so the real choice is between CP (Consistency-first) and AP (Availability-first).

When to choose CP (Consistency):

• Bank account balances (showing wrong balance = disaster)
• Inventory counts (selling something that’s out of stock)
• Leader election (can’t have two leaders)

When to choose AP (Availability):

• Social media feeds (showing a slightly stale feed is fine)
• Product catalogs (a 5-second delay on price updates is acceptable)
• Analytics dashboards (approximate numbers are useful)

The important nuance: This is a SPECTRUM, not a binary. You tune consistency on a per-operation basis. A bank might be CP for transfers but AP for “checking account history.”

Instinct: AGREE vs SURVIVE (pick your side)

6.2 The Council of Elders

The problem: 5 chinchillas need to decide: which burrow is the new food depot? They can’t all meet in person (the mountain is too big). They send messengers. But messages can be lost, delayed, or arrive out of order. How do they agree?

The solution: Majority rules. If 3 out of 5 agree, the decision is made. Even if 2 chinchillas are unreachable, the council can still function.

The principle: In a distributed system, you can make progress as long as a MAJORITY of nodes are available and agree. This majority is called a QUORUM.

The real names:

Consensus algorithms: Formal protocols for distributed agreement.

Raft (the understandable one):

1. One node is elected LEADER (through majority vote)
2. All changes go through the leader
3. Leader replicates changes to FOLLOWERS
4. Change is committed when majority of followers confirm
5. If leader dies, followers detect this (missed heartbeats) and elect a new one
6. Election requires majority vote: this prevents split-brain (two leaders)

Paxos (the original, harder to understand): Same idea, different protocol. Guarantees that any two majorities MUST overlap by at least one node: this overlap ensures consistency.

Why “quorum” is the magic number: In a cluster of N nodes:

• Quorum = floor(N/2) + 1 (majority)
• 3 nodes: quorum = 2 (can tolerate 1 failure)
• 5 nodes: quorum = 3 (can tolerate 2 failures)
• 7 nodes: quorum = 4 (can tolerate 3 failures)

Why odd numbers: With 4 nodes, quorum = 3. Same as 3 nodes (quorum = 2) but more expensive. Odd numbers give you more fault tolerance per node.

Instinct: AGREE

6.3 The Chinchilla Grapevine

The problem: 500 chinchillas on a mountain. A new chinchilla joins burrow #247. How does every other chinchilla learn about this? Sending a message to all 499 others is expensive.

The solution: Tell 3 neighbors. They each tell 3 neighbors. Information spreads exponentially: like gossip.

The principle: Peer-to-peer information sharing where each node randomly exchanges state with a few others. Eventually, everyone knows everything.

The real name: Gossip Protocol

How it works:

1. Every N seconds, pick a random peer
2. Exchange state: “Here’s what I know. What do you know?”
3. Merge the state (keep the newest information for each item)
4. Repeat forever

Mathematical guarantee: In a cluster of N nodes, gossip reaches every node in O(log N) rounds. 1,000 nodes? About 10 rounds. 1,000,000 nodes? About 20 rounds.

Used for:

• Membership detection: “Who’s in the cluster right now?” (Cassandra)
• Health checking: “Who’s alive?” (Consul)
• Data dissemination: “What’s the latest config?” (DynamoDB)

Tradeoff: Gossip is EVENTUALLY consistent. There’s a brief window where some nodes have old information. But it’s:

• Decentralized (no single point of failure)
• Scalable (each node only talks to a few others)
• Resilient (works even if many nodes are down)

Instinct: AGREE (eventually)

6.4 The Shared Seed Problem

The problem: Two chinchillas reach for the same seed at exactly the same time. Both grab it. It splits. Nobody gets a whole seed.

The principle: Concurrent access to shared resources MUST be controlled, or data corruption is inevitable.

The real names:

Distributed lock: A mechanism that ensures only one process at a time can access a resource. Like putting a flag on the seed: “MINE: back off.”

How it works (Redlock algorithm for Redis):

1. Try to acquire lock across majority of nodes (quorum again)
2. If successful, you hold the lock for a limited time (lease)
3. Do your work
4. Release the lock
5. If something goes wrong, the lease expires automatically (no permanent deadlocks)

Optimistic vs Pessimistic locking:

• Pessimistic: “I’m going to grab this seed, so everyone else WAIT.” Lock before touching.
  • Pro: No conflicts ever. Guaranteed safe.
  • Con: If you hold the lock and crash, everyone waits until timeout. Under high contention, throughput tanks.
• Optimistic: “I’ll grab the seed, but I’ll check if someone else changed it when I’m done. If they did, I’ll retry.”
  • Pro: No blocking. High throughput under low contention.
  • Con: Under high contention, lots of retries. Wasted work.

Deadlock: Chinchilla A holds seed 1, waiting for seed 2. Chinchilla B holds seed 2, waiting for seed 1. Infinite wait. Solutions:

• Lock ordering: Always acquire locks in the same order
• Timeout: Give up waiting after N seconds
• Detection: Detect cycles and kill one process

Instinct: AGREE + PROTECT

6.5 Consistent Hashing: The Seed Assignment System

The problem: 10 chinchillas, 10,000 seeds. Each chinchilla is responsible for specific seeds. If chinchilla #7 gets snatched by an eagle, you need to reassign its seeds. With naive assignment (seed_id % 10), ALL seed assignments change when you go from 10 to 9 chinchillas. Chaos.

The solution: Arrange chinchillas on a ring. Each seed hashes to a position on the ring and is assigned to the next chinchilla clockwise. When one chinchilla disappears, only ITS seeds get reassigned to the next chinchilla: everyone else is unaffected.

How it works:

1. Hash each chinchilla’s ID to a position on a circle (0-360 degrees)
2. Hash each seed’s ID to a position on the same circle
3. Each seed is handled by the first chinchilla clockwise from its position
4. When chinchilla 7 disappears, its seeds slide clockwise to chinchilla 8. Everyone else is unchanged.

Virtual nodes: What if chinchillas are unevenly distributed on the ring? One chinchilla covers 180 degrees while another covers 10 degrees. Solution: Give each real chinchilla multiple hash positions (virtual nodes). A powerful chinchilla might have 200 virtual positions, a weak one might have 50. This distributes load more evenly.

Used in: Cassandra, DynamoDB, Memcached, consistent load balancing

Naive mod (seed % N): ALL assignments shuffle when N changes. Consistent hashing: only the missing chinchilla’s seeds move.

Tradeoff: More complex to implement than naive modulo, but dramatically fewer reassignments when nodes join or leave.

Instinct: SUSTAIN + ORGANIZE