Distributed Systems Interview Questions

Model answer

Retry failures that are probably temporary: a timeout, a dropped connection, 503 Service Unavailable, 429 Too Many Requests (after the Retry-After time).

Do not retry failures that will fail again: 400, 401, 403, 404, a validation error. And never blindly retry a request that is not idempotent (charging a card, sending an SMS), because the first attempt may have succeeded even though you got no response. Make it idempotent first, with an idempotency key.

Cap the attempts (2 or 3) and the total time spent.

Model answer

Retries multiply load exactly when the service has the least to spare: three retries mean up to four times the traffic. If each layer in a chain of three retries three times, one user request can become 64 calls at the bottom. The service never gets room to recover.

• Exponential backoff with jitter: wait 100 ms, 200 ms, 400 ms, each randomised, so clients spread out instead of retrying in waves.
• A retry budget: allow retries only up to, say, 10% of normal traffic.
• Retry at one layer only, usually closest to the user, not at every hop.
• A circuit breaker: after repeated failures, stop calling and fail fast for a while.

Model answer

Each caller's timeout must be shorter than its own caller's. If A gives up after 2 seconds but B waits 5 seconds for C, B keeps working on a request nobody is waiting for, and holds threads and connections doing it.

Pass a deadline down the chain rather than fixed numbers: A says "answer by 10:00:02.000", B subtracts the time it has used and gives C what is left (gRPC does this with deadlines). When time is up, every layer stops.

Set the base value from measured latency: a little above the dependency's p99, not the library default (often unlimited or 30 seconds).

Model answer

Not in delivery. Over an unreliable network a sender cannot tell "it failed" from "it worked but the reply was lost", so it must either retry (and risk running twice) or not (and risk running zero times). This is the two generals problem.

What you can build is effectively once: deliver at least once (keep retrying) and make the receiver idempotent, so duplicates have no extra effect. An idempotency key, a unique constraint on a message id, or a naturally idempotent operation ("set status to paid") does that.

When a system advertises exactly-once (Kafka transactions, for example), it means exactly-once processing within that system, built this way internally, not across your external side effects.

Model answer

In a distributed data store, when the network partitions (some nodes cannot reach others), each side must choose: refuse requests to stay consistent, or keep answering and stay available while the sides may disagree.

Partitions are not optional in real networks, so the choice is really CP or AP *during a partition*. Banking balances lean CP; a shopping cart or a like count can lean AP and reconcile later.

A more useful follow-on is PACELC: even with no partition, you trade latency against consistency, for example by waiting for replicas to confirm a write.

Model answer

Writes go to the primary, reads to a replica, and replication is asynchronous: the replica can lag by milliseconds or, under load, seconds. The user is reading their own write from a copy that has not received it yet.

You need read-your-writes consistency for that user, not for everyone:

• After a user writes, send their reads to the primary for a short window (a few seconds, tracked in their session).
• Record the write's log position (Postgres LSN) and read from a replica only once it has replayed past it, otherwise use the primary.
• Show the new value from the write's response in the UI, so the page does not need to read it back.

Model answer

• Avoid the conflict: give each record a home region and route its writes there. Most systems do this; it costs some latency for users far from home.
• Last write wins by timestamp: simple, but silently drops one edit, and clocks on different machines disagree.
• Detect and merge: version vectors tell you two edits were concurrent; then merge in code or ask the user, the way Git does.
• CRDTs: data types designed so concurrent edits always merge the same way (counters, sets, text for collaborative editors).

Choose per kind of data: a profile can live with last-write-wins, a shared document needs a merge, money needs a single home.

Model answer

Nodes run a consensus algorithm such as Raft (etcd, Consul) or ZooKeeper's Zab. A node becomes leader only with votes from a majority, for a numbered term. Since two majorities of the same cluster always share a node, two leaders cannot both win the same term.

Split brain is two nodes both believing they are the leader, usually after a network split, and both accepting writes that later conflict. Majority quorums prevent it within the cluster; that is why clusters have odd sizes (3 or 5): a 5-node cluster survives 2 failures.

An old leader that was paused may still act after losing the job, so storage should reject writes carrying an old term or fencing token.

Model answer

Each shard counts on its own, so two shards both hand out id 1001. You can give each shard its own range or step (shard A uses odd numbers, B even), but adding a shard later means re-planning the scheme.

A single central counter avoids collisions but becomes a bottleneck and a single point of failure for every insert.

Sequential ids are also guessable: /orders/1002 invites someone to try /orders/1001, and they reveal how many orders you have.

Model answer

They are unique without coordination, which is the upside. The downside is randomness in an index: B-tree indexes like inserts in order. Random keys land all over the tree, causing page splits, a bigger index, poor cache use and slower inserts as the table grows. In MySQL InnoDB, where rows are stored in primary key order, it hurts the table itself.

They are also 16 bytes against 8 for a bigint, and that size is repeated in every index and foreign key.

The fix is a time-ordered UUID: UUID v7 (standardised in RFC 9562) starts with a millisecond timestamp, so new ids sort to the end like an auto-increment while staying unique without coordination.

Model answer

Each id is a 64-bit integer built from parts, generated locally on each machine with no coordination:

• 41 bits of timestamp in milliseconds since a custom epoch: about 69 years of ids, and ids sort by time.
• 10 bits of machine id: up to 1,024 generators, each assigned a unique number at startup.
• 12 bits of sequence: a counter within the same millisecond, so 4,096 ids per millisecond per machine. If it runs out, the generator waits for the next millisecond.

It fits in a bigint, is roughly time-ordered for indexes, and needs coordination only to hand out machine ids.

Model answer

Clocks do move backwards: NTP corrections, a VM migration. If the time goes back to a millisecond already used, the generator can produce the same ids again, with the same machine id and sequence.

Handle it in the generator: remember the last timestamp used. If the current time is earlier, either wait until the clock passes it (fine for small jumps of a few milliseconds) or refuse to generate and raise an alert for larger jumps. Some keep using the last timestamp and borrow from the sequence.

On the ops side, configure NTP to slew (adjust gradually) rather than step the clock, and make machine id assignment safe so two machines never share an id after a restart.