Production Debugging Interview Questions

Model answer

The average hides it: most requests are still fast, and a small share (about 1 in 100 or fewer) became very slow. Something affects only some requests: some hosts, some endpoints, some customers, or a resource that runs out now and then.

Start by narrowing it down:

• What changed at that time: a deploy, a config change, a traffic jump, a dependency's incident.
• Break the p99 down by endpoint, host, region and customer. One dimension usually lights up.
• Pull a few slow traces and see where the time goes: our code, the database, a downstream call, or waiting for a connection.

Model answer

• Whether those two run a different version (a canary, a half-finished deploy) or different config.
• Garbage collection pauses or memory pressure on those hosts.
• CPU steal or a noisy neighbour on the same physical machine; disk or network trouble on that host.
• Their connection pools or thread pools being exhausted while others are fine.
• The load balancer sending them more than their share (sticky sessions, uneven weights).

Mitigate first, investigate second: take the two hosts out of rotation, or replace them, and see whether p99 recovers. Keep one aside to debug if you can.

Model answer

The bottleneck is the connection pool, not the database. Requests queue for one of a few connections, and something is holding connections too long:

• A slow external call inside a transaction: the code opens a transaction, then calls a payment or HTTP API while holding the connection.
• A connection leak: an error path never returns the connection to the pool.
• Long transactions or a long-running report query sharing the same pool.
• A pool simply too small for current traffic.

Check pool metrics (active, idle, waiting, wait time), enable leak detection (HikariCP leakDetectionThreshold), and look in pg_stat_activity for connections sitting idle in transaction. Move external calls outside transactions.

Model answer

• An SLO for the user-facing thing (99% of requests under 300 ms) and alerts on error budget burn rate, so you are paged for real user impact, not every blip.
• Alerts on saturation, not just symptoms: pool wait time, queue depth, thread usage, event loop lag. These rise before latency does.
• Distributed tracing sampled enough to always have slow examples to look at.
• Canary deploys with automatic rollback when the canary's latency or errors are worse than the rest of the fleet.

And write the incident review with the timeline and what made it hard to find, then fix that.

Model answer

Look at memory after garbage collection, not total memory. A healthy service makes a sawtooth: memory climbs, GC runs, it drops back to a steady floor. A leak shows the floor itself rising with every cycle until the process hits its limit.

A cache grows too, but levels off at its maximum size. If it never levels off, the cache has no bound, which is a leak by another name.

Correlate the growth with traffic: memory rising with each request handled points at something kept per request.

Model answer

Take heap snapshots: one shortly after start, one after the memory has grown. Compare them and look at which object types grew, then follow their retainers (or the dominator tree) to see what is keeping them alive.

In Node, use --inspect with Chrome DevTools, or v8.writeHeapSnapshot(), or --heapsnapshot-near-heap-limit. In Java, jcmd <pid> GC.heap_dump and Eclipse MAT, or -XX:+HeapDumpOnOutOfMemoryError to get one when it dies.

A heap dump can pause the process and contains real user data, so take it from a host out of rotation and handle the file as sensitive.

Model answer

• Unbounded maps used as caches: a module-level Map or static HashMap that is only ever added to.
• Event listeners added per request and never removed (Node warns about this with MaxListenersExceededWarning).
• Timers and closures: setInterval callbacks that are never cleared and keep their surrounding scope alive.
• ThreadLocal values in thread pools (Java): the threads live forever, so whatever was left in them does too.
• Request data stored globally, for logging or metrics labels with unbounded values such as user ids.
• Growing queues: work accepted faster than it is processed.

Model answer

• Restart before the crash, on your terms: a rolling restart on a schedule, or let the orchestrator recycle instances on a memory threshold, so users see no errors instead of random kills mid-request.
• Bound the thing that grows: swap the unbounded map for an LRU with a maximum size. Often a one-line fix that buys time.
• Turn off the feature with a feature flag if one feature is the cause.
• Raise the memory limit temporarily if it buys hours without hiding the problem, and alert on the growth rate so you know if it gets worse.

Write it down as a known issue with an owner and a date, so the restart job does not quietly become the permanent fix.

Model answer

• Find which queries are using the CPU: pg_stat_statements sorted by total time in Postgres, the slow query log or Performance Schema in MySQL, or your cloud provider's query insights.
• Look at what is running right now (pg_stat_activity): a runaway query or a pile-up of the same one.
• Check what changed: a deploy, a new job, a traffic spike, a dropped index.

Stabilise before fixing: cancel a runaway query (pg_cancel_backend), pause a batch job, roll back the deploy, or turn on more caching. Restarting the database rarely fixes the cause and costs a cold cache.

Model answer

Now: stop the bleeding. Roll back the deploy or turn the feature off with its flag. That takes minutes and the database recovers.

Properly: run EXPLAIN ANALYZE on the query against production-sized data, add the index it needs (or rewrite the query to use an existing one), check the plan, then ship the feature again.

If rolling back is impossible, adding the index is the fix, but it has to be added without locking the table, which is its own problem.

Model answer

In Postgres, CREATE INDEX CONCURRENTLY. It builds the index without blocking inserts and updates, at the cost of taking longer and scanning the table twice. It cannot run inside a transaction, so many migration tools need it marked as non-transactional.

If it fails halfway it leaves an INVALID index that still slows writes; check for it and drop it before retrying.

In MySQL, InnoDB online DDL (ALGORITHM=INPLACE, LOCK=NONE) handles most index builds; for big or tricky changes, tools such as gh-ost or pt-online-schema-change copy the table in the background and swap it in.

Either way, run it at a quiet hour and watch replication lag and disk space, since a 500 GB table's index can be tens of gigabytes.

Model answer

• Test against production-sized data: a query that is fine on 1,000 rows can be a full scan on 100 million. A staging copy or a seeded large dataset catches it.
• Review query plans for new queries on large tables as part of code review.
• Set a statement timeout for the application's database user, so one bad query fails instead of taking the database down.
• Alert on the top queries by total time changing after a deploy, and on new sequential scans of large tables.
• Roll features out gradually behind flags, so a bad query reaches 5% of traffic first.