Caching in system design interviews

1. What a cache is

A cache is a smaller, faster store that holds copies of data whose source of truth lives elsewhere. It works because access is uneven: a small share of items gets most of the reads, so keeping those few in memory serves most requests without touching the slower store. The cost is that the copy can be out of date, and keeping it correct is most of the engineering.

2. Why interviewers ask about it

Nearly every read-heavy design needs a cache, so nearly every candidate draws one. Interviewers use it to separate a box on a diagram from an understanding of its behavior. The follow-ups are predictable: what happens on a miss, what happens on a write, what happens when the cache dies, what happens when one key gets a million reads a minute. Answering those precisely is a strong signal at any level.

3. Where caches sit

Worth saying in an interview: the database already caches. A primary-key lookup on a table whose working set fits in the buffer pool is often fast enough without an extra cache layer.

4. Read and write patterns

get(id):
  v = cache.get("user:" + id)
  if v is not null: return v
  v = db.read(id)
  cache.set("user:" + id, v, ttl = 300s + random(0, 60s))
  return v

update(id, fields):
  db.write(id, fields)
  cache.delete("user:" + id)      # delete, do not overwrite

Why delete instead of writing the new value? Two concurrent updates can reach the database in one order and the cache in the other, leaving the cache holding the older value indefinitely. A delete is the same whichever order it arrives in, and the next read loads the current row.

5. Eviction, expiry and invalidation

Production caches usually approximate these: Redis, for example, samples keys to approximate LRU or LFU rather than keeping an exact ordering. Expiry and eviction work together: TTL bounds how stale an entry can be, eviction keeps memory within its limit.

Invalidation strategies

• TTL only. Accept staleness up to the TTL. Simple and robust; right for data where a minute of staleness is harmless, such as follower counts.
• Delete on write. The writer deletes the key after updating the database. Fresh in the common case; a cache that misses the delete keeps a stale entry until its TTL, so keep a TTL as a backstop.
• Versioned keys. Put a version in the key (product:42:v7) and bump it on write; old versions are never read again and age out. Good for objects assembled from many rows.
• Change data capture. A consumer reads the database's change log and deletes affected keys. Catches writes from every service, including ones that forgot to invalidate.

One race survives delete-on-write: a reader misses and loads the old row, a writer updates the row and deletes the key, then the reader stores the old value it loaded. The TTL bounds how long that lasts. To close it fully, the cache can hand out a lease token on a miss and refuse a later set whose token was invalidated by a delete, the approach described in Facebook's memcache paper.

6. Stampedes, hot keys and sizing

Cache stampede (thundering herd). When a popular key expires, every request that misses at that moment goes to the database together. A key read 10,000 times a second with a 50 ms reload can send hundreds of identical queries before the first one finishes. Three defenses, usually combined:

• Request coalescing. Only one request per key reloads; the others wait for its result. Within one server this is a per-key in-flight map (often called single-flight); across servers, a short lock or lease in the cache itself.
• Jittered TTLs. Add a random spread (say 300 s plus up to 60 s) so keys loaded together, for example after a deploy or a cache restart, do not all expire in the same second.
• Early refresh. Refresh a key shortly before it expires, either on a schedule or probabilistically as it nears expiry, and serve the existing value meanwhile (stale-while-revalidate).

Hot keys. A single key with extreme traffic, such as a viral post, saturates the one cache node that owns it however large the cluster is. Fixes: a tiny in-process cache with a TTL of a second or two in front of the shared cache; several copies of the key under suffixes (post:9:0 to post:9:7) read at random and all deleted on write; or the CDN for public content.

catalog:      50 million items x 2 KB  = 100 GB in total
hot set:      20% of items (assumed)   = 10 million items
raw size:     10 million x 2 KB        = 20 GB
overhead:     x 1.5 for keys, pointers and fragmentation = 30 GB
with replica: x 2                      = 60 GB across the cluster

The hit rate depends on how skewed the reads really are: if the hot 20% gets 80% of reads, about 80% of requests are served from memory and the database sees the rest. Say the skew is an assumption, and offer to grow the cache if measurements show a longer tail. More on the arithmetic in capacity estimation.

When not to cache

• Data that must be current for a decision: a balance used to authorize a payment, stock at checkout. Read the source of truth; cache only for display.
• Data with little reuse: items read once, or a long tail where the hit rate would be low.
• When the database is already fast enough: an indexed lookup on a working set that fits in the buffer pool.
• Responses unique to each user and request, where the hit rate does not repay the memory and invalidation work.
• Shared caches without careful keys: a key that omits the user or tenant can serve one person's data to another.

7. How caching shows up in common prompts

8. Mistakes to avoid

• "Add Redis" with no pattern. Say cache-aside or write-through, and what happens on a write.
• No staleness bound. Every cached value needs a TTL or an invalidation path, ideally both.
• Updating the cache on write instead of deleting. Concurrent writers can leave the older value cached.
• Sizing for the whole data set. Size for the hot set and say what hit rate you expect.
• Ignoring cold starts. After a cache restart, every request is a miss. The database must survive that load, or the cache is warmed before traffic returns.
• Treating the cache as durable. If losing it loses data, it is a database and needs persistence and replication.

9. How to talk about it in the interview

• Justify the cache with a number: the read rate, the read/write ratio, the hot set.
• State the pattern, the eviction policy and the TTL in one sentence.
• Say how stale is acceptable for this data, and for whom.
• Mention the stampede and hot-key defenses before you are asked.

Practicing with Natively

Caching follow-ups come fast, so practice answering them aloud. In a mock interview, Natively transcribes the conversation live and can suggest talking points for each question as it comes in; comparing them with what you said is a quick way to spot gaps such as invalidation on write or stampede protection. If you capture a screenshot of your diagram, it can explain what the picture shows. Afterwards the transcript and notes are kept for review, so you can see which follow-ups you answered precisely and which you talked around.