System design interview: design a notification system
A notification system takes an event such as "your order shipped", decides who should hear about it and on which channels, and hands each message to a delivery provider: APNs for iOS push, FCM for Android push, an email service, an SMS gateway, or your own in-app inbox. Design it as a pipeline of queues that separates deciding from sending, check each user's preferences and quiet hours before anything goes out, and deliver at least once with idempotency keys so retries do not become duplicates.
By Evin JohnPublished 8 min read
1. Clarify the requirements
The question is broad by design. Agree on channels, triggers and which messages are urgent before drawing.
- Functional: send push (iOS and Android), email, SMS and in-app messages; accept sends from internal services and from scheduled campaigns; respect per-user, per-category channel preferences and opt-outs; render localized templates; support scheduled sends and quiet hours; report delivery status.
- Non-functional: transactional messages (one-time codes, security alerts, receipts) arrive within seconds and are never silently lost; marketing can take minutes; duplicates are rare; a broadcast to every user must not slow anything else down.
- Out of scope unless asked: running your own mail servers or SMS carrier connections, the campaign authoring UI, and engagement analytics beyond delivered and opened.
2. Estimate the scale
Average throughput is modest. The difficulty is in spikes, provider limits and the volume of status records. All figures are assumed for illustration.
| Quantity | Assumption | Result |
|---|---|---|
| Daily active users | 50 million | the audience for a full broadcast |
| Messages | 10 per active user a day, all channels | 500 million a day, about 5,800 a second on average |
| Broadcast peak | one campaign to all 50 million users within 10 minutes | 50,000,000 / 600 s ≈ 83,000 a second |
| Channel mix | 70% push, 20% in-app, 9% email, 1% SMS | 350 million push, 45 million email, 5 million SMS a day |
| Status records | about 500 bytes each, kept 30 days | 250 GB a day, about 7.5 TB retained |
| Device tokens | 2 devices per user, about 200 bytes each | 100 million tokens, about 20 GB: one partitioned table |
So the steady state fits a small cluster, broadcasts need queues that absorb a burst about 14 times the average, SMS is the costly channel to protect, and the status store is the largest data set. Capacity estimation covers how to pick these round numbers quickly.
3. Define the API
POST /v1/notifications
headers: Idempotency-Key: order-81723-shipped
body: { "userIds": ["u_42"], "category": "order_updates",
"template": "order_shipped", "data": { "orderId": "81723" },
"priority": "high", "expiresInSec": 86400 }
returns: 202 { "notificationId": "n_9f2c" }
GET /v1/notifications/n_9f2c
returns: 200 { "deliveries": [ { "channel": "push", "status": "delivered" },
{ "channel": "email", "status": "sent" } ] }
PUT /v1/users/u_42/preferences
body: { "order_updates": { "push": true, "email": true, "sms": false },
"quietHours": { "start": "22:00", "end": "07:00",
"timeZone": "America/Chicago" } }The send endpoint returns 202 Accepted, not 200: the request is durable, but delivery happens later. A repeated Idempotency-Key returns the original notificationId instead of creating a second notification.
4. Sketch the data model
| Table | Key fields | Why it is there |
|---|---|---|
notifications | notification_id, (caller, idempotency_key) unique, category, template, payload, priority, expires_at | The accepted request; the unique index rejects retried submissions |
deliveries | delivery_id, notification_id, user_id, channel, status, attempts, provider_message_id | One row per user per channel; the status tracker. Partition by user_id, expire after 30 days |
preferences | user_id, category, channel, enabled; quiet hours; IANA time zone | Read for every recipient, so it is cached |
devices | user_id, token, platform (APNs or FCM), last_seen, active | Where push goes; tokens are deactivated when providers reject them |
templates | template_id, channel, locale, version, body | Versioned so an in-flight message renders the version it was queued with |
The in-app inbox is its own store keyed by user and sorted by time. Treat it as one more channel whose provider is your own database, with a live connection to nudge open apps.
5. Fan out and deliver at least once without duplicates
Exactly-once delivery to a phone is not achievable: a provider can accept a message and the acknowledgement can be lost on the way back. The realistic goal is at-least-once delivery with idempotency checks at each hop, which shrinks the window for a duplicate to almost nothing. Walk the interviewer through where duplicates come from.
| Hop | What can go wrong | Guard |
|---|---|---|
| Producer to API | The caller times out and retries | Unique (caller, idempotency_key); a repeat returns the first response |
| Fan-out to queue | A fan-out worker crashes halfway through a chunk and the chunk is redelivered | Deterministic delivery_id = hash(notification_id, user_id, channel); inserting an existing ID is a no-op |
| Queue to sender | A sender crashes before acknowledging and the message is redelivered | Before calling the provider, move the delivery from queued to sending with a conditional update; skip if it is already sent |
| Sender to provider | The provider accepted, then the sender died before recording it | Unavoidable gap; on push, set APNs apns-collapse-id or the Android notification tag so a repeat replaces the first on the device |
Retries with backoff. Retry transient failures (timeouts, 5xx, 429 from a provider) with exponential backoff and jitter, for example 1, 2, 4, 8 seconds up to a cap, then send the message to a dead-letter queue. Use delayed retry queues rather than sleeping in the worker, so one slow message does not block the queue behind it. Do not retry permanent failures: APNs answers 410 for a token that is no longer valid, FCM reports UNREGISTERED, and an email hard bounce means the address is gone. Deactivate the token or address instead.
Expiry. Every message carries a deadline. A one-time code that is valid for five minutes should be dropped, not delivered after ten. Push providers accept an expiry as well (apns-expiration on APNs, ttl on FCM), so a message for an offline phone is not shown hours late.
6. The architecture
- Producers
- Order, auth and billing services
- Campaign scheduler
- Ingest
- Notification API (validation, idempotency)
- Notifications database
- Fan-out and routing
- Fan-out workers
- Router workers
- Preference, device and template cache
- Queues
- push-high, push-low
- email-high, email-low
- sms
- in-app
- Senders
- Push senders to APNs and FCM
- Email sender to provider
- SMS sender to gateway
- Inbox writer
- Tracking
- Delivery status store
- Provider callback receiver
- Retry and dead-letter queues
- AcceptValidate, check the idempotency key, store the notification, return 202. Nothing slow happens on this path.
- Fan outTurn a user list or segment into chunks of about 1,000 recipients each, so a 50 million user broadcast becomes 50,000 independent jobs.
- RoutePer recipient: drop channels the user turned off, check quiet hours and caps, look up device tokens, render the template in the user's locale.
- SendChannel senders pull from their queues, respect the provider's rate limit, call the provider and record the result.
- TrackProvider callbacks (email delivered, bounced, opened; SMS delivery receipts) update the delivery row.
7. Scale it
- Priority without starvation. High-priority queues get dedicated sender capacity. Pure strict priority can starve marketing forever during a busy day, so give low priority a guaranteed minimum share.
- Provider rate limits. Every provider caps throughput per account, and SMS senders are often limited per sending number. Put a token bucket per provider account in front of each sender pool, the same mechanism as a rate limiter, and let queues absorb the difference.
- Per-user caps. Limit marketing to, say, three pushes a user a day with a counter keyed by user and local date. Transactional messages bypass the cap.
- Quiet hours and time zones. Store an IANA time zone, not a UTC offset, so daylight saving changes are handled. A non-urgent message that lands in quiet hours is rescheduled to the end of the quiet period in the user's zone through a delayed queue or a job scheduler. A campaign set for "9 a.m. local" is released zone by zone (some zones are offset by 30 or 45 minutes), which also spreads the load across the day.
- Re-check at send time. Preferences can change between scheduling and sending. Read them again just before a scheduled message goes out.
- Status writes. Hundreds of millions of small writes a day suit a log-structured store partitioned by user ID, with a 30-day expiry.
8. What breaks, and what you do about it
| Failure | Effect | Mitigation |
|---|---|---|
| A provider is down or throttling | Queues for that channel grow; messages age | Circuit breaker, backoff, expiry on stale messages; a second email or SMS provider for failover (push has no alternative for a given platform) |
| A broadcast floods the system | One-time codes wait behind millions of marketing messages | Separate high-priority queues and senders; per-producer quotas at the API |
| Stale device tokens | Wasted calls and error noise | Deactivate on APNs 410 or FCM UNREGISTERED; prune tokens not seen for months |
| A campaign is triggered twice or targets the wrong segment | Millions of unwanted messages | Idempotency key per campaign run, a send to 1% first, and a kill switch that drains its queued messages |
| Email bounces and spam complaints | The sending domain's reputation drops and inbox placement suffers | Suppression list for hard bounces and complaints; honor unsubscribe immediately |
9. Trade-offs to discuss
- One queue per channel or one shared queue. Separate queues isolate a slow provider and let each pool scale on its own, at the cost of more moving parts.
- Resolve preferences at fan-out or at send time. Resolving early makes sending cheap; resolving late respects a user who opted out a minute ago. Many designs do both for scheduled messages.
- Fewer duplicates or fewer losses. Marking a delivery as sent before calling the provider prevents duplicates but can lose a message if the call then fails; marking after can duplicate. For transactional messages, prefer the occasional duplicate.
- In-app by push or pull. A live connection is instant but costs a connection per open app; fetching on open is cheaper.
10. How to talk through it in the interview
Fictional example
- Interviewer
- A customer says their phone showed "Your order shipped" twice. How could your design do that?
- Candidate
- The likely path is a sender that called FCM successfully and then crashed before marking the delivery sent, so the queue redelivered it. The idempotency key at the API and the deterministic delivery ID rule out the earlier hops.
- Interviewer
- Can you close that last gap?
- Candidate
- Not completely, because the provider's acceptance and my database write cannot be one transaction. I would set a collapse ID derived from the delivery ID, so a repeat replaces the first notification on the device instead of stacking a second one.
- Draw the queues before the providers; the pipeline is the design.
- Say "at least once plus idempotency" and show the guard at each hop.
- Mention time zones and quiet hours unprompted; they are easy to forget and interviewers notice.
Practicing this design with Natively
Treat the walk-through as a mock interview: have a friend play the interviewer on a call, or read the prompt aloud and answer it yourself. Natively transcribes the conversation live and can suggest talking points for each question as it comes in; comparing them with what you said is a quick way to spot gaps such as retry rules for permanent errors or quiet hours. You can capture a screenshot of your pipeline sketch and have it explained. Afterwards, the transcript and notes are there to review which parts you rushed.
Questions people ask
How do you avoid sending the same notification twice?
Accept that delivery is at least once and put a guard at every hop: an idempotency key on the API, a deterministic delivery ID per user and channel, and a status check before calling the provider. The one gap left is a crash after the provider accepted a message, which push collapse identifiers hide on the device.
How should a notification system handle time zones and quiet hours?
Store each user's IANA time zone rather than a UTC offset, so daylight saving is handled. When a non-urgent message falls inside quiet hours, schedule it for the end of the quiet period in that zone. Urgent messages such as security alerts usually bypass quiet hours.
Why use separate queues for each channel and priority?
Providers fail and slow down independently, so one queue per channel stops an email outage from delaying push. Separate priorities stop a marketing broadcast from holding up one-time codes. Each sender pool can then scale and be rate limited to match its provider.
Where to go next
Practice with the same assistant you would use on a call: it listens, transcribes and suggests answers from your own resume and the job description.
Facts on this page were last checked on . Product details come from the Natively app itself; if something here is out of date, tell us at hello@heynatively.com.
