Design a Merchant Catalog Update Pipeline
Merchants give Pinterest their product catalogs: scheduled feed files with up to tens of millions of items, plus real-time API updates for price and availability. Design the pipeline that ingests, validates, and publishes products as shoppable pins, keeps price and stock fresh within minutes, and tells merchants what went wrong.
Feeds come as daily full snapshots and hourly deltas (CSV/TSV/XML) plus an item-level API. Full feeds imply deletion semantics: an item missing from a full feed is gone. Most merchants have thousands of items; a few have 20M+ — a 20M-item feed cannot starve a 200-item feed. Validation rejects at item level with reasons, never the whole feed for one bad row. Per item, last writer wins by source timestamp: an old feed must not overwrite a newer API price.
Cover: idempotency and per-item ordering across the two write sources, safe full-feed deletion, fairness and backpressure across merchants, and how a price change reaches the pin within minutes.
Hints
Hint 1
One topic keyed by (merchant, item) gives per-item ordering for feed rows and API updates alike; version every write with its source timestamp and make the catalog write conditional (only if newer). Replays then become no-ops.
Hint 2
Diff before you fan out: typical daily change rate is 5–10%, so emitting only changes cuts every downstream consumer's load 10–20×, and an append-only change log makes any consumer rebuildable by replay.
Hint 3
Naive full-feed deletion is how a truncated download erases a merchant's catalog. Track last_seen_run per item, hold retirement when the candidate set exceeds a threshold, and retire in two steps: out-of-stock first, hard delete after a grace period.
Asked at
Sign in to talk to the interviewer — ask what's in scope, what scale to design for, or for a nudge when you're stuck. Conversations save to your account.