Design Visual Search
Design visual search: given a query image (or a crop of one), return visually similar pins with low latency. The index covers billions of images, and a newly created pin should be searchable within seconds.
Assume an embedding model exists; you design the serving system. Budget ~200 ms per query including embedding inference. Billions of pin embeddings at 256 floats ≈ 1 KB each means terabytes of index — sharded and quantized. Metadata filters (country, safety) must apply during retrieval, not as an afterthought.
Cover: the root/leaf retrieval architecture, how the index shards and what that costs in recall, realtime inserts into an ANN index under concurrent reads, deletions, and how you'd know retrieval quality is degrading.
Hints
Hint 1
The size forces the architecture: terabytes of embeddings can't fit one box, so a root fans out to leaf shards each holding an ANN graph over a partition, and merges top-k — lightweight scoring at the leaves, a heavier reranker at the root.
Hint 2
Apply metadata filters during graph traversal, not after: post-filtering top-k can leave you with zero results when the filter is selective.
Hint 3
Freshness comes from a separate realtime segment — new embeddings ingested via a log into an in-memory index that's periodically sealed and merged — so the big immutable index never blocks on writes.
Asked at
Sign in to talk to the interviewer — ask what's in scope, what scale to design for, or for a nudge when you're stuck. Conversations save to your account.