Design an ML Evaluation Platform
Design the evaluation platform for an AIML organisation: it runs benchmarks across many model versions, scores results with both automatic scorers and human raters, and produces comparisons people make ship decisions on. Present it without relying on a diagram — a fixed spoken structure that a listener can follow: requirements and scale numbers → the API → the data model → the write path → the read path → the bottleneck → what you would trade.
Cover: benchmark and dataset versioning; a run orchestrator that fans work across models; the scorer interface; result storage keyed for reproducibility; aggregation with confidence intervals; human-annotation intake with agreement tracking; and the two hard parts — caching so an unchanged (model, item) pair is never re-run, and versioning so a scorer change does not silently invalidate history.
Hints
Hint 1
Announce the order and number the components aloud ("three services: ingest, store, query") — that is what keeps a listener oriented without a picture.
Hint 2
Key every result by (model version, benchmark version, item id, scorer version). That single key gives you reproducibility, caching, and the ability to explain any number on a dashboard.
Hint 3
A scorer change is a new scorer version, not an edit: old results stay attached to the old version and a re-score is an explicit, auditable job. Raw agreement between raters misleads on skewed labels — track kappa.
Asked at
Sign in to talk to the interviewer — ask what's in scope, what scale to design for, or for a nudge when you're stuck. Conversations save to your account.