call(back)

Design a Metadata Catalog and Schema Registry

Design the catalog that holds databases, tables, columns, views and grants for many tenants. DDL must be transactional; query planners read it at very high rates and must never plan against a stale definition without knowing; schema changes must be observable by other systems; and a dropped table should be recoverable for a time.

Cover the data model and its versioning, the consistency of DDL, how reads reach high throughput, how changes are published, and how deletes work.

Hints

Hint 1

Model every entity with a version and keep all entities of a tenant on one shard, so a DDL transaction touching several of them stays on one partition.

Hint 2

Every commit appends to a per-tenant change log with a monotonically growing version; planners and caches subscribe and validate cached reads by version instead of re-reading.

Hint 3

Soft-delete with a retention window: drop marks the entity, undrop clears the mark, and a collector removes it after the window.

Asked at