Type the Hot Columns, Pass the Rest Through

#resources #resources/programming #resources/programming/architecture

Type the hot columns, pass the rest through

When you mirror an evolving external schema, type and index only the fields you actually query on, and carry every other field through untouched. The producer can grow its schema all it likes; your mirror neither drops the new data nor needs a migration to keep up.

The bind

The lcos CLI mirrors daily_features from Convex. On the producer that row is a big bag of about 110 fields, stored as an untyped blob; the real contract is a TypeScript interface that keeps growing as the coach learns new signals.

You're stuck between two bad options:

Hand-mirror all 110 fields in your client's types. The moment the producer adds a field, your mirror is wrong, and the worst part is the failure mode: the new field is silently dropped, not flagged. You only notice when a feature quietly returns nothing.
Store the whole thing as one opaque blob. Now you can't filter or aggregate in SQL — sleep under six hours requires deserialising every row in your language.

The move

Split the difference along the line of what you query. Type the ~16 columns you filter, sort, or aggregate on, give them real SQL columns, and flatten everything else into a passthrough map that round-trips verbatim. A test that deserialises a row with unknown fields, re-serialises it, and asserts they survived is the whole safety net.

In Rust that's a struct with typed Option hot fields plus #[serde(flatten)] extra: Map<String, Value>. New producer fields land in extra and ride along; nothing breaks, nothing is lost, and the columns you care about are still first-class for queries.

The contrast worth holding

This is the opposite move from Email Internal Model. There, mxr normalises Gmail labels and IMAP folders into one provider-agnostic Label type, because mxr owns the semantics and wants every provider to collapse to a common shape. Here, the upstream owns an evolving schema you only partly consume, so you pass through instead of normalise. The rule: normalise when you own the meaning; pass through when someone else does and you're only borrowing a slice.

Validation against code (2026-06-26)

apps/cli/crates/core/src/features.rs types 16 hot Option columns and flattens the rest into extra.
round_trip_preserves_unknown_fields deserialises, re-serialises, and asserts unknown fields survive.
The SQLite mirror lifts the hot columns into real NULL-able columns; the blob is stored verbatim alongside.

Why this generalises

Any time you mirror an evolving JSON contract you don't control — webhook payloads, event streams, third-party API responses — type the slice you act on and carry the rest. It buys forward compatibility for free and turns "they added a field" from an incident into a no-op.