Deterministic Before LLM

#resources #resources/programming #resources/programming/architecture

Deterministic before LLM

For any feature that uses an LLM, the deterministic path runs first. Regex, joins, address history, stylometry, explicit config - all the cheap, exact, reproducible work. The LLM is what runs when the deterministic path can't answer, not what runs first because it's exciting.

This is not an anti-AI stance. AI is useful, and often genuinely the right tool. The point is restraint. If traditional code can solve the problem more cheaply, more predictably, and with a clearer failure mode, use traditional code. Let the LLM earn its place.

The rule and its corollary

The rule is straightforward: don't add an LLM call for something a join can do. Tone-mismatch detection in mxr stayed deterministic on purpose — stylometry against a contact's prior messages is a histogram, not a synthesis task. If a check could be answered with a query against existing tables, look at the data first.

The corollary is harder: when the LLM is disabled, the deterministic path must still produce a useful answer or an explicit error. Silent degradation that pretends success is a bug. A feature that quietly fails-open when the LLM is unavailable is worse than one that fails-loud, because the user never learns the LLM mattered.

Why this is load-bearing

Three failure modes the rule prevents:

The "demo-driven" feature. An LLM-only check looks great in a demo and falls apart when the local LLM is off, the API key isn't set, or the rate limit kicked in. The deterministic-first version works everywhere; the LLM is the cherry.
The cost-and-latency creep. Every LLM call costs tokens and adds round-trip time. Putting one in a hot path you could've answered with a SQL query is a tax on every user every time.
The unreproducible bug. Deterministic checks fail the same way every time, with stack traces you can read. LLM checks fail differently each run, depending on prompt, model, temperature, and provider. Reproducible bugs are debuggable; LLM bugs require a vibe check.

There is a fourth failure mode hiding under all three: the system becomes impressive but opaque. That was the lesson from Building an LLM Model Router: Lessons From the Wild. The first router asked an LLM to classify the request, then pushed the result through capability scores and weights. It worked until it did not, and each patch made the system harder to explain.

The rewrite used hard deterministic rules first, embeddings for the common middle, an LLM only as the uncertain fallback, and deterministic model-bin selection at the end. The win was not only cost and latency. The real win was that a bad route now had a trace: rule hit, nearest examples, confidence, margin, selected bin. You could see the failure and fix the labeled examples. A system you can explain beats a system that merely impresses you.

What this looks like in practice

A pre-send safety pipeline in mxr has six checks. Five are deterministic (typo distance, attachment regex, reply-all sanity, PII detectors, tone stylometry). One — answer-coverage — is LLM-backed because "does this draft actually address the asks in the thread?" is genuinely a synthesis task. Even there, the LLM output is validated against the retrieved thread, and an LLM-disabled run degrades to "Info" (the check ran but couldn't conclude), not silent pass.

Same pattern applies to draft-assist: the prompt is grounded in the current thread and the user's instruction. Without semantic search available, it falls back to thread-only prompting. Without an LLM, the feature is unavailable, loud and clear.

The Notto AI Agent applies the same rule inside a model-led analysis. The
PostgreSQL parser and EXPLAIN reject invalid or unsafe SQL before a reviewer
model sees it. Code checks declared metric rules and can approve a structurally
simple query only when those checks cover the same questions the reviewer
would ask. After execution, code traces answer numbers to returned rows,
aggregates and supported derivations before the answer verifier receives the
remaining uncertain cases. The model still judges meaning. It no longer needs
to rediscover facts the runtime can establish exactly.

The model-router pattern is the same shape:

Hard rules for things code can know: image attachments, "latest news", long context, high-stakes patterns
Embedding similarity for fuzzy-but-bounded classification
LLM fallback only when confidence is low
Deterministic policy for the final side effect
A decision trace that makes bad calls inspectable

When the LLM does earn its place

The LLM is the right tool when:

The work is genuine synthesis: summarization, free-text Q&A, extracting structured commitments from unstructured prose
A deterministic version exists but produces too many false positives the LLM can prune
The output is reviewed by a human before any side effect (draft assist, briefings, suggestions)

It is the wrong tool when:

The data could answer it with a query
The work fires on every message, every keystroke, every render
The output drives a side effect with no human in the loop
The main reason for using it is that the feature feels more "AI"

The decision test

Before adding an LLM call, ask:

Can a query, parser, regex, rules engine, classifier, or embedding lookup answer this?
Would that answer be easier to test and debug?
Can the failure be traced without reading model reasoning prose?
If the LLM is disabled, does the product still do something honest?
Is the LLM doing synthesis, judgment under uncertainty, or language generation that traditional code would genuinely be bad at?

If the answer to the first four is yes and the fifth is no, do not use an LLM. This is the boring path, and it is usually the product path.

Policy before workflow before agent

The UnlikelyAI cheese classifier take-home extended this rule. An LLM does not have to be the fallback for the whole task. It can resolve one narrow fact inside a larger deterministic decision.

That architecture now lives in three focused notes:

LLM as Evidence Interpreter for the model boundary;
Policy vs Workflow vs Agent for choosing the right amount of agency;
Decision Trace Is the Explanation for audit and replay.

Governed Agent Architecture ties them together and applies them to Notto AI Agent and SQL Agent Starter.