The Gate That Said No
One day of rebuilding the retrieval system behind Ask Evgeny, published with the numbers: traces on every query, a 975-node knowledge graph (as built that day) imported from data files that already existed, authority buckets that let superseded positions describe the record without ever supporting a current claim, and three model upgrades running in shadow until their gates read the evidence. The centerpiece is the feature that failed its own launch gate — an embedding-space atlas that measured 0.555 against a 0.60 bar and stayed dark. A system whose features can block themselves is governed; a system where every demo ships because it demos well is running on vibes.
In English, please
The author runs an AI assistant on his website that answers questions using only his published writing. This essay is the build log of a one-day overhaul of that assistant's search engine — the part that decides which passages of his writing get used to answer each question.
The biggest change is record-keeping. Every question now leaves a detailed receipt: which searches ran, how long each took, and exactly why each passage was picked. Like a flight recorder for the search engine — when something goes wrong, you can replay what happened instead of guessing.
Three smarter search upgrades were installed but deliberately not switched on. They run silently in the background, writing down what they would have done, while the old system keeps answering. Only when those recordings prove an upgrade actually helps does it go live. Think of a new employee shadowing the old one before taking over the desk.
One planned feature — a visual map of all the essays arranged by meaning — had to pass a quality test before launch: the map had to reflect which essays are actually related at least 60 percent of the time. It scored 55.5 percent. So the map stays hidden, even though it looks impressive. The essay argues this is the whole point: a trustworthy system needs features that can fail their own tests and stay unshipped.
The lesson for anyone building with AI: measure first, trial changes silently before switching them on, keep old evidence labeled instead of deleting it, and let quality bars say no — even to your favorite feature.
On this page
Yesterday, Ask Evgeny was a competent RAG chatbot: hybrid retrieval, grounded answers, real citations. Today it is a governed evidence system. Every query is traced, every position carries an authority bucket, a knowledge graph rebuilds itself on every deploy, and three model upgrades sit in shadow until the data says they deserve to ship. The proudest artifact of the whole build is a feature that measured its own quality, came up short, and blocked its own launch. This is the build log, with the numbers.
Where this started
The system I described in the original build guide answered questions well. It retrieved with dense vectors (bge-m3 on Cloudflare Vectorize) fused with keyword search (D1 full-text, BM25) through reciprocal-rank fusion, and it refused to answer anything the corpus did not cover. Good bones.
But it understood nothing about its own behavior. Intent detection was a stack of hand-written regexes, each added after some eval case slipped through. Retrieval was a single shot: one embedding, one query, done. Superseded positions were caught after generation by a text filter. And there was no record of why any given answer retrieved what it retrieved. When something went wrong, the diagnosis was archaeology.
The upgrade plan went through three adversarial review rounds before a line of code was written: one machine review of the architecture against the actual source, then two rounds against an independent frontier model that scored the first draft “architecture 85 to 90 percent right, execution readiness 65 to 70 percent” and was correct on both counts. The final plan had one organizing principle: the position ledger stops being a feature and becomes the governance layer that retrieval, evidence selection, synthesis, and refusal all answer to.
Instrument before you optimize
The first thing that shipped was not a model. It was measurement.
Every uncached question now writes a retrieval trace: how long each stage took, which candidates each signal surfaced, and the exact score decomposition for every passage that made the final context — the base fusion score and the boost that moved it. Every trace carries version stamps: which embedding model, which generation model, which corpus revision, which ledger revision. A trace without versions is refused at write time.
Two rules made this trustworthy. First, the trace store never sees the question text — a hash and a length, nothing else, purged after 90 days. Second, 27 of the 159 evaluation cases were frozen as a holdout the day before any model change. The system is never tuned against them. When a change claims improvement, the holdout is the judge, and the holdout does not negotiate.
The instrumentation paid for itself within the hour. The first baseline run “failed” two leak probes — and the investigation showed the probes were stale: the essays they guarded had been published weeks earlier, which meant the production leak filter was actively stripping legitimate sentences about my own published work. A measurement system that finds a real bug on day one is not overhead.
Shadow mode is a governance stance
Three model upgrades are now running in production. None of them touches a single answer.
A cross-encoder reranker scores the top 15 candidates on every ask. A knowledge-graph leg links the question to concepts and walks one hop of typed edges. A query planner (a small model, not the big one) classifies intent, rewrites follow-up questions into standalone ones, and splits multi-part questions. All three run after the response has already been sent — they cost the visitor nothing — and log what they would have done into the trace.
When enough shadow data accumulates, each one faces a gate: the reranker is judged on ordering quality against the holdout, the planner on how often it disagrees with the regexes it wants to replace and who was right. Whatever fails the gate does not ship. The regexes stay in the codebase either way, as the fallback path for the day a model call fails.
The discipline sounds slow. It is the opposite: all three shadows shipped in one afternoon because shipping them risked nothing.
The graph was already there
The knowledge graph took no machine learning at all. It took an inventory.
A glossary of 387 terms with cross-references. A position ledger with 111 dated, receipt-backed stances. Around 300 pre-verified citable claims. Hand-curated reading paths. An accountability log of confirmed and retracted calls. Every one of these already existed as a data file, maintained by hand over months, sitting in five parallel vocabularies that never talked to each other.
One deterministic import script — no LLM anywhere in the path — unified them into 975 nodes and 1,236 typed edges: cites, defines, mentions, supersedes, refines, coins, confirmed-by, redrawn-by. The graph re-seeds itself on every deploy, diffed by content hash, so publishing an essay now grows the graph the same way it already re-trains the retrieval index. If you run a content site with any editorial discipline, you probably own a knowledge graph too. It is just scattered.
Authority buckets, not deletion
The obvious way to handle superseded positions is to suppress them at retrieval. The reviewers talked me out of it, and they were right: you need the old position to explain how the thinking changed, whether a shift was a refinement or a reversal, and why an old quote is no longer the current view.
So nothing is deleted. Every position is classified — deterministically, from ledger status fields, with no model in the path — into one of four buckets: current-authoritative, historical, record-only, internal-only. The rule that matters: record-only evidence may describe the record, but it can never support a current-view claim. The public position pages now show the full record: the current stance, the dated receipts behind it, the prior versions with refinements distinguished from reversals, and an explicit statement of where the verified record ends. Two new machine-readable tools expose the same thing to any agent that asks.
The gate that said no
The flashiest feature in the plan was a concept atlas: all 80 essays projected into two dimensions by meaning, rendered as an interactive map. Every retrieval demo has one. That is exactly the problem — a 2D projection can look profound while faithfully representing almost nothing.
So the atlas had to pass three gates before it could go live: the build must be deterministic, the clusters must not just re-draw the category folders, and the 2D neighborhoods must actually preserve the high-dimensional ones — at least 60 percent overlap between each essay’s nearest neighbors in embedding space and its nearest neighbors on the map.
It scored 0.555. The gate said no. The page exists, fully built, unlinked and unindexed, and it stays that way until the projection earns the wall it wants to hang on.
That is the whole philosophy in one number. A system whose features can block themselves is governed. A system where every demo ships because it demos well is running on vibes.
The receipts
Numbers from the eval harness, before and after, same frozen gates:
- Retrieval hit-rate: 100 percent on the working set, 91.7 percent on the untouched holdout, against an 85 percent gate. The two holdout misses are both disjoint-vocabulary synthesis questions — the exact class the shadow upgrades exist to fix, which is why those two numbers were left standing as the before mark.
- Leak probes: all closed. Out-of-corpus refusal: 100 percent on holdout.
- One “failure” in the final run was a single refusal case that flipped. Re-tested live three times: declined correctly every time. Recorded as model nondeterminism with the retest as evidence, not quietly re-run until green.
- One silent bug fixed for good: chunks up to 6,000 characters were embedding only their first 4,000 — the tail of every long section was searchable by keyword but invisible to semantic search. Repacked so every chunk fits the embedding window. Check your own pipeline for this one; it is the most invisible retrieval bug I know of.
Why this is not just about one website
The pattern transfers directly to where advertising is heading. When buying agents negotiate with selling agents, the seller’s claims stop being marketing and start being machine-checked assertions. The sellers who win that transition will be the ones who can show their work: traced decisions, versioned claims, evidence that carries its own authority classification, and features that ship behind gates instead of press releases.
I have argued for a while that AI can interpret data but cannot vouch for it — that trust has to be built as infrastructure, not asserted. This build is that argument, applied to my own product, with the measurements published. The corpus that argues for governed evidence now runs on it.