Give Your Writing a Brain

The shelf and the brain

Most bodies of writing are shelves. The essays are all there, indexed by date, searchable by keyword — and completely inert. A reader with a real question (“does this apply to a company like mine?”) has to become a librarian first: guess the right terms, skim five posts, assemble the answer themselves. The writing knows things. It just can’t tell you.

The shift is small to describe and large in effect: put a retrieval-grounded answer layer on top of the corpus. Not a chatbot that makes things up — an engine that answers from what you actually wrote, cites the source every time, and says “the writing doesn’t cover that” instead of inventing. Then expose it where people already are: inside ChatGPT and Claude as a connector, and as a chat box on the site itself.

There is one exception to “only from what you wrote”, added later, and it belongs up front rather than buried: questions about live facts the corpus was never going to hold. Step 3 covers where that starts and where it stops.

I built exactly this for this corpus — roughly eighty-five essays, a 390-term glossary, the playbooks and standards pages — over an afternoon, on infrastructure that costs roughly nothing at this traffic. This is the whole recipe, including the parts that went wrong. If you have a shelf, you can give it a brain.

The shape: one brain, two mouths

Resist the urge to make this complicated. The entire system is one small server that does four things — index, retrieve, generate, and expose — and everything else is a surface on top of it.

• Index: turn the corpus into searchable vectors + text, and re-run it automatically whenever you publish.
• Retrieve: given a question, pull the handful of passages most likely to contain the answer.
• Generate: hand those passages to an LLM under strict rules — answer only from these, cite them, refuse otherwise.
• Expose: one HTTP endpoint for programmatic use (the MCP connector), one for the site chat, one for a semantic search box.

I ran all of it on a single Cloudflare Worker: Vectorize for embeddings, D1 (SQLite) for the text and keyword index, KV for rate limits and caching, Workers AI for the embedding model. Generation started on Workers AI too and now runs on Claude Sonnet. That switch was one line of config, which is the argument for putting the provider behind a seam on day one — start on the free tier, move when the answers deserve it. Pick your own stack; the shape is what matters. Now, part by part.

Step 1 — The corpus is already structured. Chunk it that way.

Retrieval works on chunks, not whole documents — a 2,000-word essay is too coarse to match a specific question against. The instinct is to slice by a fixed character count. Don’t. Your writing already has structure: headings. Chunk on those, carry the metadata (title, URL, section heading, category, date), and each chunk arrives pre-labeled with where it came from — which is exactly what you need for citations later.

Two decisions here pay off later. First, keep a content hash per chunk so re-indexing only touches what changed — publishing one essay shouldn’t re-embed the whole corpus. Second, wire the indexer into your deploy pipeline, not a separate cron you’ll forget. When publishing is the training event, the brain is never stale and there’s nothing to maintain. That single property — the corpus updates itself — is most of why this is worth building rather than paying for a SaaS that silos your content.

Step 2 — Retrieval is a multi-lane problem

Here’s the mistake almost everyone makes: they reach for vector search alone. Embeddings are magic for meaning — they’ll connect “are my retail-media numbers real?” to an essay titled iROAS Is Not a Number, It’s a Negotiation that shares not one keyword with the question. But they’re mediocre at exact terms — acronyms, product names, a specific phrase — where old-fashioned keyword search wins. Use both, then fuse the rankings.

Fusing is simpler than it sounds. Reciprocal rank fusion just says: a chunk’s score is the sum of 1/(k + rank) across every list it appears in. No weight-tuning, no calibration — a passage that several lanes rank highly floats to the top; one that only appears deep in a single lane sinks. Take the top handful and pass them on. This one technique closes most of the gap between a demo that impresses you and a system that actually finds the right passage on a real, oddly-worded question.

Start with two lanes. This corpus now runs a third, a knowledge-graph leg added in August once its A/B cleared, and nothing about the fusion had to change to take it: same formula, same weight, one more list. That is the practical case for RRF. Adding a retrieval signal is an append, not a re-tune.

Step 3 — Grounding is the entire game

This is where a knowledge tool is won or lost. A model handed some passages and a question will happily blend what it retrieved with what it already “knows” from training — and the moment it does, you’re shipping confident, plausible, uncited fabrication under your own name. The whole discipline is forcing the model to answer only from the passages, cite them, and refuse when they don’t cover the question.

Three techniques make grounding real, not aspirational:

• Prompt rules are necessary but not sufficient. Tell the model to answer only from the passages, cite them with markers, refuse otherwise, and — for a personal corpus — speak about the author, never impersonate them. Good models mostly comply. “Mostly” is the problem, which is why the next two exist.
• Map citations server-side. The model emits markers like [S2]; your code translates those to real titles and URLs from the passages you actually retrieved. The model never writes a URL on its own authority, so it can’t invent one.
• Tier your sources, and filter the output. If some material is private (drafts, unlisted pieces), let it inform answers as “background” but tag it so it can never be cited, named, or quoted — then run a server-side filter that strips any sentence referencing it. Belt and suspenders: the prompt asks the model to keep secrets; the filter enforces it when the model forgets.

A filter you have never tested is a filter you hope works. Mine now gets a canary: seed a synthetic private chunk through the real indexing path, ask four questions engineered to drag it out — two quoting a unique token from it verbatim, two circling the same idea in words the chunk never uses. Then confirm it never appears, and delete it. The part that took three attempts to get right is the positive control. Without proving the canary was reachable in the first place, “the filter withheld it” and “retrieval never found it” produce identical output, and only one of them is a working filter. So the harness asks retrieval for the canary with the visibility filter off and requires it back before any negative check counts. If that control fails, the run is inconclusive — never a pass. Run it against throwaway resources, not your production database. I seeded production on the first attempt and had to clean up by hand.

One more piece belongs here: detecting a refusal. Models decline in prose (“the passages don’t mention…”) while still emitting citation markers, so you can’t infer refusal from “no citations.” Anchor the check on the answer’s opening — a real answer starts with substance; a refusal starts by disclaiming — and when you detect one, return a clean, on-brand “not covered” message instead of the model’s clumsy internal framing. Then log the question. That log becomes the most valuable output of the whole system.

The one exception, and its fences

Readers ask about things a corpus of essays structurally cannot know: who runs a company now, what got announced last week, who just raised. Declining every one of those is correct on the letter and useless in practice, so this engine has a live-data lane, and the boundaries around it are the interesting part.

A cheap regex decides whether a question is even about live facts — company, people, news, market phrasing. Roughly nine in ten questions never match, and those pay nothing: no extra latency, no external call. Questions that do match get one bounded lookup, which returns nothing when the provider has no good answer. That is two gates before any outside data exists, and the second one is allowed to say no.

What comes back is fenced by three rules. It arrives in its own labelled block, so the model always knows which text is yours and which is not. It gets attributed in prose, never a citation marker — [S#] means he wrote this, and nothing else is ever allowed to wear that badge. And it may never contradict what your passages say about your own views; live data can add a fact, not overrule a position.

The honest part: a question with zero corpus hits and a live block will answer instead of declining. That is a deliberate choice, not an accident, and it is the sentence that qualifies the promise at the top. The corpus stays primary; nothing outside it is ever citable as yours. If you build this, make that call explicitly rather than discovering later that your “answers only from your writing” engine quietly stopped doing that.

Step 4 — Publishing makes it smarter (the loop)

The reason to own this rather than rent it is compounding. Three loops run once it’s live, and none of them need you.

The corpus loop: publish, auto-index, done — the brain never drifts from the writing. The quality loop: a small golden set of test questions (definitional, cross-essay synthesis, “should decline” traps) you run before shipping any prompt or retrieval change, so quality is measured, not vibes-based — a tweak that quietly breaks grounding shows up as a failing case, not a customer complaint. And the gap loop: the log of unanswerable questions feeds a monthly pass that ranks genuine, repeat-asked, on-topic gaps into an essay backlog. Your readers tell you exactly what to write next — in their own words, weighted by how many asked.

A word on what the quality loop actually measures, because I had this wrong for a while. A golden case passes when the right source comes back. That is retrieval working. It says nothing about whether the passage behind [S2] supports the sentence [S2] is attached to, and a citation that points at a real source making a different point is a more expensive failure than no citation at all — it looks like evidence. Checking it needs a separate pass: hand a judge the question, the answer, and the full text behind every marker, and ask for each one whether the passage entails the specific claim. Use a different model call than the one that wrote the answer. A generator grading its own citations is not a check.

Step 5 — Positions age. A retriever alone can’t tell.

Steps 1 through 4 solve retrieval and grounding. They don’t solve time. A chunk from an essay you wrote in 2020 and a chunk from one you published last week look identical to a vector index — same shape, same treatment, no sense that one of them is a position you’ve since walked back. Ask the system “what’s your view on X” and it can hand back your old answer with total confidence, citation and all, because nothing in the pipeline knows the answer changed.

Half the fix is cheap and I should have done it on day one. Every chunk already carried a publication date from Step 1, and nothing read it. Two changes: put the date in the passage header the model sees, and tell it that any date it states must come from one of those headers. That alone kills the most common time failure, which is not citing an old view — it is stating a year the corpus never said. Then read the question for temporal intent. “In 2024” is a point in time; “recently” and “these days” mean lately; “originally” means the opposite. Weight passages accordingly.

Two cautions from getting this wrong in public. Keep the weighting small — my first attempt used a 2.2× boost, which was strong enough to drag a personal essay into a technical answer purely because it was recent, so it now sits inside the same band as the doc-type boosts rather than overpowering them. And test it on the scores your system actually produces: my first tests used a tidy 0 to 1 range, passed, and told me nothing, because real fused scores are not shaped like that.

That handles time as a ranking problem. It does not handle the case where you changed your mind, and no amount of date-weighting will, because both essays are equally real and equally yours. That needs a second, much smaller layer sitting on top of the chunk index: a curated ledger of positions — named stances on named concepts, each with a date, a verbatim receipt, and a status. Not every sentence you’ve written qualifies; a position is a claim worth tracking over time, something you might revise. When a later piece changes your mind, the ledger doesn’t overwrite the old row — it adds a new one and links them, so both survive with a label: current, refined, reversed, or retracted.

The rule that makes this trustworthy: nothing gets to declare itself current. A row’s lifecycle is computed, not typed in. If a later row supersedes it, it’s superseded. If you’ve explicitly pulled the claim, it’s retracted. Otherwise it’s held. That sounds pedantic until you see what it prevents — an author, or an agent drafting on the author’s behalf, marking a position “current” by habit, days after it stopped being true.

Grounding this in verbatim text turned out to be the hard part, not the schema. A quick pressure test came back rough: draft a batch of position rows, adversarially check every receipt against the actual source, and roughly four in ten were wrong — quoted from a summary instead of the essay itself, dated to the wrong publish date, or citing a stance that wasn’t really the author’s own. None of it was malicious; all of it would have shipped silently. The fix is the same one Step 3 already argued for: don’t trust the model to self-police. A small script greps every receipt against the real corpus text, checks that dates move forward in the right order, and refuses to build if anything doesn’t match. The ledger only ships once the validator is clean.

That last sentence was true of the script long before it was true of the pipeline. The validator sat as a manual command, and every time anyone ran it by hand it flagged the same row: a receipt reading “‘No Fluff’ offers” where the essay says “No Fluff Advisory offers”. One substitution, in a field whose only job is to be verbatim, on a row already stamped verified. Almost nobody ran it. A check nothing runs is a check you don’t have, so it’s a blocking deploy step now.

At answer time, a matched position becomes a second input alongside the retrieved passages — a short, dated block the model is told outranks anything a passage merely implies. And because a prompt rule is still just a rule, a second check runs after generation: if the answer quotes a retracted or superseded position without saying so, it gets flagged and corrected before it goes out. Same belt-and-suspenders logic as the leak filter in Step 3, aimed at a different failure mode.

The payoff is public, not just internal: a small set of pages, Evolution of Thinking, lays out the dated timeline for a tracked concept in one place — what was said, when it changed, and why. Including the times it was wrong. A ledger that only shows the wins is marketing wearing a research costume; this one keeps the corrections in.

Two mouths: the connector and the chat box

One brain, two ways in — and the second one is the interesting one.

The obvious surface is a chat box on your site: a floating button, a panel, questions to the same engine, cited answers rendered with links. Keep it lazy — the button is static HTML and the server is only called when someone actually asks, so it costs your page-load budget nothing.

The less obvious, higher-leverage surface is a remote MCP server — the Model Context Protocol, the emerging standard that lets any AI client call external tools. Expose your brain as an MCP endpoint and it drops into ChatGPT, Claude, or any MCP client as a connector: now people query your entire body of work from inside the assistant they already use, without visiting your site at all. Tools like ask_evgeny (grounded answer), search_writings, get_essay, get_glossary_term. This is distribution most content sites never get — your writing, available at the point of thought, in someone else’s tool. Gate it with a free key issued by email, and the distribution channel doubles as a lead list.

Worth deciding per surface: whether to stream. Streaming took time-to-first-text from 13.5 seconds to 5.8, which on a knowledge tool is the difference between waiting and reading. But it is opt-in per request here, and the on-site chat box does not use it, on purpose. A cited answer is checked as a whole — the citations resolve at the end, and a leak filter that runs over the finished text cannot run over a sentence that has already left the building. The standalone destination streams because speed wins there. The widget waits because correctness does.

The four bugs that actually cost me an afternoon

Every clean architecture diagram hides the debugging. Here’s the honest part — the specific things that broke, because you’ll hit versions of them too:

1. The vector store rejected my IDs. Long essay slugs blew past the index’s 64-byte ID limit and the whole batch failed. Fix: hash the slug into a fixed-width ID. The lesson generalizes — don’t use human-readable strings as primary keys in systems with length caps; hash and keep the readable version in your own metadata.
2. Verbose pages buried the essays. My reference pages are keyword-dense, so raw retrieval kept surfacing standards docs over the thought-leadership essays that actually answered the question. Fix: a light doc-type boost — essays and their summaries up, reference pages down — so the writing you’re proud of wins on thematic questions while pages still win on direct lookups.
3. Graceful refusals counted as answers. The model would decline and emit citation markers, so my “did it answer?” check (based on citation count) called a polite “I don’t know” a successful answer. Fix: detect refusal from the answer’s opening sentence, not its citations, and route it to the gap log.
4. The vocabulary-mismatch miss. A question phrased with none of an essay’s words (“privacy as an engine, not a brake”) sat right at the retrieval cutoff. Fix: widen the candidate pool before fusion and give the model a few more passages of context, so a borderline-but-correct source still gets a seat at the table.

None of these show up in a weekend demo. All of them show up the first time a real person asks a real question. Budget the afternoon.

What changed in August 2026

The system above kept running. The August 2026 update was a semantic-discovery upgrade — a reranker, a knowledge graph, a query planner — and the part worth copying is not the feature list but the discipline around shipping it. Seven lessons:

• Instrument before you optimize. Every uncached ask now writes a retrieval trace: per-stage latency, each passage’s score split into base relevance × boost, and version stamps for what was live at the time. The trace keeps a hash of the question, never the raw text, and expires after 90 days. You cannot A/B what you never measured; before the traces, every tuning argument was an anecdote.
• Freeze a holdout. The golden set from Step 4 now has 174 cases. 33 of them are frozen as a holdout: stratified, never tuned against, read only when deciding whether a change ships. A test set you optimize toward stops being a test set.
• Ship model changes in shadow. Three candidate upgrades shipped dark, running after each response was already sent and logging what they would have done: a cross-encoder reranker (bge-reranker-base), a one-hop knowledge-graph retrieval leg, and a query planner on llama-3.3-70b. Zero added latency for the user, zero change to ranking, until the shadow data justified switching one on. One has since earned it. The graph leg went live on 11 August — hit-rate identical on both eval splits, median latency 10.4s against a 10.5s baseline — and now runs inside the request as a third fusion signal. The reranker and the planner are still dark, and the old regex gates stay in place as the fallback path.
• Turn curation into a graph. 1,028 nodes and 1,281 typed edges at the time of writing, imported deterministically from files the site already had: the glossary, the position ledger, citable claims, curated cross-links. No LLM anywhere in the import path, and the graph re-seeds on every deploy. Most sites already own their knowledge graph; it is just scattered across data files.
• Let gates say no. An embedding-space atlas failed its own neighborhood-preservation gate, scoring 0.555 against a 0.60 bar, and stayed dark. A feature that can block itself is what separates governance from vibes.
• Buckets over deletion. Superseded positions still get retrieved, but classified: current-authoritative, historical, or record-only. Record-only evidence may describe the record; it may never support a claim about the current view. Same logic as Step 5, pushed down into retrieval itself.
• Fix the silent gap. Chunks ran up to 6,000 characters, but the embedding model only read the first 4,000 — the tail of every long chunk was invisible to semantic search, and nothing errored. Repacking every chunk to fit the model’s window fixed it. Check your own embed-input truncation; it is the most invisible retrieval bug there is.

Most of this shipped dark on purpose: measurement first, gates second, features only when the data clears them.

What it costs, and when it’s worth it

At this scale — a few thousand chunks, low-hundreds of questions a month — the honest number is small: embeddings on a serverless AI tier, generation metered per token against a frontier model, a vector index and a database both inside free-or-cheap allowances, one worker. It was near zero while generation ran on the free tier, and it stayed in single-digit dollars a month after the switch. Per-IP and per-key rate limits plus an answer cache keep a viral day from becoming a surprise bill. You can start on a free model tier and flip to a premium one with a config change if the answers deserve it.

It’s worth building when three things are true: you have a corpus (dozens of pieces, not five), the content is evergreen enough that answers stay useful, and you’d rather own the intelligence layer than rent one that silos your writing behind someone else’s login. If you’re there, the whole thing is an afternoon and a handful of files.

The attention economy taught us to publish and hope the archive gets found. This is the other move: make the archive answer. You already did the hard part — you wrote the things. Giving them a brain is mostly plumbing, and the plumbing is worth it, because a shelf that can talk back is a fundamentally more useful thing than a shelf.

Want to see it running? Ask this corpus anything — the chat box in the corner, or add it to your own ChatGPT or Claude as a connector. The same brain also powers a standalone destination: Yevgeny.ai. For the position ledger specifically — the dated record of what changed and when — see Evolution of Thinking.

This build is also a practice area: the same entity, citable-content, and machine-surface engineering, applied to AdTech, MarTech, and data companies — AI Visibility & GEO Advisory.