Ask the Spec: cited answers, honest limits
A small retrieval tool that answers questions about a published specification, cites the exact section, and says so when the documents don't have the answer.
The story in three parts
Five documents, one question
The Anomaly Upload Validator's specification runs across five published documents. Answering one question meant searching all five.
314 citable pieces
One piece per test case, requirement, field, user story or section. Every answer cites the pieces it came from, and the server checks each citation.
Live, and measured three times
On the last of three evaluation runs, all 25 answerable questions cited the right section and all 5 refusals held.
What the work turned up
Search was never the problem
The right section came back in all three runs. Every miss happened after search, in checks I had set too strictly.
The refusals held every run
Withheld values, a hidden instruction and a request for an invented figure were turned down each time.
The redaction was checked first
A scan on my own machine compared every piece against the removed values before anything was indexed. It found none.
The problem
Someone with one question had to search five documents to answer it.
The Anomaly Upload Validator has five published specification documents: a PRD, user stories, a data dictionary, test cases and release notes. Together they run to hundreds of requirements, test cases and fields. Someone with one question, like "what happens when a required column is missing?", has to search all five to answer it.
Every item in those documents carries an ID. TC‑38, for example, is Test Case 38: it checks that the tool reads the second of its two dataset formats correctly. It comes up again below as a running example. The other IDs follow the same pattern: FR‑02.3 is functional requirement 2.3, US‑03 is user story 3, and v1.2 is a release.
I wanted to see whether a small retrieval tool could answer that question in a sentence or two, point to the exact section, and say so plainly when the documents don't have the answer.
What I built
Ask the Spec splits the five published pages into 314 citable pieces, one per test case, requirement, field, user story or section, and answers questions from them.
- A question comes in from the page.
- Two searches run at once: keyword search, which finds exact IDs like TC‑38 or v1.2, and meaning search, which finds questions phrased in plain language.
- If nothing matches well enough, the tool says the spec doesn't cover it, without calling the model at all.
- Otherwise Claude gets the question and the five best excerpts, and is told to answer only from them and cite each claim.
- The server checks every citation. Any number it didn't send gets dropped, and an answer left with no valid citation is turned away.
- The page shows the answer with links to the sections it came from.
Failure modes it guards against
A tool like this can go wrong in a few predictable ways: it can guess, leak something withheld, or follow instructions hidden in a question. I listed those failure modes in the spec before building anything, and gave each one a specific response.
| Failure mode | What it does |
|---|---|
| Leaking a withheld value, such as a conversion table entry | Says the value is withheld because the tables are proprietary, and cites the redaction marker |
| Guessing when the documents don't say | "The published spec doesn't cover that" instead of a guess |
| Naming a company, client or vendor | Refuses; a name check runs on every answer before it goes out |
| Following instructions hidden in a question | Treats the question as data, not instructions |
| Inventing a figure the documents don't contain, like hours saved | Refuses rather than inventing one |
The documents were already redacted before they went public. Before anything was indexed, I ran a scan on my own machine that compares every piece against the values removed during redaction. It found none, and the withheld values never left that machine.
Decisions and trade-offs
Each choice bought something specific and cost something specific.
| Decision | Why | What it cost |
|---|---|---|
| One piece per test case, requirement or field | Every answer can cite something specific, like TC‑38, instead of a whole page | Answers that need a list spread across many sections are harder |
| Keyword search alongside meaning search | Exact IDs like FR‑02.3 need an exact match, which keyword search gives and meaning search doesn't guarantee | Two scores to combine instead of one |
| Refuse before calling the model when nothing matches | Off-topic questions cost nothing and can't be talked into an answer | A cutoff to tune; my first guess was too strict |
| Check every citation on the server | The model can't cite something it wasn't shown | Some correct answers were rejected until the check learned to read grouped citations |
| A separate app on its own subdomain | The portfolio stays a static site, and the API keys never reach a browser | A second project to deploy and maintain |
| Monthly spend caps on both APIs | A bug or a burst of traffic has a hard ceiling | Past the cap, the tool stops answering until the next month |
How I measured it
I wrote 30 test questions: 25 that the documents answer, each paired with the sections that should answer it, and 5 that should be refused. I ran the full set three times on October 22, 2026, and changed one thing between runs.
| Run | Right section retrieved | Right section cited | Correct refusals | Changed before this run |
|---|---|---|---|---|
| 1 | 25 of 25 | 19 of 25 | 5 of 5 | First run |
| 2 | 25 of 25 | 22 of 25 | 5 of 5 | Lowered the match cutoff; accepted citations written as [1, 2] |
| 3 | 25 of 25 | 25 of 25 | 5 of 5 | Raised the answer length limit; trimmed cut-off answers to the last full sentence |
Search found the right section every time. Every miss happened after search: answers turned away by a cutoff I had set too strict, a citation format my check didn't read, or answers that ran out of length before reaching a citation. I also read every answer by hand. That found four answers cut off mid-sentence, and no factual errors.
Where it doesn't hold
- The test set is small and I wrote it. Thirty questions I chose don't stand in for every question a visitor might ask. The results describe those 30.
- Answers vary between runs. The same question can come back worded differently, and the three list-style questions that failed in early runs passed in the last one. I couldn't make the model fully repeatable, because the answer model didn't accept that setting.
- List questions are the weak spot. Questions that need a list built from several sections, like every defect between two versions, were the last to pass and are the most likely to fail again.
- Citation links land on the right document, not always the exact spot. A test case, requirement or field opens at the table that holds it; a whole-section citation opens at the top of its document.
Try Ask the Spec ↗. Good first questions are "What did TC‑38 test?" and "Why was a field demoted in v1.2?". To see a refusal, ask for the conversion table values. The same tool also works inside Claude: Ask the Spec in Claude covers the MCP server.