You can now describe RAG and build a small version with visible chunks. The remaining question is harder: how do you know the answers are any good — and how do you keep them good when documents change?

Three checks, not one vibe

"It cited something" is not a quality test. Judge every answer on three separate checks, because each fails differently:

1. Retrieval quality — did the system fetch relevant material? The right chunks, from the right version, with the right permissions, for this question. Check this in the retrieval log before reading the answer. 2. Answer faithfulness — does the answer stay within that material? Every factual claim should trace to a supplied chunk. Anything the model added from memory or imagination fails here, even if it sounds right. 3. Answer usefulness — did it answer the person's actual question clearly? A faithful quotation of the wrong policy section is still a bad answer. The response should resolve the asked question in language the reader can act on.

A handbook answer that retrieves the current remote-work section, quotes the two-day rule accurately, and tells the employee exactly how to request days passes all three. An answer that retrieves last year's policy fails check one. One that invents an exception fails check two. One that dumps three policy pages without answering fails check three.

Score them separately in your test log. Conflating them produces the familiar shrug: "the AI was kind of wrong."

Build a small evaluation set

Five questions beat five hundred unjudged chats. For each test question, record four fields:

• The question as a real user would ask it.
• The expected source (document, section, version).
• An acceptable answer (what counts as correct and grounded).
• An unacceptable claim (the plausible invention or overreach to watch for).

Example from a Sonariq-style corpus:

Include one question the corpus cannot answer. The correct behavior is a clean decline (below), and any invented answer fails automatically. Run this set after every corpus or prompt change. Evaluation is how you notice that a "small improvement" broke grounding.

Filled example: five-question sheet with scoring columns

Copy this table for your feature. Score R / F / U per row (pass–fail each); a row passes only if all three pass.

R = retrieval quality, F = faithfulness, U = usefulness. Question 5 passes only on a clean decline with named checked sources and zero invented content.

Citations are product evidence, not decoration

A citation should take the reader to the actual supporting material: the exact section, page, or filing passage behind the claim — not a homepage, not a search page, not a footnote-shaped ornament.

Teach your system this standard: a source link that cannot be opened and verified is not a citation. Each claim in the answer should map to a visible chunk with a working link, title, and date. When the reader clicks, they land where the answer came from.

Tool formats for this idea differ, and that is fine. Anthropic's citations documentation shows one implementation of grounded answers with linked evidence. Treat it as an example of the durable rule, not the rule itself: the format is tool-specific, but preserving and showing the source is permanent. Whatever stack you use, keep the chunk ID, source URL, and retrieved text in your log so any citation can be audited later.

"I cannot find that" is a valid answer

Every RAG system needs a designed decline. When retrieval returns nothing relevant — or only fragments below your threshold — the answer should say so plainly:

This is not a failure mode to hide. It is the behavior that makes every other answer believable. A system that never declines will eventually invent, and one confident invention destroys more trust than ten honest declines.

Set the threshold in advance: minimum relevance, required source coverage, and which topics always decline. Test the decline with your out-of-corpus evaluation question, and log declines the same way you log answers — they are evidence about corpus gaps.

Documents change: plan the refresh

A retrieval index is a snapshot, and snapshots rot. Policies get revised, manuals get new versions, filings get superseded, research notes get corrected. Without change management, your RAG system becomes a confident archive of last year's truth.

Give every document three lifecycle fields and one process:

• Expiry or review date. When must this chunk be rechecked?
• Version. Which revision is indexed, and which replaced it?
• Retirement. When a document is superseded, remove or clearly supersede its chunks — do not leave two versions competing in similarity search.

Then run a small loop: source freshness check, retrieval check, answer check, citation check, update or fix. Assign an owner and a cadence — monthly for policies, per-filing for Sonariq inputs, per-release for manuals. The owner from Lesson 23.2's plan is not ceremonial; they decide what "current" means.

Enforce permissions before the prompt

Access control belongs in retrieval, not in the model's good intentions. Never retrieve material simply because the model can technically see it. Filter by user, role, and document visibility before assembling the prompt, as introduced in Class 14 — what the model never receives, it cannot reveal.

This matters twice: once for privacy (contractor versus employee handbook sections, one client's research versus another's), and once for evaluation (test each role's question set against that role's visible corpus). A RAG demo that runs as an admin and ships to everyone is not finished.

Practical exercise: build a five-question evaluation sheet

Make a five-question test sheet for your proposed RAG feature. For each question include the expected source, an acceptable answer, and an unacceptable claim. Make question five one the system should decline, and write the exact decline wording plus which sources were checked.

Finish line: a five-question sheet you could hand to a colleague, who could run all five questions and score retrieval, faithfulness, and usefulness separately.

Verify: run the decline question (question five) yourself. Does the system refuse cleanly, name what it checked, and invent nothing? If it answers anyway, the decline threshold is not enforced.

Common failure mode: scoring "it cited something" as passing — a decorative footnote to a homepage while the answer drifts beyond the retrieved chunks. Every claim must map to a visible chunk with a working section-level link, or the row fails.

Check your understanding

1. What are the three separate quality checks, and why must they be scored independently? 2. What four fields belong in each evaluation row, and why include one unanswerable question? 3. What makes a link a real citation rather than decoration? 4. Why is "I cannot find that in the available sources" a valid — and trust-building — answer? 5. What three lifecycle fields plus one process keep a retrieval index from rotting?

Part VI synthesis: from raw data to grounded answers

Look back at the path you walked:

Class 18 → see the public data universe and turn it into product ideas
Class 19 → choose sources that are legal, usable, current, and fit
Class 20 → store records durably in tables that survive growth
Class 21 → ask precise questions with SQL
Class 22 → retrieve by meaning with embeddings and vectors
Class 23 → answer with visible evidence through RAG

The durable habits travel with you: preserve source URL, owner, date and period, unit, and retrieval time. Prefer read-only prototypes before write paths. Keep one checkable artifact per lesson — source card, source map, schema sketch, query plan, retrieval plan, evaluation sheet. Teach concepts before dashboards, and never call a visible website a production API.

Forward, two parts build directly on this one. Part VIII turns a dependable RAG workflow into an agent capability: a tool the agent can call with the same filters, logging, and citations you built here. Later application work turns grounded answers into reports, alerts, and research products readers pay for — because the evidence is inspectable, not asserted.