Class 38 taught you to catch drift — when the AI quietly changes the task. Class 39 taught you to see debt — when quick patches leave no understandable shape. Both turn dangerous at the same moment: when a confident explanation substitutes for evidence.

That is what this final class is about. Not whether the AI sounds right — whether you can prove it.

Why "hallucination" is not quite the right word

"Hallucination" suggests one thing: the model dreamed something up. In software work, that covers only a fraction of what goes wrong. Here are five failures that all sound identical in chat — confident, fluent, plausible — but need different fixes:

1. Invented method. The model names an API function that never existed. The fix is current documentation plus a test call. 2. Stale version. The method was real — in 2022. The library has since renamed it, and the model quoted the old tutorial. The fix is checking the version you actually installed. 3. Wrong structure. The code is valid JavaScript, but your project stores source dates under record.publishedAt, not record.date. The model assumed a shape that is not present. The fix is opening the real files. 4. Valid but wrong requirement. The approach works but violates what you needed — fetching live sources on every load when your rule was approved-sources-only. The fix is re-reading your requirement, not the docs. 5. Looks right, behaves wrong. The page renders, no error appears — but a missing date silently drops the source, or a timezone shift shows yesterday's brief as today's. The fix is a behavioral test with edge-case input.

One word cannot cover all five. That is why this course uses a more actionable phrase: unverified autonomous decision-making — allowing a system to make or present a consequential decision without checking the relevant evidence.

Once you see it that way, the response stops being vague distrust and becomes a practical question: *what would prove this claim?*

Four claim types, four kinds of proof

Every technical claim the model makes falls into one of four buckets, each with its own proof:

Notice the pattern: the evidence always lives *outside* the chat. In a file, in the docs, in a test run, in a log. If the only source for a claim is the conversation that produced it, you have no evidence yet.

Walkthrough: the date field that might not exist

Take the research dashboard. You ask for source dates, and the model replies: "Done — the provider API returns published_date, and I have wired it through."

A beginner accepts that and moves on. You run the check instead:

1. Docs first. Open the provider's current documentation for that endpoint. Does published_date appear — spelled exactly that way, in your API version? Note the page and date checked. 2. Test request. Make a limited test request with a non-secret credential in a safe environment. Inspect the returned JSON yourself. Is the field present, null, or nested elsewhere? 3. Absent plan. Whatever the answer, define what the app does when the date is missing: show "Date unavailable," fall back to retrieval date, or hide the citation? Write that decision into the task file before the model writes another line.

Audit the repo with a second model

Before a vibe-coded system carries consequential data, run a repository audit as evidence practice:

1. Give a second, capable model read-only authority: inspect, do not change. 2. Require file-specific findings: exact file and line or section for every claim, ranked by impact, each with a verification step. 3. Verify every finding yourself against the actual code. The reviewer will be wrong about some things. 4. Resolve the real issues, then retest the affected behavior.

A second model catches a different set of mistakes because it did not write the original code. But its report is still a claim, not proof — step 3 is what makes it evidence.

Compare models with a scorecard, not loyalty

For any recurring job — copywriting, code review, test generation, source extraction — run the same sanitized task through two or more candidates under the same acceptance rubric, and score accuracy, constraint adherence, useful output, time, and cost.

Expect a split result. A small, fast model frequently wins a narrow, well-specified job. A larger reasoning model tends to win architectural or security-sensitive work, where a wrong decision multiplies downstream. The scorecard decides per job — not brand loyalty or price. Keep it in a short MODEL-NOTES.md, and refresh it when the work or the models change.

Know the escalation line

There is a boundary where an LLM audit is no longer sufficient. When the application is public and handles sensitive data, money, payments, accounts, private documents, or powerful integrations, commission a qualified independent penetration test — a controlled attempt to find weaknesses an attacker could exploit — as part of release.

The LLM audit prepares the ground: it cleans the repository and maps the attack surface. Professional testing finds attack paths that static reading can miss. Repeat it after material changes; security is not a one-time sticker.

Everything you already learned, reconnected

This evidence habit is where the earlier parts meet:

• Markdown holds the research and the decision — the claim, the source, the verdict.
• JSON shows the actual data shape — what the API really returned, not what the chat said it returns.
• The tool/API returns evidence — a test request with observable output.
• Logs show what ran — the background job, the deployment, the error.
• An agent can gather all of the above — but gathering is not evaluating. You still decide whether the evidence proves the claim.

Four prompts that ask for proof

Keep these verbatim. They turn any confident answer into an evidence request:

• "Which file proves that? Quote the relevant function and path."
• "Which official documentation page supports this API behavior?"
• "What command or test did you run? Show the output — and what it does not prove."
• "What assumption are you making because you could not inspect the real system?"

The last one matters most. An honest "I assumed the field exists" is worth more than a confident paragraph — it tells you exactly where to look. For security-sensitive findings, compare them against the weakness categories in the OWASP Top 10 before deciding severity.

Practical exercise: write an evidence record

Pick one live AI technical claim in your project. Create EVIDENCE-RECORD.md:

# Evidence record: source publication date

- Claim: the provider API returns `published_date` per record.
- Source/test: provider docs page (URL, checked today) + test request in safe env.
- Observed result: field present on 8/10 records; null on 2.
- Confidence: medium — present but not guaranteed.
- Next check: render "Date unavailable" for nulls; preview with both cases before merge.

Finish line: an evidence record whose supporting material a second person can inspect without replaying your chat.

Verify: hand the record to a fresh AI session with only the repo. Can it reach the same verdict? If it needs your memory of the conversation, the record is incomplete.

Common failure mode: recording the model's statement as the source ("AI confirmed the field exists"). Recovery: replace it with the outside artifact — the file, the doc, the output.

Check your understanding

1. Name two failures "hallucination" hides, and say how their fixes differ. 2. What does *unverified autonomous decision-making* mean — and who is responsible for the verification? 3. For an API claim, what two pieces of evidence do you need before building on it? 4. Why does a second model catch mistakes the first model missed — and why must you still verify its report? 5. What system conditions escalate you from an LLM audit to a professional penetration test?