September 12, 2026
EVALUATE THE WORK, NOT THE AGENT'S CONFIDENCE

Your loop now stops on purpose, asks before consequential actions, and runs inside enforced budgets. One vulnerability remains, and it is the subtlest: the agent can do everything right procedurally and still hand you wrong work — wrapped in fluent, confident prose.
Models narrate well. They summarize their own runs persuasively, cite sources that sound real, and describe validation they never performed. None of that is evidence. Evidence is what you can check without asking the model whether it did a good job.
Narration is not proof
Treat every agent self-report as a claim, not a result. "I retrieved five sources and all checks pass" proves nothing until you see the five retrievals in the tool log and the check outputs beside them. This is Lesson 30.2's distinction at the finish line: simulated diligence and real diligence read identically in chat.
Real proof lives in five places, all outside the model's narration:
1. Retrieved sources. The actual items fetched, with names, timestamps, and raw content or links — not the agent's summary of them. 2. Tool logs. Inputs, outputs, errors, and retries for every call. The log shows the two empty retrievals the summary omitted. 3. Output schema. The required fields, types, and counts the task demands. A packet missing its source list fails no matter how elegant the prose. 4. Citations. Each factual claim traceable to a retrieved source. An uncited claim is a draft note, not a finding. 5. Acceptance tests. Mechanical checks the output either passes or fails — field presence, source match, duplication scan — runnable without model judgement.
If you cannot point to these five, you are grading confidence. Confidence is free and unlimited. Evidence is scarce and checkable — which is exactly why it matters.
A scorecard for agent work
Different tasks need different checks, but six dimensions cover nearly every agent output in this course. Turn them into a scorecard for each job:
| Dimension | Question it answers | Observable check |
|---|---|---|
| Factual accuracy | Is each claim backed by a source? | Every claim cites a retrieved item; spot-check two citations by opening them |
| Source quality | Were the right sources used, and only those? | All citations come from the approved list; no unvetted URLs |
| Formatting | Does the output match the required shape? | Schema check: fields present, types correct, file in the right path |
| Duplication | Is this new, or a repeat? | Similarity scan against recent packets; duplicates flagged, not re-filed |
| Safety boundary | Did the run stay inside its permissions? | Log review: no unapproved tools, no writes outside scope, gates respected |
| Completion | Did it finish or stop honestly? | Packet present with minimum counts, or a missing-info note with named gaps |
Score each dimension pass or fail — no partial credit on safety or citations. A packet with beautiful formatting and one invented citation fails. That strictness is the point: the scorecard exists to catch the failures confidence hides.
Keep scorecards task-specific and short. A research packet needs citation and duplication checks; a ticket classifier needs accuracy and completion checks; every task needs safety and formatting. Three to six checks per task is plenty. Twenty checks means none of them get run.
Worked example: the draft validator before GitHub handoff
Return to the editorial agent from Lesson 30.2. Before its review PR can even be opened, a small validator — ordinary code, no model judgement — runs six mechanical gates:
DRAFT VALIDATOR — article package
[ ] required fields present (title, date, packet ref, author, sources)
[ ] every claim has a citation pointing at the approved packet
[ ] cited sources match retrieved items (no invented references)
[ ] formatting matches article schema (headings, file path, filename date)
[ ] duplication scan: similarity below threshold vs. last 30 articles
[ ] safety: no publish/send/delete calls in tool log; PR targets review branch
→ all pass: open review PR with validator report attached
→ any fail: return to agent with the failing check named, max 1 fix cycle
Notice the design. The validator does not judge writing quality — a human reviewer does that at the PR. It judges checkable properties a script can verify in seconds. Citation present or absent. Field present or missing. Similarity score above or below threshold. Tool log clean or dirty.
Only a validator-passing draft reaches GitHub, and it arrives as a review PR — never a direct push to main, never a silent publish. The handoff packet contains the draft, the validator report, the source packet reference, and the tool log excerpt. The human merges (Lesson 30.2's gate) with everything needed to review in one place. Failed drafts loop back once for a named fix; a second failure escalates to the owner with the validator output attached. The loop improves the work without negotiating its own standards.
Practical exercise: define three observable pass/fail checks
Take your agent from Lessons 30.1–30.3 and write its evaluation:
1. Pick the three dimensions from the scorecard that matter most for your task. 2. Write each as a pass/fail check an independent person — or a script — could run without asking the agent anything. Name the artifact examined (log, file, citation) and the exact bar ("all three claims cite packet items" beats "sources look good"). 3. State what happens on failure: one named fix cycle, then escalation to the owner with the failing check attached.
Finish line: three written pass/fail checks plus a failure path, added to your agent card or loop-budget file.
Verify: take a real or sample output and run your own checks against it by hand. If a check requires you to trust the agent's summary instead of opening a log, file, or citation — rewrite it.
Common failure mode: checks like "output is high quality." Observable how, and by whom? Replace adjectives with artifacts: named fields, counted citations, a similarity number, a clean log.
Check your understanding
1. Why is the agent's own summary of its run insufficient as evidence? Name two places you would look instead. 2. List the five sources of proof and say what each one rules out. 3. Pick two scorecard dimensions and explain what failure looks like in each for a research packet. 4. In the validator example, what does the script check — and what does it deliberately leave for the human reviewer? 5. Your agent fails the same check twice. What happens next, and why only one fix cycle before escalation?
ARTICLE DISCUSSION
JOIN THE
CONVERSATION.
Got a question, a take, or a better way to do this? Log in and leave a comment.
