ByeBuy.ai
BUILD YOUR ESCAPE ROUTE · ✦ CURSOR · HOST IT · ◫ SUPABASE · CONNECT IT · ↯ RELAY · BUILD YOUR ESCAPE ROUTE · ✦ CURSOR · HOST IT · ◫ SUPABASE · CONNECT IT · ↯ RELAY ·
CURRICULUM
← BYEBUY NOTES

September 12, 2026

BUILD AN EVIDENCE LOOP BEFORE YOU TRUST AUTOMATION

Build an Evidence Loop Before You Trust Automation

Lesson 40.1 taught you to check one claim: sort it, find the outside proof, record the verdict. Reliable AI development turns that check into a loop you run on every change.

The loop

State the intended behavior
  → name the evidence that would prove it
  → let the AI inspect/propose/change within bounds
  → run or inspect the evidence
  → compare observed result to intended behavior
  → keep, revise, or escalate

Read it as a contract. You state the outcome and the proof *before* the AI acts — so "it works" can later be compared against something you wrote down. The AI works inside the bounds from Class 38. Then you judge the gap between intended and observed. Three exits: keep it, revise it with a new bounded task, or escalate it.

Try it on the dashboard's fallback: intended behavior ("every citation shows a date or the fallback string"), evidence named upfront ("preview with dated, undated, and malformed records plus the export test"), bounded change, observed run, comparison, decision.

A green check is not always the right check

The loop fails silently when the evidence proves the wrong thing. A test can pass while testing nothing that matters:

  • The brief page renders — but exposes another user's saved research. Green render, wrong check; the missing evidence was an access-control probe with two accounts.
  • The deployment completes — but the required environment variable was never set in production, so every source fetch fails quietly. Green deploy, wrong check; the missing evidence was a real request against the live configuration plus a log read.

Before accepting any passing result, ask: *does this evidence match the actual risk?* Rendering proves layout. It does not prove privacy, freshness, cost, or recovery. Name the risk first (Lesson 36.3's risk levels help), then choose the check that touches it.

The verification matrix

Use this as your default menu. Left column is the floor; right column is what higher stakes add:

ChangeMinimum evidenceHigher-stakes additional evidence
Text/layout changeBrowser preview at normal and narrow widthAccessibility and real-content review
Data transformationKnown input/output examples, including missing dataAutomated test plus source-sample comparison
API/tool integrationOfficial docs plus safe request/responseRate-limit, error, auth, and cost behavior
Background/agent workflowRun record, logs, saved output, stop behaviorRetries, alerting, permission review, failure simulation
Public/production actionPreview and explicit approvalRollback plan, monitoring, post-release check

Two habits make the table work. First, always include the ugly input: the undated source, the empty watchlist, the expired credential. Second, escalate the row when the boundary crossings from Class 35 appear — real users, real data, money, automation, strangers on the internet. The change did not get bigger; its blast radius did.

Make the agent report honestly

A useful coding agent reports five things after every task: what it changed, why, what it ran, the output, and what remains unverified. Require that shape in the task brief, and reward the last item most. A concise "I could not test the scheduled brief because the job runner needs a production credential I was not given" is worth more than an invented all-clear — it tells you the exact next evidence to arrange.

Treat "I could not test this because X" as a normal, professional result. It keeps the loop honest: the comparison step records *unverified* instead of pretending the gap is closed, and the next task closes it in a test environment or with human eyes.

Release evidence for high-stakes apps

When the app handles real users, private data, money, or production systems, the evidence loop grows a paper trail. Before release, save in the project record:

  • code-review findings and how each was resolved,
  • dependency and security scan results where available,
  • test and preview evidence for the critical paths,
  • verified access-control behavior (who can read and write what, proven with two roles),
  • the scope and results of any independent penetration test.

This is not bureaucracy. It is the difference between "the AI said it was fine" — a memory — and a record a second person, an auditor, or your future self can re-walk. If any item is missing, the release waits or the scope shrinks. That is the loop's escalate exit doing its job.

Escalate normally, not dramatically

Escalation sounds alarming; in practice it is four calm moves. When evidence is insufficient for the stakes:

1. Stop widening the AI's permissions or scope — no new files, no new keys, no production access to "just check." 2. Get more information — the real doc, the real payload, the real log line. 3. Shrink the blast radius — a test environment, a fixture, a copy of the data, a smaller task. 4. Bring the right judge — a domain expert, a security professional, or simply a slower, more deliberate human review.

Never escalate by giving the model *more* autonomy to compensate for missing proof. Uncertainty is the signal to narrow, not to widen.

The six-class synthesis

Part IX began with a fast conversation and ends with a habit that makes speed trustworthy. Hold the whole arc in one chain:

Use vibe coding to explore
  → graduate responsibility deliberately
  → direct the AI with bounded jobs
  → measure delivery, not just generation
  → stop drift and repair shape early
  → trust evidence over confidence

Class 35 gave you permission to explore quickly. Class 36 gave you bounded jobs, model hierarchy, and orchestration. Class 37 taught delivery over generation. Class 38 put rails against drift; Class 39 gave one-seam repairs. This class closes the set: earlier speed is unsafe without proof, and proof stays fast once it is a loop rather than a rescue.

Forward into Part X

Instinct is not a control system. Part X turns this part's habits into machinery: Git and GitHub for recoverable history, specifications for durable intent, plans before code, and small tasks that a human and an AI can inspect together. Every evidence record, task file, and build loop you wrote here becomes the input those tools protect. Preview the shape now: a branch per bounded job, a review per diff, a checkpoint per validated step — the evidence loop, made permanent.

Practical exercise: write your build loop

For one upcoming change — real or imagined on the research dashboard — create BUILD-LOOP.md:

# Build loop: show fallback for undated sources

- Outcome: every citation shows a date or "Date unavailable"; export unchanged.
- Scope: citation component + research-record mapping only. No auth, nav, or schema changes.
- Checks: preview with dated / undated / malformed records; run existing export test.
- Evidence: screenshots + test output + changed-file list, saved beside this file.
- Checkpoint: save recoverable version before and after; note remaining unknowns.
- Escalation: if the API shape differs from docs, stop, update the task, re-approve before coding.

Finish line: a complete one-page build loop ready to hand to a coding AI — outcome, scope, checks, evidence, checkpoint plan, and escalation trigger on a single page.

Verify: give the file to a fresh session with repo access. Can it state the goal, the boundaries, and the proof without asking you to re-explain? If it invents scope, the loop is under-specified.

Common failure mode: listing "tests pass" as the evidence without saying which test, which input, and what it does not cover. Recovery: name the exact check per row of the matrix above.

The weekly 15-minute review: keep the loop alive

Evidence, routing notes, and roadmaps decay after week one unless something refreshes them. Once a week — fifteen minutes, same day — run this ritual:

1. Debt (5 min): open PROJECT-SHAPE.md. Anything new? Promote one candidate cleanup to a REFACTOR-CARD.md, retire anything already fixed. 2. Routing (3 min): open MODEL-NOTES.md. Which model won which job this week? Update the project model map; drop anything that needed major correction twice. 3. Roadmap (4 min): open ROADMAP.md. Check off validated phases with their evidence, unblock or re-scope the next one, shrink anything that grew. 4. Evidence folder (3 min): every EVIDENCE-RECORD.md and DONE.md has its artifact attached (test output, screenshot, log)? Anything cited but missing gets re-attached or marked unknown.

Save the result as a dated line in the roadmap (2026-09-12 review: …). That line is how the next AI session knows what is settled and what moved.

Finish line: four files touched, one dated review line, zero "I think we decided…" mysteries left for Monday.

Check your understanding

1. Recite the six steps of the evidence loop without looking. Which step must happen *before* the AI changes anything? 2. Give one example of a green check that is the wrong check, and name the missing evidence. 3. For an API integration at higher stakes, what four behaviors go beyond docs-plus-request? 4. What five things should an agent report — and why is an honest "cannot test" valuable? 5. Trace the six-class synthesis in your own words: what does each stage add to the one before?

ARTICLE DISCUSSION

JOIN THE
CONVERSATION.

0 COMMENTS

BYEBUY ACCOUNT ACCESS

Sign in

Use your account to save routes and make the catalogue yours.

Enter your email and we’ll send a secure sign-in link and code.

NEW ROUTES ADDED WEEKLY · 9,235 CATALOGUE ENTRIES · BUILD · DEPLOY · QUERY · STACK · SAY BYE TO BUY · NEW ROUTES ADDED WEEKLY · 9,235 CATALOGUE ENTRIES · BUILD · DEPLOY · QUERY · STACK · SAY BYE TO BUY ·