September 12, 2026
AI CAN WRITE TESTS; YOU STILL DECIDE WHAT MUST BE PROVEN

You now know the portfolio: fast rule checks, seam-crossing integration tests, browser journeys, human exploration, production smoke. Writing all of that by hand is slow — which is exactly why AI help is welcome here. But speed without judgment produces green checkmarks that prove nothing. This lesson is about staying the judge.
The productive pattern: plan, then implement, then fail-before-fix
Give the AI everything it needs and forbid it from rushing. The input bundle is specific: the existing spec, the task card with acceptance checks, the route contract, the bug report if there is one, and the relevant files — not the whole repo.
Then demand this order:
1. Propose the minimum test plan first. "Before writing any code, propose the smallest set of tests for this card: each as Given/When/Then, with type, fixtures, and what it does not prove." Minimum matters. An unbounded AI writes forty brittle tests; a bounded one writes the five that match the risk. 2. Implement the tests, then explain how each would fail before the fix. This is the fail-before-fix question: "for each test, describe the bug that makes it go red." A test with no answer is decoration. For a bug fix, the test should actually run red against the current code before the repair goes in — the reproduction Lesson 52.1 insists on. 3. Report raw results, not confidence. "Show the command you ran and paste the output" beats "all tests pass, the feature is solid." You review evidence; the GitHub pull-request review flow exists precisely so a human checks the diff, the checks, and the claims before merging.
A prompt with this shape works across Research Desk slices:
Given SPEC.md, TASK-CARD.md, and API-CONTRACT.md for watchlist save:
1. Propose the minimum test plan (Given/When/Then + type + fixtures).
2. Wait for my approval.
3. Then implement the tests, run them, and paste raw results.
4. For each test, state what bug would make it fail and what it does not prove.
Do not change production code until the plan is approved.
Independent review: don't let the AI grade its own homework
Here is the danger, stated plainly: when the same AI writes the production code and the tests, the test can encode the same wrong assumption as the implementation. Both agree; both are wrong; all checks pass.
The textbook failure: an AI adds a watchlist test that replaces the authorization helper with a mock returning true — "to keep the test focused." The suite is green. Any user can edit anyone's watchlist, and the suite will stay green forever, because the test disabled the very rule it claimed to check. The fix is a route-plus-policy integration test with two real identities and no mocked authorization — exactly the Journey 2 from Lesson 50.3.
So require independent review on test changes, proportional to risk:
- A second model or a human reads the test diff alongside the feature diff, especially when author and test-writer are the same agent.
- The reviewer asks: does any mock neutralize the rule under test? Is there a two-identity case? Do the assertions check what was written and who owns it, or just the status code?
- Route the review through the normal pull-request gate — the reviewer checks files, runs, and screenshots, not the author's summary.
One model investigates, another implements, a third — or you — reviews. That separation is cheap. A permission hole in production is not.
Gates: run what the risk earns
Not every change needs every gate. Six gates, chosen proportionally to the Part X risk tier:
| Gate | What it catches | When to require it |
|---|---|---|
| Lint + type check | Syntax, formatting, type errors, dead imports | Every change; fast and automatic. |
| Unit tests | Broken small rules (Lesson 50.2) | Every logic change. |
| Integration tests | Seam failures: route plus auth plus policy (Lesson 50.3) | Any route, permission, or schema change. |
| Browser tests | Broken user journeys in a real browser (Playwright) | User-visible flows and auth boundaries. |
| Preview + manual check | Visual, copy, keyboard, mobile, surprise | Anything a person sees; attach a screenshot. |
| Production smoke | Deployed reality: URL live, sign-in works, critical write succeeds | After deploy, always (tiny and fast). |
A copy change earns lint plus a preview glance. A watchlist ownership change earns all six. The task card should name its gates in advance, so "we skipped the browser test" is a visible decision with an owner — not something discovered after the incident.
The "prove it" prompt
Keep one reusable prompt for the end of any AI-built change. It forces the worker to surface untested assumptions instead of reassuring you:
Prove it. List every assumption this change relies on that no test checks.
For the riskiest untested assumption, generate the right test, run it,
and report: command run, raw output, what the test proves,
and what remains untested. Do not summarize with confidence statements.
Good answers sound like: "Assumption: the test database enforces the same row policy as production — unverified. I added a two-identity denial test; command npm test watchlist.policy output 3 passed; it proves the route denies User B; it does not prove the production policy matches; manual check needed on preview." That paragraph is worth more than a hundred auto-generated assertions.
Exercise: ship a PR that proves itself
Take your watchlist slice branch and write the pull-request description the course requires. It must link four things:
1. Changed requirement — spec section and task card ID. 2. Tests run — which gates ran, with commands and raw results (paste the output, not "all green"). 3. Manual / preview evidence — preview URL, screenshot of the saved confirmation and the denied state, keyboard and mobile notes. 4. Remaining risk — what is still untested and who owns it ("live provider quota behavior untested; owner: you; revisit when paid refresh ships").
Finish line: a PR description with all four links, reviewed independently — comments from the reviewer visible, per the GitHub review flow — before merge.
Verify fast: pick one claim in the description and re-run it yourself: the two-identity denial test, the preview URL after sign-out. Common failure: links point to the author's chat summary instead of the spec, the test output, and the preview. Relink to the artifacts.
Check your understanding
1. Why must the AI propose the test plan before implementing? 2. What does "fail before fix" mean, and what does it catch? 3. Describe the mocked-auth-true failure and its two-identity fix. 4. Which gates does a permission change earn versus a copy change?
Next: the part comes together
Testing was the last piece of the trust stack: frontend and backend boundaries, architecture, auth, security review — and now proof. Class 51 turns to what happens when proof fails in the wild: logs, errors, and observability, the trail that turns "save is broken" into a diagnosable event. And the Part XI capstone will demand all of it at once — spec, contract, threat model, test plan, and evidence — for one slice you can actually deploy.
ARTICLE DISCUSSION
JOIN THE
CONVERSATION.
Got a question, a take, or a better way to do this? Log in and leave a comment.
