You know the families. You can research a release on Hugging Face. Now comes the decision: which model becomes your default for a real task?

Do not decide by ranking, price, or excitement. Decide by running the same test on every candidate and writing down why the winner won. That record is what lets you defend the choice next month when a new release appears.

By the end of this lesson, you can run a fair three-model comparison — one task, one context package, one rubric — and record the result in a comparison table.

The fair comparison workflow

The method is deliberately boring. Boring is what makes it fair.

1. One task. Pick a single bounded job from your real work. Not "help me code." Something like: "Refactor this 80-line checkout component to extract the pricing logic, keep all existing behavior, and pass the existing tests." If the task is vague, the results are incomparable.

2. One fixed context package. Every model gets the same inputs: the same files, the same instructions, the same constraints. Save them as a small folder or prompt file so you can re-run the test later. This is the context discipline from Class 09 applied to evaluation — the smallest useful file set, identical every time.

3. One evaluation rubric, written before you run anything. Decide in advance what good looks like:

4. Separate generation from review. Run all candidates first, then review all outputs together. Do not grade model A, adjust the prompt, then grade model B on the improved prompt. Same test, then judgment.

Fix task + context + rubric
  → run model A → save output
  → run model B → save output
  → run model C → save output
  → review all three against the rubric
  → record the default and why

Where the three signals fit

You have three kinds of evidence. Use each for what it is good at.

• Arena (from Class 08): a useful external signal for building your short list. Filter by the task domain, check votes and price, and pick candidates. It is not a verdict on your task.
• Hugging Face (from Lesson 11.2): the facts about open-weight candidates — license, size, context, hardware, intended use. It tells you what you are allowed and able to run.
• OpenRouter + OpenCode (from Class 10): your optional real-world test bench. Same repository, same prompt, same acceptance criteria, routed to different models. Log which model and route produced each output so the comparison is reproducible.

No single signal decides. The short list comes from Arena and Hugging Face; the decision comes from your test.

The scorecard

Score each model 1–3 per dimension (3 is best), then add a one-line evidence note. The numbers force a comparison; the notes make it honest.

Below the table, write the decision record:

## Decision
- Default for [task]: [model + provider route]
- Runner-up and when to use it: [model + situation]
- Rejected because: [one sentence with evidence]
- Re-test when: [new release, price change, or task change]
- Date and context package version: [date + folder or commit]

That last line matters. When Qwen, DeepSeek, or any other lab ships a new version, you do not start over. You re-run the same package and update the record.

Worked shape of a comparison

Imagine your task is the checkout refactor above. Your candidates: a Qwen coding model, a DeepSeek reasoning model, and your current American default — all reached through OpenRouter routes you logged.

You run all three on the identical file set. Qwen finishes fastest and cheapest but needs ten minutes of cleanup. DeepSeek is slower, costs slightly more, and passes tests untouched. The default is excellent but costs five times more per run. Your rubric says correction time matters more than raw speed for this task, so DeepSeek becomes the default for refactors while the cheaper model handles routine edits.

The details will differ for your task. The shape stays the same: same test, scored dimensions, recorded reason.

Exercise: build a comparison table for three models

Choose one real task and three models. Include at least one Chinese-family model from Lessons 11.1–11.2.

1. Write the task in one sentence with acceptance criteria ("passes existing tests," "under 200 words," "valid JSON"). 2. Freeze the context package: list the exact files and instructions every model will receive. 3. Write the rubric before running anything — at minimum quality, correction time, speed, and cost. 4. Run (or, if budget is tight, plan in full detail) the comparison: model names, provider routes, and where outputs will be saved. 5. Fill in the scorecard and write the five-line decision record above.

Finish line: a comparison table plus decision record you could show a teammate. The default model is named with its provider route, and the reason cites your evidence.

Verify: could someone else re-run your test from the context package and rubric alone? If not, the package is not fixed enough yet.

Common failure mode: changing the prompt between runs because "this model needs different instructions." That may be true in production, but during evaluation it destroys comparability. Note prompt sensitivity in the reliability row; do not silently fix it mid-test.

Check your understanding

1. What are the four parts of a fair comparison, and why must the rubric come first? 2. Why should you review all outputs together instead of grading each run immediately? 3. What role does each signal play: Arena, Hugging Face, OpenRouter/OpenCode? 4. What goes in a decision record, and why does it include a re-test condition? 5. Why is cost per run only one of six dimensions rather than the whole decision?

You can now evaluate Chinese model families the way you evaluate any models: capability against your task, economics per useful result, licensing you have actually read, and hosting through a provider you chose deliberately. That is the outcome of Class 11.

Next, Class 12 turns the cost row of your scorecard into a full operating skill: estimating API spend, comparing cost per useful result instead of sticker price, and putting guardrails around agent spending before the bill surprises you.