Queues (26.3) promised that a crashed message returns to the line and runs again. That sounds reassuring until the retry itself causes the damage.

The double-notify failure

A worker calls a slow payment-status API. The API takes too long, the worker hits its timeout and gives up — but the API actually completed the charge a second later. The message retries. The worker, having recorded nothing about the first attempt, charges again and sends the customer two "payment confirmed" emails. The job *ran* twice. It *worked* zero times — it produced a wrong the customer can see.

Every term in this lesson lives inside that story:

• Retry: running the job again after failure. Necessary and dangerous.
• Timeout: how long the worker waits before declaring "this attempt failed." Too short causes false failures; too long blocks the line.
• Duplicate: the same effect recorded twice — two rows, two emails, two charges.
• Dead-letter (failure queue): where a message goes after exhausting its retries — parked visibly for inspection, not deleted, not retried forever.
• Idempotency: designing a job so repeating it does not accidentally duplicate the result. Same job, same effect, no matter how many times it runs.

The plain-language version: *running* means the code executed; *working* means the intended real-world effect happened exactly once and is recorded.

The core pattern: four moves

Safe background work follows four moves, in order:

1. Stable ID. Give each job an identifier derived from its intent, not from the attempt: charge-order-9917, fred-UNRATE-2026-06, report-acme-2026-09-12. Attempt 1 and attempt 4 share the ID; the system can recognize "I have seen you before." 2. Effect recorded. Before (or atomically with) performing the outside effect, record it: "notification N sent," "row R written," "charge C confirmed." The record is the memory the timed-out worker lacked. 3. Safe retry. On retry, check the record first: if the effect already happened, skip it and report success. Retry with bounded count and growing delays (backoff: e.g., 3 retries at 1, 5, 15 minutes) — never infinite, never hammering. 4. Human surface. After retries are exhausted, park the job in the dead-letter queue and show it to a person: what failed, what was already done, and what decision is needed. Silent endless retry is how one bug becomes a thousand duplicate emails overnight.

Unsafe vs. safe: daily market ingestion

The same daily job, two designs. Source: FRED series UNRATE. Goal: one row per month.

Unsafe version: the worker fetches the value and appends a row every run. First run writes June's row. The API hiccups, the message retries, the worker appends June again. Two rows, charts double-count, nobody knows which is real. Every retry is a coin flip on data corruption.

Safe version: the row's identity is (source, series, obs_date) — fred/UNRATE/2026-06. The worker checks for that key first: exists → update the one intended row (or skip if unchanged); missing → insert exactly one row. Five retries produce one row. The schedule from 26.2 can fire twice, the queue from 26.3 can deliver twice, and the data stays correct. That is idempotency in one sentence: *key the effect by intent, then upsert instead of append.*

retry arrives (same stable ID)
  → look up effect record by ID
  → already done? report success, do nothing
  → not done? perform once, record, report success
  → retries exhausted? park in dead-letter + alert human

Observability: how you see repeats behaving

Safe retries are observable retries. Three layers:

• Logs tell what happened on one run: start, attempt number, stable ID, outcome, duration. One line per attempt, always including the ID.
• Metrics show patterns over time: job success rate, retry rate, queue depth, dead-letter count. A retry rate jumping from 1% to 20% is the early warning.
• Alerts tell a person when normal failure becomes a real problem: dead-letter non-empty for 30 minutes, same job failing five times, freshness older than the schedule allows. Alerts go to someone with the power to pause the job — which leads to the boundary.

The boundary: retries are a product decision

Do not silently retry actions that could charge money, publish publicly, alter production data broadly, or make a market or trading decision. A retry policy for "resize this thumbnail" can be generous; a retry policy for "execute this trade" needs a human in the loop and, frankly, a lawyer-shaped conversation before any automation exists.

Concretely: charges, public posts, bulk deletes, outbound emails to customers, and anything resembling an order get *narrow* retry budgets, mandatory duplicate checks, and fast escalation to a person. "The queue retried it" is never an acceptable explanation for a double charge. Choosing what retries, how often, and when a human decides — that is product design, not plumbing.

Exercise: write JOB-SAFETY.md

For one background job in your project, fill this card:

# JOB-SAFETY.md — <job name>
- Stable job ID: <how the ID is built, e.g. source/date/series or order-id>
- Success condition: <the exact test — row exists? email accepted? API confirmed?>
- Retries + backoff: <e.g. 3 retries at 1m, 5m, 15m, then dead-letter>
- Duplicate protection: <effect record key + check-before-act rule>
- Human-review condition: <e.g. dead-letter non-empty, 3rd failure, any charge involved>
- Proof log line: <one example line with timestamp, job ID, attempt, outcome>

Example proof line to imitate:

2026-09-12T07:01:22-04:00 job=fred/UNRATE/2026-06 attempt=2 outcome=success rows_upserted=1 note=duplicate-delivery-skipped

Finish line: a committed JOB-SAFETY.md with all six fields and a realistic proof line.

Verification: replay the double-notify story against your card — the first attempt times out after the effect happened, the retry arrives. Walk the card's rules and confirm the retry reports success without repeating the effect. Then confirm the dead-letter path: after the last allowed retry, where does the job sit and who gets told?

Common failure mode: a card whose "success condition" is "the script finished." Finishing is running. Success is the recorded real-world effect — the row, the receipt, the confirmed delivery.

Check your understanding

• Why does a timeout followed by a blind retry produce duplicates even when every component "works"?
• What makes fred/UNRATE/2026-06 a better job ID than attempt-uuid-8f3a?
• Name one action in your project that must never silently retry, and what should happen instead.

Next

Safe jobs survive repeats. But who notices when a job stops running entirely? Lesson 26.5 builds the operating view — logs, errors, uptime, metrics, and backups — so silence itself becomes a signal.