September 12, 2026
LOGS, ERRORS, UPTIME, AND BACKUPS — HOW YOU KNOW IT IS WORKING

Deployment feels like the finish line. The site loads on your machine, the deploy log says success, you close the laptop. Three days later a user writes: "your data is a week old." Deployment was the *beginning*. This lesson is the rest: how you know the product is healthy while you sleep.
Loaded once is not healthy
A healthy product does four things continuously: it serves requests, it runs its important jobs, it reads and writes data correctly, and it surfaces failures before users become the monitoring system. "The site loaded once" proves exactly one of those, for one moment.
Learn the four signals by name:
- Logs are timestamped records of what the app or worker says happened — one line per meaningful event, always with the job or request ID. They answer "what exactly occurred?"
- Error tracking groups failures: where they happened, how often, and which release introduced them. Sentry is one current example — it collects exceptions from your app with stack traces and release tags. It is an example, not a mandate; any grouped error reporting counts.
- Uptime check is an outside request that periodically verifies a public URL or critical endpoint is reachable. It answers "can anyone reach us?"
- Metrics are measurements over time: slow requests, queue depth, job failures, storage growth, spend. They answer "are we drifting toward trouble?"
No single signal suffices. You need all four the way a car needs a speedometer, a fuel gauge, warning lights, and mirrors.
The story uptime alone misses
The public research page loads fine. The uptime check is green, week after week. But the nightly refresh worker — the cron job from 26.2 feeding the queue from 26.3 — has been failing for three days: the source changed its response format, every run lands in the dead-letter queue from 26.4, and nobody set an alert on it.
Result: uptime green, data stale. Every visitor reads last week's numbers presented as today's. The fix was never "more uptime checks." It was two missing signals: job-success (did last night's refresh write fresh rows?) and data-freshness (how old is the newest observation the page displays?). A healthy system checks the *outputs users rely on*, not just the front door.
Put a freshness line on every data page — "FRED UNRATE as of June 2026, refreshed Sep 12" — and you turn every visitor into a backup freshness monitor for free.
The minimum ops board
A first product needs one small operations board — a page, a doc, or a dashboard — with these rows:
| Signal | What it shows | Healthy looks like |
|---|---|---|
| Latest deployment | Version + time + what changed | Known version, deployed deliberately |
| Last successful scheduled job | Each cron job's latest job_runs success + timestamp | Every job succeeded within its schedule window |
| Error count | Grouped errors since last deploy | Near zero new errors; known ones tracked |
| Public health check | Uptime probe on the homepage + one critical endpoint | Green, with latency recorded |
| Queue backlog | Unprocessed + dead-letter depth | Near zero backlog; dead-letter empty |
| Database / storage growth | Size trend + % of plan limit | Steady growth, headroom, retention enforced |
| Cost alert | Spend vs. budget threshold | No surprise bill; alert fires before overrun |
Seven rows. Check them on a rhythm — daily for a live product, weekly for a prototype. The board's job is to make "worker dead for three days" impossible to miss: the scheduled-job row goes red on day one.
Backup vs. restore: the hopeful file
A backup is a recoverable copy. A restore proves the copy can recover something useful. An untested backup is only a hopeful file — it might be empty, encrypted with a lost key, or three schema versions behind. Schedule restores (even partial ones to a scratch environment) the way you schedule the jobs themselves.
What must be recoverable, and from where:
- Project code → from GitHub or GitLab. The repository is the source of truth; a deployed server is never the only copy. (Class 25's rule.)
- Database records → from provider backups and periodic exports. Know the retention window and who can trigger a restore.
- Uploaded files → from object-storage retention and versioning. Know the bucket policy, the retention period, and the deletion path from your
FILE-LIFECYCLE.mdthinking (25.5). - Secrets → through controlled account ownership and the deployment platform's environment settings — never a copy in the repo, never a screenshot in chat. Losing an API key must mean rotating it, not excavating it from git history.
Write the owner next to each row: a named person or account, not "someone."
The incident rhythm
When the board goes red, follow one rhythm — detect, inspect, contain, recover, document:
1. Detect: the alert or the freshness line says something is wrong. Note the time. 2. Inspect logs: find the failing job ID, the first failing attempt, and the error. Logs before guesses. 3. Contain harmful automation: pause the cron schedule or the queue consumer before it writes more bad data or sends more wrong emails. Stopping the bleeding outranks understanding the wound. 4. Restore or replay safe work: restore from backup or replay the idempotent jobs from 26.4 — safe *because* they are idempotent. 5. Document the resolution: one paragraph — what broke, what fixed it, what signal will catch it next time. Future-you (or a fresh AI session reading the runbook from 24.3) will be grateful.
Exercise: write HEALTH-CHECK.md
Create HEALTH-CHECK.md with one row per item below. For one row (at minimum), paste the exact evidence that proves health today — a timestamp, a log line, a dashboard reading.
# HEALTH-CHECK.md — <product>
- Public URL: <url + uptime probe location + last green time>
- Latest deployment: <version/commit + date + where to see it>
- Latest scheduled job: <job name + last success timestamp + output row>
- Critical worker/queue: <queue name + backlog depth + dead-letter depth today>
- Database backup owner: <provider + retention + owner + last restore test date>
- Object-storage recovery owner: <bucket + versioning/retention + owner>
- Error destination: <where errors group + count since last deploy>
- Alert recipient: <who gets woken + through which channel>
Finish line: a HEALTH-CHECK.md with every row filled and verification evidence pasted for at least one row (e.g., "queue depth 0 at 2026-09-12 09:00 ET, dead-letter 0, screenshot/link").
Verification: pick the scheduled-job row and answer: if the job failed silently tonight, which row turns red first, and who is told by morning? If the answer is "nobody until a user complains," add the missing alert before calling it done.
Common failure mode: a board where every row says "check dashboard" with no links, names, or timestamps. A health check without evidence is a wish. Paste the proof.
Check your understanding
- Why can uptime be green while the product is broken? Name the two signals that would have caught the stale-data story.
- What is the difference between a backup and a restore, and why does it matter?
- Your queue consumer starts writing corrupt rows at 3 AM. What is the first correct action — fix the bug, or pause the consumer? Why?
Next
You can now run work in the background (26.1), on a clock (26.2), through a line (26.3), safely repeated (26.4), and visibly healthy (26.5). Class 27 adds the last trigger type: webhooks — work that starts because something *happened* elsewhere, a doorbell rather than an alarm clock.
ARTICLE DISCUSSION
JOIN THE
CONVERSATION.
Got a question, a take, or a better way to do this? Log in and leave a comment.
