September 12, 2026
BUILD YOUR FIRST DATA SOURCE MAP

You can now tell a dataset from an API (19.1), score a single source on five questions (19.2), and see past "free" (19.3). Real products break the single-source habit immediately. Sonariq — our running financial-research example — needs company filings *and* macro context *and* company metadata *and* the user's research criteria *and* its own saved notes. Five needs, five different shapes, five different owners.
The artifact that holds them together is the data source map: one table where every product question points at an evaluated source, with its owner, boundary, and fallback written down. It is the bridge from choosing sources to storing them — Class 20's database work.
The map columns
One row per product question. Nine columns, no blanks:
| Column | What to write | Why it matters |
|---|---|---|
| Product question | The user-facing question in plain words | Starts from need, not vendor |
| Data category | Filings / metadata / macro / criteria / notes (Class 18's categories) | Shows the shape of the answer |
| Candidate source | Exact dataset + provider | Prevents "some API" vagueness |
| Original owner | Who vouches for the fact | Provenance survives handoffs |
| Access path | API / MCP / file / export + docs URL | Builder and agent fetch identically |
| Freshness | Update schedule + latency + what "current" means here | Kills stale-data bugs early |
| Storage decision | Cache / store copy / reference live (preview of Class 20) | Decides what the database must remember |
| Credential location | Key name / env var / none (never the secret itself) | Secrets stay out of the map |
| Fallback | What happens when this source is missing, late, or changed | Every row has a plan B |
The map is also agent context. An AI coding agent with this file will fetch from the named provider through the named path. An agent without it will invent a provider, grab the nearest unofficial endpoint, or silently swap sources mid-task. The map is a guardrail disguised as documentation.
Sonariq's map (worked example)
| Product question | Category | Candidate source | Owner | Access / docs | Freshness | Storage | Credential | Fallback |
|---|---|---|---|---|---|---|---|---|
| What did the company disclose? | Filings | SEC EDGAR filing facts (JSON) | SEC | API via server/CLI | Filed date + retrieval time kept | Store filing + evidence items | None (public, fair-access) | Show cached filing with date; block new claims |
| Which company is this? | Metadata | EDGAR company tickers/concepts | SEC | API / bulk files | Slow-changing; refresh weekly | Store company record | None | Manual ticker confirm; no auto-merge |
| What is the economic backdrop? | Macro | FRED series (e.g., unemployment, rates) | Fed St. Louis | API key | Series release schedule + vintage | Store observations + series ID + units | FRED_API_KEY env | Hide macro panel with "as of" date; never invent |
| What does the user care about? | Criteria | User-provided watchlist + thresholds | User | App form / upload | User-edited | Store as app-owned records | Session auth | Keep last saved criteria; mark stale |
| What did we conclude? | Notes | App-owned research notes (Markdown) | Your app | Internal store | Written per research run | Store reports + evidence links | App auth | Read-only cached report; no regeneration |
Two details worth copying. First, storage already anticipates Class 20: filings and observations are stored with provenance; slow metadata is cached; user criteria and notes are app-owned truth. Second, fallbacks degrade visibly — "show cached with date," "hide panel," "block new claims" — never silent invention. A map row without a fallback is a promise the source cannot keep.
Copy-ready DATA-SOURCES.md — 3-row minimum (Sonariq-style, no blanks):
# DATA-SOURCES.md — Sonariq prototype
# Terms/docs last checked: 2026-09-11 — re-check at build time
| Product question | Category | Candidate source | Owner | Access / docs | Freshness | Storage | Credential | Fallback |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| What did the company disclose? | Filings | SEC EDGAR companyfacts JSON (`data.sec.gov/api/xbrl/companyfacts/CIKxxxx.json`) | SEC — https://www.sec.gov/search-filings/edgar-application-programming-interfaces | API via server/CLI (no browser CORS), User-Agent required, ≤10 req/sec fair-access | Filed date + retrieved_at stored per fact | Store filing + evidence items with CIK, form, filed date | None (public, fair-access) | Show cached filing with date; block new claims if missing |
| What is the economic backdrop? | Macro | FRED series UNRATE + CPIAUCSL via https://fred.stlouisfed.org/docs/api/fred/ | Federal Reserve Bank of St. Louis | API key, `series_id`, `units`, `seasonal_adjustment` kept | Monthly release (UNRATE first Friday); keep vintage + retrieved_at | Store observations + series ID + units + vintage | `FRED_API_KEY` env (never secret in repo) | Hide macro panel with "as of DATE"; never invent value |
| What does the user care about? | Criteria | User watchlist + thresholds (app form) | User | App form / upload — internal | User-edited; mark `updated_at` per edit | Store as app-owned records | Session auth | Keep last saved criteria; mark stale if >30 days |
Candidates often start as discoveries in the ByeBuy Data directory — including macro context from the World Bank Indicators API or live weather-style time series via Open-Meteo for non-financial products. Discovery there; evaluation on source cards (19.2); commitment here.
Practical exercise
Create DATA-SOURCES.md for a real project — Sonariq, a trip planner, a price-watch tool, or your own idea — with at least three rows.
1. Write each row's product question first, then fill every column. No blanks; "unknown — must check" is allowed once per row at most. 2. Each row must name the source owner, the exact use (which question it answers), the access boundary (key? fair-access? license limit?), and the fallback. 3. Refresh the terms/docs links at build time and date the check.
Finish line: a committed DATA-SOURCES.md with 3+ complete rows, each naming owner + use + boundary + fallback.
Verify: for each row, ask: (a) could an agent fetch this from the access path alone? (b) if the source went down today, does the fallback keep the app honest? Two "yes" answers per row or the row is not done.
Common failure mode: three rows pointing at the same convenient wrapper ("we will use X for everything"). Different questions usually need different owners. If every row names the same provider, re-ask question 1 of 19.2: who is actually close to each fact?
Check your understanding
1. Why does the map start from product questions rather than providers? 2. An agent suggests a faster unofficial feed not on the map. What do you do — and which map column settles it? 3. Why does the credential column name the *location* of the secret, never the secret?
Transition: into storage
Class 19's outcome is now yours: you can find candidates, name what you found, judge each one with evidence, see past price, and map several needs into one documented layer.
That layer needs a home. A source map says what enters the product; a database preserves what the product must remember — with structure that survives growth. Class 20 turns the map's "storage decision" column into real tables.
ARTICLE DISCUSSION
JOIN THE
CONVERSATION.
Got a question, a take, or a better way to do this? Log in and leave a comment.
