antist.ai
all scenarios

Drawing set · L0–L3

run a data system

A data system is a machine that keeps doing something while nobody watches: fetching sources on a schedule, cleaning what comes back, storing it and publishing conclusions. Its failures are quiet — a source changes its format, a job silently skips, and the page keeps showing yesterday's numbers as if they were today's. This drawing set maps the shapes such a system can take, using this site's own world-intelligence radar as the worked example.

Design philosophy

A data system is a promise that its numbers are true and current. Showing nothing is honest; showing stale data as fresh is a lie told automatically.

  1. Silence is the enemy

    The dangerous failure is not a crash but a job that skips, returns empty, and reports success. Treat an empty result as an alarm until proven otherwise.

  2. Keep the raw material

    Store what the source actually returned before you transform it. Logic can be fixed and rerun; data you did not keep is gone.

  3. Make claims checkable

    Every conclusion should point to its sources and its collection time, and predictions should be settled in public. Trust comes from being checkable, not from being confident.

Collecting for yourself? A scheduled script writing to files is enough. Move to a hosted schedule and a database when other people rely on the output, and to queues only when one run no longer fits in its time limit.

SheetL0

Personal collector

Used byOnly you
Per month$0

①Why move up

You keep checking the same sources by hand and want a machine to do it. The only reader is you, so a missed run costs nothing but a gap.

②Architecture

A script runs on a timer on your own machine, fetches a few sources, and writes one dated file per run. The raw responses are saved next to the cleaned results.

Δ New at this level

  • + A scheduled script
  • + One dated file per run
  • + Raw responses kept beside results

Shapes on the map: Scheduled script → files

③Tools

Use whatever language has the best parser for your sources. Prefer official feeds and APIs over scraping pages — they break less and their terms are clearer.

④Code and tests

  • Make every run write a line saying how many items it got; zero is a warning, not a success.
  • Keep a few saved raw responses and test the parser against them after every change.
SheetL1

Public feed

Used byReaders who rely on the output
Per month$0–5

①Why move up

Other people read your page or subscribe to your digest. Your laptop being closed can no longer mean missing data, and a wrong number now misleads someone else.

②Architecture

The schedule moves to a hosted platform that runs the collector every hour, writes to a database and serves a public page. Every item is validated against an expected shape before it is stored, and every run records its own health.

Δ New at this level

  • + A hosted schedule
  • + A database instead of files
  • + Validation at the point of entry
  • + A health record for every run

Shapes on the map: Hosted cron + database + site (radar's shape)

③Tools

Choose a platform where the schedule, the database and the website live together; fewer moving parts means fewer silent failures. Check the platform's per-run time and CPU limits before you design the job.

Also worth knowing — not verified by this site

④Code and tests

  • Validate every incoming item against a schema; reject and log what does not fit instead of storing it.
  • Show the collection time next to every number on the public page.
  • Alert when a source returns zero items or a run does not happen at all.
SheetL2

Money and trust

Used byPeople who act on your conclusions
Per month$5–25

①Why move up

Readers make decisions from your output, subscribers may pay, and some sources carry terms of use or personal information. A wrong number or a leaked field now has consequences.

②Architecture

Conclusions go into an append-only ledger with their sources and dates, and forecasts are settled by code when they come due. Raw responses are archived in cheap storage, sources' terms are recorded per source, and personal data is filtered before anything is stored.

Δ New at this level

  • + An append-only ledger of claims
  • + Automatic settlement of forecasts
  • + A raw archive in object storage
  • + Source terms and a personal-data filter

Shapes on the map: Hosted cron + database + site (radar's shape)

③Tools

Keep judgments and bookkeeping separate: a model may write the analysis, but settling whether a forecast came true must be plain code that anyone can rerun.

④Code and tests

  • Never edit a ledger entry; corrections are new entries that point to the old one.
  • Settle forecasts with deterministic code, not with a model.
  • Run a poisoning test: feed items containing keys or personal data and confirm none reach the database.
SheetL3

Scale

Used byMany sources, many consumers
Per month$50–500+

①Why move up

Hundreds of sources no longer fit in one run's time limit, some fail every hour, and people want to ask questions across years of history.

②Architecture

Collection is split into small jobs on a queue, each retried on its own and safe to run twice. The raw archive becomes the source of truth, and an analytics layer is rebuilt from it rather than edited by hand.

Δ New at this level

  • + A queue of small idempotent jobs
  • + Per-source retries with backoff
  • + The raw archive as source of truth
  • + A rebuildable analytics layer

Shapes on the map: Queue-based pipeline · Warehouse + analytics layer

③Tools

Add the queue before the warehouse. Most data systems hit the time limit of a single run long before they have questions a single database cannot answer.

Also worth knowing — not verified by this site

④Code and tests

  • Make every job idempotent so a retry can never create a duplicate.
  • Be able to rebuild any derived table from the raw archive with one command, and practice it.
  • Track freshness per source and page someone when a source goes stale.

The architecture map

Every shape this can take. Most exist so you know what not to build yet.

Scheduled script → files

Personal collection; one machine; you read the output yourself

$0 · Minimal

△ Someone else depends on the output, or your machine being off means missing data

Hosted cron + database + site (radar's shape)

A public page or feed others rely on; runs of a minute or less

$0–5/mo · Low–mid

△ One run no longer fits in the platform's time or CPU limit, or sources must be fetched in parallel

Queue-based pipeline

Many sources, retries, steps that must survive partial failure

$5–50/mo · Mid–high

△ Analysts need to ask questions across years of history, not just read the latest page

Warehouse + analytics layer

Large history, many consumers, questions nobody planned for

$50–500+/mo · High

△ This is the top floor. If one database still answers your questions, stay below

Concepts you'll actually need

One plain sentence and one metaphor each. Click any tool module for the same treatment.

Idempotency
Running the same job twice gives the same result as running it once, so retries never create duplicates.
≈ Pressing the lift button five times still calls one lift.
Freshness
Every number shown should carry when it was collected, and the system should say so loudly when data is stale.
≈ A best-before date on every carton, not just on the shop door.
Schema drift
A source quietly changes its format; if the collector does not check, it keeps saving empty or wrong fields.
≈ A supplier who changes the box size without telling you — your shelves still fill, with the wrong thing.
Ledger
An append-only record of claims and their outcomes, so conclusions can be checked against what actually happened.
≈ A weather station that writes every forecast on the wall and marks each one right or wrong.