# The as-issued archive

**Since:** migration 212 (written 2026-10-28). **Writer:** `lib/archive/as-issued.ts`. **Tables:** `as_issued_polls`, `as_issued_products` — append-only, service-role only.

## What it is

Every federal product the live workers read, kept as it was received: one row per distinct body, with the time the product says it was issued, the time the database received it, and a SHA-256 of the canonical body. A second table records every poll, including the quiet ones, so "nothing new at 14:05Z" is a fact in the record rather than an absence.

A product is one item of a feed, not the feed body: an alert version, an earthquake revision, a storm's entry in `CurrentStorms.json`, a best-track file revision, an outlook category polygon, a fire perimeter. Items are hashed one by one, so a poll that saw 480 alerts of which 3 changed stores 3 rows.

## Why it exists

Until 2026-10-28 the platform kept nothing as received. Each worker parsed its feed into a curated `events.raw_data` and overwrote it on the next poll. The record of what NWS, USGS, NHC, SPC and NIFC said at the moment they said it existed only upstream, and upstream revises, retires and replaces. The one exception was the trigger evaluation snapshot (migration 067, `lib/ils/source-snapshot.ts`), which captures evidence only for the products a trigger happened to look at.

The archive is the layer under that snapshot. It is the raw material for the warning performance ledger (product at issue vs federal post-event measurement), the event-to-outcome dataset, and the live half of the loss track record. Its value is elapsed time: nobody who starts later can reconstruct what Calchis received and when, and no later feed read can replace a body that upstream has since revised.

## What it is not

- **Not unique as a capture.** NWS text products are archived publicly elsewhere (the Iowa Environmental Mesonet keeps years of them). The asset is the joined, scored, exposure-attached record built on this archive, not the capture by itself. Say so in any data room.
- **Not a backup of `events`.** The curated rows are derived; the archive is the source they were derived from.
- **Not yet readable by anyone but the service role.** A public, verifiable read path (a product by its hash; a poll ledger by day) is a later migration, once there is a record worth reading.

## Sources hooked (2026-10-28)

| `source` | Module and hook | Product | `product_id` | `issued_at` |
|---|---|---|---|---|
| `nws_alerts` | `nws-alert-pages.ts` — covers all six `/alerts/active` feeds (storm, tropical, heat/wind, fire weather, flood from 2026-10-28; winter from 2026-11-07 — until then the winter module read one page on its own and nothing it received was archived, which this table did not say) | alert version | alert `id` | `properties.sent` |
| `usgs_earthquakes` | `usgs.ts`, before the M2.5 filter | earthquake revision | feature `id` | `properties.updated` |
| `nhc_current_storms` | `noaa-nhc.ts` | storm entry | storm `id` | `lastUpdate` |
| `nhc_atcf_best_track` | `atcf-radii.ts` `fetchBestTrack` | whole b-deck file revision | storm id | null (the file carries none) |
| `spc_day1_outlook` | `noaa-spc.ts`, before the ENH+ filter | category polygon | `ISSUE:LABEL` | `ISSUE` |
| `nifc_perimeters` | `nifc.ts`, prescribed burns included | perimeter revision | IRWIN id, else incident name | `poly_DateCurrent` |

A half-read NWS feed (a page failed, or more than six pages) is not archived: the poll row would claim a count the read did not have.

### The outcome side (second pass, 2026-10-29)

| `source` | Module and hook | Product | `product_id` | `issued_at` |
|---|---|---|---|---|
| `nws_dat_surveys` | `nws-damage-assessment.ts`, per page, per layer | survey feature | `properties.globalid` | `surveydate` |
| `usgs_stn_hwms` | `usgs-stn.ts` `fetchStnHwms` | high-water mark, **personal fields removed** | `hwm:<id>` | `flag_date` |
| `usgs_shakemap_products` | `usgs-shakemap.ts` `fetchShakeMapProduct` | ShakeMap product entry | `<event>:<product id>` | `updateTime` |
| `usgs_shakemap_contours` | `usgs-shakemap.ts` `fetchShakeMapMmiContours` | `cont_mmi.json`, whole | `<event>:cont_mmi:<version>` | `process-timestamp` |
| `nhc_wind_swath` | `nhc-wind-swath.ts` | best-track bundle (zip), **base64 in `content_text`** | `<storm>_best_track.zip` | null |
| `nhc_forecast_cone` | `nhc-forecast-cone.ts` | advisory cone bundle (zip), base64 | the archive file name | `ADVDATE` when the bundle parsed |
| `nhc_surge` | `noaa-surge.ts` | surge GeoJSON, whole | storm id | null |
| `nasa_firms` | `nasa-firms.ts`, before the confidence filter | hotspot row (raw CSV line) | `lat,lng,acq_date,acq_time,satellite` | acquisition time |
| `openfema_declarations` | `historical-sync` worker, both reads | declaration row, **`lastRefresh` removed** | row GUID | `declarationDate` |

Two deliberate departures from "as received", each visible in the stored body: STN rows lose the surveyors' names and the free-text fields (`STN_REMOVED_FIELDS`; the body lists what went under `_archive_removed_fields`) — the platform has never copied those and an append-only archive is the wrong place to start; and OpenFEMA rows lose `lastRefresh`, which the dataset restamps on every reload, so that 70,000 unchanged rows do not return as new bodies each time. Binary bundles are stored base64 in `content_text` and hashed as that text. The FIRMS URL is stored with `<key>` in place of the API key.

**Not yet hooked:** NWC inundation (read on demand for flood rings) and the ShakeMap radii path (`fetchShakeMapRadii`, cached, reads the same contour file).

## Rules the writer keeps

- **Called after parse, with everything the body held, before any filter.** The archive is what was received, not what was kept.
- **Never throws, never changes the caller's result.** A failed write is logged under `AS_ISSUED_ARCHIVE` and the worker goes on.
- **Awaited, not fire-and-forget.** A serverless function may end before a dangling promise settles.
- **`issued_at` is what the product says, or null.** It is never guessed. `received_at` is set by the database.
- **Only sources the rights register permits.** `ARCHIVE_SOURCES` names each source's register id; `tests/lib/archive/as-issued.test.ts` checks `storage_permitted: "yes"` for every one.
- **Never from a test.** CI runs the unit suite with real database credentials, and the archive is append-only, so a feed test that did not mock `fetch` would write its fixture into the record for good. Under Vitest the writer returns `archive disabled under test` unless `AS_ISSUED_ARCHIVE_IN_TESTS=1`, which only the archive's own test sets (against a stubbed client). Feed tests that count `fetch` calls mock the module as well.

## Cost

Per-product rows with hash dedup. Expected order of magnitude at the 2026-10 cadences: NWS alert versions a few MB a day, USGS revisions about 1–2 MB a day, NIFC a few changed perimeters a day, SPC and NHC small — roughly 3–4 GB a year before TOAST compression, on a 5.6 GB database. Poll rows: about 2 million a year across the hooked sources, ~100 bytes each. Supabase storage beyond the plan's allowance is billed per GB-month; at these volumes that is cents, not dollars. Each read adds one hash lookup and one or two inserts to its worker, of the order of 100–300 ms.

Watch `as_issued_polls` for `items_new` that never drops to zero on a source (a body that carries a per-poll timestamp inside the item would store every poll) and `pg_total_relation_size('as_issued_products')` monthly.

## Continuity

**Since:** migration 224 (written 2026-10-31). **Function:** `archive_continuity(start, end)`. **Reader:** `lib/archive/continuity.ts`, `GET /api/v1/archive/continuity?month=YYYY-MM` (public), a line in the archive section of `/track-record`.

The archive is worth what its continuity is worth: a product a feed published while nobody was reading is a product the archive does not hold, and nothing can put it back. `as_issued_polls` records every read, quiet ones included, so the claim is measured rather than made. For each source in a month: the reads, the usual interval between them (the source's own median; since migration 226 reads less than 90 seconds apart are one read, or a paginated fetch would make the usual interval zero and every real interval a gap), the 95th percentile, the longest interval and when it began, and how many intervals ran past three times the usual. A source is `continuous` when none did, `gapped` when one did, `silent` when it has not been read for three times its usual interval as of now, `single_read` when there is nothing to compare.

What a gap says: that the platform did not read the feed for that long. A feed that was down, a worker that did not run, a deploy and a Vercel incident all read the same here; the availability record (`docs/source-availability.md`) is where they are told apart. The usual interval is measured, not typed: several workers write the NWS alert source, so its reads are a minute apart and no one worker's cadence describes it.

```sql
SELECT * FROM archive_continuity(date_trunc('month', now()), now());
```

## Verifying a body

```sql
-- what did Calchis hold for this alert, and when did each version arrive?
SELECT received_at, issued_at, content_hash, byte_length
FROM as_issued_products
WHERE source = 'nws_alerts' AND product_id = 'urn:oid:2.49.0.1.840.0....'
ORDER BY received_at;

-- was the feed read at 14:05Z, and what did it hold?
SELECT fetched_at, received_at, http_status, items_seen, items_new
FROM as_issued_polls
WHERE source = 'nws_alerts' AND fetched_at BETWEEN '2026-10-28T14:00Z' AND '2026-10-28T14:10Z';
```

The hash is `sha256(JSON.stringify(body with keys sorted at every level))` for JSON, `sha256(text)` for text; `canonicalJson()` in the writer is the reference.

**Every body is sealed, and the seal is independently timestamped (since 2026-11-06).** The daily determination-log digest (`docs/digest-anchoring.md`) includes the hash of every body received that day as the entry `aip1-<content_hash>`, beside the determination ids, and the digest's root is anchored through OpenTimestamps. So `GET /api/v1/archive/products/{hash}` returns `sealed`: the period, the root, the Merkle inclusion proof and the anchor's state, or `not_sealed` for a body received today. What that proves: the hash — and so the body — existed by the block's time, on evidence a reader can check against the public ledger without trusting Calchis. What it does not prove: that the body was received at `received_at` to the minute (that is the archive's own word, bounded above by the seal), or that the product was correct. The archive's claim was always "as received, when received"; the seal makes the *when* a bound a stranger can verify.