# Calchis Data Lineage

> Every data element traces back to a named source. This document is the provenance proof an acquirer's engineering team reads during due diligence.
>
> Last updated: 2026-10-21

---

## 0. What is actually stored

Every `Records` row in the per-dataset entries below describes the **upstream**
dataset — how much the agency publishes. It is not a count of rows in this
database, and the two differ by orders of magnitude for several sources.

That distinction matters more than the individual numbers. A reader who takes
`~3 million` beside the earthquake catalog as a description of what backs the
seismic scoring would be wrong by a factor of about a thousand.

Audited against each upstream API and this database on **2026-08-26** by
`scripts/verify-historical-coverage.ts`, which asks both independently and
never reads an ingest script's own report. Every figure is a measurement.

| Dataset | Expected | Stored | State |
|---|---:|---:|---|
| NFIP claims | 2,706,674 (1978–2024) | 2,706,673 | Complete |
| USGS earthquake catalog | 26,594 (US scope) | 26,600 | Complete |
| NOAA storm events | 1,750,000 floor | 1,765,885 | Complete. 1996-onward by design — NCEI only records all 48 event types from then |
| FEMA declarations | 5,200 floor | 5,218 | Complete, and drives the scorer's frequency sub-scores |
| IBTrACS hurricane tracks | 127,000 floor | 127,139 | Complete |
| NIFC fire perimeters | 38,000 floor | 38,471 | Complete |
| NOAA MRMS hail | 340,000 floor | 340,739 | Complete |
| MS building footprints | ~85,000 US tracts | 84,044 tract rows | Complete at tract level |
| NHC ATCF a-deck advisories (CARQ + OFCL) | ~1,500 storms 1998–2025, AL/EP/CP (`docs/historical-advisories.md`) | 0 — not loaded yet | Migration 228 written 2026-10-31; load with `scripts/historical-ingest/run-nhc-adeck.ts` (about an hour), which exits 1 unless every storm's stored count matches what it parsed |

**This table previously described a platform far emptier than the one that
exists.** Until this revision it reported NFIP claims as "Never ingested, 0"
and the earthquake catalog at "8.3%", which was true when audited on
2026-08-11 and stopped being true on 2026-08-19 when the backfill ran. A
lineage document that understates the data is not a safe error — it is the
document an acquirer reads first, and it was describing a gap that had been
closed a week earlier.

### Derived reference datasets

Static geography and lookup tables generated from published sources by scripts
in `scripts/`, committed to the tree, and regenerable. They carry no rows in
the database; the generator is the provenance.

| Dataset | Source | License | Generated by | Used for |
|---|---|---|---|---|
| `lib/geo/coastline.ts` | Natural Earth 50m coastline | Public domain | `scripts/derive-coastline.ts` | Distance to coast — hurricane coastal proximity. Ocean only; Natural Earth keeps lakes in a separate layer, so the Great Lakes are correctly not coast |
| `lib/geo/neighbour-land.ts` | Natural Earth 50m admin-0 (Canada, Mexico, Cuba, The Bahamas, Russia rings near US territory) | Public domain | `scripts/derive-us-outline.ts` | Coverage — a point strictly inside a neighbouring sovereign's land is refused before the outline's tolerance strip is consulted (known-debt #116) |
| `lib/geo/state-bounds.ts` | 175,873 rows of `critical_infrastructure` | Platform data | `scripts/derive-state-bounds.ts` | Resolving a US state from coordinates, which decides which perils are scored |
| `lib/geo/flood-zones.ts` | 2,503,717 NFIP claims carrying a zone and coordinates | Platform data (FEMA source) | `scripts/derive-flood-zones.ts` | Neighbourhood FEMA flood zone. 0.1° cells because NFIP rounds claim coordinates to one decimal for policyholder privacy — a neighbourhood, **not** a parcel-level FIRM lookup |
| `lib/modeling/generated/hazus-hurricane-wind-curves.ts` | NHERI SimCenter DLML commit `662999759cc59bf9cfc099d4ff5a5814a7bf01c8`, `src/dlml/data/hurricane/building/portfolio/Hazus v5.1 original/loss_repair.csv` (sha256 `6f023f82c4ab56e57d0d20becf72e0c46a13cf4a80ebd28c3ac50f6bb83de222`), via the committed 40-row verbatim extract `scripts/data/dlml-hurricane-loss-extract.csv`; each curve checked against the raw `huDamLossFun.xlsx` (sha256 `f1b01169…`) | BSD-3 (SimCenter tabulation); FEMA values are federal public-domain work product — position closed 2026-09-24 after no reply from the Hazus help desk (docs/data-rights-audit.md) | `scripts/generate-hazus-hurricane-wind-curves.ts` (byte-equal test) | Hurricane building-loss curve per construction type and terrain roughness (model 2.3, M-27); seeded by migration 153 |
| `lib/modeling/generated/hazus-earthquake-pga-fragilities.ts` | NHERI SimCenter DLML commit `192d28020cafaa4a3bba9fb7340bedef593780f7`, `src/dlml/data/seismic/building/portfolio/Hazus v6.1/fragility.csv` (sha256 `4746c127…`) and `consequence_repair.csv` (sha256 `2ea57a63…`), via the committed verbatim extracts `scripts/data/dlml-seismic-lf-fragility-extract.csv` (34 rows) and `dlml-seismic-lf-repair-extract.csv` (2 rows) | BSD-3 (SimCenter tabulation); FEMA values are federal public-domain work product — same position as the hurricane curves | `scripts/generate-hazus-earthquake-pga-fragilities.ts` (byte-equal test) | Earthquake damage: lognormal equivalent-PGA fragilities per building type and design level x repair-cost ratios per occupancy (model 2.5, #79 step 1); seeded by migration 170 |
| `lib/modeling/generated/nshm-2023-wus-ruptures.ts`, `nshm-2023-wus-sections.ts` | USGS nshm-conus 6.2.0 (commit `a69e117b`), `active-crust/fault/wus-system/branch-avg/ruptures.csv` (52 MB, sha256 `d5bd221d…`) and `sections.geojson` (sha256 `4fc45683…`); ruptures rarer than 1e-8/yr dropped (130,491 of 525,684 kept, 99.9% of the rate); gzipped and base64-encoded in the modules with payload hashes | Public domain (USGS); citation carried | `scripts/fetch-nshm-fault-system.ts` (payload-hash test; the 52 MB source is not committed) | Earthquake fault sources (model 3.0, #79 remaining fix 2): every rupture at its published rate, distance to the rupture trace |
| `lib/modeling/generated/bssa14-pga-coefficients.ts` | USGS nshmp-lib 1.8.5 (commit `44728a7d`), `src/main/resources/gmm/coeffs/BSSA14.csv` (verbatim copy `scripts/data/nshmp-lib-BSSA14.csv`, sha256 `710d717e…`); functional form from `BooreEtAl_2014.java` in the same release | Public domain (USGS); paper cited | `scripts/generate-bssa14-coefficients.ts` (byte-equal test; hand-evaluated equations in `tests/lib/modeling/bssa14.test.ts`) | Active-crust site intensity (model 3.1): BSSA14 PGA on rock with its aleatory variability, then Worden (2012) to MMI |
| `lib/geo/terrain-roughness-tracts.ts` | Hazus Hurricane TM 5.1 (July 2022) Tables 4-13/4-14 (`lib/geo/hazus-nlcd-roughness.ts`) × `tract_land_cover` NLCD 2021 fractions (read-only export, sha256 `23c3a5a8…`) × Census 2020 Gazetteer tract internal points (sha256 `a49f3f3b…`) | Public domain (US Gov) | `scripts/derive-terrain-roughness.ts` (offline; inputs too large to commit) | Terrain roughness z0 of 40,871 tracts in the 20 states with a Hazus column; Hawaii has no NLCD CONUS cover and falls to the open-terrain default |
| `us_faults` (table) | USGS Quaternary Fault and Fold Database, 2020 release, DOI 10.5066/F7S75FJM | Public domain (US Government work) | `scripts/ingest-usgs-faults.ts`, migration 094 | Distance to mapped Quaternary faults — earthquake fault proximity |

**A license decision worth recording.** The GEM Global Active Faults dataset is
considerably easier to obtain than the USGS database and was rejected: it is
CC-BY-SA 4.0, and a filtered derivative is adapted material carrying the same
ShareAlike obligation. That is an encumbrance inside a codebase being prepared
for acquisition. The USGS database is a work of the United States Government
and is public domain, and is also the database the scorer's own component label
names.

---

## 0.1 Provenance Column Coverage (audited 2026-08-12)

CLAUDE.md's data standard requires that **every ingested record** stores
`source`, `source_url`, `ingested_at` and `confidence_score`. Audited against
the migrations rather than assumed:

| Table | source | source_url | ingested_at | confidence_score |
|---|---|---|---|---|
| `events` | yes | yes | yes | yes |
| `historical_declarations` | yes | **085** | yes | **085** |
| `historical_storm_events` | yes | **085** | yes | **085** |
| `historical_hurricane_tracks` | yes | **085** | yes | **085** |
| `historical_fire_perimeters` | yes | **085** | yes | **085** |
| `critical_infrastructure` | yes | **095** | yes | **095** | *(absent from this audit until 2026-08-26 — the table was created in 027 with only source and ingested_at, and being unlisted is why nothing caught it)*
| `historical_hail_events` | yes | **085** | yes | **085** |
| `historical_nfip_claims` | yes | **085** | yes | **085** |
| `osint_items` | `source_type` | `url` | **085** | `credibility` |
| `tract_building_stats` | yes | yes | yes | n/a |

**085** = added by `supabase/migrations/085_ingested_record_provenance.sql`,
which is **written but NOT YET APPLIED**. Until it is applied, only `events` and
`tract_building_stats` satisfy the standard in full, and a historical row can be
traced to the ingest script that wrote it but not to the record it came from.

`osint_items` satisfies the intent under different column names — `url` is the
source document and `credibility` is the confidence score. Only the ingest
timestamp was genuinely absent.

### Record-level vs dataset-level provenance

These are different guarantees and the distinction is kept explicit rather than
blurred:

- **Record-level** — the URL addresses the individual public record, so a third
  party can check one row against the source. Applies to USGS earthquakes (ComCat
  event page), NOAA storm events and MRMS hail (NCEI event detail page), and FEMA
  declarations (FEMA disaster page).
- **Dataset-level** — no per-record public page exists, so the URL identifies the
  exact file or service the row was parsed from. Applies to IBTrACS track points,
  NIFC perimeter features, and NFIP claims.

## 1. Live Data Sources

### 1.1 USGS Earthquake Feed

| Field | Value |
|-------|-------|
| Official Name | USGS Earthquake Hazards Program — Real-Time GeoJSON Feed |
| Endpoint | `https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_day.geojson` |
| Data Format | GeoJSON FeatureCollection |
| Authentication | None |
| Update Frequency | Real-time (polled every 5 minutes via cron) |
| Worker Route | `/api/workers/usgs` |
| Data Source Library | `lib/data-sources/usgs.ts` |
| Supabase Tables | `events`, `consequence_rings` |
| Fields Extracted | `id`, `mag`, `place`, `time`, `url`, `coordinates` (lat/lng/depth) |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | No — USGS global feed is ingested in full. Global earthquakes are relevant for US insurance interest and SEO. Display may be filtered to US for non-earthquake perils. |
| Magnitude Filter | M >= 2.5 (events below 2.5 are excluded at ingestion) |
| Deduplication | `source_id` = USGS event ID (unique constraint on `events.source_id`) |

### 1.2 NASA FIRMS VIIRS

| Field | Value |
|-------|-------|
| Official Name | NASA Fire Information for Resource Management System — VIIRS S-NPP Near Real-Time |
| Endpoint | `https://firms.modaps.eosdis.nasa.gov/api/area/csv/{KEY}/VIIRS_SNPP_NRT/USA/1` |
| Data Format | CSV |
| Authentication | API key (free registration at earthdata.nasa.gov) |
| Update Frequency | ~15 minutes from satellite overpass (polled every 15 minutes via cron) |
| Worker Route | `/api/workers/firms` |
| Data Source Library | `lib/data-sources/nasa-firms.ts` |
| Supabase Tables | `events` |
| Fields Extracted | `latitude`, `longitude`, `brightness`, `frp` (fire radiative power), `acq_date`, `acq_time`, `confidence`, `satellite` |
| Confidence Score | Derived from VIIRS confidence flag: high=1.0, nominal=0.7, low=0.3 (averaged across cluster) |
| US-Only Filter | Yes — endpoint scoped to `/USA/` |
| Clustering | Individual hotspots are clustered within 0.045 degrees (~5 km). Each cluster becomes one `wildfire` event with aggregated FRP. |
| Deduplication | `source_id` = `firms_{date}_{lat_grid}_{lng_grid}` (lat/lng rounded to 1 decimal) |

### 1.3 NIFC WFIGS Interagency Perimeters

| Field | Value |
|-------|-------|
| Official Name | National Interagency Fire Center — Wildland Fire Interagency Geospatial Services (WFIGS) Current Perimeters |
| Endpoint | `https://services3.arcgis.com/T4QMspbfLg3qTGWY/arcgis/rest/services/WFIGS_Interagency_Perimeters_Current/FeatureServer/0/query?where=1%3D1&outFields=*&f=geojson` |
| Data Format | GeoJSON (ArcGIS REST API) |
| Authentication | None |
| Update Frequency | Twice daily (polled every 12 hours via cron, at :07 past the hour to stay off the top-of-hour load; 45 s timeout with one retry) |
| Worker Route | `/api/workers/nifc` |
| Data Source Library | `lib/data-sources/nifc.ts` |
| Supabase Tables | `events` |
| Fields Extracted | `IncidentName`, `GISAcres`/`DailyAcres`, `PercentContained`, `FireDiscoveryDateTime`, `IrwinID`, `POOState`, `FireCause`, geometry (perimeter polygon) |
| Confidence Score | 1.0 (authoritative federal interagency source) |
| US-Only Filter | Yes — NIFC data is US-only by definition |
| Filters | Prescribed burns excluded (`FireCause = "Prescribed Fire"`, `IncidentTypeCategory = "RX"`, or name matching `/\bRX\b/i` or `/\bprescribed\b/i`) |
| Deduplication | `source_id` = IRWIN ID when available, otherwise `nifc_{name_slug}_{year}` |

### 1.4 NOAA NHC — Active Tropical Cyclones

| Field | Value |
|-------|-------|
| Official Name | NOAA National Hurricane Center — Current Storms |
| Endpoint | `https://www.nhc.noaa.gov/CurrentSurges.json` |
| Data Format | JSON |
| Authentication | None |
| Update Frequency | Every 6 hours during active storm season (polled every 30 minutes via cron) |
| Worker Route | `/api/workers/nhc` |
| Data Source Library | `lib/data-sources/noaa-nhc.ts` |
| Forecast cone | `lib/data-sources/nhc-forecast-cone.ts` — per advisory, the public GIS bundle `https://www.nhc.noaa.gov/gis/forecast/archive/{stormid}_5day_{adv}.zip` (shapefiles: `_5day_pgn` cone polygon, `_5day_lin` track, `_5day_pts` forecast positions with TAU/VALIDTIME/MAXWIND/GUST). Stored on the event row as `forecast_cone`, `forecast_track`, `forecast_points`, `forecast_advisory`, `forecast_issued_at`, `forecast_source_url` (migration 160). An intermediate advisory (e.g. `20A`) has no bundle; the numeric advisory's is used. A bundle that cannot be read leaves the previous advisory's cone in place. |
| Supabase Tables | `events` |
| Fields Extracted | `id`, `name`, `classification` (HU/TS/TD/SS/SD), `intensity` (max wind kt), `pressure` (mb), `latitude`, `longitude`, `movementDir`, `movementSpeed`, `lastUpdate`, `windRadii` (34/50/64kt per quadrant NE/SE/SW/NW), `publicAdvisory` |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | No — Atlantic and Eastern Pacific storms are ingested. These are the basins relevant to US hurricane risk. |
| Deduplication | `source_id` = NHC storm ID (e.g., `al092024`) |
| Notes | Returns 404 outside hurricane season or when no storms are active; this is expected behavior and not logged as an error. |

### 1.5 NOAA SLOSH — Storm Surge

| Field | Value |
|-------|-------|
| Official Name | NOAA NHC Sea, Lake, and Overland Surges from Hurricanes (SLOSH) Model |
| Endpoint | `https://www.nhc.noaa.gov/storm_graphics/api/{STORM_ID}_MOST_LIKELY_FLOODING_wSurge.json` (per storm) |
| Data Format | GeoJSON FeatureCollection |
| Authentication | None |
| Update Frequency | Per NHC advisory (polled every 30 minutes via cron) |
| Worker Route | `/api/workers/noaa-surge` |
| Data Source Library | `lib/data-sources/noaa-surge.ts` |
| Supabase Tables | `events` (fields: `surge_geometry`, `surge_max_ft`) |
| Fields Extracted | Surge polygons with `SURGE_FT` property, max surge height |
| Confidence Score | 1.0 when NHC SLOSH data available; reduced confidence for Saffir-Simpson fallback estimates |
| US-Only Filter | Yes — NHC surge products are US coastal only |
| Fallback | When NHC SLOSH GeoJSON is unavailable, surge is estimated from Saffir-Simpson category using published midpoint values (e.g., CAT 3: 9-12 ft surge, 5-15 km inland). Fallback geometry is a circular polygon centered on the storm. |
| Source Attribution | `source` field on `SurgeEstimate` is `"nhc_slosh"` or `"estimated"` — always distinguishable |

### 1.6 NOAA AHPS — Flood Alerts

| Field | Value |
|-------|-------|
| Official Name | NOAA/NWS Active Flood Alerts |
| Endpoint | `https://api.weather.gov/alerts/active?event=Flood%20Warning,Flash%20Flood%20Warning,Flood%20Watch,Coastal%20Flood%20Warning` |
| Data Format | GeoJSON (OGC-compliant NWS API) |
| Authentication | None (User-Agent header required per NWS API policy) |
| Update Frequency | Real-time (polled every 10 minutes via cron) |
| Worker Route | `/api/workers/noaa-ahps` |
| Data Source Library | `lib/data-sources/noaa-ahps.ts` |
| Supabase Tables | `events` |
| Fields Extracted | `id`, `event` (alert type), `severity` (Extreme/Severe/Moderate/Minor), `certainty`, `urgency`, `headline`, `areaDesc`, `onset`, `expires`, `description`, `instruction`, geometry (polygon) |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | Yes — NWS alerts are US-only by definition |
| Deduplication | `source_id` = NWS alert ID |
| Severity Mapping | NWS severity mapped to 0-10 scale: Extreme=9.0, Severe=7.0 (8.0 for flash), Moderate=5.0 (6.0 for flash), Minor=3.0 |
| Gauge identifier (since 2026-10-21) | `/api/workers/gauge-enrich` reads the alert's AWIPS id and P-VTEC, finds the matching segment of the office's text product (`https://api.weather.gov/products/types/{type}/locations/{office}`) and stores its H-VTEC NWS Location Identifier as `raw_data.nwsli` (e.g. `ASTF1`, St Johns River near Astor), with `hvtec` and `nwsli_source: nws_product_hvtec`. Matched on the event (office, phenomenon, significance, tracking number), never on the gauge name; `00000` (areal warnings) and anything unresolved store no identifier (`lib/data-sources/nws-hvtec.ts`). Not yet read by trigger binding, which still uses `gauge_key` |

### 1.6a NOAA National Water Center — Modeled Inundation Extent (EXPERIMENTAL)

| Field | Value |
|-------|-------|
| Official Name | NWC National Water Model analysis inundation extent (and CREST FLASH inundation extent) |
| Endpoint | `https://maps.water.noaa.gov/server/rest/services/nwm/ana_inundation_extent/FeatureServer/0/query` (FLASH: `.../flash/flash_inundation_extent/FeatureServer/0/query`) |
| Authentication | None |
| Update Frequency | Hourly analysis upstream; read once when a flood warning's consequence zones are built |
| Worker Route | `/api/workers/noaa-ahps` (through `computeRings`) |
| Data Source Library | `lib/data-sources/nwc-inundation.ts` |
| Supabase Tables | `consequence_rings` (zone 1 of a flood warning; `parameters.experimental: true`, `reference_time`, `streamflow_cfs_max`, `truncated`) |
| Fields Extracted | polygon geometry per stream reach, `feature_id`, `streamflow_cfs`, `reference_time` |
| Confidence Score | Modeled, labelled EXPERIMENTAL by the NWC — carried on every ring |
| US-Only Filter | Yes — CONUS layers only (AK/HI/PRVI layers exist upstream and are not read) |
| Scope Rule | Only inside an NWS warning polygon (server-side intersection); never for a watch; never with no polygon. FLASH is wired but not read — it timed out at 60 s per query on 2026-09-23. |
| Rights | Public domain (US Government work); register row "NWC inundation extent" in `docs/data-rights-audit.md` |

### 1.7 NOAA SPC — Severe Weather

| Field | Value |
|-------|-------|
| Official Name | NOAA Storm Prediction Center — Active Watches & Day 1 Convective Outlook |
| Endpoints | Watches: `https://www.spc.noaa.gov/products/watch/ActiveWW.json` / Outlook: `https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson` |
| Data Format | GeoJSON |
| Authentication | None |
| Update Frequency | Continuous (polled every 5 minutes via cron) |
| Worker Route | `/api/workers/noaa-spc` |
| Data Source Library | `lib/data-sources/noaa-spc.ts` |
| Supabase Tables | `events` |
| Fields Extracted | Watches: `WATCH_NUM`, `WATCH_TYPE`, `ISSUED`, `EXPIRES`, `MAX_HAIL`, `MAX_GUSTS`, `STATES`, geometry. Outlooks: `LABEL` (risk category), `VALID`, `EXPIRE`, geometry. |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | Yes — SPC data is US-only by definition |
| Outlook Filter | Only Enhanced (ENH, severity 6.0) and above outlooks are ingested. Marginal and Slight risk are too common for event-level tracking. |
| Deduplication | Watches: `spc-watch-{num}`. Outlooks: `spc-outlook-{label}-{date}`. |

### 1.8 NWS Severe-Convective Warnings (SCS / hail)

| Field | Value |
|-------|-------|
| Official Name | NWS Active Alerts — Severe Thunderstorm & Tornado Warnings |
| Endpoint | `https://api.weather.gov/alerts/active?event=Severe Thunderstorm Warning,Tornado Warning` |
| Data Format | GeoJSON (impact-based warning polygons) |
| Authentication | None (User-Agent required) |
| Update Frequency | Continuous (polled every 5 minutes via cron) |
| Worker Route | `/api/workers/nws-storm-alerts` |
| Data Source Library | `lib/data-sources/nws-storm-alerts.ts` |
| Supabase Tables | `events`, `consequence_rings` |
| Fields Extracted | `event`, warning polygon geometry, `parameters.maxHailSize` (in), `parameters.maxWindGust` (mph), `parameters.tornadoDetection`, `expires`, `areaDesc`. Severity = strongest of hail/wind/tornado, floored by warning type. |
| Purpose | The OBSERVED / actively-warned complement to SPC forecast outlooks (§1.7). Captures hail size — the key SCS insurance metric — with a real polygon. |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | Yes — NWS covers the US + territories by definition |
| Deduplication | `{nws-alert-id}`. Warnings not in the active feed / past `expires` / older than 3h are retired to historical each run. |

### 1.9 NWS Winter-Storm / Freeze Warnings

| Field | Value |
|-------|-------|
| Official Name | NWS Active Alerts — Winter Storm / Ice Storm / Blizzard / Extreme Cold / Hard Freeze |
| Endpoint | `https://api.weather.gov/alerts/active?event=Winter Storm Warning,Winter Storm Watch,Ice Storm Warning,Blizzard Warning,Extreme Cold Warning,Wind Chill Warning,Hard Freeze Warning` |
| Data Format | GeoJSON (polygon when present; many winter warnings are zone-based with `geometry:null`) |
| Authentication | None (User-Agent required) |
| Update Frequency | Every 15 minutes via cron (winter warnings evolve slower than convective) |
| Worker Route | `/api/workers/nws-winter-alerts` |
| Data Source Library | `lib/data-sources/nws-winter-alerts.ts` |
| Supabase Tables | `events` (event_type `winter_storm`, added in migration 079) |
| Severity | By warning type: Ice/Blizzard 8.0, Winter Storm Warning 6.5, Extreme Cold 6.0, Hard Freeze 5.0, Watch 4.5; +1 for NWS "Extreme". |
| Geometry Fallback | Zone-based warnings (no polygon) placed at the state centroid (marker only), not dropped. |
| Purpose | Closes the winter peril gap — a distinct major insurance loss driver (Winter Storm Uri 2021 ≈ $15B). |
| Confidence Score | 1.0 (authoritative government source) |
| US-Only Filter | Yes — NWS covers the US + territories |
| Deduplication | `{nws-alert-id}`; a readable SEO `slug` is set on every event. Warnings not in the feed / past `expires` / older than 48h are retired to historical. |

### 1.10 GDELT DOC 2.0

| Field | Value |
|-------|-------|
| Official Name | GDELT DOC 2.0 API — Global Document Search |
| Endpoint | `https://api.gdeltproject.org/api/v2/doc/doc` |
| Data Format | JSON |
| Authentication | None |
| Update Frequency | Every 15 minutes (polled every 15 minutes via cron) |
| Worker Route | `/api/workers/gdelt` |
| Data Source Library | `lib/data-sources/gdelt.ts` |
| Supabase Tables | `osint_items` |
| Fields Extracted | `url`, `title`, `seendate`, `socialimage`, `domain`, `language`, `sourcecountry` |
| Confidence Score | Domain-based: `.gov`/`.mil` = 1.0, verified media (Reuters, AP, BBC, NYT, WaPo, CNN, NPR, ABC) = 0.85, all others = 0.60 |
| US-Only Filter | Query-level — `sourcecountry:US` appended for earthquake and severe weather queries. Hurricane and wildfire queries are not US-filtered since international coverage is relevant. |
| Content Moderation | All articles pass through `lib/moderation/filter.ts` before storage. See Section 4. |
| Deduplication | `osint_items.url` has a UNIQUE constraint. Duplicate URLs are silently skipped. |

### 1.11 Supplementary OSINT Sources

| Source | Library | Purpose |
|--------|---------|---------|
| Google News RSS | `lib/data-sources/google-news-rss.ts` | Supplementary news feed. Free, keyless. Fetched via RSS XML parsing. |
| NWS Active Alerts (per state) | `lib/data-sources/gov-feeds.ts` | Government alert feed for OSINT enrichment. Credibility 1.0. |
| FEMA Recent Declarations | `lib/data-sources/gov-feeds.ts` | Recent federal disaster declarations for OSINT enrichment. |

---

### 1.12 Observed footprints — NWS DAT, USGS ShakeMap, NHC wind swath (migration 203)

What federal agencies say an event actually DID, stored apart from the
modeled `consequence_rings` in `observed_footprints` and served with
`kind: "observed"` on every feature by `GET /api/v1/events/{id}/footprints`
(and the MCP `event_footprints` tool). One worker,
`/api/workers/observed-footprints`, reads all three; each fails on its own.
Every row carries `source`, `source_url` (the exact published record),
`observed_at`, `ingested_at` and `confidence_score`. Upsert key
`(source, source_record_id)`, so a re-surveyed track or a new ShakeMap
version replaces its row. Until 204 is applied the worker skips (logged once
per process) and the route answers `observed_available: false`.

| Field | Value |
|-------|-------|
| Official Name | NWS Damage Assessment Toolkit — DamageViewer FeatureServer |
| Endpoint | `https://services.dat.noaa.gov/arcgis/rest/services/nws_damageassessmenttoolkit/DamageViewer/FeatureServer` — layer 0 damage points, 1 damage lines (tracks), 2 damage polygons; `query?where=stormdate >= CURRENT_TIMESTAMP - INTERVAL '14' DAY&outFields=*&f=geojson`, paged by 2,000. Confirmed live 2026-10-17 (ArcGIS 11.5, SR 4326): the last 30 days held 445 points, 33 tracks, 11 polygons |
| Data Format | GeoJSON (Point / LineString / Polygon) |
| Authentication | None |
| Update Frequency | Re-read every run (polled every 30 minutes via cron); records with a storm date in the last 14 days |
| Worker Route | `/api/workers/observed-footprints` |
| Data Source Library | `lib/data-sources/nws-damage-assessment.ts` |
| Supabase Tables | `observed_footprints` (`source = 'nws_dat'`) |
| Fields Extracted | `globalid` (key), `event_id` (the NWS name, e.g. "Francis Peak Tornado"), `efscale`/`efnum`, `maxwind`/`windspeed` (mph), `length` (mi), `width` (yd), injuries, fatalities, `stormdate`/`starttime`, `wfo`/`office`, `qc`, damage indicator and degree of damage on points. -99 is the service's "not assessed" and is stored as null |
| Event link | By space and time to a `tornado` / `severe_weather` warning issued from 3 h before to 1 h after the storm time, track start within 25 km of the warning polygon's box (`lib/observed-footprints/match.ts`); otherwise stored unmatched (`event_id` NULL) |
| Confidence Score | 0.9 when the service marks the record `qc = 'Y'`, else 0.7 — DAT is preliminary until NCEI Storm Data, by the service's own statement |

| Field | Value |
|-------|-------|
| Official Name | USGS ShakeMap — MMI contours (`download/cont_mmi.json`) |
| Endpoint | ComCat detail `https://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/{id}.geojson` → `products.shakemap[0].contents["download/cont_mmi.json"].url`. Confirmed live 2026-10-17 on nc75382936 (M5.6 Redwood Valley, CA; ShakeMap v8, automatic, ten contours MMI 2–6.5) |
| Update Frequency | Polled every 30 minutes via cron, for US (`country = 'US'`) USGS earthquakes M5+ that started in the last 14 days, ten per run |
| Worker Route | `/api/workers/observed-footprints` |
| Data Source Library | `lib/data-sources/usgs-shakemap.ts` (`fetchShakeMapProduct`, shared with the earthquake ring radii) |
| Supabase Tables | `observed_footprints` (`source = 'usgs_shakemap'`), one row per event per MMI level, the contour lines as published (MultiLineString — never closed into areas) |
| Event link | Exact: the USGS event id is the event's `source_id` |
| Confidence Score | 0.85 reviewed map, 0.75 automatic — ShakeMap interpolates stations and felt reports with ground-motion-model fill, so it is observation-constrained, not a direct measurement at every point |

| Field | Value |
|-------|-------|
| Official Name | NOAA NHC best-track GIS — cumulative wind swath (`{ID}_windswath.shp`) |
| Endpoint | `https://www.nhc.noaa.gov/gis/best_track/{stormid}_best_track.zip` (linked from CurrentStorms.json `bestTrackGIS.zipFile`). Same zip/shapefile readers as the forecast cone (migration 160). Confirmed live 2026-10-17 on al142024 (Milton: 64, 64, 50, 34 kt records) and the then-active ep212026 |
| Update Frequency | Polled every 30 minutes via cron, for Atlantic (`al…`) storms active or monitoring in the last 3 days, and (since 2026-10-19) eastern and central Pacific (`ep…`, `cp…`) storms whose swath reaches US territory (`lib/geo/on-us-territory.ts`) |
| Worker Route | `/api/workers/observed-footprints` |
| Data Source Library | `lib/data-sources/nhc-wind-swath.ts` |
| Supabase Tables | `observed_footprints` (`source = 'nhc_best_track'`), one row per swath record (`RADII` 34/50/64 kt, `STARTDTG`/`ENDDTG`) |
| Event link | Exact: the NHC storm id is the event's `source_id` |
| Confidence Score | 0.8 — an analysis of observed winds (operational best track), revised in the post-season Tropical Cyclone Report |

**USGS STN high-water marks** (migration 207, live 2026-10-21)

| Field | Value |
|-------|-------|
| Official Name | USGS Short-Term Network (STN) Flood Event Viewer — high-water marks |
| Endpoint | `https://api.waterdata.usgs.gov/stnservices/events/` (event list) and `https://api.waterdata.usgs.gov/stnservices/hwms/filteredhwms/?event={id}` (marks). No key |
| Update Frequency | Polled every 30 minutes via cron, for STN riverine-flood and hurricane events started in the last 90 days (`STN_LOOKBACK_DAYS`) |
| Worker Route | `/api/workers/observed-footprints` |
| Data Source Library | `lib/data-sources/usgs-stn.ts` |
| Supabase Tables | `observed_footprints` (`source = 'usgs_stn'`, `product_type = 'usgs_stn_hwm'`), one row per mark: point, surveyed elevation, vertical datum, height above ground, quality class, mark type, environment. Surveyors' names and free-text notes are not stored |
| Event link | `name_season`: a hurricane deployment ("2024 Helene") links to the Calchis storm of that name and season, and never when two match; `space_time`: a riverine mark links to the flood warning whose polygon contains it during the event; otherwise `unmatched` |
| Confidence Score | From STN's quality class, Calchis's own ordering: Excellent 0.95, Good 0.9, Fair 0.8, Poor 0.65, VP / unknown 0.5 |

The first mark arrived on 2026-10-21 (Marsh Creek, Centre County, PA; STN event "2026 September PA Flood") and is unmatched: the nearest flash flood warning polygon passes 0.2 km from it without containing it, and containment is the rule.

## 2. Historical Datasets

All historical datasets are US-scoped. No global historical tables exist in the MVP. Each dataset is ingested via a one-time manual script (`scripts/historical-ingest/`) and refreshed periodically by the `historical-sync` worker (`/api/workers/historical-sync`, cron: 1st of each month).

### 2.1 FEMA Disaster Declarations

| Field | Value |
|-------|-------|
| Official Source | FEMA OpenFEMA API |
| Endpoint | `https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries` |
| Records | ~68,000 |
| Time Period | 1953 to present |
| Geographic Scope | US counties |
| Data Format | JSON (paginated at 1,000 records) |
| Ingest Script | `scripts/historical-ingest/run-fema-declarations.ts` |
| Supabase Table | `historical_declarations` |
| Key Fields | `fema_disaster_number` (unique), `declaration_type` (DR/EM/FM), `state`, `county`, `fips_code`, `incident_type`, `declaration_date`, `incident_begin`, `incident_end`, `lat`, `lng` |
| Refresh | Daily, `app/api/workers/historical-sync` (03:00 UTC; from the newest declaration held less 30 days, keyed `<type>-<number>` by `lib/historical/fema-declaration-key.ts`) |

### 2.2 NOAA Storm Events

| Field | Value |
|-------|-------|
| Official Source | NOAA National Centers for Environmental Information (NCEI) Storm Events Database |
| Endpoint | `https://www.ncei.noaa.gov/pub/data/swdi/stormevents/` (bulk CSV) |
| Records | ~1.8 million |
| Time Period | 1950 to present |
| Geographic Scope | US |
| Data Format | CSV (compressed, annual files) |
| Ingest Script | `scripts/historical-ingest/run-noaa-storm-events.ts` |
| Supabase Table | `historical_storm_events` |
| Key Fields | `noaa_event_id` (unique), `year`, `event_type`, `state`, `county`, `begin_date`, `end_date`, `lat`, `lng`, `deaths_direct`, `deaths_indirect`, `injuries_direct`, `property_damage_usd`, `crop_damage_usd`, `narrative` |
| Refresh | Re-ingested by hand with the script above (no scheduled sync; the historical-sync worker refreshes FEMA declarations only) |

### 2.3 IBTrACS Hurricane Tracks

| Field | Value |
|-------|-------|
| Official Source | NOAA NCEI International Best Track Archive for Climate Stewardship (IBTrACS) v04 |
| Endpoint | `https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r01/` |
| Records | ~100,000 track points |
| Time Period | 1842 to present |
| Geographic Scope | North Atlantic basin only (MVP scope) |
| Data Format | CSV/NetCDF |
| Ingest Script | `scripts/historical-ingest/run-ibtracs.ts` |
| Supabase Table | `historical_hurricane_tracks` |
| Key Fields | `ibtracs_id`, `storm_name`, `basin` (default `NA`), `season`, `track_point_time`, `lat`, `lng`, `max_wind_kt`, `min_pressure_mb`, `category` |
| Unique Constraint | `(ibtracs_id, track_point_time)` |
| Refresh | Annual sync |

**Not every row is an NHC record.** IBTrACS adds its own position every three
hours between NHC's six-hourly HURDAT2 fixes, and the ingest keeps no flag
telling them apart (roughly half the rows in a backtest read are
interpolations). Every backtest counts NHC's records only
(`lib/modeling/backtest/nhc-records.ts`, since 2026-10-21): rows at 03, 09, 15
and 21 UTC exactly are set aside unless HURDAT2 lists a record at that time and
place (`lib/modeling/generated/hurdat2-three-hourly-records.ts`, generated by
`scripts/fetch-hurdat2-three-hourly-records.ts`: 214 records 1851–2025, all
matching a stored row from 1900 on). The location history panel's count of
major hurricanes near a point counts NHC's fixes too (since 2026-10-21,
known-debt #139); the Florida blog series states both counts.

### 2.4 NIFC Historical Fire Perimeters

| Field | Value |
|-------|-------|
| Official Source | NIFC GeoMAC / WFIGS Historical Perimeters |
| Endpoint | NIFC ArcGIS Feature Service (historical layer) |
| Records | ~25,000 |
| Time Period | 1984 to present |
| Geographic Scope | US |
| Data Format | GeoJSON (ArcGIS REST) |
| Ingest Script | `scripts/historical-ingest/run-nifc-perimeters.ts` |
| Ingest Library | none — the script reads the service itself (`lib/historical/fire-key.ts` dedupes); an unused `lib/historical/nifc-perimeters.ts` was removed 2026-10-21 |
| Supabase Table | `historical_fire_perimeters` |
| Key Fields | `fire_name`, `year`, `acres`, `state`, `agency`, `discovery_date`, `containment_date`, `centroid_lat`, `centroid_lng`, `geometry` |
| Refresh | Annual sync |

### 2.5 NOAA Storm Events hail reports (ingested by `run-mrms-hail.ts`)

Corrected 2026-10-14. This section described radar data from NSSL's
Multi-Radar Multi-Sensor system. The ingest has never read MRMS: it reads the
NCEI Storm Events CSVs, `event_type = Hail` — reports from spotters, the
public and NWS staff. The script and table names are historical. Reports
follow population, so the densest cells of the hail grid built from them are
cities in hail country (Oklahoma City, Dallas–Fort Worth, Wichita, Denver).
Since model 4.2 (2026-10-14) the hail SCORE no longer reads these reports
except as the state fallback outside radar coverage: it reads NEXRAD Hail
Detection Algorithm signatures from NOAA NCEI's Severe Weather Data Inventory
(`scripts/derive-hail-radar-days.ts`, `lib/geo/hail-radar-days.ts`).

| Field | Value |
|-------|-------|
| Official Source | NOAA NCEI Storm Events Database, hail reports |
| Records | 340,739 |
| Time Period | 1996 to present |
| Geographic Scope | US states |
| Data Format | Annual CSV.gz (StormEvents_details) |
| Ingest Script | `scripts/historical-ingest/run-mrms-hail.ts` |
| Supabase Table | `historical_hail_events` |
| Key Fields | `event_date`, `lat`, `lng`, `max_hail_size_inches`, `state`, `county` |
| Refresh | Annual sync |

### 2.6 USGS Earthquake Catalog

| Field | Value |
|-------|-------|
| Official Source | USGS Advanced National Seismic System (ANSS) Comprehensive Catalog (ComCat) |
| Endpoint | `https://earthquake.usgs.gov/fdsnws/event/1/query` |
| Records (ComCat overall) | ~3 million, all magnitudes, global |
| Records (what we ingest) | **27,135** M4.0+ events across the five US regions below — measured against the FDSNWS `/count` endpoint with our exact query envelope on 2026-08-11, not estimated |
| Per-region breakdown | CONUS 8,801 · Alaska 16,349 · Hawaii 904 · Puerto Rico/USVI 399 · Guam 682 |
| Magnitude floor | M4.0 (`minmagnitude=4.0`) |
| Time Period | 1900–2024 |
| Geographic Scope | CONUS + Alaska + Hawaii + US territories, selected by bounding box rather than by place label |
| Data Format | GeoJSON / CSV |
| Ingest Script | `scripts/historical-ingest/run-usgs-catalog.ts` |
| Ingest Library | `lib/historical/usgs-earthquake-catalog.ts` |
| Supabase Table | `historical_storm_events`, with `event_type = 'Earthquake'` and `source = 'usgs_catalog'` |
| Completeness check | The ingest queries `/count` per region and compares against what it stored; it exits non-zero on any shortfall, and reports UNVERIFIED rather than passing if `/count` is unreachable |
| Refresh | Annual sync |

> **Current state (2026-08-11): this table is NOT fully ingested.** 2,257 of
> 27,135 events are present (8.3%). CONUS holds nothing after 1939 and Alaska
> nothing after 1999, caused by transient fetch failures in an earlier run that
> were indistinguishable from quiet years. Re-running the ingest is the remedy.
> Any figure derived from this table — earthquake frequency, return periods —
> should be treated as unsupported until that run completes and reports 100%.

### 2.7 FEMA NFIP Claims (Aggregate)

> Duplicate of 2.8 below, which is the authoritative entry for this dataset.
> Kept only so existing links resolve. See 2.8 for the endpoint, fields, and
> current ingest state.

### 2.8 FEMA NFIP Claims (OpenFEMA API)

| Field | Value |
|-------|-------|
| Official Source | FEMA National Flood Insurance Program (NFIP) Claims |
| Endpoint | `https://www.fema.gov/api/open/v3/NfipClaims` |
| Superseded endpoint | `https://www.fema.gov/api/open/v2/FimaNfipClaims` — **depDate 2026-10-15** per FEMA's own `/api/open/v1/DataSets` metadata. The successor is RENAMED, not re-versioned, so `/v3/FimaNfipClaims` returns 404 and reads as "no successor exists". |
| API Key Required | No (public API) |
| Records | **2,724,656** individual claims (OpenFEMA `recordCount`, read 2026-08-11) |
| Geographic Scope | US (all 50 states + DC + territories) |
| Temporal Coverage | 1978-present |
| Data Format | JSON (OData v4 with pagination) |
| Ingest Method | One-time manual (`scripts/historical-ingest/run-nfip-claims.ts`). There is **no** automatic refresh — `historical-sync` does not touch this dataset, contrary to an earlier version of this row. |
| Fields Used | state, countyCode, **censusGeoid**, yearOfLoss, amountPaidOnBuildingClaim, amountPaidOnContentsClaim, **ratedFloodZone**, latitude, longitude |
| Field notes | `censusGeoid` is a 12-digit BLOCK GROUP GEOID; the 11-digit tract stored in `census_tract` is its first 11 characters. `ratedFloodZone` is the zone the property was rated in — there is no field named `floodZone` in the OpenFEMA data model, and asking for one returns HTTP 400. |
| Purpose | Flood hazard scoring validation, historical loss analysis |
| License | Public domain (US government data) |
| DB Table | `historical_nfip_claims` (migration 012) |

> **Current state (2026-08-11): this dataset is NOT ingested.** The ingest
> script's `$select` requested `floodZone`, a field absent from the OpenFEMA
> data model, so OpenFEMA rejected every request with HTTP 400 — every state,
> every page, every run. No NFIP claim has ever been stored. The query is fixed
> and verified against the live API, but the ingest still has to be run.
>
> Any figure derived from `historical_nfip_claims` — flood frequency, flood loss
> calibration — is unsupported until it is.

### 2.9 Microsoft Building Footprints

| Field | Value |
|-------|-------|
| Official Source | Microsoft US Building Footprints (GitHub release) |
| Endpoint | `https://github.com/microsoft/USBuildingFootprints` |
| Records | ~130 million structures |
| Geographic Scope | US |
| Data Format | GeoJSON (state-level files) |
| Ingest Method | One-time download |
| Purpose | Structure count and footprint area for exposure estimation (pre-ATTOM) |
| Notes | Used by `lib/property/msft-building-footprints.ts` for structure count within event zones |

---

## 3. Modeling Parameters

### 3.1 Source Chain

Every record in the `model_parameters` table traces to a specific published source through a mandatory chain:

```
Research Paper / Government Publication
       |
       v
model_parameters.source_description  (full citation with table/figure/equation reference)
model_parameters.source_paper_id     (FK to research_papers when applicable)
model_parameters.validation_note     (how the value was validated against observed data)
       |
       v
hazard_scores.parameters_used[]      (array of model_parameters IDs used in computation)
hazard_scores.score_components        (JSONB breakdown of all sub-scores)
```

### 3.2 FEMA Declaration Frequency (scoring input)

When a peril-specific frequency parameter does not cover a location's geography
scope, the hazard scorer uses the state's real federal disaster-declaration
history instead of a hardcoded regional guess.

| Attribute | Detail |
|-----------|--------|
| Official Source | FEMA OpenFEMA Disaster Declarations Summaries v2 |
| Supabase Table | `historical_declarations` (~5,200 rows, 1953–present) |
| Access Library | `lib/historical/declaration-frequency.ts` (`getDeclarationFrequency`) |
| Granularity | **State** (rows carry `state`/`fips_code` but no lat/lng) |
| Peril Mapping | Fire→wildfire, Flood→flood, {Hurricane, Typhoon, Tropical Storm, Coastal Storm}→hurricane, Tornado→tornado. "Severe Storm" is never mapped; hail and earthquake are excluded by design. |
| Calibration | `model_parameters.fema_decl_full_scale_rate` (per peril, US-National) — the most-exposed state's annual rate that saturates the frequency sub-score. |
| Consumers | `lib/modeling/hazard-scorer.ts` (frequency sub-scores); `app/api/v1/locations/history/route.ts` (Workbench "Federal Disaster Declarations" count, also state-based). |
| Refresh | Monthly via `historical-sync` worker; frequency cached 7 days. |

### 3.3 Parameter Sources by Domain

| Parameter Domain | Primary Source | Publication |
|------------------|---------------|-------------|
| FEMA declaration frequency (fallback) | FEMA | OpenFEMA Disaster Declarations Summaries v2, 1953–present (state-level counts) |
| Earthquake frequency (a/b values) | USGS | ComCat catalog; NSHM 2023 (Petersen et al., 2024, Earthquake Spectra) |
| Earthquake intensity prediction | USGS | Atkinson & Wald (2007), Seismological Research Letters 78(3), 362-368 |
| Earthquake damage thresholds | FEMA | Hazus v6.1 Technical Manual, Ch. 5, Tables 5-19/5-21 (Sd in inches) |
| Earthquake damage ratios | FEMA | Hazus v6.1 Technical Manual, Ch. 15, Table 15.2 |
| Hurricane wind decay | NOAA | Kaplan & DeMaria (1995, 2001), J. Applied Meteorology |
| Hurricane return periods | NOAA NHC | NHC State Hurricane Strikes (1851-2004), nhc.noaa.gov/paststate.shtml |
| Hurricane damage ratios (deterministic category path) | FEMA | Hazus Hurricane Model continuous wind-speed curves (huDamLossFun.xlsx). The `hazus_w1_cat*` category-binned values are Calchis interpolations (known debt #131). |
| Hurricane building-loss curves (stochastic and live-event paths, model 2.3) | FEMA via NHERI SimCenter | Hazus 5.1 hurricane wind loss functions as tabulated by NHERI SimCenter DLML (BSD-3), commit 662999759cc59bf9cfc099d4ff5a5814a7bf01c8, `Hazus v5.1 original/loss_repair.csv` (sha256 6f023f82c4ab56e57d0d20becf72e0c46a13cf4a80ebd28c3ac50f6bb83de222); one representative building per type; migration 153 |
| Hurricane terrain roughness | FEMA | Hazus Hurricane Model Technical Manual 5.1 (July 2022), §4.4.2, Tables 4-13 and 4-14 (z0 by NLCD class and state), over NLCD 2021 tract fractions |
| Storm surge estimates | NOAA NHC | Pre-2010 SSHWS ranges (NHC removed surge from scale in 2010) |
| Wildfire frequency | NIFC | Historical statistics 1984-2024 (ESTIMATED — needs FRAP verification) |
| Wildfire climate adjustment | Academic | Abatzoglou & Williams (2016), PNAS 113(42), 11770-11775 (2x area, 1.4x frequency inference) |
| WUI structure loss | Academic | PNAS (2025): 10% (2002-2012) → 32% (2013-2022). 25% period average. |
| Flood depth-damage | USACE | EGM 01-03 (Dec 2000), Table 1: One Story, No Basement. Verified 2026-03-27. |
| NFIP loss statistics | FEMA | NFIP Fact Sheet May 2023 ($66K avg claim); GAO-04-401T (repetitive loss 25-38%) |
| Hail frequency | NOAA SPC | SPC hail CSVs 2000-2024: 1.16/10K km²/yr for ≥2" in Plains. Verified 2026-03-28. |
| Hail damage thresholds | IBHS | Hail Research Program 2012-2020 (published findings; no IBHS or PCS data held — damage-ratio anchors are ESTIMATED) |
| Tornado frequency | NOAA SPC | SPC tornado CSV 2000-2024: Plains 0.26, SE 0.52 per 10K km²/yr. Verified 2026-03-28. |
| Compound event amplification | Engineering estimate | Zscheischler (2020) provides typology only; 1.35×/1.50× are our estimates, gated. |

### 3.4 Verification Status

As of 2026-03-28:

| Status | Count | Notes |
|--------|-------|-------|
| VERIFIED | 39 | GR a/b-values, A&W 2007 IPE (8), K&D wind decay (2), Hazus EQ damage ratios (4), Hazus EQ Sd thresholds (4), hurricane return periods (4), surge heights (3), flood depth-damage (3), NFIP stats (2), wildfire (2), tornado (2), hail frequency (1), IBHS threshold (1), Hazus complete damage (1), CA GR a-value (1) |
| NEEDS_VERIFICATION | 5 | Hurricane W1 cat1-5 damage ratios — Hazus uses continuous curves, values are defensible interpolations |
| ESTIMATED | 7 | Wildfire CA freq (needs FRAP query), climate multiplier (inference), hail damage ratios (industry composite), TX flood depth, compound amplifiers (gated) |

All ESTIMATED and NEEDS_VERIFICATION parameters are clearly labeled in the codebase. Compound amplifiers are gated behind `COMPOUND_SCORING_ENABLED` and do not affect production scores.

### 3.5 Versioning and Audit Trail

- **Active parameter lookup:** Only parameters where `superseded_at IS NULL` are used in scoring. The unique index `idx_model_params_active` enforces one active value per `(peril, geography_scope, parameter_name)` combination.
- **Parameter updates:** When a parameter changes, the old record receives a `superseded_at` timestamp. A new record is inserted. The `model_update_log` table records the old value, new value, change rationale, and supporting paper IDs.
- **Geography scope:** MVP parameters use US state/region codes only: `US-CA`, `US-TX`, `US-FL`, `US-Southeast`, `US-Gulf`, `US-Pacific-Northwest`, `US-Mountain`, `US-Plains`, `US-Northeast`, `US-National`. No `global` scope in MVP.

---

## 4. Data Quality Controls

### 4.1 OSINT Credibility Scoring

Implemented in `lib/moderation/filter.ts`. Every OSINT item receives a credibility score before storage:

| Domain Type | Credibility | Rule |
|-------------|-------------|------|
| `.gov` / `.mil` domains | 1.0 | Government sources are authoritative by default |
| Verified major media (Reuters, AP, BBC, NYT, WaPo, CNN, NPR, ABC News) | 0.85 | Established editorial standards |
| All other domains | 0.60 | Unknown provenance |

Display rules:
- Items with `credibility < 0.50` are never shown to users (filtered at query time via index: `idx_osint_event` has `WHERE credibility >= 0.50`)
- Items with `quarantined = true` are never shown

### 4.2 Content Moderation Pipeline

1. **Keyword blocklist check** — titles containing quarantine keywords (`hoax`, `fake`, `conspiracy`, `plandemic`, `false flag`, `government cover`, `deep state`, `chemtrail`) are quarantined at ingestion and never displayed.
2. **Credibility scoring** — domain-based scoring as described above.
3. **User reporting** — 3 user reports on any OSINT item triggers auto-quarantine (`osint_reports` table tracks reports, `osint_items.report_count` incremented).
4. **Language filter** — non-English articles from GDELT are excluded at ingestion.

### 4.3 Data Source Failure Tracking

The `data_sources` table tracks operational health of every ingestion pipeline:

| Column | Purpose |
|--------|---------|
| `last_fetched` | Timestamp of most recent fetch attempt |
| `last_success` | Timestamp of most recent successful fetch |
| `records_ingested` | Cumulative count of records ingested |
| `consecutive_failures` | Counter incremented on each failure, reset to 0 on success |

Each worker upserts to `data_sources` at the start of every run with `last_fetched`. On success, `last_success` is updated and `consecutive_failures` is reset. On failure, `consecutive_failures` is incremented.

Alerting: Sentry alerts fire when `consecutive_failures >= 3` for any source. A broken source never affects any other source — all workers are independent.

The `/api/v1/health` endpoint exposes current `data_sources` status for monitoring.

### 4.4 Event Deduplication

Every event record has a `source_id` column with a UNIQUE constraint. The deduplication strategy varies by source:

| Source | `source_id` Format | Strategy |
|--------|---------------------|----------|
| USGS | USGS event ID (e.g., `us7000abcd`) | Upsert — existing events have severity/status updated |
| NASA FIRMS | `firms_{date}_{lat_1dp}_{lng_1dp}` | Grid-based — hotspots within same 0.1-degree cell on same day merge |
| NIFC | IRWIN ID or `nifc_{name}_{year}` | Authoritative ID when available, name-based fallback |
| NOAA NHC | NHC storm ID (e.g., `al092024`) | Upsert — position, intensity, and wind radii updated each advisory |
| NOAA AHPS | NWS alert ID | Upsert — alert status updated |
| NOAA SPC | `spc-watch-{num}` or `spc-outlook-{label}-{date}` | Watch number or outlook category per day |
| GDELT | Article URL (on `osint_items.url` UNIQUE constraint) | URL-level deduplication — duplicate URLs silently skipped |

---

## 5. Data Flow Architecture

```
External API / Dataset
       |
       v
Data Source Library (lib/data-sources/*.ts)
  - Fetches raw data with AbortSignal.timeout(10_000)
  - Parses and normalizes to DisasterEvent / OsintItem types
  - Applies US geographic filter where applicable
  - Returns [] on any failure (never throws)
       |
       v
Worker Route (app/api/workers/*/route.ts)
  - Authenticates via CRON_SECRET bearer token
  - Upserts to data_sources tracking table
  - Upserts events to events table (source_id dedup)
  - Computes consequence rings where applicable
  - Updates event_timeline for active events
  - Returns HTTP 200 always ({ error, processed } on failure)
       |
       v
Supabase (Postgres + PostGIS)
  - Tables: events, consequence_rings, osint_items, event_timeline, etc.
  - PostGIS geometry storage and spatial indexing
  - RLS enforced on all tables (see Section 5.2)
  - Real-time subscriptions available for live updates
       |
       v
API Endpoints (app/api/v1/*)
  - GeoJSON output (OGC compliant)
  - Rate limited per minute in middleware (in-process, lib/rate-limit/memory-limiter.ts); daily API quota in Postgres
  - Response caching via lib/cache.ts (process memory + Vercel Runtime Cache; see cache TTLs below)
       |
       v
Client (CalchisMap.tsx, OsintPanel.tsx)
```

### Cache TTLs

| Key Pattern | TTL | Purpose |
|-------------|-----|---------|
| `events:live` | 60s | Live event feed |
| `rings:{eventId}` | 300s | Consequence ring geometry |
| `sitrep:free:{eventId}` | 7,200s (2 hr) | AI sitrep for free tier |
| `sitrep:pro:{eventId}` | 900s (15 min) | AI sitrep for pro tier |
| `hazard:score:{lat}:{lng}:{peril}` | 86,400s (24 hr) | Hazard score (lat/lng rounded to 2dp) |
| `portfolio:exposure:{id}` | 300s | Portfolio exposure calculation |
| `trigger:eval:{triggerId}:{eventId}` | 120s | ILS trigger evaluation |
| `event:timeline:{eventId}` | 300s | Post-event status timeline |

---

## 6. Compliance and Audit

### 6.1 Record Provenance

Every ingested record stores:

| Field | Column | Purpose |
|-------|--------|---------|
| Data source | `source` | Which data source library produced it (e.g., `usgs`, `noaa_nhc`, `nasa_firms`) |
| Original ID | `source_id` | ID from the external system — enables deduplication and back-reference |
| Original URL | `source_url` | URL to the original data for human verification |
| Ingestion timestamp | `ingested_at` | UTC ISO 8601 timestamp of when Calchis stored the record |
| Credibility | `confidence_score` | 0.0-1.0 assessment of source reliability |

### 6.2 Row-Level Security

RLS is enabled on every table in the database. Policies follow two patterns:

**Public data tables** (events, consequence_rings, osint_items, ai_situation_reports, historical_*, model_parameters, hazard_scores, research_papers, event_timeline):
- `public_read`: `SELECT` allowed for all users
- `service_write`: All operations restricted to `service_role` (workers and server-side only)

**User-owned tables** (user_locations, portfolios, ils_triggers):
- Users can only read, insert, update, and delete their own records (`auth.uid() = user_id`)
- `portfolio_locations` access controlled via portfolio ownership join
- `ils_trigger_events` readable by trigger owner via join

### 6.3 SOC 2 Readiness

| Feature | Implementation | Table |
|---------|---------------|-------|
| Alert audit trail | Every alert sent is logged with hashed user ID (SHA256, never raw), event ID, ring zone, channel, delivery status, and timestamp | `alert_log` |
| Model change audit | Every parameter update records old value, new value, rationale, and supporting paper IDs | `model_update_log` |
| Data source health | Continuous tracking of fetch success/failure rates per source | `data_sources` |
| Credential isolation | Service role key, API keys, and secrets are server-only — never exposed in client bundle | `middleware.ts`, environment variable separation |
| Rate limiting | All `/api/*` routes are rate-limited per minute in middleware (in-process since 2026-10-21; Upstash before) | `middleware.ts` |

### 6.4 Coordinate and Timestamp Standards

- All coordinates are WGS84 (EPSG:4326)
- All timestamps are UTC ISO 8601
- All API output uses OGC-compliant GeoJSON where spatial data is involved
- PostGIS spatial indexes (`GIST`) on all lat/lng columns for efficient geographic queries

---

## 7. Database Migrations

| Migration | Scope | Tables Created |
|-----------|-------|----------------|
| `001_initial.sql` | Core live event tables | `data_sources`, `events`, `consequence_rings`, `osint_items`, `user_locations`, `ai_situation_reports`, `osint_reports`, `alert_log` |
| `002_historical.sql` | US historical data | `historical_declarations`, `historical_storm_events`, `historical_hurricane_tracks`, `historical_fire_perimeters`, `historical_hail_events` |
| `003_modeling.sql` | Hazard/risk modeling | `model_parameters`, `model_update_log`, `hazard_scores`, `portfolios`, `portfolio_locations` |
| `004_research.sql` | Research pipeline | `research_papers` |
| `005_ils.sql` | ILS trigger monitoring | `ils_triggers`, `ils_trigger_events`, `event_timeline` |
| `006_subscriptions.sql` | Stripe billing | Subscription-related tables |
| `007_reports.sql` | Report generation | Report-related tables |
| `008_stochastic.sql` | Stochastic event sets | `stochastic_event_sets`, `stochastic_events`, `portfolio_ep_curves` |
| `009_alert_radius.sql` | Alert radius configuration | Alert radius fields |
| `203_observed_footprints.sql` | Observed footprints (NWS DAT, USGS ShakeMap, NHC wind swath), kept apart from the modeled rings | `observed_footprints` |

---

## 8. Worker Schedule (vercel.json)

| Worker | Route | Schedule | Source |
|--------|-------|----------|--------|
| USGS Earthquakes | `/api/workers/usgs` | Every 2 min | USGS |
| NASA FIRMS | `/api/workers/firms` | Every 15 min | NASA |
| NOAA NHC | `/api/workers/nhc` | Every 30 min | NOAA |
| NOAA Storm Surge | `/api/workers/noaa-surge` | Every 30 min | NOAA |
| NOAA Flood Alerts | `/api/workers/noaa-ahps` | Every 10 min | NOAA |
| NOAA Severe Weather | `/api/workers/noaa-spc` | Every 5 min | NOAA |
| NIFC Perimeters | `/api/workers/nifc` | Every 12 hr | NIFC |
| GDELT OSINT | `/api/workers/gdelt` | Every 15 min | GDELT |
| AI Sitrep Generation | `/api/workers/ai-sitrep` | Every 2 hr | Internal (Anthropic API) |
| ILS Trigger Evaluation | `/api/workers/trigger-eval` | Every 2 min | Internal |
| Research Paper Ingest | `/api/workers/research-ingest` | Weekly (Mon 2am) | Semantic Scholar |
| Historical Data Sync | `/api/workers/historical-sync` | Monthly (1st, 3am) | Multiple |
| Location Alert Eval | `/api/workers/location-alerts` | Triggered | Internal |
| OSINT AI Enrichment | `/api/workers/osint-ai` | Triggered | Internal |
| Report Generation | `/api/workers/report-gen` | Triggered | Internal |
| Observed Footprints | `/api/workers/observed-footprints` | Every 30 min | NWS DAT, USGS ShakeMap, NOAA NHC |

All workers authenticate via `Authorization: Bearer {CRON_SECRET}`. All workers return HTTP 200 regardless of success or failure, with error details in the response body.

---

## 9. Data Licensing and Attribution

| Source | License | Attribution Required |
|--------|---------|---------------------|
| USGS | Public domain (US Government work) | Recommended citation per USGS policy |
| NASA FIRMS | Public domain | NASA Earthdata citation |
| NOAA (NHC, AHPS, SPC, SLOSH) | Public domain (US Government work) | NOAA source attribution |
| NIFC | Public domain (US Government work) | NIFC/WFIGS attribution |
| GDELT | Open access (academic project) | GDELT Project citation |
| IBTrACS | Public domain | NOAA NCEI citation |
| FEMA OpenFEMA | Public domain | FEMA attribution |
| NOAA NCEI Storm Events | Public domain | NOAA NCEI citation |
| Microsoft Building Footprints | Open Data Commons Open Database License (ODbL) | Attribution required |
| Semantic Scholar | API terms of use | Allen Institute for AI citation |
| Mapbox | Commercial license | Attribution control required (Mapbox ToS) |

All government data sources (USGS, NOAA, FEMA, NIFC, NASA) are US federal government works and are not subject to copyright. Commercial use is permitted without restriction.