← Back to Home

Prediction Scorecard

How accurate Creek Intelligence's forecasts have been lately. Experimental — self-grading, updated daily.
Updated 2026-09-10 06:30 UTC · scoring window: last 45 days

Every forecast the system makes is recorded and later graded against what the river actually did — this page is that report card. It currently scores 82 physics crest predictions, 1044 empirical likelihood forecasts, and 155 recession countdowns (0 still maturing). Updated daily; treat it as experimental.

Where the truth comes from: USGS keeps revising its provisional gauge readings for months. The gauge archives these grades are checked against were last re-pulled from USGS on 2026-09-10 05:20 UTC (trailing 120 days, with approval status), and every settled grade whose readings are not yet USGS-approved is re-graded nightly; a grade that moves keeps its first verdict on the record. Last pass (2026-09-10 06:05 UTC): rise: 499 checked, 0 verdicts moved last night, 172 re-graded so far, 0 final; recession: 1872 checked, 162 verdicts moved last night, 778 re-graded so far, 0 final; physics: 321 checked, 0 verdicts moved last night, 107 re-graded so far, 192 final; ponca: 736 checked, 0 verdicts moved last night, 678 re-graded so far, 0 final; empirical: 1154 checked, 0 verdicts moved last night, 0 re-graded so far, 0 final. USGS is retiring the data service these gauge feeds use; its replacement has been running alongside it since 2026-09-04: 135 hourly checks, 27247 readings compared between the two services with 0 value differences, and 23145 readings compared against what the live feeds stored with 10 differences. 30 readings were seen by the new service before the old one and 0 the other way round (timing, not disagreement). Right now 1 gauge series returns no values at all (07055875/00060) — see the data-health table below.

Physics Rise Predictors

Predicts the crest a gauge will reach from rainfall + antecedent moisture. Since 2026-09-08 every hourly run is a claim — a rise, or no rise — and both are graded: detection (rises the engine called) and false alarms side by side, then how close the crest and its timing were when a called rise happened.

Sample: 82 records · 1 events · 2 on the new grading · last 45 days · truth as of 2026-09-10 05:20 UTC · events counted on v2 records only (fixed 18-h horizon, from 2026-09-04)

New grading from 2026-09-04: each prediction is checked over a fixed 18-hour window after it was issued (before, the window closed 2 h after the predicted peak, so a later crest was never seen). It is graded on three separate questions: did the gauge rise at all, how far off was the crest when it did, and how far off was the timing. Predictions whose crest was not above the current level are no longer archived as claims.

Claims (from 2026-09-08): every hourly run is archived as a claim — a rise, or no rise (quiet / rain too light) — and graded 18 h later against the gauge. Detection = rises the engine called, out of every rise that happened (a no-rise claim followed by a rise is a miss). False alarms = rise predictions the gauge did not follow. Events = episodes of consecutive hourly runs.

Empirical Forecast Engine

Each hourly forecast is a calibrated probability that the gauge rises to the next level within the promised window (12 h on the fast creeks, 30 h on the slow ones). Scored as probabilities: Brier = mean squared error (0 is perfect); skill (BSS) = improvement over always forecasting the long-run rate (>0 is skill); obs/forecast = how many rises happened per rise forecast (1.0 is calibrated); reliability = what actually happened when the engine said X%. Probabilistic scoring began 2026-09-02; earlier records are not comparable and are not shown.

Sample: 1044 records · 2 events · 23 claims / 1021 no-rise calls · last 45 days · truth as of 2026-09-10 05:20 UTC · events = claim episodes (issuances >= 8 % within 3 h of each other); non-claims do not form episodes

"No rise indicated" calls (forecast below 8%) are graded separately on their miss rate: 1021 such hourly calls in the window, 0.0% of which were followed by a rise to the target level. Real claims (8% and above): 23, of which 0 verified.

All basins pooled: 1044 forecasts, 0 rises, Brier 0.00059, skill None, obs/forecast None, AUC None.

Reliability (when the engine said X%, how often did the rise come?)

Recession Countdowns

Predicts how long until a falling river drops to each threshold. HIT% = reached the level near the predicted time; MAE = average timing error (hours). Newly launched — most predictions are still maturing.

Sample: 161 records · 5 events · last 45 days · truth as of 2026-09-10 05:20 UTC · events = recession episodes per gauge/target

An episode is one run of hourly countdowns for a gauge and target (a new episode starts after a 3-hour gap); it counts as reached if any countdown in it was. "Last call" is the timing error of the final countdown issued before the crossing.

Ponca AI Rainfall Event Analysis

When rain arms it, predicts how big the Ponca gauge will get — a class (Fizzle/Moderate/High/Flood), a typical-peak band, and a flood-risk %. Scored per armed EVENT since 2026-09-02: the first call and the most-informed ('mature', most rain so far) call of each event are graded against the crest the event actually reached; the flood-risk % is scored as a probability (Brier, 0 is perfect; 'shown' is the number on the card, 'raw' the analog match alone). After the crest the card hides the % unless a second, higher crest is coming, so post-peak calls are graded only on whether a higher crest came. Events, not 15-minute re-issues, are the sample size.

Sample: 304 records · 5 events · last 45 days · truth as of 2026-09-10 05:20 UTC

Last 45 days — 5 armed event(s), 304 fifteen-minute calls, 0 flood(s).

All time since 2026-06-08 — 12 event(s), 2 flood(s).

Reliability of the shown flood-risk % on rising calls (304 fifteen-minute rows — rows within one event are near-duplicates, so this shows calibration shape, not sample size): 0-10%: said 1% → 0% flooded (n=232), 10-25%: said 14% → 0% flooded (n=72).

Recent events — first call, mature call, actual

Radar Nowcast (Ponca card)

While rain is falling, the Ponca card reads the last 40 minutes of weather radar, moves the picture forward, and says roughly how much more rain lands on the creeks above Ponca in the next two hours (or that the rain is about done). Each radar frame's estimate is graded against the rain the next two hours actually delivered, using the card's own hourly rain accounting. Bias = estimate minus actual (negative means radar under-called); 'more coming' = the card said 0.1 in or more was on the way; 'about done' = it said little was left. The 'past storms like this went on to a real rise' perspective line is graded against whether the event rose at all.

Last 45 days — 303 radar frames graded across 5 rain event(s) (0 not gradable).

Verified = the call was right when made; caught = share of the real ≥0.1″ two-hour periods the card had called in advance.

By storm motion (estimates ≥0.1″): <8 mph (near-stationary): n=4, bias 0.248″, actual÷est 0.39; 8-20 mph: n=11, bias 0.072″, actual÷est 0.69; >=20 mph: n=13, bias 0.103″, actual÷est 0.33. Slow-moving storms grow over the watershed and deliver more than radar advection suggests; fast movers pass through and deliver less.

“Past storms like this went on to a real rise” line (12 lines in 4 events): “more often than not” (63%): said 10 times, the event rose 0% of them; “about half the time” (52%): said 2 times, the event rose 0% of them.

All time since 2026-07-11 — 494 radar frames graded across 7 rain event(s) (0 not gradable).

Verified = the call was right when made; caught = share of the real ≥0.1″ two-hour periods the card had called in advance.

By storm motion (estimates ≥0.1″): <8 mph (near-stationary): n=9, bias 0.15″, actual÷est 0.56; 8-20 mph: n=21, bias 0.09″, actual÷est 0.69; >=20 mph: n=15, bias 0.1″, actual÷est 0.33. Slow-moving storms grow over the watershed and deliver more than radar advection suggests; fast movers pass through and deliver less.

“Past storms like this went on to a real rise” line (24 lines in 6 events): “more often than not” (63%): said 20 times, the event rose 0% of them; “about half the time” (53%): said 4 times, the event rose 0% of them.

Buffalo Rise Engine (per-gauge nowcast)

For each Buffalo mainstem gauge, predicts a coming rise (slight / moderate / large) from local rain + upstream propagation, with a timing window. Graded on whether the gauge actually rose, and within the predicted window. Predictions issued in the last 45 days, with the all-time totals beside them; predictions only fire during rain events, so this fills in over time.

Sample: 243 records · 16 events · last 45 days · truth as of 2026-09-10 05:20 UTC

Predictions issued in the last 45 days: 243 graded records, 16 events (10 rose). All-time: 499 graded records, 40 events (21 rose, 52%).

Since 2026-09-03 the local-rain layer serves a rise probability and a typical rise band (25th–75th percentile, cfs) instead of a size word. "Band held" = the share of real rises that landed inside the band (target about 50%); Brier = mean squared error of the probability (0 is perfect, 0.25 is a coin flip).

A "rise" here means peak >= 1.25 x v0 and peak - v0 >= clamp(0.10 x v0, 10, 30) cfs — the same rule the engine uses to decide a predicted rise has arrived. Records graded before 2026-09-03 used a fixed +30 cfs floor, which read some real low-water rises as misses; the 45-day window mixes both vintages until mid-October. An event = a run of records less than 3 h apart.

Typical actual rise by predicted category: slight: ~27.2 cfs (n=53), moderate: ~28.2 cfs (n=36), large: ~12.2 cfs (n=2). (Categories come from rainfall, so this is how they map to real gauge rises — calibration that accrues over time.)

Recent rise predictions — predicted vs. actual

Downstream Propagation (Ponca → Pruitt → St. Joe)

Retired 2026-09-02. The Ponca card's own downstream model duplicated the wave tracker's reaches with weaker validation, so the card now narrates the wave tracker's bands and arrival windows for Pruitt, St. Joe and Harriet, and those are graded in the Routed Waves section below. The four events this section had graded included two synthetic host tests, which have been removed.

Retired — graded under Routed Waves below.

Routed Waves (wave tracker)

When an upstream gauge crests, the wave tracker predicts the crest size and arrival window at each downstream gauge. Every confirmed wave is graded after its window closes: did a real rise arrive, was the crest inside the predicted band, and did it arrive inside the window. Waves are rare events — this fills in slowly.

Sample: 1 records · 1 events · last 90 days · truth as of 2026-09-10 05:20 UTC · one grade per confirmed wave-target

Waves graded: 1 (significant: 1); crest inside the likely band 1/1; inside the wide band 1/1; arrived inside the window 0/1; median predicted/actual crest 1.0×. Alerts issued: 0; floods with no alert: 0.

Watershed Alerts (Q-bucket triggers)

The rain-triggered WATCH/WARNING words on the calibrated creeks (Upper Buffalo, Richland Main, Upper Cossatot, Upper Big Piney, Mulberry), graded against the creek's own gauge: an alert verifies when the creek reached its floatable floor within 24 h; a rise with rain and no alert is a miss. Same recipe as the leave-one-year-out numbers on the Watersheds page.

Sample: 22980 records · 2 events · last 90 days · truth as of 2026-09-10 05:20 UTC · records = 15-min Tier-1 decisions; events = alert episodes (+ rises with no alert)

Alert episodes: 2 (verified 1, false alarms 1, still open 0); rises with no alert: 2; median lead from first WARNING 3.2 h.

Rises with no alert: Upper Cossatot 2026-07-15 18:17Z (rain 0.77" of 1.20"); hailstone 2026-06-27 19:47Z (rain 0.86" of 1.15")

Neural Net Predictions (Ponca & Boxley)

A neural network projects the level hour by hour from 48 h of 1 km radar rainfall (Ponca 6 h ahead, Boxley 4 h). Every hourly issuance is graded against what the gauge actually did: MAE = average error of the median; IN-BAND% = how often the likely range contained the truth (target ~50%); SKILL = error reduction vs assuming the level doesn't change, on hours where it moved.

0 hourly issuance(s) logged and maturing. Each settles ~8 hours after it is made; skill numbers become meaningful once the river actually moves.

Neural Net Predictions (Cossatot)

The Cossatot pixel net projects the level hour by hour from 48 h of 1 km radar rainfall (6 h ahead); crossing probabilities come from its companion gradient-boosted ensemble. Graded the same way as the Buffalo net: MAE = average error of the median; IN-BAND% = how often the likely range contained the truth (target ~50%); SKILL = error reduction vs assuming the level doesn't change, on hours where it moved.

0 hourly issuance(s) logged and maturing. Each settles ~8 hours after it is made; skill numbers become meaningful once the river actually moves.

Data health

Every grade above is checked against a USGS gauge feed. A single parameter can go silent while its site keeps reporting (USGS stops publishing discharge when the stage falls below its rating's lowest measured point); this table shows the age of the newest USGS reading per gauge and parameter, and whether a stage-derived stand-in is being served in its place.

21 series checked at 2026-09-10 06:30 UTC — 1 silent, 0 lagging.

Coverage

What the scorecard grades today, and what is still being wired into the loop: