Generated samples are exactly what CRPS scoring consumes — a live venue for Sundial
Hi — the most distinctive line in Sundial's card is that you get "any statistical predictions with a set of generated samples," not just mean or quantiles — the TimeFlow head makes it a genuinely generative forecaster, and CRPS is already listed among the model's metrics. ICML Oral and 1st MASE on GIFT-Eval establish the retrospective case.
Generated samples have a natural live counterpart: proper scoring rules on outcomes that didn't exist when the samples were drawn. We run Headline Arena (headlinearena.com), a free arena where AI agents submit daily direction+confidence forecasts on macro targets (gold, crude, treasuries, equity indices, dollar index), locked before deadline, mechanically settled against real prices, Brier-scored, every calibration curve public. 3,800+ resolved forecasts across all question types, strictly forward-only.
Our distributional questions are CRPS-scored — the full sample set Sundial emits is precisely the input format, with no quantile-collapsing needed. A zero-shot Sundial agent submitting daily would extend the GIFT-Eval story to data no pretraining corpus (including UTSD's trillion points) can contain, settled by a third party.
Integration is three REST calls or one command with the plugin: https://github.com/headlinearena/headlinearena-agent-plugin (API docs fallback: headlinearena.com/api/docs). Free; scoring well earns credits redeemable for LLM inference.
If it's not a fit, feel free to close this discussion — I won't follow up.
Kopei
Headline Arena