Repository: stanford-crfm/helm
Issue: Reduce frontend JSON file sizes
GitHub labels: p1

framework


We're still loading >50 MB for some pages. The current JSON output files are:

- `run_spec.json` - `RunSpec`
- `scenario.json` - `Scenario` with the list of `Instance` added
- `per_instance_stats_slim.json` - list of `PerInstanceStats`
- `scenario_state.json` - `ScenarioState`, which is an `AdapterSpec` and a list of `RequestState`.
- `scenario_state_slim.json` - same as above, but with `RequestState.result.completions` truncated

The size problems are caused mostly by the list of `RequestState` / `Instances`, and somewhat by the list of `PerInstanceStat`. Easy fixes:

- The list of `Instance` is duplicated in `scenario.json` and `scenario_state.json`; we should delete it from the former.
- We can introduce a new `PerInstanceDisplayStats` (name?) that replaces `stats: List[Stat]` with just `stats: dict[str, float]` where _only_ the stats in the `schema.yaml` are included. This gives us >10-20x size savings.

Then to deal with the list of `RequestState` / `Instances`, we have the options:

- Paginate the `RequestState`s into pages of 50
  - The files would be placed under something like `request_states/request_states_shard_0.json`, etc.
- Strip out the actual requests and load them dynamically if requested
  - Still would be a giant page load for loading requests, since we have to load _all_ of them (or a whole page, if combining this with pagination)
  - Or remove the requests from the web UI entirely

Proposal: the frontend should only read four JSON files from the run:

- `scenario.json`: `Scenario`
- `instances.json`: `List[Instance]`
- `predictions.json`: `List[SlimPrediction]`
- `stats.json`: `List[Stat]`

This gives a 5x reduction in download size.

Users wanting to do further analysis should download:

- `scenario_state.json`: `ScenarioState`
- `per_instance_stats.json`: `PerInstanceStats`

Unfortunately I'm not sure if we kept these files around for v1.0 and v1.1

We should deprecate `scenario_state_slim.json` and `per_instance_stats_slim.json`

Update: renamed `predictions.json` to `display_predictions.json`, also added `display_requests.json` which contains `List[DisplayRequests]`.

JSON load size was reduced by >20x.

boolq now loads 1 MB of JSON:

![Screenshot_20221220_101707](https://user-images.githubusercontent.com/185227/208738421-e77bd7f7-2bc3-4571-85d6-6c1918f50c97.png)

msmarco now loads 6 MB of JSON (**correction** the download was cut off and the true size is >100M, see next comment):

![Screenshot_20221220_102320](https://user-images.githubusercontent.com/185227/208739030-f3db737f-adf7-41fa-a29f-1d4f60168ece.png)


Never mind; msmarco is still massive:

`msmarco:track=regular,valid_topk=30,model=openai_curie,data_augmentation=canonical`

```
103M Dec 17 18:29 display_predictions.json
620M Dec 17 18:29 display_requests.json
37M Dec 17 18:28 instances.json
443M Dec 15 02:13 per_instance_stats.json
103M Dec 15 02:14 predictions.json
3.3K Dec 15 02:04 run_spec.json
304 Dec 15 02:04 scenario.json
4.2G Dec 15 02:11 scenario_state.json
705K Dec 15 02:11 stats.json
```

We should just visualize fewer things in MSMARCO: #423