Repository: stanford-crfm/helm
Issue: Wishlist for paper
GitHub labels: p2

Paper

- [x] Additional stuff for robustness team (and fairness; but I don't think this is a priority). This will be addressed by #758. 
- [x] Add ablations: Extra seeds, RunExpander for number of in-context example. We will need to finalize these, but I think the priority will be instead figuring out what we want to show, since it seems like several things are not about getting new results but understanding things using what we have given what I list below. So for now, this is adequately handled.

Here is an initial proposal for visualizations (and requisite new runs) for the paper:

1. **Tradeoffs.** Figure with `M - 1` subplots of acc. vs robustness, fairness, bias, toxicity, efficiency, where `M` is num. metrics. Each subplot has x-axis of acc. and y-axis the other metric. The points can either be all models on all scenarios, or all models on a scenario we choose. This will be reminiscent of accuracy-on-a-line (though it might not be a line). **Requires no new runs.**
2. **Correlations.** To complement the Figure in 1, we include a `M x M` table where `M` is num. metrics. Each cell of the table is Pearson and Spearman correlations (with bolding to indicate significance at p = 0.05), where these correlations could either be computed as follows: (i) over all (model, scenario) points or (ii) the average over scenario over all model points. (ii) is probably better. Looks like Table 2 in this old paper of mine: https://aclanthology.org/2020.emnlp-main.649.pdf. **Requires no new runs.**
3.  **Learning curves.** Figure with ~4 subplots. We choose ~4 prototypical scenarios. For each subplot, the x-axis is the number of training examples. The y-axis is performance (probably has to be just 1 metric to be visualizable, e.g. accuracy). We have `k` learning curves where `k` is num. models, i.e. each point is `(num. train examples, acc; model)`. **Requires selection of prototypical scenarios and associated runs, which is directly invoking the RunExpander.**
4. **Scale.** Similar to Figure 1, but now the x-axis is scale (either in params or training co2 cost), y-axis is the metric, and the points can be for some or all scenarios. **Requires no new runs.**
5. **Public vs. Private.** `M` subplots, x-axis is scenarios (some or all), y-axis is performance, and we have bars for (i) best public model on that scenario, (ii) performance of public model with best average across scenarios, and (iii)/(iv) being the same for private. Some nuance on what counts as public vs. private, mainly on whether the APIs of OpenAI/AI21 get counted. Could theoretically have 3 categories which matches how I described in the Models section, i.e. free and open but no compute (e.g. OPT, BLOOM), open and compute but not free (OpenAI, AI21), free and compute but not open (MT-NLG, Anthropic). 

We could also have something on stability that varies number of seeds, but not sure this is needed; we should more generally just discuss the variance as we discuss various results from our 3 runs. We could also vary prompting details less clear of an axis of variation/feels less fundamental. At any rate, I think these 5 visualizations are compelling top-level things that are strong top-level matters.