Repository: stanford-crfm/helm
Issue: All models do (often considerably) better for all fairness perturbations
Root cause: Validation Gap

This seems very implausible, but happens consistently for all models evaluated thus far (across Anthropic, OpenAI text-curie-001, AI21) for all the QA scenarios I looked at (listed below). 

So we should figure out if there is some error (otherwise this is a fairly stunning result I totally don't understand...) as it seems to be the case that systematically (i.e. all models, all scenarios) do worse for robustness perturbations but better for fairness perturbations compared to unperturbed accuracy. 

Places to look:
1. Fairness perturbations (we have looked at these before and they seemed right, idk if something changed {and somehow this doesn't break for robustness but only fairness})
2. Perturbing references (wouldn't explain something like BoolQ, but at any rate should check)
3. If 1 and 2 pass, then we are supplying the desired inputs to the LM. Then we should verify the predictions of the LM are post-processed as desired, referring to 4 and 5.
4. Computation of both the fairness-related metrics and also accuracy itself is correct in the presence of augmentations. (Not sure if this is related to skipping unperturbed instances; not quite sure why that would be different for fairness vs. robustness but maybe one gets applied a lot more than the other or something)
5. Visualization of metrics

[NaturalQuestions](https://crfm-models.stanford.edu/static/benchmarking.html?scenarioGroup=NaturalQuestions)
[BoolQ](https://crfm-models.stanford.edu/static/benchmarking.html?scenarioGroup=BoolQ)
[QuAC](https://crfm-models.stanford.edu/static/benchmarking.html?scenarioGroup=QuAC)
[NarrativeQA](https://crfm-models.stanford.edu/static/benchmarking.html?scenarioGroup=NarrativeQA)


@teetone We should get this sorted before we do the OpenAI run since in the worst case that there is something wrong with fairness-related augmentations, we would need to fix that and it would change the queries.

Some observations:
* All the fairness perturbations are consistently lower than or equal to the stat with `perturnation=identity`.
* From better to worse, the ordering of perturbations is as follows: `identity >= person_name > gender > dialect`. The order follows the expected numbers of tokens perturbed.

Related issue #597

The order you said seems reasonable and it correlating with num. tokens also seems reasonable and a good thing to highlight in the paper. 

What is the source for the error in visualization? The priority should be ensuring there is no error in the queries (so this does not block our ability to do runs), and then to fix visualization. At that point we can just go through all the scenarios and look at the results in the table format for the purposes of analysis/paper-writing.

I think the issue is in how the "None" perturbation is computed. Perhaps we can continue with the following steps:
* Complete a few local runs and inspect the prompts / stats
* Ensure that the numbers match what we have on the website
* Ensure that the identity perturbation stat is computed correctly
* Figure out how perturbation=None stat is computed 

I agree that one issue might be the presence of `perturbation=None` metrics that skew things.

But also, note that since https://github.com/stanford-crfm/benchmarking/pull/589 we are not evaluating instances that remain unchanged by perturbations. This does come up in fairness perturbations, since we don't always have things like "person name" applying. Thus, by default, the perturbation metric (`F1 perturbation="name"`) only reflects instances where this perturbation applies and thus could be inherently different than the identity one.

More generally, whether or not we should be including unperturbed inputs when computing metrics is a design choice:
- Including unperturbed might be useful for gauging the overall performance of the model. So asking, "how robust is the model to names?", a valid answer would be [names show up in 20% of the inputs and performance drop by 50% when changing names, meaning, performance drops by] => [10%].
- Not including might also be useful since we are directly evaluating the model on inputs that matter. Thus, we can directly see how well the model performs on inputs where names are actually present.

What do we think in general? Happy to chat more live or involve more people in the decision.

(Thinking a bit about it, if we go with the second, I should probably also record the performance of the model on the unperturbed versions of the relevant inputs.)

Yup, I agree both numbers bear some signal and should be recorded. 

For the main tables displayed in the paper/website, I think we should note the number of examples the metric is effectively computed over and then report the second one as the default.

Yeah, makes sense. The number of examples is already recorded in `num_instances` but will add the original accuracy as a metric.

Yup (i meant we should visualize it in table, i.e.)


F1 (N =1k)    |     Robustness (easy; N=800)    |      Robustness (hard; N = 900) ... fairness ... bias ... toxicity ... efficiency

As the column names or something like that

Can we revisit this after the metric cleanup that we just merged?

I also just submitted #723 which captures the performance of the relevant inputs before the perturbation is applied (and thus we don't have to worry about whether the set of Instances which we perturb is easier/harder to begin with).

I think this is resolved.

Looking at the several scenarios, all models do worse on fairness than accuracy. Of course, this is guaranteed by the fact we visualize the worst metric, whereas before (i.e. when I wrote this message) we were compare acc on the perturbed examples vs acc on the original examples. But we do see considerable degradation in several cases. 

However, looking at the runs view for AI21 models on BoolQ gives me some doubts: https://crfm-models.stanford.edu/static/benchmarking.html?runs=undefined&runSpec=.*boolq.*ai21.*canonica.*

Specifically, the results for gender and for person name seem to be better than the unperturbed numbers (whereas the synonym/typo numbers are clear worse, and the dialect are marginally worse). 

@dilarasoylu @dtsip Can you verify I am interpreting the numbers right and see if this is more generally true? It of course is fine if this is a true finding, but seems very implausible, so making sure we are looking at the right metric/etc., i.e. that there is no mistake, would be appreciated!


@rishibommasani this should be an artifact of having a different number of instances for each perturbation.

If I look at the json file for one of the runs you linked (e.g., [here](https://crfm-models.stanford.edu/static/benchmark_output/runs/latest/boolq:model=ai21_j1-jumbo,dataaugmentation=canonical/stats.json)), I parse it as follows:
```
synonym (1000 instances): 0.79 [original] 0.70 [perturbed] 0.63 [worst]
typos (1000 instances): 0.79 [original] 0.72 [perturbed] 0.66 [worst]
dialect (1000 instances): 0.79 [original] 0.77 [perturbed] 0.73 [worst]
gender (132 instances): 0.86 [original] 0.86 [perturbed] 0.86 [worst]
person_name (74 instances): 0.86 [original] 0.81 [perturbed] 0.81 [worst]
```

This looks perfectly reasonable to me, but we might want to think a bit about how we should visualize it.