Repository: stanford-crfm/helm
Issue: Generalize train example sampling for `generation` adapter to multi-label
Root cause: Unimplemented Feature Gap

The [train example sampling](https://github.com/stanford-crfm/helm/blob/913895386ff08034443bf9b768999be01a55e5ee/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L96) for the `generation` adapter (which actually lives in `InContextLearningAdapter`) works by sampling an example from the most frequent class, then the next most frequent class, etc. Unfortunately it assumes that [each example has only one correct reference](https://github.com/stanford-crfm/helm/blob/913895386ff08034443bf9b768999be01a55e5ee/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L128-L129). I would like to generalize this to allow multiple references.

The natural generalization would be to still use the label frequency counting method, but counting all references instead of just the first. Unfortunately, in practice this could lead to sampling only training examples with a single label, which means the model never sees a demonstration of multiple labels ("A, C, D"), and never sees a demonstration for no label ("n/a").

cc @JoelNiklaus

Makes sense. So this would add two examples that we should include. Maybe it makes sense to give first priority to the most frequent single class, second to "No Label", third to one example with multiple labels and then the subsequent priorities to the less frequent single classes.
What do you think?

cc @dtsip since we were chatting about train example selection last week

I agree with the general idea, but it's not safe to assume that we get >= 3 train examples for any given scenario.

Yes, that's true. This is why i would suggest to have them only as priority 2 and 3 to make sure that we have one example with an actual label for sure. But maybe it makes more sense to have an example with multiple labels first priority?

What if we just count all correct references?

```
ref_to_instances: Dict[str, List[Instance]] = defaultdict(list)
for instance in instances:
  for ref in instance.correct_refs:
    ref_to_count[ref.text].append(instance)
```

and then sort references according to frequency, and pick a random instance, making sure we don't include an Instance twice?

> making sure we don't include an Instance twice?

- This means we'll almost never sample an unlabeled train example, which _do_ we want (so that the model knows that unlabeled examples are possible).
- Also means that we don't give special weighting to multi-label examples, which perhaps we do want?
- When we pick an instance, we have to remove it from the instance list for every other of its label in `ref_to_instances`.

Also, whatever we pick should preserve existing behavior for single-label examples, for reproducing existing evaluations.

Also noting there's a bug [here](https://github.com/stanford-crfm/helm/blob/913895386ff08034443bf9b768999be01a55e5ee/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py#L150-L152): the comment suggests we break when we run out of unlabeled examples, but actually we break when any label has run out of examples. I don't want to change this because it may change what instances get sampled, which would make our existing evaluations hard to reproduce.

Despite the caveats, I do agree that @dtsip's solution is the most reasonable generalization...