Repository: stanford-crfm/helm
Issue: Adapter to handle multiple choice where we make one request per choice
Root cause: Unimplemented Feature Gap

Currently, all options are provided in the prompt (MMLU-style).  But for HellaSwag (where references are continuations of the input), much better to use "Causal LM" style.  Currently, this is special-cased for CommonsenseQA, but this should be made more generic.  Should switch BLIMP over to using this too.

Making this P2 since it is front-end/shouldn't affect the results as long as it doesn't mess with caching. 

This should be a `p1` because it changes results and we need to recache.

@percyliang Thanks to @yuhui-zh15's fantastic work, this is now addressed in terms of the capacity to do this nicely.

What remains is to decide what scenarios we run with which adaptation method and to modify the conf to reflect. Thoughts? Currently we have it set to the joint approach as before (e.g. https://github.com/stanford-crfm/benchmarking/blob/91df2960971889742136e72129e392dc72614e40/src/benchmark/presentation/run_specs.conf#L31). 

I think all LM-like tasks like HellaSwag where the reference is meant to be a continuation of the input, we should do SEPARATE (the version that works best according to the literature), and for everything else, we do JOINT.

@teetone Have you launched the run/we should make this change in the conf.

@percyliang To be precise, do you mean separate_calibrated instead of spearate_original? The former does considerably better from Yuhui's reporting in #604. I am also not quite sure I know what the condition you are saying is. For HellaSwag, sure, it is clear, since in general the input ends with some sentence and the answer is the following sentence(s). But for something where the input ends with a question, the choices could be follow-ups naturally but it is less unequivocal. For now, I have noted those as Joint and described my reasoning below. 

We then need to decide on the specific changes. 
- [x] BBQ - Joint (Input ends with an explicit question)
- [x] MMLU - Joint (Would follow MMLU paper; have not looked at all subset but in `abstract_algebra` you have something like "Find the order of some finite ring.")
- [x] HellaSwag - Separate
- [x] OpenbookQA - Separate
- [x] TruthfulQA - ? (Ends in a question, but seems like the answers are natural continuations in some cases). See screenshot
![image](https://user-images.githubusercontent.com/47439426/179915345-3743192b-2605-47d4-923f-0495850b2a68.png)
- [x] LSAT_QA - Joint (Input ends with explicit question, though it could be continuation?)
- [x] BLiMP - @teetone are we running this currently?
- [x] Legal_Support - Joint (seems like separate doesn't make sense. Independently, while on the topic, @neelguha, does it make sense that the question is in the instruction (i.e. at the start) rather than as a prefix right before the model has to output the answer? Seems like the model has to do some extra work to remember what question it needs to answer, though I could see this making sense if the point is to prime the model to think about this while reading the passage.)

Also noting this may significantly increase token expenditure, since the majority of our prompts is the inputs in in-context examples (afaik), and now for every test instances we have increased these tokens from `5 x avg. input length` to `5 x num_answer_choices x avg.input_length`, which is a four-fold increase for HellaSwag, for example. This shouldn't be quite as bad as just scaling the number of tokens per scenario by `num_answer_choices` but should not be far off afaik. 



**TL;DR: Here are the action items.**
- [x] @percyliang @rishibommasani - Converge to the adaptation method for every scenario listed above.
- [x] @rishibommasani - Update `run_specs.conf` once the decisions are made.
- [ ] @teetone - Run with the updated `conf` (or just run the stuff that changes, seems like it will only be 2-3 things changing, so not too big of a deal). 

**Independent matters.**
- [x] @YianZhang - We are currently not running BLiMP because of the priorities, but maybe we should update to run at least something. Can we pick a few things to maximize diversity/coverage of linguistic phenomena based on how stuff was categorized in the BLiMP paper and update the conf priorities to reflect this?  Edit: @rishibommasani made some decisions on this with noted justifications in #659. 
- [x] @neelguha - Is the prompt formatting for LegalSupport what we want (see https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=legal_support%3Amodel%3Dai21_j1-jumbo as an example)? This is the last moment to finalize, so should make sure. While you are at it, can you all address the TODO in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_specs.py#L1425 where it says `@Neel`?

@rishibommasani Sorry for the delayed response! Completely missed these notifications. 

>  Legal_Support - Joint (seems like separate doesn't make sense. Independently, while on the topic, @neelguha, does it make sense that the question is in the instruction (i.e. at the start) rather than as a prefix right before the model has to output the answer? Seems like the model has to do some extra work to remember what question it needs to answer, though I could see this making sense if the point is to prime the model to think about this while reading the passage.)

I thought it would work better for the same reason–defining the the task up front and priming the model with the "purpose" of the passages before it actually process them. If you think it's better to switch however, I can do it. I don't have a good sense of which works better for this particular dataset. 

> @neelguha - Is the prompt formatting for LegalSupport what we want (see https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=legal_support%3Amodel%3Dai21_j1-jumbo as an example)? This is the last moment to finalize, so should make sure. While you are at it, can you all address the TODO in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/run_specs.py#L1425 where it says @Neel?

Prompt looks good to me. With regards to the TODO, I picked 3 because these samples tend to be a bit longer (I know 5 is generally the default). Do you want me to justify it in a comment in the code, or is this enough justification?


@neelguha Thanks, this sounds great. Can you replace the TODO in the code with the justification, both so (i) deviations in the code will be clearer to anyone using it in the future and (ii) so we can make sure to document these in the paper? You can tag me in the PR, thanks!!

Based on our meeting today, we settled on the principle that we default to `Joint` unless the input is an incomplete phrase that the answers complete.
1. OpenbookQA and HellaSwag, which require one to (sometimes) complete an unfinished phrase, will use the `Separate` approach. Further, since we are worried about acc. being too low, they will use the `Calibrated` variant from the literature, which also does considerably better in @yuhui-zh15's experiments in #604. 
2. For the remainder, since there is nothing incomplete. 

These changes are reflected in #659, which will close this issue.