Repository: stanford-crfm/helm
Issue: Unexpected or unjustified adaptation parameters
Root cause: Documentation Deficiency

In my pass over all the adapter specs, I identified several specs with unexpected/unjustified adaptation parameters. 

The authors of each of these scenarios should modify these specs to resolve the @ tagging in https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/presentation/run_specs.conf as in #312. 
To identify, simple `CTRL+F` for @your_name in the `.conf` file.


@michiyasunaga - Several scenarios for knowledge team
@neelguha - Wiki and Legal Help scenarios
@hyren - Wiki
@tonywu95 - Several scenarios for reasoning team
@ezelikman - GSM
@friedeggs - MATH; also the numeracy adapter spec looks quite different/doesn't specify a temperature, etc., but maybe that is intentional
@dorarad - BABI, LSAT
@suzgunmirac - Dyck
@lxuechen - Copyright
@bnewm0609 - Disinformation
@ameliahardy @AshwinParanjape @inescgu - Empathetic Dialogue

**Of especial importance is scenarios with a non-zero temperature.** If the temperature should be non-zero, please state why it is set the way it is as a justification. Or, if it should be zero, please change. This relates to #244. 
**Please check the box once resolved and comment below**

- [x] Wiki - @michiyasunaga @neelguha @hyren 
- [x] Synthetic Reasoning and Synthetic Reasoning Natural - @tonywu95 (will resolve on Thursday)
- [x] GSM - @ezelikman 
- [x] Numeracy - @friedeggs (will resolve on Thursday)
- [x] Copyright - @lxuechen 
- [x] Disinfo - @bnewm0609 

**Of especial importance is scenarios with no stop sequences** If there should be no stop sequences or something other than `"\n"`, barring language modeling scenarios, please state why it is set the way it is as a justification. Or, if it should be `["\n"]`, please change. This relates to #242. 
**Please check the box once resolved and comment below**

- [x] GSM - @ezelikman 
- [x] Numeracy - @friedeggs (will resolve on Thursday)
- [x] MATH - @friedeggs (will resolve on Thursday)
- [x] LSAT - @dorarad  (will be resolved in #434)
- [x] Copyright - @lxuechen 
- [x] Disinfo - @bnewm0609 
- [x] Code - @q-hwang (will resolve on Thursday)
- [x] XSUM + CNN/DM - @fladhak 
- [x] Empathetic Dialogues - @ameliahardy @AshwinParanjape @inescgu (changed in the `interactive-mode` branch)


I think both MATH and numeracy scenarios set temperature to 0.0 and have at least "\n" as a stop sequence. @rishibommasani I see your comments by the MATH adapter spec definition and can add justifying comments. (One of them is the same question flagged by @teetone earlier that we were discussing in #141.) I think for numeracy, the reason it looks non-standard is because it's using a helper function defined in the scenario (where it's also used so that's why it's there), so you'll see the temperature and stop sequence are defined there. Is there anything else I'm missing? The run_specs.conf file has a number of TODOs left by you on deciding which subtasks to run (I also have those questions) and I agree on deciding based on results. The [other two](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/presentation/run_specs.conf#L143-L144) I can discuss with @tonywu95. But yes is there anything else besides adding the two lines of justifying comments to do before the run on Wednesday ie tomorrow?

@rishibommasani About GSM8K:
- In the verifier paper they use one temperature for test@1 (greedy) and another temperature for test@100 (t=0.7). The paper "Self-Consistency Improves Chain of Thought Reasoning in Language Models" also showed that taking several higher-temperature (t=0.7) samples with reasoning and then selecting the answer via majority vote is far better than greedy sampling. What do you think we should do in this situation? It's not a standard QA scenario since coming up with the open-ended reasoning is a key part of it.
- We do 3 examples for few-shot since the GSM8K problems are sometimes long and this ensures that there should be fewer than 1024 tokens. Is that a reasonable justification?
- We use two newlines as the stop sequence because the model can output multiple lines as part of its answer. As far as I know, there's no standard. Is that a reasonable justification?

## Disinformation
* `temperature=0.7` for `reiteration` comes from the implementation of the CSET evaluation's [narrative amplification](https://github.com/georgetown-cset/GPT3-Disinformation/blob/main/Narrative_Amplification/Pull_Climate_Skepticism.ipynb) setting, and the temperature for `wedging` comes from their implementation of [narrative wedging](https://github.com/georgetown-cset/GPT3-Disinformation/blob/main/Narrative_Wedging/NarrativeWedging_Christian.ipynb).
* no stop sequences: during development, we noticed that some generations began with `\n`, so including it as a stop token would lead to an empty generation. I can revert this, but some generations will be empty when they shouldn't be. Should we prefer empty generations to hitting the token limit?
* Justifications and other standardizations are in #318. 

## Knowledge team

Submitted a PR (https://github.com/stanford-crfm/benchmarking/pull/319) that addresses the TODO items for knowledge team. 
In particular, the reason we use temperature=1.0 for wikifact is we found that with temperature=0 the model generates highly similar answers for the top k predictions, and we need temperature=1.0 to get more diverse answers.


The copyright scenario aspect will be addressed in PR #314; more is described there.

For code: the stop sequences for humanEval are taken from the openai paper. It is to prevent the further generation of irrelevant classes/functions so that the code is executable as a single function. For APPS, it is for similar reason, except that @lxuechen manually selected the stop tokens since they are not mentioned in the paper.

@q-hwang @lxuechen - Cool, can we specify these reasons in the comments in the code and submit a PR (that also gets rid of the current TODOs)? You can tag me as a reviewer since it should be very fast to merge. One you submit the PR, can you check off the items in the above checklist. 

@lxuechen - Cool, I am going to check off copyright then! 






@bnewm0609 - Looks like this is being resolved in PR #318. For stop sequences, I think it is fine to do the more token-intensive option, but still document we did this intentionally/why in the code (i.e. we didn't just accidentally forget to supply stop sequences). In general, I think it is fine for us to do different things as long as we state the reasons clearly in the code (and eventually the paper), so if someone disagrees with our implementation, it is at least clear why we made certain design decisions. 

@ezelikman 
1a. By default, it might be simpler if we just take a single sample as the standard configuration in the benchmark and we separately do ablations/analyses we want to do (in the paper). Since I think simplicity will be pretty important for adoption of the benchmark (e.g. consider how SuperGLUE or ImageNet are very straightforward for run), but we should definitely do deeper analyses we want.

1b. For the base/default/standard configuration, shouldn't temperature be 0 not 1 then for greedy (I didn't check the paper, so sorry if this is a silly question)?
 
2. Using fewer in-context examples is fine for context-length reasons; can you document this justification in the code?
3. 2 new lines makes sense; can you document this justification in the code?

**Action items:**
Once you add/change the justifications in `run_specs.conf` (and remove the existing TODOs that tag you in the file), can you submit a PR + check off the associated scenarios in the above checklist? You can assign me as a reviewer and it should be a very fast merge. 
 
 

@friedeggs 
Ah sorry, I looked through it quickly. I guess the things I intended for were:
1. MATH has additional non-standard stop sequences. They seem sensible, but can you replace the TODO on that line with the justification for why they were added?
2. For numeracy, that makes sense, I had not realized. With that said, for uniformity, can we move the adapter_spec into the main `run_specs` file? If there is a reason we cannot, that's fine, but maybe we should at least note it in the run_specs file, since afaik it is the only spec with an adapter_spec specified elsewhere. 
3. Minor: I think `delimiter` is misspelled in some places (as `delimeter`). 
4. Resolving the rest based on results makes sense to me, thanks!

**Action items:**
If there are any changes (probably need to update the comments minimally, at least to replace TODOs with justifications), can you submit a PR + check off the associated scenarios in the above checklist? You can assign me as a reviewer and it should be a very fast merge.

Empathetic Dialogue stop sequence change is done in the `interactive-mode` branch (CC @ameliahardy @AshwinParanjape @inescgu)

Commenting re: checking off tasks:
- numeracy settings do indeed use `temperature=0` and `stop_sequences=['\n']`
- added justification comment for MATH stop_sequences (Reproduces [setting from official MATH codebase](https://github.com/hendrycks/math/blob/main/modeling/evaluate_gpt3.py#L31))