Repository: stanford-crfm/helm
Issue: GLM results are worse than reported in https://arxiv.org/pdf/2210.02414.pdf
Root cause: Documentation Deficiency

we find opt > glm but glm says they are better. Look into this

Any ideas?

Also, we should look at why UL2 is underperforming for us.

**GLM vs OPT**
GLM evaluates on following:
1. Big-Bench - no OPT results, and we don't eval either
2. MMLU - no OPT results; for specific subsets we jointly eval on:
abstract algebra - .28 (us) > .24 (GLM paper)
college chemistry - .28 (us) < 34 (GLM paper)
computer security - .4 (us) < .61 (GLM paper)
econometrics - .263 (us) = .263 (GLM paper)
US foreign policy - .35 (us) < .61 (GLM paper)
3.  LAMBADA - we dont eval
4. The Pile - no OPT results, and we don't eval GLM (idk why @teetone?)

**Takeaway.** There is no relevant datasets that evaluate OPT vs GLM that we eval on (the only examples in paper afaict are LAMBADA, StereoSet, CrowS-Pairs). The bigger issue potentially is on 2 of 5 MMLU subsets, we are much worse (computer security, US foreign policy), noting that the overall average is .448 for GLM vs .315 for us here: https://nlp.stanford.edu/helm/?suite=v7&group=mmlu



Aside: UL2 - TBD with semicolon + [NLG], tracked in different issue

Go read the paper carefully and figure out what they did with prompting.

I tried to match their prompts for MMLU as closely as possible. Example prompt:


```
The following are multiple choice questions about college chemistry.
	
Which of the following statements about the lanthanide elements is NOT true? (A) The most common oxidation state for the lanthanide elements is +3. (B) Lanthanide complexes often have high coordination numbers (> 6). (C) All of the lanthanide elements react with aqueous acid to liberate hydrogen. (D) The atomic radii of the lanthanide elements increase across the period from La to Lu. Answer: (D) The atomic radii of the lanthanide elements increase across the period from La to Lu.
	
A 0.217 g sample of HgO (molar mass = 217 g) reacts with excess iodide ions according to the reaction shown above. Titration of the resulting solution requires how many mL of 0.10 M HCl to reach equivalence point? (A) 1.0 mL (B) 10 mL (C) 20 mL (D) 50 mL Answer: (C) 20 mL
	
Predict the number of lines in the EPR spectrum of a solution of 13C-labelled methyl radical (13CH3•), assuming the lines do not overlap. (A) 4 (B) 3 (C) 6 (D) 24 Answer: (A) 4
	
3 Cl−(aq) + 4 CrO_4^2−(aq) + 23 H+(aq) → 3 HClO2(aq) + 4 Cr3+(aq) + 10 H2O(l). In the reaction shown above, Cl−(aq) behaves as (A) an acid (B) a base (C) a catalyst (D) a reducing agent Answer: (D) a reducing agent
	
Which of the following lists the hydrides of group-14 elements in order of thermal stability, from lowest to highest? (A) PbH4 < SnH4 < GeH4 < SiH4 < CH4 (B) PbH4 < SnH4 < CH4 < GeH4 < SiH4 (C) CH4 < SiH4 < GeH4 < SnH4 < PbH4 (D) CH4 < PbH4 < GeH4 < SnH4 < SiH4 Answer: (A) PbH4 < SnH4 < GeH4 < SiH4 < CH4
	
Which of the following is true for Br2 at standard temperature and pressure? (A) It is a colorless gas. (B) It is a red-brown volatile liquid. (C) It is a colorless volatile liquid. (D) It is a yellow metallic solid. Answer: (
```

Changes:

- Add parens around answer choices and put them all on a single line
- Include both the letter and the full answer for the correct answers in the in-context examples
- Have the prompts end with `Answer: (`


 The numbers look more reasonable now, but we’re underperforming for `us_foreign_policy` and `computer_security`.

![glm_mmlu](https://user-images.githubusercontent.com/16793796/200964750-c622cf46-ddcf-4618-95fa-15b7b2076914.png)


@teetone did you upper-case US (since its the only thing we do worse on) for foreign policy?

> @teetone did you upper-case US (since its the only thing we do worse on) for foreign policy?

@rishibommasani Do you mean in the instructions, so:

`The following are multiple choice questions about us foreign policy.`

becomes

`The following are multiple choice questions about US foreign policy.`

That's interesting..did they mention this in the GLM paper?


um no, but I think using the capital version makes more sense? I guess idk what they implement, but using lower case seems wrong (even if it might be the natural outcome of programmatic substitution). I.e. I would expect given they probably really care about the numbers that they manually looked at the individual prompts and capitalized US (no clue though)

> um no, but I think using the capital version makes more sense? I guess idk what they implement, but using lower case seems wrong (even if it might be the natural outcome of programmatic substitution). I.e. I would expect given they probably really care about the numbers that they manually looked at the individual prompts and capitalized US (no clue though)

Yeah, that's a good point. I will try it out.

> um no, but I think using the capital version makes more sense? I guess idk what they implement, but using lower case seems wrong (even if it might be the natural outcome of programmatic substitution). I.e. I would expect given they probably really care about the numbers that they manually looked at the individual prompts and capitalized US (no clue though)

It doesn't seem to help. I got the exact same numbers.

awesome, thanks for checking. And hurray, good to hear models are robust to case! (John Hewitt anecdotally told me models like BERT could be very sensitive to this specific type of casing) 

@percyliang @teetone I am going to close this for now then?