Repository: stanford-crfm/helm
Issue: Compare GPT-J offline v2 results to GooseAI's GPT-J
Root cause: Documentation Deficiency


To resolve this, I looked at 9 scenarios that provide coverage of all generic scenarios, looking at the comparisons between Goose and Together for GPT-J. I also looked at Together for GPT-NeoX.

Ordering: model=gooseai/gpt-j-6b | model=together/gpt-j-6b | model=together/gpt-neox-20b

QA:
1. [BoolQ](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*boolq.*gpt-(j%7Cneox).*) 
quasi_exact_match on valid	0.647	0.644	0.644
quasi_exact_match on valid with identity(robustness=false, fairness=false)	0.661	0.649	0.665
2. [QuAC](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*quac.*gpt-(j%7Cneox).*)
Did not load
3. [HellaSwag](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*hella.*gpt-(j).*)
exact_match on valid	0.242	0.256	0.258
4. [(MMLU abstract algebra)](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*mmlu%3Asubject%3Dabstract_algebra.*gpt-(j%7Cneox).*)
exact_match on valid	0.286	0.238	0.286
exact_match on valid with identity(robustness=false, fairness=false)	0.364	0.364	0.273
5. [NaturalQuestions (closed book)](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*natural_qa%3Amode%3Dclosedbook.*gpt-(j%7Cneox).*)
exact_match on valid	0.003	0.064	0.085
f1_score on valid with identity(robustness=false, fairness=false)	0.102	0.165	0.2

Summarization:
1. [CNN/DM](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*cnn.*gpt-(j%7Cneox).*)
rouge-2 on test	0.154	0.158	0.144
2. [XSUM](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*xsum.*gpt-(j%7Cneox).*)
rouge-2 on test	0.053	0.084	0.078

IR:
1. [MS MARCO (regular)](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*marco.*regular.*True%2Cuse.*gpt-(j%7Cneox).*) 
RR@20	0.1	0.062	0.094
2. [MS MARCO (TREC)](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*marco.*trec.*True%2Cuse.*gpt-(j%7Cneox).*)
NDCG@20	0.293	0.215	0.268


**TL:DR:** For GPT-J, the model predictions are not identical (Input 15 is the first such instance for BoolQ - see [here](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=.*boolq.*gpt-j.*)). 
The performance can vary by a nontrivial amount (e.g. 0.08 on NDCG in IR), and is not consistent (e.g. Goose does better for MS MARCO regular but NDCG does better for TREC). 
I had hoped to use GPT-NeoX to give some guidance, expecting it to generally better and the delta with it to be potentially larger than the GPT-J variants. This is not true in any particularly consistent way; NeoX is bet marginally better if at all given these 9 evaluations. And it is not closer to the other Together model (not that there is a specific reason it should be). 

So, seems like the models are similar but not really the same or epsilon away from being the same. 


In general, GPT-J (both variants) and GPT-Neox (Together) seem to do quite bad. E.g. HellaSwag/MMLU is close to chance afaik, natural questions seems pretty bad; I have worked on summarization and such low ROUGE-2s are terrible (though the summaries are occasionally better than scores suggest) and idk IR well but seems like IR performance is also bad. 

"All the models" corresponds to just the 3 listed here, right?

yup, updated the comment!

Update: We had planned to also compare results on language modeling scenarios, but we did not run these on the Goose models (at least in the most recent runs; if we have older results, I can go find them but didn't see either on the website or on the cluster (looked at `/u/scr/nlp/crfm/benchmarking/benchmarking/benchmark_output/runs/latest` and `/u/scr/nlp/crfm/benchmarking/benchmarking/benchmark_output/runs/08-09-2022`, of which the latter has empty output folders for the goose models for language modeling scenarios). 

So, I think we can close this issue in the sense that it seems clear that the models can be considerably different (Percy also provided further evidence on Slack). Or if we want to do something specific to further fix the issue, we can discuss at the meeting.

Ok, I am closing this since we know they are different. And we will instead verify a different more pragmatic condition, i.e. that our results match up with metrics measured in the GPT-J better (i.e a particular special case of item 2 in #746). 

Tagging @percyliang just in case he thinks we should do something else.