Repository: stanford-crfm/helm
Issue: questions about NarrativeQA from bug bash on 5-10
Root cause: Documentation Deficiency

- [ ] Which real NarrativeQA result are we comparing against? Are the metrics reported in the paper comparable to the ones we have here?
- [ ] Most of the model predictions seem correct, and are rephrasing of the “ground truth” answer. Is there any way to account for this?
- [ ] Why is the “num_tokens on test” increasing (varying) so much between models
- [ ] Should there be a “bleu” that is an average of bleu_1, 2, 3, 4?

Typo in the ground truth answer of input 1 (see https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=narrative_qa%2Cmodel%3Dai21_j1-jumbo).

@mertyg Not sure what changes are needed, if any, so to start it would be good to get your thoughts on these items. 

Thanks for pinging @rishibommasani ! Below are some thoughts:

**Comparison:**
I have been looking into what might be meaningful comparisons we can think about. Couldn’t find papers that evaluate LLMs on NarrativeQA, but below are some papers:

0. As a reference here, our evaluations achieve: Rouge-L : 0.676 ([ai21_j1-jumbo](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=narrative_qa%2Cmodel%3Dai21_j1-jumbo)), 0.701([openai_davinci](https://crfm-models.stanford.edu/static/benchmarking.html?runSpec=narrative_qa%2Cmodel%3Dopenai_davinci))
1. [https://arxiv.org/pdf/2202.12359.pdf](https://arxiv.org/pdf/2202.12359.pdf) uses Rouge-L as the evaluation metric. Their model (UnifiedQA) is specialized for question answering tasks, their best model achieves 0.674
2. Paperwithcode shows Masque([https://aclanthology.org/P19-1220.pdf](https://aclanthology.org/P19-1220.pdf)) to be the SOTA, and it achieves 0.599 Rouge-L.
3.  The original NarrativeQA paper ([https://arxiv.org/pdf/1712.07040.pdf](https://arxiv.org/pdf/1712.07040.pdf)) reports a few baselines (IR/Seq2Seq models); we get much better performance, however I’m not sure if these are meaningful comparisons.

Overall, I’m still looking for a paper that evaluates LLMs on NarrativeQA, I hope I can find it. Also let me know if there is anyone who might know more about this. But looking at other papers I’d say our results are in a reasonable range for the task.

**Rephrasings of the ground truth answer:** I agree with this, we also discussed similar things earlier at some point with @dtsip in general for QA tasks/robustness. Among the papers I have looked at, I could not find one that talks about / addresses this issue, and I am not sure if there is anything we can do here. 

**Bleu**: In papers, I have only seen Bleu-1 and Bleu-4 being reported ([https://arxiv.org/pdf/1712.07040.pdf](https://arxiv.org/pdf/1712.07040.pdf), [https://aclanthology.org/P19-1220.pdf](https://aclanthology.org/P19-1220.pdf)) for this task. 

**Num_tokens on test**: I am not sure exactly, but is this specific to NarrativeQA, though? Looking at natural_qa or news_qa, I think we observe the same behavior, which makes me think that this has more to do with different models having different tokenizers.

**Typo**: I just verified it is a typo in the dataset and not a processing error on our part. (see the file [https://github.com/deepmind/narrativeqa/blob/master/qaps.csv](https://github.com/deepmind/narrativeqa/blob/master/qaps.csv))

We see the same rephrasing issue for other QA scenarios. Metrics such as F1 help a bit because they ask if we retrieve roughly the right words. I don't think there is anything we can do that is better in general without involving humans.

Ok this seems fine, I think we can close this. Thanks for all the info @mertyg and @dtsip!

Some parting thoughts:
1. Similar to what we see in summarization (@fladhak @esdurmus), it seems that these models often do the task well but get less credit from word-overlap metrics compared to task-specific models finetuned on the task. Seems like a general point we should we make as the field starts comparing more general few-shot models to finetuned/strongly task-specialized models.
2. Seems like the results are pretty compelling if they exceed specialized QA models. Should definitely note that somewhere.