Repository: stanford-crfm/helm
Issue: MSMARCO: error when calculating `InformationRetrievalMetric`
Root cause: Validation Gap

The logic [here](https://github.com/stanford-crfm/benchmarking/blob/main/src/benchmark/multiple_request_metrics.py#L329) we assume there is at least one token in the completion, but there are examples of responses where there aren't any tokens:

### Example 1

RunSpec:

`"msmarco:task=passage,model=openai/text-curie-001,track=trec,use_qrels_passages=True,use_topk_passages=True,valid_topk=30": {status: "READY", priority: 1}`

Request:

`Request(model='openai/text-curie-001', prompt="Passage: Search for foreclosed homes for sale in Mokena, Will County, Illinois.\nQuestion: what county is mokena in?\nPrompt: Does the passage above answer the question?\nAnswer: Yes\n\nPassage: Original conversation. User: What happens to Bob Ewell at the end of the novel? A. We do not know what happens to him. B. He goes to prison for threatening Scout and Jem. C. He commits suicide. D. He is stabbed by Boo Radley Weegy: A. He is stabbed by Boo Radley cinprincess07|Points 10|User: Which character does the mockingbird best represent?. We do not know what happens to him. B. He goes to prison for threatening Scout and Jem. C. He commits suicide. D. He is stabbed by Boo Radley Weegy: A. He is stabbed by Boo Radley cinprincess07|Points 10|User: Which character does the mockingbird best represent?\nQuestion: what is boo!\nPrompt: Does the passage above answer the question?\nAnswer: No\n\nPassage: Mini Bio (1) Mehmet Oz was born on June 11, 1960 in Cleveland, Ohio, USA as Mehmet Cengiz Oz. He is known for his work on The Dr. Oz Show (2009), You: The Owner's Manual (2005) and Today (1952). He has been married to Lisa Oz since June 29, 1985. They have four children.\nQuestion: what is doctor oz's birth name\nPrompt: Does the passage above answer the question?\nAnswer: Yes\n\nPassage: Results. A thyroid-stimulating hormone (TSH) blood test is used to check for thyroid gland problems. The normal values listed here-called a reference range-are just a guide.These ranges vary from lab to lab, and your lab may have a different range for what's normal. thyroid-stimulating hormone (TSH) blood test is used to check for thyroid gland problems.\nQuestion: what is the test called for thyroid\nPrompt: Does the passage above answer the question?\nAnswer: No\n\nPassage: Pontamine fast scarlet 4B: a new fluorescent dye for visualising cell wall organisation in radiata pine tracheids. 1  Jimmy ThomasAffiliated withSchool of Biological Sciences, The University of CanterburyCentral Wood Testing Laboratory, The Rubber Board.\nQuestion: tracheids are part of _____.\nPrompt: Does the passage above answer the question?\nAnswer:", temperature=0.0, num_completions=1, top_k_per_token=1, max_tokens=100, stop_sequences=['\n'], echo_prompt=False, top_p=1, presence_penalty=0, frequency_penalty=0, random=None)`


Response:

`RequestResult(success=True, completions=[Sequence(text='', logprob=0, tokens=[])], cached=True, request_time=0.29157495498657227, error=None)`


### Example 2

RunSpec:

`"msmarco:task=passage,model=openai/text-curie-001,track=trec,use_qrels_passages=False,use_topk_passages=True,valid_topk=30": {status: "READY", priority: 1}`

Request:

`Request(model='openai/text-curie-001', prompt="Passage: Search for foreclosed homes for sale in Mokena, Will County, Illinois.\nQuestion: what county is mokena in?\nPrompt: Does the passage above answer the question?\nAnswer: Yes\n\nPassage: Original conversation. User: What happens to Bob Ewell at the end of the novel? A. We do not know what happens to him. B. He goes to prison for threatening Scout and Jem. C. He commits suicide. D. He is stabbed by Boo Radley Weegy: A. He is stabbed by Boo Radley cinprincess07|Points 10|User: Which character does the mockingbird best represent?. We do not know what happens to him. B. He goes to prison for threatening Scout and Jem. C. He commits suicide. D. He is stabbed by Boo Radley Weegy: A. He is stabbed by Boo Radley cinprincess07|Points 10|User: Which character does the mockingbird best represent?\nQuestion: what is boo!\nPrompt: Does the passage above answer the question?\nAnswer: No\n\nPassage: Mini Bio (1) Mehmet Oz was born on June 11, 1960 in Cleveland, Ohio, USA as Mehmet Cengiz Oz. He is known for his work on The Dr. Oz Show (2009), You: The Owner's Manual (2005) and Today (1952). He has been married to Lisa Oz since June 29, 1985. They have four children.\nQuestion: what is doctor oz's birth name\nPrompt: Does the passage above answer the question?\nAnswer: Yes\n\nPassage: Results. A thyroid-stimulating hormone (TSH) blood test is used to check for thyroid gland problems. The normal values listed here-called a reference range-are just a guide.These ranges vary from lab to lab, and your lab may have a different range for what's normal. thyroid-stimulating hormone (TSH) blood test is used to check for thyroid gland problems.\nQuestion: what is the test called for thyroid\nPrompt: Does the passage above answer the question?\nAnswer: No\n\nPassage: In this study, P4B was investigated as a label for the secondary cell walls of radiata pine tracheids. Although the data show some significant differences to Anderson et al. (2010), P4B was confirmed as an excellent stain for cellulose in pine tracheids, notably for the S1 and S3 layers.\nQuestion: tracheids are part of _____.\nPrompt: Does the passage above answer the question?\nAnswer:", temperature=0.0, num_completions=1, top_k_per_token=1, max_tokens=100, stop_sequences=['\n'], echo_prompt=False, top_p=1, presence_penalty=0, frequency_penalty=0, random=None)`

Response:

`RequestResult(success=True, completions=[Sequence(text='', logprob=0, tokens=[])], cached=True, request_time=0.29445886611938477, error=None)`


### Diff between the two example requests:

![Screen Shot 2022-06-23 at 9 49 50 AM](https://user-images.githubusercontent.com/16793796/175352605-ef437b46-d6e7-4c34-9567-b02925a0b718.png)



@dilarasoylu Would it work if we just change the check to:
```
if request_state.result and request_state.result.completions and request_state.result.completions[0].tokens:
```