Repository: EleutherAI/lm-evaluation-harness
Issue: Clarification on Multiple Choice Question Handling in LM-Harness
Root cause: Documentation Deficiency

I am currently exploring the internal workings of the LM-Harness library and am considering contributing to its documentation to help clarify some of its mechanisms for future users.

I have a query regarding the handling of multiple-choice questions (MCQs) within the library. From what I gather, each MCQ is broken down into separate inputs corresponding to each choice, and each is then evaluated by the model. Could you please confirm if my understanding is correct, and elaborate on how exactly the library processes these inputs to select the most probable answer?

To illustrate my question with an example:

```
Context: "For a physics exam:"
Question: "What is the speed of light?"
Choices:

A. "300,000 km/s"
B. "150,000 km/s"
C. "299,792 km/s" (Correct)
D. "None of the above"
```

As I understand it, the library constructs individual requests for each choice as follows, using [task.py:972](https://github.com/EleutherAI/lm-evaluation-harness/blob/34cded30e6c0f13ad682f37f932e1573a36cc410/lm_eval/api/task.py#L972)

- "For a physics exam: What is the speed of light? A: 300,000 km/s"
- "For a physics exam: What is the speed of light? B: 150,000 km/s"
- "For a physics exam: What is the speed of light? C: 299,792 km/s"
- "For a physics exam: What is the speed of light? D: None of the above"

Each of these inputs is then fed into the model. Could you clarify whether the library simply extracts the first token's response, or if there's a more complex mechanism at play involving the vocabulary to determine which choice is the most likely correct answer?

Then the answers are fed into the model.

> For a physics exam: What is the speed of light? A: 300,000 km/s -> model

and output might look like this?

`{'created': 1721875422, 'model': 'llama3.1', 'usage': {'prompt_tokens': 82, 'completion_tokens': 58, 'total_tokens': 140}, 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': "That's correct! The speed of light in a vacuum is approximately 300,000 kilometers per second (km/s), which is equivalent to about 186,282 miles per second (mi/s). This is a fundamental constant in physics, denoted by the letter c. Well done!", 'function_call': None}, 'finish_reason': 'stop', 'logprobs': {'tokens': ['That', "'s", ' correct', '!', ' The', ' speed', ' of', ' light', ' in', ' a', ' vacuum', ' is', ' approximately', ' ', '300', ',', '000', ' kilometers', ' per', ' second', ' (', 'km', '/s', '),', ' which', ' is', ' equivalent', ' to', ' about', ' ', '186', ',', '282', ' miles', ' per', ' second', ' (', 'mi', '/s', ').', ' This', ' is', ' a', ' fundamental', ' constant', ' in', ' physics', ',', ' den', 'oted', ' by', ' the', ' letter', ' c', '.', ' Well', ' done', '!', '
'], 'token_logprobs': [-0.33203125, -0.6328125, -0.7578125, -0.043945312, -0.0022125244, -0.00019073486, -3.2186508e-06, -7.1525574e-06, -0.38867188, -0.22558594, -8.821487e-06, -0.000111579895, -0.034179688, -0.0012207031, -0.5234375, -3.0994415e-05, -1.180172e-05, -0.036865234, -4.4107437e-06, -1.9669533e-05, -0.003967285, -0.004547119, -5.722046e-06, -1.9140625, -1.3203125, -0.011108398, -0.625, -3.2186508e-06, -2.328125, -0.00016212463, -0.0390625, -2.2411346e-05, -0.043945312, -8.8214874e-05, -5.2452087e-06, -0.0003414154, -0.43554688, -0.005218506, -1.0609627e-05, -0.1328125, -1.1796875, -0.58203125, -0.021240234, -0.00064468384, -0.004699707, -0.20214844, -0.0022125244, -0.8359375, -0.21386719, -5.9604645e-07, -0.0040893555, -0.0001745224, -0.1796875, -0.16601562, -0.30273438, -0.89453125, -0.00013256073, -0.035888672, -0.13671875], 'token_ids': [4897, 596, 4495, 0, 578, 4732, 315, 3177, 304, 264, 29302, 374, 13489, 220, 3101, 11, 931, 41668, 824, 2132, 320, 16400, 2754, 705, 902, 374, 13890, 311, 922, 220, 9714, 11, 16544, 8931, 824, 2132, 320, 8318, 2754, 570, 1115, 374, 264, 16188, 6926, 304, 22027, 11, 3453, 9437, 555, 279, 6661, 272, 13, 8489, 2884, 0, 128009]}}]}`

Additionally, I am curious about how the library calculates the accuracy (ACC) from the log probabilities obtained for each choice. Specifically, are we considering the probability of the first token, or are we summing the probabilities across tokens? Could you direct me to the specific lines of code in the LM-Evaluation-Harness that handle this calculation, and provide a brief explanation of the method used?



Hi! I think this discussion should answer your question: https://github.com/EleutherAI/lm-evaluation-harness/issues/942

with respect to #2135, likelihood based tasks aren't supported with chat-completion endpoints as yet. I have made a PR to make that clearer. Also I don't think it's possible with the api you are using as it doesn't seem to support an [echo](https://platform.openai.com/docs/api-reference/completions/create#completions-create-echo) like arg where the logprobs of the _prompt_ (rather than the model generations) are returned. 



@baberabb Thank you for the response and for pointing to the references. I wanted to confirm: if the API includes an `echo` feature (returning the log probabilities of the prompt as well), will it be compatible with the `lm-harness`? Is there anything else we need to include in the API output to ensure compatibility with the LM harness?

yeah thats the main one. And not many APIs support echo with chat inputs tbf. together is the only one I know of. I don't think even vllm supports it with their chat-completions endpoint iirc.

@baberabb could you confirm if @monk1337  is right about the question formatting?
If yes, does that mean that harness will receive these options (phrases) individually and you get the likelihood of each option and then , the highest likelihood becomes the correct answer?

@baberabb I have the same question as @monk1337 & @owos about expected payload formatting to get loglikelihoods for all choices. Can you please confirm? 
I am trying to evaluate my model hosted on a rest endpoint. In `_create_payload` function the `messages` argument only contains the question and does not include the option. It does get called as many times as the choices are for each question, but each time with only the question. This is counterintuitive to the explanation of how models get evaluated for multiple choice tasks. Any insight is highly appreciated. Thanks! 

@pallavis-cdns related discussion: https://github.com/EleutherAI/lm-evaluation-harness/pull/1425#issuecomment-2602674569