Repository: EleutherAI/lm-evaluation-harness
Issue: Mismatch Between Prompt Format and Expected Choices in LogiQA Dataset
Root cause: Interface Contract Mismatch

In the file lm_eval/tasks/logiqa/utils_logiqa.py and lm_eval/tasks/logiqa/logiqa.yaml, there seems to be a mismatch between the expected choices for the LogiQA dataset and the format of the prompt presented to the language model.
* Prompt Format: The prompt, however, is structured with options labeled as "A.", "B.", "C.", and "D.".
* Expected Choices: The code defines the possible choices as [0, 1, 2, 3].

The Llama 3B model performs significantly better when choices align with the prompt format, using letters ["A", "B", "C", "D"] instead of numbers.

Using four RTX A6000 GPUs, the results are as follows.
| Choices | Llama-3-8B Acc(%) | Llama-3-8B-Instruct Acc(%) |
|---|---|---|
| ["0", "1", "2", "3"] | 27.50 | 27.65 |
| ["A", "B", "C", "D"] | 39.78 | 41.32 |
___
Possible Solutions:
* Option 1: Change the choices to ["A", "B", "C", "D"] in the code.
* Option 2: Modify the prompt to present the choices as numbers ("0.", "1.", "2.", "3.").

I Think this issue originates from the original dataset itself: https://github.com/lgw863/LogiQA-dataset/tree/master .
In the original dataset, the labels are written in lowercase letters.
However, in the current lm-evaluation-harness, the labels are not even lowercase letters; they are represented as numbers.

As [the paper](https://arxiv.org/pdf/2007.08124) clarifies, the prompts used are in uppercase letters.
Therefore, I believe that changing the numerical choices to uppercase choices is a more reasonable option than changing the prompts.

So, I will create a pull request to address this issue.
The proposed change will align the expected choices with the prompt format,  Change the choices to ["A", "B", "C", "D"] in the code.
Correcting it should lead to more accurate assessments of language model performance on this task.

Let me know if you'd like any further refinements!

Hi! So the way it's set up now is that choices are indexing the actual choice strings. So the model is prompted with the full string rather than just the letter corresponding to the answer choices, for example:
```
<Passage>
<Question>
A. <Chioce A>
B. <Choice B>
C. <Choice C>
Answer: <Choice A>
```
We could add a variant with just the letter continuations. Are you aware if this approach has been used for LogiQA anywhere else recently?

https://github.com/EleutherAI/lm-evaluation-harness/pull/1888

@baberabb 
~~Well.. The issue I've identified is that the current task setup uses 0, 1, 2, 3 as labels, which means the model is considering the logits of the tokens corresponding to 0, 1, 2, 3 as its answers. I think this is an implementation error and that, like other tasks (e.g., [MMLU](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu)), the labels should be handled as letters "A", "B", "C", "D".~~

Oh, I misunderstood. You were right that doc["options"] was the continuation.
Still, we could add a separate task, like MMLU added continuation to the A, B, C, D questions, but as you said, without a reference to LogiQA, it doesn't seem meaningful.