Repository: EleutherAI/lm-evaluation-harness
Issue: Strange negative log likelihoods when predicting single token options
Root cause: Algorithmic Error

I tried to evaluate several models on several multiple choice QA datasets (including MMLU and some custom datasets) and I realized some strange results in the resulting negative log likelihoods for the options. Specifically, i observed one or several of the following in many questions:

- Multiple options have exactly the same likelihood
- Multiple options have exactly the same decimal number (whole 15 decimal points) with different  whole numbers
- Multiple options have the same decimal number up to some number of digits  and they differ at smaller decimal locations (e.g., -2.7487025260925293 and -1.7487024068832397)
- Multiple options have the same decimal number starting after some number of digits (This one is particularly strange since I cannot think of a natural mathematical operation that will yield numbers with different initial n decimal digits followed by the same digits such as -2.2487025260925293 and -2.7487025260925293)

I am wondering if this is a normal and expected behavior that can easily be explained or due to a bug in the code. It seems that this  does not happen if the targets are multiple tokens such as the ones in mmlu_flan_n_shot_loglikelihood datasets (e.g., target is ' (A)' instead of ' A') 

Below is an example result I obtained for google/gemma-7b model on mmlu_anatomy dataset.

> {
>     "doc_id": 29,
>     "doc": {
>       "question": "Which statement concerning the atrioventricular bundle is correct?",
>       "subject": "anatomy",
>       "choices": [
>         "It is formed of autonomic nerve processes and is the only electrical connection between the atria and the ventricles.",
>         "It is formed of Purkinje fibres and is the only electrical connection between the atria and the ventricles",
>         "It is formed of autonomic nerve processes and forms one of several electrical connections between the atria and ventricles.",
>         "It is formed of Purkinje fibres and forms one of several electrical connections between the atria and ventricles."
>       ],
>       "answer": 1
>     },
>     "target": 1,
>     "arguments": [
>       [
>         "The following are multiple choice questions (with answers) about anatomy.\nWhich statement concerning the atrioventricular bundle is correct?\nA. It is formed of autonomic nerve processes and is the only electrical connection between the atria and the ventricles.\nB. It is formed of Purkinje fibres and is the only electrical connection between the atria and the ventricles\nC. It is formed of autonomic nerve processes and forms one of several electrical connections between the atria and ventricles.\nD. It is formed of Purkinje fibres and forms one of several electrical connections between the atria and ventricles.\nAnswer:",
>         " A"
>       ],
>       [
>         "The following are multiple choice questions (with answers) about anatomy.\nWhich statement concerning the atrioventricular bundle is correct?\nA. It is formed of autonomic nerve processes and is the only electrical connection between the atria and the ventricles.\nB. It is formed of Purkinje fibres and is the only electrical connection between the atria and the ventricles\nC. It is formed of autonomic nerve processes and forms one of several electrical connections between the atria and ventricles.\nD. It is formed of Purkinje fibres and forms one of several electrical connections between the atria and ventricles.\nAnswer:",
>         " B"
>       ],
>       [
>         "The following are multiple choice questions (with answers) about anatomy.\nWhich statement concerning the atrioventricular bundle is correct?\nA. It is formed of autonomic nerve processes and is the only electrical connection between the atria and the ventricles.\nB. It is formed of Purkinje fibres and is the only electrical connection between the atria and the ventricles\nC. It is formed of autonomic nerve processes and forms one of several electrical connections between the atria and ventricles.\nD. It is formed of Purkinje fibres and forms one of several electrical connections between the atria and ventricles.\nAnswer:",
>         " C"
>       ],
>       [
>         "The following are multiple choice questions (with answers) about anatomy.\nWhich statement concerning the atrioventricular bundle is correct?\nA. It is formed of autonomic nerve processes and is the only electrical connection between the atria and the ventricles.\nB. It is formed of Purkinje fibres and is the only electrical connection between the atria and the ventricles\nC. It is formed of autonomic nerve processes and forms one of several electrical connections between the atria and ventricles.\nD. It is formed of Purkinje fibres and forms one of several electrical connections between the atria and ventricles.\nAnswer:",
>         " D"
>       ]
>     ],
>     "resps": [
>       [
>         [
>           -2.6825902462005615,
>           false
>         ]
>       ],
>       [
>         [
>           -2.6825902462005615,
>           false
>         ]
>       ],
>       [
>         [
>           -1.6825902462005615,
>           false
>         ]
>       ],
>       [
>         [
>           -0.6825901865959167,
>           true
>         ]
>       ]
>     ],
>     "filtered_resps": [
>       [
>         -2.6825902462005615,
>         false
>       ],
>       [
>         -2.6825902462005615,
>         false
>       ],
>       [
>         -1.6825902462005615,
>         false
>       ],
>       [
>         -0.6825901865959167,
>         true
>       ]
>     ],
>     "acc": 0.0
>   },


Below is how called lm_eval:

> SHOTS=0
> MODEL=google/gemma-7b
> lm_eval --model hf \
>         --model_args pretrained=${MODEL},parallelize=True \
>         --tasks 'mmlu_anatomy' \
>         --batch_size auto \
>         --num_fewshot $SHOTS \
>         --output_path output-MMLU/${MODEL} \
>         --log_samples \
>         --limit 50 





Replied in #1672 !