Repository: EleutherAI/lm-evaluation-harness
Issue: Squadv2 PostProcessing Issues with unanswerable questions
Root cause: Validation Gap

Hello all,

first of all thanks to the authors for maintaining this awesome resource for benchmarking!

I'd like to evaluate a few models on squadv2 but scores for unanswerable questions (both f1 + EM) keep reading 0.0 no matter which model I use. Logging 100 samples show me that many unanswerable questions are indeed recognized as such. However, in answer post-processing the target is set to " unanswerable" instead of "unanswerable" (no whitespace) which seems to break post-processing for the mentioned scores.

For instance, Llama-3.3-70B:
```
import transformers
tok = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")
tok("unanswerable")
#{'input_ids': [128000, 359, 9399, 481], 'attention_mask': [1, 1, 1, 1]}
 tok(" unanswerable")
#{'input_ids': [128000, 653, 9399, 481], 'attention_mask': [1, 1, 1, 1]}
```
so including whitespace results in different tokens
and indeed when running:
```
lm_eval --model hf --model_args "pretrained=meta-llama/Llama-3.3-70B-Instruct,parallelize=True,enable_thinking=False,think_end_token=</think>" --tasks "squadv2" --batch_size "auto"  --max_batch_size 32 --apply_chat_template --output_path "[PATH_TO_DIR]" --system_instruction "You are a helpful assistant tasked with extractive questions answering. The user will provide you some context and a question. If the context contains an answer to the question, respond only with a word for word citation of the relevant passage. Otherwise, respond only with 'unanswerable'. What follows are 3 examples of the task." --num_fewshot "3" --fewshot_as_multiturn --log_samples --limit 100
```
gives the following scores:
```
| Tasks |Version|Filter|n-shot|   Metric   |   | Value |   |Stderr|
|-------|------:|------|-----:|------------|---|------:|---|------|
|squadv2|      3|none  |     3|HasAns_exact|↑  |62.2222|±  |   N/A|
|       |       |none  |     3|HasAns_f1   |↑  |83.2540|±  |   N/A|
|       |       |none  |     3|NoAns_exact |↑  | 0.0000|±  |   N/A|
|       |       |none  |     3|NoAns_f1    |↑  | 0.0000|±  |   N/A|
|       |       |none  |     3|best_exact  |↑  |74.0000|±  |   N/A|
|       |       |none  |     3|best_f1     |↑  |82.1310|±  |   N/A|
|       |       |none  |     3|exact       |↑  |28.0000|±  |   N/A|
|       |       |none  |     3|f1          |↑  |37.4643|±  |   N/A|
```

and yields (note model gave wrong answer for first example)
```
entry['target']  unanswerable
entry['filtered_resps'] ['Dyrrachium', ['-36.75', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f028604f3c001a3ff825', 'prediction_text': 'Dyrrachium', 'no_answer_probability': 1.0956650033262367e-16}, {'id': '5ad3f028604f3c001a3ff825', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f028604f3c001a3ff825', 'prediction_text': 'Dyrrachium', 'no_answer_probability': 1.0956650033262367e-16}, {'id': '5ad3f028604f3c001a3ff825', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-24.5', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff86f', 'prediction_text': 'unanswerable', 'no_answer_probability': 2.289734845645553e-11}, {'id': '5ad3f187604f3c001a3ff86f', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff86f', 'prediction_text': 'unanswerable', 'no_answer_probability': 2.289734845645553e-11}, {'id': '5ad3f187604f3c001a3ff86f', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-29.5', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff870', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.5428112031918877e-13}, {'id': '5ad3f187604f3c001a3ff870', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff870', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.5428112031918877e-13}, {'id': '5ad3f187604f3c001a3ff870', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-29.875', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff871', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.0603575991523846e-13}, {'id': '5ad3f187604f3c001a3ff871', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff871', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.0603575991523846e-13}, {'id': '5ad3f187604f3c001a3ff871', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-30.625', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8ef', 'prediction_text': 'unanswerable', 'no_answer_probability': 5.008774637843506e-14}, {'id': '5ad3f350604f3c001a3ff8ef', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8ef', 'prediction_text': 'unanswerable', 'no_answer_probability': 5.008774637843506e-14}, {'id': '5ad3f350604f3c001a3ff8ef', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-23.375', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8f0', 'prediction_text': 'unanswerable', 'no_answer_probability': 7.05287985111216e-11}, {'id': '5ad3f350604f3c001a3ff8f0', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8f0', 'prediction_text': 'unanswerable', 'no_answer_probability': 7.05287985111216e-11}, {'id': '5ad3f350604f3c001a3ff8f0', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-23.375', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8f1', 'prediction_text': 'unanswerable', 'no_answer_probability': 7.05287985111216e-11}, {'id': '5ad3f350604f3c001a3ff8f1', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8f1', 'prediction_text': 'unanswerable', 'no_answer_probability': 7.05287985111216e-11}, {'id': '5ad3f350604f3c001a3ff8f1', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target']  unanswerable
entry['filtered_resps'] ['unanswerable', ['-27.375', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f4b1604f3c001a3ff951', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.2917800047859961e-12}, {'id': '5ad3f4b1604f3c001a3ff951', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f4b1604f3c001a3ff951', 'prediction_text': 'unanswerable', 'no_answer_probability': 1.2917800047859961e-12}, {'id': '5ad3f4b1604f3c001a3ff951', 'answers': {'text': [], 'answer_start': []}}]
------

```
As can be seen, 'no answer' probability is extremely low even when the model generated the correct response...

Changing https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/squadv2/task.py starting line 100:
from
```
    def doc_to_target(self, doc):
        answer_list = doc["answers"]["text"]
        if len(answer_list) > 0:
            answer = answer_list[0]
        else:
            answer = "unanswerable"
        return " " + answer
```
to

```
    def doc_to_target(self, doc):
        answer_list = doc["answers"]["text"]
        if len(answer_list) > 0:
            answer = " " + answer_list[0]
        else:
            answer = "unanswerable"
        return answer
```
(removing the whitespace in post-processing; also remove it in line 133!)
and then rerunning the test gives these scores:

```
| Tasks |Version|Filter|n-shot|   Metric   |   | Value |   |Stderr|
|-------|------:|------|-----:|------------|---|------:|---|------|
|squadv2|      3|none  |     3|HasAns_exact|↑  |62.2222|±  |   N/A|
|       |       |none  |     3|HasAns_f1   |↑  |83.2540|±  |   N/A|
|       |       |none  |     3|NoAns_exact |↑  | 0.0000|±  |   N/A|
|       |       |none  |     3|NoAns_f1    |↑  | 0.0000|±  |   N/A|
|       |       |none  |     3|best_exact  |↑  |74.0000|±  |   N/A|
|       |       |none  |     3|best_f1     |↑  |81.4167|±  |   N/A|
|       |       |none  |     3|exact       |↑  |28.0000|±  |   N/A|
|       |       |none  |     3|f1          |↑  |37.4643|±  |   N/A|
```
and yields

```
entry['target'] unanswerable
entry['filtered_resps'] ['Dyrrachium', ['-4.75', 'False']]
entry["NoAns_exact"] [{'id': '5ad3f028604f3c001a3ff825', 'prediction_text': 'Dyrrachium', 'no_answer_probability': 0.008651695203120634}, {'id': '5ad3f028604f3c001a3ff825', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f028604f3c001a3ff825', 'prediction_text': 'Dyrrachium', 'no_answer_probability': 0.008651695203120634}, {'id': '5ad3f028604f3c001a3ff825', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.006195068359375', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff86f', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9938240815112875}, {'id': '5ad3f187604f3c001a3ff86f', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff86f', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9938240815112875}, {'id': '5ad3f187604f3c001a3ff86f', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.002410888671875', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff870', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9975920151861242}, {'id': '5ad3f187604f3c001a3ff870', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff870', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9975920151861242}, {'id': '5ad3f187604f3c001a3ff870', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.000148773193359375', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f187604f3c001a3ff871', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9998512378728234}, {'id': '5ad3f187604f3c001a3ff871', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f187604f3c001a3ff871', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9998512378728234}, {'id': '5ad3f187604f3c001a3ff871', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.0004825592041015625', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8ef', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.999517557208865}, {'id': '5ad3f350604f3c001a3ff8ef', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8ef', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.999517557208865}, {'id': '5ad3f350604f3c001a3ff8ef', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.047119140625', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8f0', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.95397373379264}, {'id': '5ad3f350604f3c001a3ff8f0', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8f0', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.95397373379264}, {'id': '5ad3f350604f3c001a3ff8f0', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-0.1318359375', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f350604f3c001a3ff8f1', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.8764847815685415}, {'id': '5ad3f350604f3c001a3ff8f1', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f350604f3c001a3ff8f1', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.8764847815685415}, {'id': '5ad3f350604f3c001a3ff8f1', 'answers': {'text': [], 'answer_start': []}}]
------
entry['target'] unanswerable
entry['filtered_resps'] ['unanswerable', ['-1.0728836059570312e-06', 'True']]
entry["NoAns_exact"] [{'id': '5ad3f4b1604f3c001a3ff951', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9999989271169696}, {'id': '5ad3f4b1604f3c001a3ff951', 'answers': {'text': [], 'answer_start': []}}]
entry["NoAns_f1"] [{'id': '5ad3f4b1604f3c001a3ff951', 'prediction_text': 'unanswerable', 'no_answer_probability': 0.9999989271169696}, {'id': '5ad3f4b1604f3c001a3ff951', 'answers': {'text': [], 'answer_start': []}}]
------

```
So, overall scoring seems unaffacted by the change (slight derivation in 'best_f1') while "No answer" probability is still low for the incorrectly answered unanswerable question but now seems to work much better for the others. However, the corresponding scores still sit at 0.0 and I don't get why. 

Unfortunately, github doesn't seem to allow me to upload the sample files here. However, I'm happy to send them directly to whoever is interested - just let me know!

Any help here is greatly appreciated!

The samples would be really useful, if you could post a pastebin link to your samples that will help massively. Unfortunately I think I may have discovered the reason for the unusual squad behaviour (but will have to confirm when I look over the samples) and that is I think squad might simply ignore chat templates for the last request in the context:

The way the harness constructs "requests" (the inputs used to either generate responses or calculate logprobs) is in a few steps:
1. create the prefix (which may include multiple shots) and apply the relevant chat templating
2. construct the final request, which takes the prefix context and adds the final stuff we're testing

Unfortunately the Squad task is a bit of a "leftover" from before we refactored all tasks to use yaml configs instead of python classes. As such I think (2) doesn't actually apply the chat template correctly for Squad because that uses custom code to build the requests instead of relying on a yaml config or using helper functions combined with a yaml config.

So honestly I'm surprised it's even getting decent scores for the answerable questions too because often chat models freak out when not utilised with the correct template throughout. But I'm fairly sure that's why the logprobs are so low for "unanswerable" because the template gets applied for the few shots, but then skipped for the actual questions we're tryna answer here.

(There's also more to it than that, like issues with the delimiter which your "fix" sorta helps but sorta breaks too, but again, when I can see the samples I might have an idea on how we might be able to fix the squad task!)

Hi Avelina, thanks for showing interest in the issue!
Here is the pastebin: https://pastebin.com/QFmBnVXX
Unfortunately, it allowed only a maximum filesize of 512 kb so I could only upload the first 25 samples. Is that sufficient?
It looks like chat templates are applied correctly during preprocessing (as you said in (1)), but regarding (2) I have found that some additional post-processing to the target of unanswerable questions is applied after evaluation has already taken place. Specifically, when looking at `process_results` ( https://github.com/EleutherAI/lm-evaluation-harness/blob/d021bf846218f1bb3bdc0603864789329476d464/lm_eval/tasks/squadv2/task.py#L139 ) continuations are checked against gold answers, which in the case of unanswerable questions is just an empty string. I have thus applied some post-processing to the continuation with:
```
        predictions = {
            "id": doc["id"],
            "prediction_text": "" if (continuation == "unanswerable") or (no_answer_probability > 0.95) else continuation,
            "no_answer_probability": no_answer_probability,
        }
```
which yields
```
    "squadv2": {
      "alias": "squadv2",
      "exact,none": 61.0,
      "exact_stderr,none": "N/A",
      "f1,none": 70.46428571428572,
      "f1_stderr,none": "N/A",
      "HasAns_exact,none": 62.22222222222222,
      "HasAns_exact_stderr,none": "N/A",
      "HasAns_f1,none": 83.25396825396825,
      "HasAns_f1_stderr,none": "N/A",
      "NoAns_exact,none": 60.0,
      "NoAns_exact_stderr,none": "N/A",
      "NoAns_f1,none": 60.0,
      "NoAns_f1_stderr,none": "N/A",
      "best_exact,none": 75.0,
      "best_exact_stderr,none": "N/A",
      "best_f1,none": 83.13095238095238,
      "best_f1_stderr,none": "N/A"
    }
```
It's a hack for sure, but seems to get the job done. I hope you find that information helpful. Let me know if I can do anything else!