Repository: EleutherAI/lm-evaluation-harness
Issue: Afrobench MasakhanePOS issue with prompt 5
Root cause: Validation Gap

I'm not yet sure what the issue is, but here is the error trace:
```
[2025-08-23 12:09:02,946][__main__][INFO] - Detected nested config group 'eval'. Unwrapping configuration.
[2025-08-23 12:09:02,951][__main__][INFO] - Run mode: evaluate
[2025-08-23 12:09:02,955][sallm.evaluation.run][INFO] - Evaluating task-pack 'masakhapos_xho' with 5 tasks …
[2025-08-23 12:09:03,153][lm_eval.models.huggingface][INFO] - Using device 'cuda:0'
[2025-08-23 12:09:03,156][lm_eval.models.huggingface][INFO] - Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
[2025-08-23 12:09:04,091][lm_eval.evaluator][INFO] - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
[2025-08-23 12:09:04,091][lm_eval.evaluator][INFO] - Using pre-initialized model
[2025-08-23 12:09:20,913][lm_eval.evaluator][INFO] - masakhapos_xho_prompt_5: Using gen_kwargs: {'do_sample': False, 'until': ['
', '
']}
[2025-08-23 12:09:20,914][lm_eval.evaluator][WARNING] - Overwriting default num_fewshot of masakhapos_xho_prompt_5 from None to 0
[2025-08-23 12:09:20,914][lm_eval.evaluator][INFO] - masakhapos_xho_prompt_4: Using gen_kwargs: {'do_sample': False, 'until': ['
', '
']}
[2025-08-23 12:09:20,914][lm_eval.evaluator][WARNING] - Overwriting default num_fewshot of masakhapos_xho_prompt_4 from None to 0
[2025-08-23 12:09:20,914][lm_eval.evaluator][INFO] - masakhapos_xho_prompt_3: Using gen_kwargs: {'do_sample': False, 'until': ['
', '
']}
[2025-08-23 12:09:20,914][lm_eval.evaluator][WARNING] - Overwriting default num_fewshot of masakhapos_xho_prompt_3 from None to 0
[2025-08-23 12:09:20,914][lm_eval.evaluator][INFO] - masakhapos_xho_prompt_2: Using gen_kwargs: {'do_sample': False, 'until': ['
', '
']}
[2025-08-23 12:09:20,914][lm_eval.evaluator][WARNING] - Overwriting default num_fewshot of masakhapos_xho_prompt_2 from None to 0
[2025-08-23 12:09:20,914][lm_eval.evaluator][INFO] - masakhapos_xho_prompt_1: Using gen_kwargs: {'do_sample': False, 'until': ['
', '
']}
[2025-08-23 12:09:20,914][lm_eval.evaluator][WARNING] - Overwriting default num_fewshot of masakhapos_xho_prompt_1 from None to 0
[2025-08-23 12:09:20,918][lm_eval.api.task][INFO] - Building contexts for masakhapos_xho_prompt_5 on rank 0...
100%|██████████| 601/601 [00:00<00:00, 2254.92it/s]
[2025-08-23 12:09:21,219][lm_eval.evaluator_utils][INFO] - Task: ConfigurableTask(task_name=masakhapos_xho_prompt_5,output_type=generate_until,num_fewshot=0,num_samples=601); document 0; context prompt (starting on next line):    
Given the following sentence, identify the part of speech (POS) for each word. Use the following POS tag set: 
NOUN: Noun (person, place, thing), 
VERB: Verb (action, state), 
ADJ: Adjective (describes a noun), 
ADV: Adverb (modifies a verb, adjective, or adverb), 
PRON: Pronoun (replaces a noun), 
DET: Determiner (introduces a noun), 
ADP: Adposition (preposition or postposition), 
CCONJ: Conjunction (connects words, phrases, clauses)
PUNCT: Punctuation, 
PROPN: Proper Noun, 
AUX: Auxiliary verb (helper verb), 
SCONJ: Subordinating conjunction 
PART: Particle, 
SYM: Symbol, 
INTJ: Interjection, 
NUM: Numeral, 
X: others. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label key only from the POS tag set provided
Your response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. 

Sentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] 
Output: 
(end of prompt on previous line)
target string or answer choice index (starting on next line):
['VERB', 'NOUN', 'PRON', 'VERB', 'NOUN', 'PUNCT']
(end of target on previous line)
[2025-08-23 12:09:21,219][lm_eval.evaluator_utils][INFO] - Request: Instance(request_type='generate_until', doc={'id': '0', 'tokens': ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'], 'upos': [16, 0, 11, 16, 0, 1]}, arguments=("Given the following sentence, identify the part of speech (POS) for each word. Use the following POS tag set: \nNOUN: Noun (person, place, thing), \nVERB: Verb (action, state), \nADJ: Adjective (describes a noun), \nADV: Adverb (modifies a verb, adjective, or adverb), \nPRON: Pronoun (replaces a noun), \nDET: Determiner (introduces a noun), \nADP: Adposition (preposition or postposition), \nCCONJ: Conjunction (connects words, phrases, clauses)\nPUNCT: Punctuation, \nPROPN: Proper Noun, \nAUX: Auxiliary verb (helper verb), \nSCONJ: Subordinating conjunction \nPART: Particle, \nSYM: Symbol, \nINTJ: Interjection, \nNUM: Numeral, \nX: others. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label key only from the POS tag set provided\nYour response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. \n\nSentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] \nOutput: ", {'do_sample': False, 'until': ['
', '
']}), idx=0, metadata=('masakhapos_xho_prompt_5', 0, 1), resps=[], filtered_resps={}, task_name='masakhapos_xho_prompt_5', doc_id=0, repeats=1)
[2025-08-23 12:09:21,219][lm_eval.api.task][INFO] - Building contexts for masakhapos_xho_prompt_4 on rank 0...
100%|██████████| 601/601 [00:00<00:00, 2495.01it/s]
[2025-08-23 12:09:21,494][lm_eval.evaluator_utils][INFO] - Task: ConfigurableTask(task_name=masakhapos_xho_prompt_4,output_type=generate_until,num_fewshot=0,num_samples=601); document 0; context prompt (starting on next line):    
Annotate each word in the provided sentence with the appropriate POS tag. The annotation list is given as: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The input sentence will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided
Your response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. 

Sentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] 
Output: 
(end of prompt on previous line)
target string or answer choice index (starting on next line):
['VERB', 'NOUN', 'PRON', 'VERB', 'NOUN', 'PUNCT']
(end of target on previous line)
[2025-08-23 12:09:21,494][lm_eval.evaluator_utils][INFO] - Request: Instance(request_type='generate_until', doc={'id': '0', 'tokens': ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'], 'upos': [16, 0, 11, 16, 0, 1]}, arguments=("Annotate each word in the provided sentence with the appropriate POS tag. The annotation list is given as: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The input sentence will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided\nYour response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. \n\nSentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] \nOutput: ", {'do_sample': False, 'until': ['
', '
']}), idx=0, metadata=('masakhapos_xho_prompt_4', 0, 1), resps=[], filtered_resps={}, task_name='masakhapos_xho_prompt_4', doc_id=0, repeats=1)
[2025-08-23 12:09:21,494][lm_eval.api.task][INFO] - Building contexts for masakhapos_xho_prompt_3 on rank 0...
100%|██████████| 601/601 [00:00<00:00, 2482.82it/s]
[2025-08-23 12:09:21,770][lm_eval.evaluator_utils][INFO] - Task: ConfigurableTask(task_name=masakhapos_xho_prompt_3,output_type=generate_until,num_fewshot=0,num_samples=601); document 0; context prompt (starting on next line):    
Acting as a isiXhosa linguist and without making any corrections or changes to the text, perform a part of speech (POS) analysis of the sentences using the following POS tag label annotation ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The input will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided
Your response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. 

Sentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] 
Output: 
(end of prompt on previous line)
target string or answer choice index (starting on next line):
['VERB', 'NOUN', 'PRON', 'VERB', 'NOUN', 'PUNCT']
(end of target on previous line)
[2025-08-23 12:09:21,770][lm_eval.evaluator_utils][INFO] - Request: Instance(request_type='generate_until', doc={'id': '0', 'tokens': ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'], 'upos': [16, 0, 11, 16, 0, 1]}, arguments=("Acting as a isiXhosa linguist and without making any corrections or changes to the text, perform a part of speech (POS) analysis of the sentences using the following POS tag label annotation ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The input will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided\nYour response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. \n\nSentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] \nOutput: ", {'do_sample': False, 'until': ['
', '
']}), idx=0, metadata=('masakhapos_xho_prompt_3', 0, 1), resps=[], filtered_resps={}, task_name='masakhapos_xho_prompt_3', doc_id=0, repeats=1)
[2025-08-23 12:09:21,770][lm_eval.api.task][INFO] - Building contexts for masakhapos_xho_prompt_2 on rank 0...
100%|██████████| 601/601 [00:00<00:00, 2474.80it/s]
[2025-08-23 12:09:22,047][lm_eval.evaluator_utils][INFO] - Task: ConfigurableTask(task_name=masakhapos_xho_prompt_2,output_type=generate_until,num_fewshot=0,num_samples=601); document 0; context prompt (starting on next line):    
You are an expert in tagging words and sentences in isiXhosa with the right POS tag. 

Please provide the POS tags for each word in the isiXhosa sentence. The input is a list of words in the sentence. POS tag label set: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided
Your response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. 

Sentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] 
Output: 
(end of prompt on previous line)
target string or answer choice index (starting on next line):
['VERB', 'NOUN', 'PRON', 'VERB', 'NOUN', 'PUNCT']
(end of target on previous line)
[2025-08-23 12:09:22,047][lm_eval.evaluator_utils][INFO] - Request: Instance(request_type='generate_until', doc={'id': '0', 'tokens': ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'], 'upos': [16, 0, 11, 16, 0, 1]}, arguments=("You are an expert in tagging words and sentences in isiXhosa with the right POS tag. \n\nPlease provide the POS tags for each word in the isiXhosa sentence. The input is a list of words in the sentence. POS tag label set: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the POS tag label set provided\nYour response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. \n\nSentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] \nOutput: ", {'do_sample': False, 'until': ['
', '
']}), idx=0, metadata=('masakhapos_xho_prompt_2', 0, 1), resps=[], filtered_resps={}, task_name='masakhapos_xho_prompt_2', doc_id=0, repeats=1)
[2025-08-23 12:09:22,047][lm_eval.api.task][INFO] - Building contexts for masakhapos_xho_prompt_1 on rank 0...
100%|██████████| 601/601 [00:00<00:00, 2546.14it/s]
[2025-08-23 12:09:22,316][lm_eval.evaluator_utils][INFO] - Task: ConfigurableTask(task_name=masakhapos_xho_prompt_1,output_type=generate_until,num_fewshot=0,num_samples=601); document 0; context prompt (starting on next line):    
Please provide the POS tags for each word in the input sentence. The input will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the tag label set: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. 
Your response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. 

Sentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] 
Output: 
(end of prompt on previous line)
target string or answer choice index (starting on next line):
['VERB', 'NOUN', 'PRON', 'VERB', 'NOUN', 'PUNCT']
(end of target on previous line)
[2025-08-23 12:09:22,316][lm_eval.evaluator_utils][INFO] - Request: Instance(request_type='generate_until', doc={'id': '0', 'tokens': ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'], 'upos': [16, 0, 11, 16, 0, 1]}, arguments=("Please provide the POS tags for each word in the input sentence. The input will be a list of words in the sentence. The output format should be a list of tuples, where each tuple consists of a word from the input text and its corresponding POS tag label from the tag label set: ['ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ, 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X']. \nYour response should include only a list of tuples, in the order that the words appear in the input sentence, including punctuations, with each tuple containing the corresponding POS tag label for a word. \n\nSentence: ['Ndiyabazi', 'abalandeli', 'bethu', 'balambele', 'ntoni', '.'] \nOutput: ", {'do_sample': False, 'until': ['
', '
']}), idx=0, metadata=('masakhapos_xho_prompt_1', 0, 1), resps=[], filtered_resps={}, task_name='masakhapos_xho_prompt_1', doc_id=0, repeats=1)
[2025-08-23 12:09:22,316][lm_eval.evaluator][INFO] - Running generate_until requests
Running generate_until requests: 100%|██████████| 3005/3005 [3:55:06<00:00,  4.69s/it]  
Error executing job with overrides: []
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/lmbanr001/masters/sallm/src/main/sallm/main.py", line 99, in <module>
    main()
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
           ^^^^^^
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
            ^^^^^^^^^^
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
        ^^^^^^^^^^^^^^^^
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/scratch/lmbanr001/.conda/envs/sallm-ner/lib/python3.12/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lmbanr001/masters/sallm/src/main/sallm/main.py", line 91, in main
    run_eval(config)
  File "/home/lmbanr001/masters/sallm/src/main/sallm/evaluation/run.py", line 29, in run
    results: Dict = evaluate_pack(pack, model_cfg, out_root, overrides)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lmbanr001/masters/sallm/src/main/sallm/evaluation/harness.py", line 63, in evaluate_pack
    results = lm_eval.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/evaluator.py", line 342, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/evaluator.py", line 683, in evaluate
    task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/evaluator_utils.py", line 114, in calculate_aggregate_metric
    self.agg_metrics[metric_key] = agg_fn(items)
                                   ^^^^^^^^^^^^^
  File "/home/lmbanr001/.local/lib/python3.12/site-packages/lm_eval/tasks/afrobench/masakhapos/prompt_5/utils.py", line 33, in acc_score
    unzipped_list = list(zip(*items))
                         ^^^^^^^^^^^
TypeError: 'float' object is not iterable
lmbanr001@srvrochpc001 ~/masters/sallm$ 
```

All other prompts are returning the expected (golds, preds) tuples, but for some reason a float is being returned in this edge case. The obvious fix is just letting acc_score in utils.py handle this, but I don't understand the issue to be comfortable submitting a PR for that solution yet.

For more context, here is my arguments:
```
tasks:
  - masakhapos_xho_prompt_1
  - masakhapos_xho_prompt_2
  - masakhapos_xho_prompt_3
  - masakhapos_xho_prompt_4
  - masakhapos_xho_prompt_5
fewshot: 0
batch_size: 8
apply_chat_template: true

```

Here is my harness wrapper:
```
from __future__ import annotations

import json
from pathlib import Path
from typing import Dict

import numpy as np
import torch
import lm_eval
from lm_eval.models.huggingface import HFLM
from peft import PeftModel
from transformers import AutoTokenizer
from tokenizers.decoders import ByteLevel

from sallm.config import ModelEvalConfig
from sallm.evaluation.config import TaskPack


def _safe_json_encoder(obj):
    if isinstance(obj, (np.integer, np.floating)):
        return obj.item()
    if isinstance(obj, np.ndarray):
        return obj.tolist()
    if isinstance(obj, torch.Tensor):
        return obj.tolist() if obj.ndim else obj.item()
    if isinstance(obj, torch.dtype):
        return str(obj)
    return str(obj)


def evaluate_pack(
    pack: TaskPack,
    model_cfg: ModelEvalConfig,
    out_dir: Path,
    overrides: Dict[str, Dict],
) -> Dict:
    pack_over = overrides.get(pack.name, {})
    task_list = pack_over.get("tasks", pack.tasks)
    fewshot = int(pack_over.get("fewshot", pack.fewshot))
    batch_size = int(pack_over.get("batch_size", pack.batch_size))
    apply_chat_template = pack_over.get("apply_chat_template", pack.apply_chat_template)

    tok = AutoTokenizer.from_pretrained(model_cfg.checkpoint, trust_remote_code=True)
    tok.backend_tokenizer.decoder = ByteLevel()

    hf_model = HFLM(
        pretrained=model_cfg.checkpoint,
        tokenizer=tok,
        peft=model_cfg.peft_adapter,
        device=model_cfg.device,
        dtype=model_cfg.dtype,
        trust_remote_code=True,
    )

    genconf = hf_model.model.generation_config
    genconf.eos_token_id = tok.eos_token_id
    genconf.pad_token_id = tok.pad_token_id or tok.eos_token_id

    if model_cfg.merge_lora and isinstance(hf_model.model, PeftModel):
        hf_model._model = hf_model.model.merge_and_unload()

    try:
        results = lm_eval.simple_evaluate(
            model=hf_model,
            tasks=task_list,
            num_fewshot=fewshot,
            batch_size=batch_size,
            apply_chat_template=apply_chat_template,
            log_samples=True,
            write_out=True,
        )
    except TypeError:
        raise

    out_dir.mkdir(parents=True, exist_ok=True)
    out_path = out_dir / f"{pack.name}.json"
    with out_path.open("w") as f:
        json.dump(results, f, indent=2, default=_safe_json_encoder)

    return results

```


My code runs perfectly for multiple lm eval harness tasks, just having an issue with this pos task and specifically prompt 5. The short term solution will be to comment out prompt 5 and see what happens, so I'll report if any error occurs.

As I anticipated, this is a problem across all prompts