Repository: EleutherAI/lm-evaluation-harness
Issue: Creating a new task with data in chat format (openai)
Root cause: Validation Gap

Hello,

I'm creating a new task and my data splits are stored in jsonl files with the relevant part in the 'conversations' column in the openai chat format.

An example of one row:

{...., 'conversations':[{'role':'user','content':'......'},{'role':'assistant','content':'......'}]}

I tried the following commands:

```
lm_eval --model hf \
--model_args pretrained=/data/tir/projects/tir7/user_data/lmaben/llm-weights/outputs/llamafactory/llmfac_qwen2_5_1_5b_instruct-full-lr1e-6-24kcontext-grad_accum32/checkpoint-266 \
--tasks code_edit \
--device cuda:0 \
--batch_size auto:2 \ 
--apply_chat_template True \
--limit 4
```

```
lm_eval --model hf \
--model_args pretrained=/data/tir/projects/tir7/user_data/lmaben/llm-weights/outputs/llamafactory/llmfac_qwen2_5_1_5b_instruct-full-lr1e-6-24kcontext-grad_accum32,revision=step266
--tasks code_edit \
--device cuda:0 \
--batch_size auto:2 \ 
--apply_chat_template True \
--limit 4
```


The Error I run into is:

> Traceback (most recent call last):
>   File "/data/tir/projects/tir7/user_data/lmaben/miniconda3/envs/code-edit-bench/bin/lm_eval", line 8, in <module>
>     sys.exit(cli_evaluate())
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/__main__.py", line 388, in cli_evaluate
>     results = evaluator.simple_evaluate(
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/evaluator.py", line 308, in simple_evaluate
>     results = evaluate(
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
>     return fn(*args, **kwargs)
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/evaluator.py", line 562, in evaluate
>     metrics = task.process_results(
>   File "/home/lmaben/lm-evaluation-harness/lm_eval/api/task.py", line 1619, in process_results
>     result_score = self._metric_fn_list[metric](
>   File "/home/lmaben/evaluate/src/evaluate/module.py", line 455, in compute
>     self.add_batch(**inputs)
>   File "/home/lmaben/evaluate/src/evaluate/module.py", line 520, in add_batch
>     batch = self.selected_feature_format.encode_batch(batch)
>   File "/data/tir/projects/tir7/user_data/lmaben/miniconda3/envs/code-edit-bench/lib/python3.10/site-packages/datasets/features/features.py", line 2027, in encode_batch
>     encoded_batch[key] = [encode_nested_example(self[key], obj, level=1) for obj in column]
>   File "/data/tir/projects/tir7/user_data/lmaben/miniconda3/envs/code-edit-bench/lib/python3.10/site-packages/datasets/features/features.py", line 2027, in <listcomp>
>     encoded_batch[key] = [encode_nested_example(self[key], obj, level=1) for obj in column]
>   File "/data/tir/projects/tir7/user_data/lmaben/miniconda3/envs/code-edit-bench/lib/python3.10/site-packages/datasets/features/features.py", line 1352, in encode_nested_example
>     return schema.encode_example(obj) if obj is not None else None
>   File "/data/tir/projects/tir7/user_data/lmaben/miniconda3/envs/code-edit-bench/lib/python3.10/site-packages/datasets/features/features.py", line 528, in encode_example
>     return int(value)
> ValueError: invalid literal for int() with base 10: '`\nCode changes will be provided in the form of a draft. You will need to apply the draft to the original code.\nThe original code will be enclosed within `<original_code>` tags.\nThe draft will be e



I've attached the yaml config. It looks like the tokenizer and chat template are not applied. Would appreciate any help. It would also be great if I could be pointed to a config that deals with chat data.

```
task: code_edit

dataset_path: json
dataset_kwargs:
  data_files:
    train: /home/lmaben/lm-evaluation-harness/lm_eval/tasks/code_edit/data/filtered_data/train.jsonl
    validation: /home/lmaben/lm-evaluation-harness/lm_eval/tasks/code_edit/data/filtered_data/val.jsonl
    test: /home/lmaben/lm-evaluation-harness/lm_eval/tasks/code_edit/data/filtered_data/test.jsonl

training_split: train
test_split: test
validation_split: validation

doc_to_text: "{{conversations[0]}}"
doc_to_target: "{{conversations[1]}}"

output_type: generate_until
generation_kwargs:
  max_new_tokens: 10000
  temperature: 0.0


metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
  - metric: accuracy
    aggregation: mean
    higher_is_better: true
  - metric: f1
    aggregation: mean
    higher_is_better: true
  - metric: precision
    aggregation: mean
    higher_is_better: true
  - metric: recall
    aggregation: mean
    higher_is_better: true
  - metric: wer
    aggregation: mean
    higher_is_better: false

```

Thanks in advance.