Repository: EleutherAI/lm-evaluation-harness
Issue: Caught jinja2.exceptions.UndefinedError: 'context' is undefined when dealing with japanese_leaderboard
Root cause: Validation Gap

# Traceback is below
```bash
Traceback (most recent call last):
  File "/opt/miniconda3/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/__main__.py", line 449, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/evaluator.py", line 264, in simple_evaluate
    task_dict = get_task_dict(
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 635, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 426, in load_task_or_group
    collections.ChainMap(
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 428, in <lambda>
    lambda task: self._load_individual_task_or_group(task),
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 410, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 326, in _load_individual_task_or_group
    return _load_task(task_config, task=name_or_config)
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 286, in _load_task
    task_object = ConfigurableTask(config=config)
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/api/task.py", line 928, in __init__
    test_text = self.doc_to_text(test_doc)
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/api/task.py", line 1294, in doc_to_text
    text_string = utils.apply_template(doc_to_text, doc)
  File "/data/tujie/lm-eval/lm-evaluation-harness/lm_eval/utils.py", line 533, in apply_template
    return rtemplate.render(**doc)
  File "/opt/miniconda3/lib/python3.10/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
  File "/opt/miniconda3/lib/python3.10/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 5, in top-level template code
  File "/opt/miniconda3/lib/python3.10/site-packages/jinja2/environment.py", line 490, in getattr
    return getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'context' is undefined
```
# Runing Script
```bash
lm_eval --model vllm --model_args pretrained="/data/tujie/qwen-ckpts/Qwen3-30B-A3B",max_model_len=8192,tensor_parallel_size=8,dtype=auto,data_parallel_size=1 --tasks ceval-valid,mmlu,japanese_leaderboard --batch_size 4 --trust_remote_code
```
# Background
I manually downloaded all the datasets of japanese_leaderboard from huggingface, and change the yaml with
```bash
dataset_path: "csv"
dataset_kwargs:
    data_files:
        train: "/nfs-158/tujie/lm-eval/japanese_leaderboard/xlsum_ja/data/train.csv"
        validation: "/nfs-158/tujie/lm-eval/japanese_leaderboard/xlsum_ja/data/validation.csv"
        test: "/nfs-158/tujie/lm-eval/japanese_leaderboard/xlsum_ja/data/test.csv"
```
But I got the problem showing above, why this happened?
And I have another question,

![Image](https://github.com/user-attachments/assets/4d80b0a4-5274-49c3-b1de-3d759417ed31)

Should I use the python file in the picture to preprocess json files befor I load_dataset into the disk? If so, how to take advantage of it ?

By the way, if i change the tasks in `lm_eval/tasks/japanese_leaderboard/_ja_leaderboard.yaml`:
```bash
group: japanese_leaderboard

task:
  - ja_leaderboard_mgsm
  - ja_leaderboard_xlsum
  - ja_leaderboard_xwinograd

metadata:
  version: 1.0
```

I got the below problems:

![Image](https://github.com/user-attachments/assets/3f6c0c88-4cd7-46b7-afd5-e4648a9b059b)

And the final result is:

![Image](https://github.com/user-attachments/assets/eab033de-d4f9-43c9-b425-23c3e746cc68)

Which is absolutely not ok.

How to solve there problems?

Appreciate the help, it had haunted me for days!!

Hi! The first error is that the `context` field isn't in the dataset, and the second also suggests your dataset isn't structured as the config expects. Generally for datasets with processing scripts, the processing script transforms the dataset structure.

I think for your use case, the simplest way to get around this would be to save the dataset(s) using `save_to_disk` ([HF doc](https://huggingface.co/docs/datasets/en/process#save)), if you can, and modify the loading method here

https://github.com/EleutherAI/lm-evaluation-harness/blob/82a99365abdc801a11d41095182a4da506591aec/lm_eval/api/task.py#L991-L995

to call `load_from_disk` (pass the file path using `dataset_path` in the yaml) method rather than `load_dataset`.

Hi, @baberabb Gratefully thanks for the solution before. I have succeeded in loading local datasets by passing the `script.py` to the `data_path` in yaml. But I confronted problems when i try to use `f1` score to evaluate the datasets, specifically `jsquad`,  I ran to issue: 
```bash
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
```
So I change the definition of `f1` in `api.metrics.py` by adding `average="macro"`, problem solved but the evaluation result is **0.0000**. 
```python
@register_aggregation("f1")
def f1(items):
    # this is different from f1-score which uses default binary avg
    unzipped_list = list(zip(*items))
    golds = unzipped_list[0]
    preds = unzipped_list[1]
    fscore = sklearn.metrics.f1_score(golds, preds, average="macro")

    return fscore
```

Another problem occures when I try to build a new metric , I have already add code below to the `api.metrics.py`:
```python
@register_aggregation("balanced-mean")
def balanced_mean(arr):
    # each entry is of the form (acc score, class label)
    # first group the results
    by_class = defaultdict(list)
    for acc, label in arr:
        by_class[label].append(acc)

    # calculate class averages
    avgs = []
    for key, vals in by_class.items():
        avgs.append(sum(vals) / len(vals))

    # average the class values
    return sum(avgs) / len(avgs)
    
@register_metric(
    metric="balanced-acc",
    higher_is_better=True,
    output_type=["loglikelihood", "multiple_choice"],
    aggregation="balanced-mean",
)
def acc_fn(items):  # This is a passthrough function
    return items

```

And I use it on dataset `jnli` yaml,

```bash
metric_list:
  - metric: balanced-acc
    aggregation: balanced-mean
    higher_is_better: true
```
But I found the `jnli evaluation` disappeared in my final table.

The evaluation result is:

In conclusion, I have three questions:
1. how to deal with the zero f1 score;
2. how to add custom metric;
3. how to solve the zero rouge.

Sorry to disturb you so much, gratefully for the solution!