Repository: EleutherAI/lm-evaluation-harness
Issue: [Bug] `fewshot_config.doc_to_text` is ignored in v0.4.9.2, breaking MMLU-Pro CoT fewshot examples
Root cause: Configuration Error

## 🐛 Bug Description

In version `0.4.9.2`, the `fewshot_config.doc_to_text` and `fewshot_config.doc_to_target` settings in task YAML configurations are no longer respected. This causes tasks like `mmlu_pro`,`mmlu_prox`,`mmlu-pro-plus` to generate fewshot examples without chain-of-thought (CoT) content.

## Versions

- **Broken**: `0.4.9.2`
- **Working**: `0.4.9.1`

## Steps to Reproduce
### config.yaml
```yaml
model: hf
model_args:
  pretrained: "Qwen/Qwen2.5-7B-Instruct"
  dtype: bfloat16

tasks:
  - mmlu_pro

batch_size: 32

device: "cuda:0"

apply_chat_template: true

output_path: "xxx"
log_samples: true

limit: 3
```
### prompt
```bash
# Using v0.4.9.2
lm-eval run --config config.yaml
```

## Expected Behavior

Fewshot examples should include the CoT reasoning from `cot_content` field, as defined in `fewshot_config.doc_to_text: !function utils.fewshot_to_text`:

```
Question:
Find the longest common subsequence of the following two strings: AABBC BABCAB.
Options:
A. ABAB
B. ABBC
...
Answer: Let's think step by step. We need to find the longest common subsequence (LCS)...
...The answer is (B).
```

## Actual Behavior

Fewshot examples only contain question and options, missing the CoT content:

```
Question:
Find the longest common subsequence of the following two strings: AABBC BABCAB.
Options:
A. ABAB
B. ABBC
...
Answer: Let's think step by step. B
```

## Root Cause Analysis

In `v0.4.9.1`, `ContextSampler` (in `lm_eval/api/samplers.py`) respects `fewshot_config` overrides:

```python
# v0.4.9.1 - lm_eval/api/samplers.py lines 37-57
if (
    self.config.fewshot_config is not None
    and self.config.fewshot_config.get("doc_to_text", None) is not None
):
    self.doc_to_text = partial(
        self.task.doc_to_text,
        doc_to_text=self.config.fewshot_config.get("doc_to_text", None),
    )
# Similar for doc_to_target
```

In `v0.4.9.2`, `ConfigurableTask.fewshot_context` was refactored (in `lm_eval/api/task.py`). The new implementation directly calls `self.doc_to_text(fs_doc)` without checking `fewshot_config`:

```python
# v0.4.9.2 - lm_eval/api/task.py lines 1126-1129
for fs_doc in self.sampler.sample(...):
    q, c, a = (
        self.doc_to_text(fs_doc),  # ← ignores fewshot_config.doc_to_text
        self.doc_to_choice(fs_doc) if self.config.doc_to_choice else None,
        self.doc_to_target(fs_doc),  # ← ignores fewshot_config.doc_to_target
    )
```

The new `ContextSampler` in `v0.4.9.2` no longer stores or uses `fewshot_config` overrides.

## Affected Tasks

Any task that relies on `fewshot_config.doc_to_text` or `fewshot_config.doc_to_target`, including:

- `mmlu_pro` (CoT examples)
- `mmlu_prox`
- `mmlu-pro-plus`
- Potentially other tasks with custom fewshot formatting

## MMLU-Pro Task Config Reference

```yaml
# lm_eval/tasks/mmlu_pro/_default_template_yaml
fewshot_config:
  sampler: first_n
  doc_to_text: !function utils.fewshot_to_text  # ← This is ignored in v0.4.9.2
  doc_to_target: ""
```

## Suggested Fix

In `ConfigurableTask.fewshot_context`, check for `fewshot_config` overrides before calling `doc_to_text`/`doc_to_target` on fewshot documents:

```python
# In fewshot_context method, when iterating fewshot examples:
fewshot_doc_to_text = (
    self.config.fewshot_config.get("doc_to_text")
    if self.config.fewshot_config else None
)
fewshot_doc_to_target = (
    self.config.fewshot_config.get("doc_to_target")
    if self.config.fewshot_config else None
)

for fs_doc in self.sampler.sample(...):
    if fewshot_doc_to_text is not None:
        q = fewshot_doc_to_text(fs_doc) if callable(fewshot_doc_to_text) else self.doc_to_text(fs_doc, doc_to_text=fewshot_doc_to_text)
    else:
        q = self.doc_to_text(fs_doc)
    # Similar for doc_to_target
```

## Workaround

A monkey patch can be applied at runtime to restore the expected behavior:

```python
import json
from functools import partial


def patch_fewshot_context():
    from lm_eval.api.task import ConfigurableTask
    from lm_eval.api.utils import Message, maybe_delimit, multiturn_to_singleturn

    original_fewshot_context = ConfigurableTask.fewshot_context

    def patched_fewshot_context(
        self,
        doc,
        num_fewshot,
        system_instruction=None,
        apply_chat_template=False,
        fewshot_as_multiturn=False,
        chat_template=None,
        gen_prefix=None,
    ):
        fewshot_doc_to_text = None
        fewshot_doc_to_target = None

        if self.config.fewshot_config is not None:
            fewshot_doc_to_text = self.config.fewshot_config.get("doc_to_text", None)
            fewshot_doc_to_target = self.config.fewshot_config.get("doc_to_target", None)

        if fewshot_doc_to_text is None and fewshot_doc_to_target is None:
            return original_fewshot_context(
                self, doc, num_fewshot, system_instruction,
                apply_chat_template, fewshot_as_multiturn, chat_template, gen_prefix
            )

        messages = []
        tgt_delim, few_delim = self.config.target_delimiter, self.config.fewshot_delimiter
        chat_template_fn = (
            partial(chat_template, add_generation_prompt=not gen_prefix)
            if chat_template else None
        )

        description = self.resolve_field(doc, self.config.description) or ""
        system_prompt = maybe_delimit(system_instruction, description, few_delim)
        if system_prompt:
            messages.append(Message("system", system_prompt))

        if num_fewshot > 0:
            for fs_doc in self.sampler.sample(
                n=num_fewshot,
                eval_doc=doc if self.config.fewshot_split == self.config.test_split else None,
            ):
                if fewshot_doc_to_text is not None:
                    q = fewshot_doc_to_text(fs_doc) if callable(fewshot_doc_to_text) else self.doc_to_text(fs_doc, doc_to_text=fewshot_doc_to_text)
                else:
                    q = self.doc_to_text(fs_doc)

                c = self.doc_to_choice(fs_doc) if self.config.doc_to_choice else None

                if fewshot_doc_to_target is not None:
                    if fewshot_doc_to_target == "":
                        a = ""
                    elif callable(fewshot_doc_to_target):
                        a = fewshot_doc_to_target(fs_doc)
                    else:
                        a = self.doc_to_target(fs_doc, doc_to_target=fewshot_doc_to_target)
                else:
                    a = self.doc_to_target(fs_doc)

                if self.multiple_input:
                    q = c[q] if isinstance(q, int) and c else q
                    c = None

                messages += self.build_qa_turn(
                    q=q, c=c, a=a,
                    gen_prefix=gen_prefix,
                    tgt_delim=tgt_delim,
                    few_delim=few_delim,
                )

        q = self.doc_to_text(doc)
        c = self.doc_to_choice(doc) if self.config.doc_to_choice else None

        if self.multiple_input:
            return self.multiple_input_context(
                messages, gen_prefix, c,
                chat_template=chat_template_fn if apply_chat_template else None,
                fewshot_as_multiturn=fewshot_as_multiturn,
            )

        messages += self.build_qa_turn(
            q=q, c=c,
            gen_prefix=gen_prefix,
            tgt_delim=tgt_delim,
            few_delim="",
        )

        if apply_chat_template and chat_template_fn:
            res = (
                [m.to_dict() for m in messages]
                if fewshot_as_multiturn
                else multiturn_to_singleturn(messages)
            )
            res = chat_template_fn(res)
        else:
            res = "".join(m.to_text() for m in messages)

        return res

    ConfigurableTask.fewshot_context = patched_fewshot_context


import lm_eval

patch_fewshot_context()

# Then run evaluation as normal
results = lm_eval.simple_evaluate(...)
```

## Environment

- Python version: 3.12.12
- lm-eval version: 0.4.9.2
- OS: Ubuntu 24.04

Hi! Thanks for reporting this! And sorry, this was an oversight on my part. This PR should fix this.

Thanks!!!