Repository: EleutherAI/lm-evaluation-harness
Issue: openai/gpt-oss-20b AIME25 eval is much lower than reported in paper
Root cause: Configuration Error

Hi,

As per title.

OpenAI reports AIME25 results in https://arxiv.org/pdf/2508.10925

Running:

```bash
CUDA_VISIBLE_DEVICES=1 lm_eval \
  --model hf \
  --model_args '{"pretrained":"openai/gpt-oss-20b","dtype":"auto","chat_template_args":{"reasoning_effort":"low"},"enable_thinking": true}' \
  --device "cuda" \
  --gen_kwargs max_gen_toks=4048 \
  --tasks aime25 \
  --apply_chat_template \
  --batch_size 16
```

I get

```
Running generate_until requests:   0%|          | 0/30 [00:00<?, ?it/s]max_gen_toks here 4048 context_enc torch.Size([16, 868])
generated text: torch.Size([16, 4286])

Running generate_until requests:   3%|▎         | 1/30 [01:59<57:43, 119.42s/it]max_gen_toks here 4048 context_enc torch.Size([14, 159])
generated text: torch.Size([14, 3948])

Running generate_until requests:  57%|█████▋    | 17/30 [03:55<02:35, 11.97s/it]
Running generate_until requests: 100%|██████████| 30/30 [03:55<00:00,  7.86s/it]
[2025-11-13 15:04:36] INFO evaluation_tracker.py:280: Output path not provided, skipping saving results aggregated
hf ({'pretrained': '/models/openai_gpt-oss-20b', 'dtype': 'auto', 'chat_template_args': {'reasoning_effort': 'low'}, 'enable_thinking': True}), gen_kwargs: (max_gen_toks=4048), limit: None, num_fewshot: None, batch_size: 16
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.2667|±  |0.0821|
```

running on `torch==2.9.0+rocm6.4`, `transformers==4.57.1`, MI325.

This seems to be much smaller than the reported `37.1` in the paper.

<img width="1022" height="636" alt="Image" src="https://github.com/user-attachments/assets/a8817cd2-c0d6-4894-a2e5-04a7bd40886d" />

AIM25 uses `num_fewshot: 0` so I don't think `--fewshot_as_multiturn` would change anything here.

Is it a known issue?

Thank you!

Noticed that `max_gen_toks` is specified at https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/tasks/aime/aime25.yaml#L25, and actually limiting to 4k tokens hurts, there is one answer which is truncated due to this, specifically for this question:

```

system
You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-11-15

Reasoning: low

# Valid channels: analysis, commentary, final. Channel must be included for every message.

user
Question: Let the sequence of rationals $x_1,x_2,\dots$ be defined such that $x_1=\frac{25}{11}$ and\[x_{k+1}=\frac{1}{3}\left(x_k+\frac{1}{x_k}-1\right).\]$x_{2025}$ can be expressed as $rac{m}{n}$ for relatively prime positive integers $m$ and $n$. Find the remainder when $m+n$ is divided by $1000$.
Answer:

assistant
```

Additionally, one needs to specify `"think_end_token":200008` to get good results.

Eventually:

```bash
LOGLEVEL=debug CUDA_VISIBLE_DEVICES=7 lm_eval \
  --model hf \
  --model_args '{"pretrained":"/models/openai_gpt-oss-20b","dtype":"auto","chat_template_args":{"reasoning_effort":"low"},"enable_thinking": true,"think_end_token":200008}' \
  --device "cuda" \
  --tasks aime25 \
  --apply_chat_template \
  --fewshot_as_multiturn \
  --batch_size 16
```

yields

```
hf ({'pretrained': '/models/openai_gpt-oss-20b', 'dtype': 'auto', 'chat_template_args': {'reasoning_effort': 'low'}, 'enable_thinking': True, 'think_end_token': 200008}), gen_kwargs: (max_gen_toks=4048), limit: None, num_fewshot: None, batch_size: 1
|Tasks |Version|Filter|n-shot|  Metric   |   |Value |   |Stderr|
|------|------:|------|-----:|-----------|---|-----:|---|-----:|
|aime25|      0|none  |     0|exact_match|↑  |0.4333|±  | 0.092|
```

More in line with the paper.

Let me close this issue.