Repository: EleutherAI/lm-evaluation-harness
Issue: Inadequate default max_length and max_gen_toks
Root cause: Configuration Error


The current default settings max_length=2048 and max_gen_toks=256 are way too low for tests like MMLU Pro.

Some requests in 5-shot MMLU Pro get over 5K tokens.

On top of that the LLM is encouraged to "Think step by step." so the responses can easily get over 256 tokens.

With the default settings the test results cannot be deemed adequate.

There is also no mention of either of these parameters in the documentation. 

Here are some results of mmlu_pro_math for Qwen2.5-72B-Instruct-GPTQ-Int4 with different max_gen_toks:

The default (max_gen_toks=256)

```
lm_eval \
  --model local-completions \
  --tasks mmlu_pro_math \
  --batch_size 16 \
  --model_args model=Qwen2.5-72B-Instruct,base_url=http://localhost:8000/v1/completions,max_retries=3,tokenizer=Qwen/Qwen2.5-72B-Instruct,max_length=8192
```

|       Tasks       |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-------------------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
| - math            |      1|custom-extract|     5|exact_match|↑  |0.5899|±  |0.0134|


```
lm_eval \
  --model local-completions \
  --tasks mmlu_pro_math \
  --batch_size 16 
  --model_args model=Qwen2.5-72B-Instruct,base_url=http://localhost:8000/v1/completions,max_retries=3,tokenizer=Qwen/Qwen2.5-72B-Instruct,max_length=8192,max_gen_toks=1024
```

|Tasks|Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|math |      1|custom-extract|     5|exact_match|↑  |0.7794|±  |0.0113|

```
lm_eval \
  --model local-completions \
  --tasks mmlu_pro_math \
  --batch_size 16 
  --model_args model=Qwen2.5-72B-Instruct,base_url=http://localhost:8000/v1/completions,max_retries=3,tokenizer=Qwen/Qwen2.5-72B-Instruct,max_length=8192,max_gen_toks=2048
```

|Tasks|Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|math |      1|custom-extract|     5|exact_match|↑  |0.7861|±  |0.0112|


To summarize the MMLU Pro Math test went from 0.5899 (max_gen_toks=256) to 0.7861 (max_gen_toks=2048) !

Hi! Thanks for the detailed report! The [official](https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/ef9890a4db3fd2c4fc50342606aac21093edb03f/evaluate_from_local.py#L18C1-L18C15) evaluation script also has it at 2048. Would you be willing to make a PR? Just need to add the `max_gen_toks` to the [config](https://github.com/EleutherAI/lm-evaluation-harness/blob/61b63da78d50ffdc4386c3845eacee42e4add4b1/lm_eval/tasks/mmlu_pro/_default_template_yaml#L18), increment the version and add the changelog to the readme.

I'll also see about making the default `max_length` for api models more intuitive as well, making it more in-line with what we [use](https://github.com/EleutherAI/lm-evaluation-harness/blob/61b63da78d50ffdc4386c3845eacee42e4add4b1/lm_eval/models/huggingface.py#L415) for HF or vllm.

Hello,

Added the PR.

With the new settings there are log entries when starting the job showing the global default values: 

```
INFO     [models.api_models:115] Using max length 2048 - 1
...
WARNING  [models.api_models:664] Some contexts exceeded (max length: (2047) - max_gen_toks (2048). They were left truncated.
```

This probably should be addressed