Repository: EleutherAI/lm-evaluation-harness
Issue: Recommended way of setting `max_new_tokens`
Root cause: Documentation Deficiency

Hi,

I am implementing some task for which models are expected to only produce a few digits. However, base models sometimes ramble and produce many many more tokens, making evaluation needlessly expensive, as these responses will always be incorrect. I would thus like to enforce some small number of max generation tokens. What is the recommended way of enforcing this within a task's .yaml?

Setting `max_new_tokens` within `generation_kwargs`, e.g., 
```yaml
generation_kwargs:
  until:
    - "\n\n"
    - "."
    - "\nQuestion:"
  do_sample: false
  temperature: 0.0
  max_new_tokens: 5
```
leads to a warning at evaluation time when using hf models
```
Both `max_new_tokens` (=5) and `max_length`(=8247) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Running generate_until requests:   0%|                                                                                                                                                                                                                                                  | 1/10000 [01:21<225:14:35, 81.10s/it]Both `max_new_tokens` (=5) and `max_length`(=8246) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)
Running generate_until requests:   0%|                                                                                                                                                                                                                                                   | 2/10000 [01:21<93:16:39, 33.59s/it]
```
and an error when using vllm:
```
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:00<00:00, 121887.76it/s]
2025-02-24:16:25:38,450 INFO     [lm_eval.evaluator:517] Running generate_until requests
Running generate_until requests:   0%|                                                                                                                                                                                                                                                              | 0/10000 [00:00<?, ?it/s][rank0]: Traceback (most recent call last):
[rank0]:   File "/lustre/home/rolmedo/vllm/bin/lm_eval", line 8, in <module>
[rank0]:     sys.exit(cli_evaluate())
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/__main__.py", line 388, in cli_evaluate
[rank0]:     results = evaluator.simple_evaluate(
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/evaluator.py", line 308, in simple_evaluate
[rank0]:     results = evaluate(
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/utils.py", line 422, in _wrapper
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/evaluator.py", line 528, in evaluate
[rank0]:     resps = getattr(lm, reqtype)(cloned_reqs)
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 418, in generate_until
[rank0]:     cont = self._model_generate(
[rank0]:   File "/lustre/home/rolmedo/lawma-eval/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 240, in _model_generate
[rank0]:     sampling_params = SamplingParams(max_tokens=max_tokens, stop=stop, **kwargs)
[rank0]: TypeError: Unexpected keyword argument 'max_new_tokens'
Running generate_until requests:   0%|                                                                                                                                                                                                                                                              | 0/10000 [00:01<?, ?it/s]
[rank0]:[W224 16:26:44.329068192 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
```

Hi! you can set `max_gen_toks` in the `generation_kwargs`!

Great, many thanks!