Repository: EleutherAI/lm-evaluation-harness
Issue: Inconsistent Behavior with max_tokens, Post-Processing, and Cache Options
Root cause: Documentation Deficiency

I've encountered several issues when running benchmarks on MMLU-Pro that may affect reproducibility and expected behavior. The issues are as follows:

1. **Default `max_tokens` Handling:**  
   - **Observation:** When the `max_tokens` argument is not provided in `model_args` or `gen_kwargs`, the current implementation still sends a max_tokens value to the API. https://github.com/EleutherAI/lm-evaluation-harness/blob/41b952f30c39abd8d2802a429119fb50c63b6a04/lm_eval/models/api_models.py#L71
   - **Expected Behavior:** If no `max_tokens` is specified, no batch size should be passed to the API, allowing the API to use its own default value.
   - **Impact:** The model wouldn't finish the answer within this number, and it's hidden so deep that can confuse the users. Using a default value here makes no sense.

2. **Post-Process Function Inconsistency:**  
   - **Observation:** The post-processing function in the MMLU-Pro template does not match the official implementation as found [here](https://github.com/TIGER-AI-Lab/MMLU-Pro).
   - **Expected Behavior:** The template should be consistent with the official implementation to ensure consistent evaluation results.
   - **Impact:** Divergence in post-processing can lead to discrepancies in benchmark results and potentially incorrect interpretations of model performance.

3. **Confusing Cache Arguments (`--use_cache` and `--cache_requests`):**  
   - **Observation:** The `--cache_requests` flag, when enabled, caches the payloads of requests, but it does so in a location indicated by the `--use_cache` argument.
   - **Expected Behavior:** The naming and behavior of these arguments should be more intuitive. Ideally, `--cache_requests` should directly imply where and how the caching is done, or the two flags should be unified/renamed for clarity.
   - **Impact:** The current setup can confuse users about where cached data is stored and how caching is managed, leading to potential misuse or debugging difficulties.

## Suggested Fixes

- **Batch Size:** Modify `lm_eval` to omit sending a batch size parameter if it is not explicitly provided, letting the API handle the default.
- **Post-Process Function:** Update the MMLU-Pro template to align the post-process function with the official implementation.
- **Cache Options:** Review and refactor the caching arguments for clarity—consider renaming or consolidating the flags to reduce confusion.


Hi! Thanks for the feedback! Couple of thoughts:
1. `max_gen_toks` is usually defined in the task config, and we set 256 as a default fallback throughout the library. This also ensures that users do not waste API credits unnecessarily. I'll see about adding a warning or making this more explicit.
2. Good catch! Would you be willing to make a PR?
3. These two caching arguments serve distinct purposes. `use_cache` stores the model outputs during generation, so if an interruption occurs, we can resume from the last sample rather than regenerating all previous samples. Meanwhile, `cache_requests` stores the preprocessed inputs, allowing for faster restart of the evaluation. I'll update the [interface](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md) doc to make this clear.