Repository: EleutherAI/lm-evaluation-harness
Issue: `o1-preview` support
Root cause: Unimplemented Feature Gap


It seems that current version of `lm_eval` does not supported `o1` series models.

When I execute
```sh
lm_eval\
    --model openai-chat-completions\
    --model_args model="o1-preview"\
    --apply_chat_template\
    --tasks ${task}\
    --verbosity DEBUG\
    --log_samples\
    --output_path ${log_path}

```

I got:

```sh
2024-10-30:10:47:19,626 WARNING  [api_models.py:349] API request failed with error message: {
  "error": {
    "message": "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "unsupported_parameter"
  }
}. Retrying...
Traceback (most recent call last):
  File "/home/zi/anaconda3/envs/robench/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/zi/harness-4-robench/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/zi/harness-4-robench/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zi/harness-4-robench/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
  File "/home/zi/harness-4-robench/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/home/zi/harness-4-robench/lm_eval/evaluator.py", line 500, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/zi/harness-4-robench/lm_eval/models/api_models.py", line 571, in generate_until
    outputs = retry(
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/home/zi/harness-4-robench/lm_eval/models/api_models.py", line 352, in model_call
    response.raise_for_status()
  File "/home/zi/anaconda3/envs/robench/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.openai.com/v1/chat/completions
```

The error is `lm_eval` will send a default variable of `max_tokens` via openai's API while o1 doesn't support this parameter. 

I doesn't define `max_tokens` in `model_args` explictly, so how can I disable this option?







The PR should fix it! (you also need to add `--apply_chat_template` for the chat-completions API)

Thanks for your contributions~ It helps me a lot.

Also, I find that in `o1`, `temperature` cannot be set to `0.0` and `stop` is disabled.

If not, we meet the error:

```sh
2024-10-30:21:00:47,838 WARNING  [api_models.py:347] API request failed with error message: {
  "error": {
    "message": "Unsupported value: 'temperature' does not support 0.0 with this model. Only the default (1) value is supported.",
    "type": "invalid_request_error",
    "param": "temperature",
    "code": "unsupported_value"
  }
```

and 

```sh
"error": {
    "message": "Unsupported parameter: 'stop' is not supported with this model.",
    "type": "invalid_request_error",
    "param": "stop",
    "code": "unsupported_parameter"
  }
```

My dirty code to address it is,

```python
    def _create_payload(
        self,
        messages: List[Dict],
        generate=False,
        gen_kwargs: dict = None,
        seed=1234,
        **kwargs,
    ) -> dict:

        gen_kwargs.pop("do_sample", False)
        if "max_tokens" in gen_kwargs:
            max_tokens = gen_kwargs.pop("max_tokens")
        else:
            max_tokens = gen_kwargs.pop("max_gen_toks", self._max_gen_toks)
        if "temperature" in gen_kwargs:
            if gen_kwargs["temperature"] == 0.:
                gen_kwargs["temperature"] = 1
        stop = gen_kwargs.pop("until", ["
"])
        if not isinstance(stop, (list, tuple)):
            stop = [stop]
        return {
            "messages": messages,
            "model": self.model,
            "max_completion_tokens": max_tokens,
            # "temperature": temperature,
            # "stop": stop[:4],
            "seed": seed,
            **gen_kwargs,
        }
```
in the end of `lm_eval/models/openai_completions.py` based on your PR.








Close because the issue is addressed.