Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Discussion: Update GPT eval models
Root cause: Architectural Constraint

Some benchmarks (such as ActivityNet, VideoChatGPT, and many others) use `gpt-3.5-turbo-0613` for evaluation, but this model has been discontinued by OpenAI. One quick fix would be to switch to `gpt-3.5-turbo`, but I would also like to bring up the discussion whether to switch all `gpt-3.5-turbo` to `gpt-4o-mini` as its performance is better and is 3 times cheaper.

After the discussion, I'm happy to submit the PR to make the change. @Luodian @kcz358 

I would surely suggest using `gpt-4o-mini`.

We should inform users about the eval models change after certain PR, likely we should specify the changed datasets in this issue (the eval models used before/after).

I'll pin this issue and link to the PR for visibility if the PR is created.



In case this is a helpful datapoint, I tested 4o-mini perf for video eval vs. turbo-0613 and I got significantly different (worse often) results. Just something to note for consistency!

I want to start the discussion on this again, the old model is quite expensive (`gpt-4-0613	$30.00	$60.00`), and there could be a significant reduction in evaluation cost by switching to GPT-5
```
Model	Input	Cached input	Output
gpt-5	$1.25	$0.125	$10.00
gpt-5-mini	$0.25	$0.025	$2.00
gpt-5-nano	$0.05	$0.005	$0.40
```
are there any plans on switching?

E.g. a single eval pass on mmvet costs 5$ with the current price

<img width="345" height="415" alt="Image" src="https://github.com/user-attachments/assets/872359e2-a055-49aa-a30e-b38416bff8b2" />

<img width="340" height="418" alt="Image" src="https://github.com/user-attachments/assets/4e3209a7-815f-4847-b72d-fa6d81f9b03e" />

@Luodian 

Additionally, is there any way we can switch the model to a cheaper one so we can run the ablations internally to see which model is better relatively during training runs? The comparative eval would of course be with the standard model but this would allow us to reduce the eval cost significantly

Hi I would update with a more detail script on how to set the eval models for our updated `llm_judge` API.

Basically, you can set these variables in env. I think `gpt-5-nano` may be enough for the judge.

```sh
API_TYPE = os.getenv("API_TYPE", "openai") # by default I think we could use `async_openai` for large concurrency.
GPT_MODEL = os.getenv("MODEL_VERSION", "gpt-4o-2024-11-20") # it follows OpenAI's model naming rule.
```

The relevant code is here:

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/ed6a095baf5181b5635f947ade9c1b7f48146830/lmms_eval/llm_judge/providers/async_openai.py#L13

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/ed6a095baf5181b5635f947ade9c1b7f48146830/lmms_eval/llm_judge/factory.py#L28

Yes, I tried to set this env variable, but we can see in the OpenAI Console that it is always using the same legacy GPT-4 model (gpt-4-0613), do you know why this could be the case? 

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/ed6a095baf5181b5635f947ade9c1b7f48146830/lmms_eval/tasks/mmvet/utils.py#L22

Its seems like it does not take the env variable or the default. We are experimenting with `mmvet` and consistently use this legacy model for some reason

Are you in the latest branch? I think I didnt meet this error in current `main`

Indeed, it was the old version in our training env. After upgrading to the newest version It uses the correct model but I suddenly get this error fro the openai provider:
```
Running lmms-eval...
2025-08-20 15:26:27.848 | INFO     | evaluation:cli_evaluate:311 - Verbosity set to INFO
2025-08-20 15:26:30.951 | INFO     | evaluation:cli_evaluate_single:398 - Evaluation tracker args: {'output_path': 'results/'}
2025-08-20 15:26:30.951 | INFO     | evaluation:cli_evaluate_single:487 - Selected Tasks: ['mmvet']
Resize to max side len: False
Resize to max side len: False
2025-08-20 15:26:30.960 | INFO     | lmms_eval.evaluator:simple_evaluate:161 - Setting random seed to 0 | Setting numpy seed to 0 | Setting torch manual seed to 0
Model version: gpt-4o-2024-11-20
Model version: gpt-4o-2024-11-20
Model version: gpt-4o-2024-11-20
Model version: gpt-4o-2024-11-20
2025-08-20 15:26:31.658 | INFO     | lmms_eval.evaluator:evaluate:402 - Running on rank 0 (local rank 0)
2025-08-20 15:26:31.660 | INFO     | lmms_eval.api.task:build_all_requests:427 - Building contexts for mmvet on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 218/218 [00:00<00:00, 19270.34it/s]
2025-08-20 15:26:31.672 | INFO     | lmms_eval.evaluator:evaluate:490 - Running generate_until requests
Model Responding: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 218/218 [03:19<00:00,  1.09it/s]
Postprocessing:   0%|                                                                                                                                   | 0/218 [00:00<?, ?it/s]2025-08-20 15:29:52.126 | WARNING  | lmms_eval.llm_judge.providers.openai:evaluate:81 - Attempt 1/5 failed: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2025-08-20 15:30:02.333 | WARNING  | lmms_eval.llm_judge.providers.openai:evaluate:81 - Attempt 2/5 failed: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2025-08-20 15:30:12.521 | WARNING  | lmms_eval.llm_judge.providers.openai:evaluate:81 - Attempt 3/5 failed: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2025-08-20 15:30:22.747 | WARNING  | lmms_eval.llm_judge.providers.openai:evaluate:81 - Attempt 4/5 failed: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2025-08-20 15:30:32.939 | WARNING  | lmms_eval.llm_judge.providers.openai:evaluate:81 - Attempt 5/5 failed: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
2025-08-20 15:30:32.939 | ERROR    | lmms_eval.llm_judge.providers.openai:evaluate:85 - All 5 attempts failed
2025-08-20 15:30:32.939 | ERROR    | utils:get_chat_response:60 - Error: Error code: 404 - {'error': {'message': 'Invalid URL (POST /v1/chat/completions/chat/completions)', 'type': 'invalid_request_error', 'param': None, 'code': None}}
```
This seems to be a problem with the specific task (mmvet), since the same code / model works for hallusion_bench_image

I think for some certain tasks, the author (or in our early dev) hardcoded some model version in the evaluation so that this env var is not taking effect. This is done so that we can reproduce their result or align with their pipeline. Maybe we will update this slowly as we test on these benchmarks and we also welcome fix from the community.

I encountered this problem when using gpt-5-mini：
[36mlmms_eval.llm_judge.providers.azure_openai[0m:[36mevaluate[0m:[36m79[0m - [33m[1mAttempt 1/5 failed: Error code: 400 - {'error': {'message': "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.", 'type': 'invalid_request_error', 'param': 'max_tokens', 'code': 'unsupported_parameter'}}[0m [32m2025-10-04 07:52:32[0m | [33m[1mWARNING [0m |

Is there any way to solve this?

> I think for some certain tasks, the author (or in our early dev) hardcoded some model version in the evaluation so that this env var is not taking effect. This is done so that we can reproduce their result or align with their pipeline. Maybe we will update this slowly as we test on these benchmarks and we also welcome fix from the community.我认为对于某些特定任务，作者（或我们早期的开发人员）在评估中硬编码了某些模型版本，导致此环境变量无法生效。这样做是为了让我们能够重现他们的结果或与他们的流程保持一致。也许我们会在这些基准测试中慢慢更新这个问题，也欢迎社区提出修复建议。

Does the current LLM Judge support GPT5? I tried using GPT5 but encountered an error.


If you change the payload parameters from `max_tokens` to `max_completion_tokens`, does the problem solve? I am a bit lost from the different changing version of parameters in openai generate api.