Repository: open-compass/VLMEvalKit
Issue: How to evaluate MathVista with GPT-like closed-source api?
Root cause: Interface Contract Mismatch

Hi, I tried the following command and got the response successfully.

```
>>>messages = [{'role': 'user', 'content': 'what is your name?'}]
>>> openai_client = openai.OpenAI(api_key='xxx',base_url="https:/xxx.com/api/openai/v1")
>>> response = openai_client.chat.completions.create(model='gpt-3.5-turbo', messages=messages)
>>> print(response)
ChatCompletion(id=None, choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='I am an AI digital assistant and do not have a personal name. You can refer to me as Assistant.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None))], created=None, model=None, object=None, service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=22, prompt_tokens=10, total_tokens=32, completion_tokens_details=None, prompt_tokens_details=None, cache_creation_input_tokens=None, cache_read_input_tokens=None))
```

However, when I set key and api-base as environment variables, 

```
export OPENAI_API_KEY=xxx
python run.py --data MathVista_MINI --model Qwen2.5-VL-7B-Instruct --verbose
```
and update gpt.py
```
APIBASES = {
    # 'OFFICIAL': 'https://api.openai.com/v1/chat/completions',
    'OFFICIAL': 'https://xxx/api/openai/v1',
}
```
it doesn't work.

The error message:

> Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|█████████▉| 999/1000 [1:51:41<00:06,  6.46s/it]
> Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|██████████| 1000/1000 [1:51:42<00:00,  4.89s/it]
> Infer Qwen2.5-VL-7B-Instruct/MathVista_MINI, Rank 0/1: 100%|██████████| 1000/1000 [1:51:42<00:00,  6.70s/it]
> [2025-05-23 02:35:33] INFO - RUN - run.py: main - 398: {'nproc': 4, 'verbose': True, 'retry': 3, 'model': 'gpt-4o-mini'}
> [2025-05-23 02:35:33] ERROR - misc.py: load_env - 219: Did not detect the .env file at /checkpoint/binary/train_package/.env, failed to load. 
> [2025-05-23 02:35:33] INFO - ChatAPI - gpt.py: __init__ - 147: Using API Base: https://xxx.com/api/openai/v1; API Key: sk-xxx
> [2025-05-23 02:35:34] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:34] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:34.268+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:34] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:34.268+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:42] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:42] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:42.988+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:42] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:42.988+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:46] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:46] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:46.323+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:46] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:46.323+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:47] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:47] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:47.390+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:47] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:47.390+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:51] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:51] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:51.178+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:51] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:51.178+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:52] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:35:52] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:35:52.337+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:35:52] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:35:52.337+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:01.070+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:01] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:01.070+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:01] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:01.228+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:01] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:01.228+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:08] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:08] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:08.129+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:08] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:08.129+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:15.114+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:15] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:15.114+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:15] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:15.223+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:15] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:15.223+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:23] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:23] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:23.494+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:23] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:23.494+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:25] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:25] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:25.175+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:25] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:25.175+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:29] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:29] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:29.783+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:29] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:29.783+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:37] ERROR - ChatAPI - gpt.py: generate_inner - 222: <class 'KeyError'>: 'choices'
> [2025-05-23 02:36:37] ERROR - ChatAPI - gpt.py: generate_inner - 223: {"timestamp":"2025-05-22T18:36:37.163+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:37] INFO - ChatAPI - base.py: generate - 257: RetCode: 404
> Answer: Failed to obtain answer via API.
> Log: {"timestamp":"2025-05-22T18:36:37.163+00:00","status":404,"error":"Not Found","message":"No message available","path":"/api/openai/v1"}
> [2025-05-23 02:36:41] ERROR - RUN - run.py: main - 478: Model Qwen2.5-VL-7B-Instruct x Dataset MathVista_MINI combination failed: MathVista evaluation requires a working OPENAI API
> 
> To debug the OpenAI API, you can try the following scripts in python:
> ```python
> from vlmeval.api import OpenAIWrapper
> model = OpenAIWrapper('gpt-4o', verbose=True)
> msgs = [dict(type='text', value='Hello!')]
> code, answer, resp = model.generate_inner(msgs)
> print(code, answer, resp)
> ```
> You cam see the specific error if the API call fails.
> , skipping this combination.
> Traceback (most recent call last):
>   File "/checkpoint/binary/train_package/run.py", line 449, in main
>     eval_results = dataset.evaluate(result_file, **judge_kwargs)
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/checkpoint/binary/train_package/vlmeval/dataset/image_vqa.py", line 229, in evaluate
>     assert model.working(), ('MathVista evaluation requires a working OPENAI API\n' + DEBUG_MESSAGE)
>            ^^^^^^^^^^^^^^^
> AssertionError: MathVista evaluation requires a working OPENAI API
> 
> To debug the OpenAI API, you can try the following scripts in python:
> ```python
> from vlmeval.api import OpenAIWrapper
> model = OpenAIWrapper('gpt-4o', verbose=True)
> msgs = [dict(type='text', value='Hello!')]
> code, answer, resp = model.generate_inner(msgs)
> print(code, answer, resp)
> ```
> You cam see the specific error if the API call fails.
> 
> [2025-05-23 02:36:44,577] [WARNING] [AiScheduler#26] [heart_beat_client_thread.py:120] HeartbeatClient: Failed to send finial message on stop status: internal_exception, detail: Internal program exception, detail: KeyError('RANK')

Hi, I guess the major problem lies in the communication method. In your navie implementation, you use `openAI` package, however, `VLMEvalKit` uses `requests.post`, this causes 404 error. There are two possible solutions:

1. modify the `gpt.py` with something like this
```
import openAI
def __init__():
    self.client = OpenAI(api_key=self.key, base_url=api_base)

def generate_inner():
    completion = self.client.chat.completions.create(
                    model=self.model,
                    messages=input_msgs,
                    max_tokens=max_tokens,
                    temperature=temperature,
                    n=1,
                    **kwargs,
                )
```

2. check `base_url` from the api providers to validate `requests.post` method, which can be verifed via
```
from vlmeval.api import OpenAIWrapper
model = OpenAIWrapper('gpt-4o', verbose=True)
msgs = [dict(type='text', value='Hello!')]
code, answer, resp = model.generate_inner(msgs)
print(code, answer, resp)
```

hope these two ways helps :)