Repository: modelscope/evalscope
Issue: vllm0.8.5测试报403错误
Root cause: Environment Incompatibility

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

使用vllm0.8.5拉起openai接口服务，使用eval的openai接口测试，始终报403错误，但用openai库单独测试正常。


## EvalScope 版本（必填）
0.17.1

## 使用的工具
- [ x] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

vllm启动代码：
```bash
mdl_path=/mdl/mdl_zoo/Qwen/Qwen3-4B

CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \
  --served-model-name Qwen3 \
   --model ${mdl_path} \
   --block-size 16 \
   --gpu-memory-utilization 0.99 \
   --swap-space 80 \
   --max-model-len 32000 \
   --enforce-eager \
   --tensor-parallel-size 2 \
   --port 8000 \
   --enable-reasoning --reasoning-parser deepseek_r1 \
```

测试代码：(其中试了localhost, 0.0.0.0, 127.0.0.1以及http, https都不行)

```bash
evalscope eval \
 --model Qwen3 \
 --generation-config '{"do_sample":true,"temperature":0.6,"max_new_tokens":2048,"chat_template_kwargs":{"enable_thinking": true}}' \
 --api-url http://localhost:8000/v1/chat/completions \
 --eval-type service \
 --datasets aime24 aime25 math_500 \
 --eval-batch-size 32
```

## 错误日志

openai接口403无权限访问错误

## 运行环境

- 操作系统：ubuntu 22.04
- Python版本：3.11

## 其他信息

如果有其他相关信息，请在此处提供。


提供一下具体的报错日志

```bash
to Error code: 403
Predicting(default):   0%|                                                                                | 0/30 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/miniconda3/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
             ^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/cli/start_eval.py", line 32, in execute
    run_task(self.args)
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/run.py", line 44, in run_single_task
    result = evaluate_model(task_cfg, outputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/run.py", line 131, in evaluate_model
    res_dict = evaluator.eval()
               ^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 460, in eval
    answers_list: list = self.get_answers(
                         ^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 194, in get_answers
    answer_ds: List[dict] = future.result()
                            ^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/mnt/miniconda3/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 123, in _get_answer
    raise e
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 115, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/models/adapters/server_adapter.py", line 60, in predict
    response = self.process_single_input(input_item, infer_cfg)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/models/adapters/server_adapter.py", line 68, in process_single_input
    response = self.send_request(request_json)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/models/adapters/server_adapter.py", line 144, in send_request
    raise e
  File "/mnt/miniconda3/lib/python3.11/site-packages/evalscope/models/adapters/server_adapter.py", line 136, in send_request
    response = self.client.chat.completions.create(**parsed_request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/openai/_utils/_utils.py", line 287, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py", line 925, in create
    return self._post(
           ^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/openai/_base_client.py", line 1249, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/miniconda3/lib/python3.11/site-packages/openai/_base_client.py", line 1037, in request
    raise self._make_status_error_from_response(err.response) from None
openai.PermissionDeniedError: Error code: 403
```

是不是和vllm0.8.5不兼容

升级到最新的vllm再试试呢，没出现过这个情况

升级到vllm 0.10.0仍然出现这个错误。新建了2个conda环境，一个装vllm，一个装evalscope，仍然存在问题

我这边无法复现这个问题

> 升级到vllm 0.10.0仍然出现这个错误。新建了2个conda环境，一个装vllm，一个装evalscope，仍然存在问题

请问您解决了吗？我报404错误

发现是端口被限制了



---原始邮件---
发件人: ***@***.***&gt;
发送时间: 2025年8月15日(周五) 晚上7:18
收件人: ***@***.***&gt;;
抄送: "Optimus ***@***.******@***.***&gt;;
主题: Re: [modelscope/evalscope] vllm0.8.5测试报403错误 (Issue #737)


Dyw08 left a comment (modelscope/evalscope#737)
  
升级到vllm 0.10.0仍然出现这个错误。新建了2个conda环境，一个装vllm，一个装evalscope，仍然存在问题
  
请问您解决了吗？我报404错误
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！