Repository: modelscope/evalscope
Issue: 用vllm起服务总是报几次retrying request问题后，就开始报 Error when calling OpenAI API: Request timed out.
Root cause: Configuration Error

## 问题描述 采用vllm推理报错

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

## 使用的工具 / Tools Used
- [ ] VLMEvalKit backend

## 执行的代码或指令
from evalscope import TaskConfig, run_task
#  start the task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='DeepSeek-R1-Distill-Qwen-1.5B',
    api_url='http://127.0.0.1:8801/v1/chat/completions',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    datasets=[
        'data_collection',
    ],
    dataset_args={
        'data_collection': {
            'local_path': 'outputs/deepseek_distill_test.jsonl'
        }
    },
    eval_batch_size=32,
    generation_config={
        'max_tokens': 30000,  # avoid exceed max length
        'temperature': 0.6,
        'top_p': 0.95,
        'n': 1,  # num of responses per sample, note that lmdeploy only supports n=1
    },
)

run_task(task_cfg=task_cfg)



## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
Getting answers:  27%|██████████████████████████████▉                                                                                    | 196/728 [29:25<3:20:40, 22.63s/it]2025-02-28 16:10:36,431 - openai._base_client - INFO - Retrying request to /chat/completions in 0.750970 seconds
2025-02-28 16:10:36,476 - openai._base_client - INFO - Retrying request to /chat/completions in 0.867721 seconds
2025-02-28 16:10:36,510 - openai._base_client - INFO - Retrying request to /chat/completions in 0.757230 seconds
2025-02-28 16:10:36,533 - openai._base_client - INFO - Retrying request to /chat/completions in 0.784805 seconds
2025-02-28 16:10:36,550 - openai._base_client - INFO - Retrying request to /chat/completions in 0.997130 seconds
2025-02-28 16:10:36,586 - openai._base_client - INFO - Retrying request to /chat/completions in 0.819820 seconds
2025-02-28 16:10:36,591 - openai._base_client - INFO - Retrying request to /chat/completions in 0.829572 seconds
2025-02-28 16:10:36,690 - openai._base_client - INFO - Retrying request to /chat/completions in 0.865757 seconds
2025-02-28 16:10:36,724 - openai._base_client - INFO - Retrying request to /chat/completions in 0.933294 seconds
2025-02-28 16:10:47,161 - openai._base_client - INFO - Retrying request to /chat/completions in 0.419880 seconds
2025-02-28 16:11:15,998 - openai._base_client - INFO - Retrying request to /chat/completions in 0.486056 seconds
2025-02-28 16:11:19,194 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:22,958 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:23,311 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:25,672 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:26,492 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:27,631 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
2025-02-28 16:11:27,679 - evalscope - ERROR - Error when calling OpenAI API: Request timed out.
```

## 运行环境 / Runtime Environment
A800 单卡 

- 操作系统 / Operating System:
  - [ ] Ubuntu

- Python3.10
- vllm 0.7.2


不固定，可能在跑了10%-30%的任意时间点发生上述错误


模型输出比较长，尝试加上`stream`参数, `timeout`加大
```python
    generation_config={       # 模型推理配置
        'max_tokens': 20000,  # 最大生成token数，建议设置为较大值避免输出截断
        'temperature': 0.6,   # 采样温度 (deepseek 报告推荐值)
        'top_p': 0.95,        # top-p采样 (deepseek 报告推荐值)
        'n': 1                # 每个请求产生的回复数量 (注意 lmdeploy 目前只支持 n=1)
    },
    stream=True,               # 是否使用流式请求，推荐设置为True防止请求超时
    timeout=60000,
)
```

@Yunnglin 感谢 确实可以

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！