Repository: modelscope/evalscope
Issue: python代码评测本地部署的DeepSeek-R1模型，出现接口请求超时的异常错误
Root cause: Configuration Error

## 问题描述 / Issue Description
可能是由于数学问题太复杂，模型推理时间过长，导致请求超时。

**测试脚本：**
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
    model='DeepSeek-R1',
    api_url='http://127.0.0.1:8801/v1/chat/completions',
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,
    datasets=[
        'data_collection',
    ],
    dataset_args={
        'data_collection': {
            'dataset_id': 'modelscope/R1-Distill-Math-Test'
        }
    },
    eval_batch_size=64,  # num of workers to seed requests
    generation_config={
        'max_tokens': 20000,  # avoid exceed max length
        'temperature': 0.6,
        'top_p': 0.95,
        'n': 5 # num of repeat for each prompt (note lmdeploy only support n=1)
    },
)

run_task(task_cfg=task_cfg)

**异常信息：**
  File "/data5/evalscope/evalscope/models/server_adapter.py", line 104, in send_request
    raise RuntimeError(f'Failed to request to remote API after {max_retries} attempts: '
RuntimeError: Failed to request to remote API after 3 attempts: 504 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2020 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!--
 /*
 * Copyright (C) 1996-2023 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
        font-family: verdana, sans-serif;
}

html body {
        margin: 0;
        padding: 0;
        background: #efefef;
        font-size: 12px;
        color: #1e1e1e;
}

/* Page displayed title area */
#titles {
        margin-left: 15px;
        padding: 10px;
        padding-left: 100px;
        background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
        color: #000000;
}
#titles h2 {
        color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
        background-color:#00ff00;
        width:100%;
}

/* Page displayed body content area */
#content {
        padding: 10px;
        background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP / Gopher directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
        margin: 0;
}

/* page displayed footer area */
#footer {
        font-size: 9px;
        padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_READ_TIMEOUT>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://10.239.121.25:8000/v1/chat/completions">http://10.239.121.25:8000/v1/cat/completions</a></p>

<blockquote id="error">
<p><b>Read Timeout</b></p>
</blockquote>


尝试通过修改request请求的timeout=3600，但还是报错。
response = requests.post(
                self.api_url, json=request_json, headers={'Authorization': f'Bearer {self.api_key}'}, timeout=3600)


请问直接请求 这个url 'http://127.0.0.1:8801/v1/chat/completions' 能通吗

> 请问直接请求 这个url 'http://127.0.0.1:8801/v1/chat/completions' 能通吗

能通，一般的问题都能够正常回答，只有复杂的数学题，需要比较长时间的推理，它就会报错。使用OpenAI设置timeout时间的话，这种需要长时间推理的问题，也能正常回复，应该是request请求的问题。
    client = OpenAI(api_key=openai_api_key, base_url=openai_api_base, timeout=3600)
    system_prompt = "Please reason step by step, and put your final answer within \\boxed{{}}."
        chat_response = client.chat.completions.create(
            model=model,
            # temperature=0.6,
            temperature=0,
            top_p=0.95,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": prompt},
            ]
        )
        response_content = chat_response.choices[0].message.content
        print(response_content)
        # 使用正则提取出jsonl格式的数据
        json_list = extract_answer(response_content)
        return json_list

我请求时没有设置超时时间，程序应该会一直等待，没遇到过这个504报错诶

https://github.com/modelscope/evalscope/blob/120e967e31c92a931cc0ba1165609f4ba1321d08/evalscope/models/server_adapter.py#L93-L97

> 我请求时没有设置超时时间，程序应该会一直等待，没遇到过这个504报错诶
> 
> [evalscope/evalscope/models/server_adapter.py](https://github.com/modelscope/evalscope/blob/120e967e31c92a931cc0ba1165609f4ba1321d08/evalscope/models/server_adapter.py#L93-L97)
> 
> Lines 93 to 97 in [120e967](/modelscope/evalscope/commit/120e967e31c92a931cc0ba1165609f4ba1321d08)
> 
>  def send_request(self, request_json: dict, max_retries: int = 3) -> dict: 
>      for attempt in range(max_retries): 
>          response = requests.post( 
>              self.api_url, json=request_json, headers={'Authorization': f'Bearer {self.api_key}'}) 
>          if response.status_code == 200:

这个requests请求，默认超时时间应该是60s，如果不设置超过60s就会报错504

添加了 `--timeout` 参数，可以拉main分支代码试试

> 添加了 `--timeout` 参数，可以拉main分支代码试试

好的，感谢！

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！

> > 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！

更新代码和配置timeout之后，还是会报错，应该跟之前错误是一样的：
测试脚本：
from evalscope import TaskConfig, run_task
from evalscope.constants import EvalType

task_cfg = TaskConfig(
model='DeepSeek-R1',
api_url='http://127.0.0.1:8801/v1/chat/completions',
api_key='EMPTY',
timeout=3600,
eval_type=EvalType.SERVICE,
datasets=[
'data_collection',
],
dataset_args={
'data_collection': {
'dataset_id': 'modelscope/R1-Distill-Math-Test'
}
},
eval_batch_size=64, # num of workers to seed requests
generation_config={
'max_tokens': 20000, # avoid exceed max length
'temperature': 0.6,
'top_p': 0.95,
'n': 5 # num of repeat for each prompt (note lmdeploy only support n=1)
},
)

异常信息：
2025-02-24 05:48:51,551 - evalscope - WARNING - Failed to request to remote API: 504 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2020 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!--
 /*
 * Copyright (C) 1996-2023 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
        font-family: verdana, sans-serif;
}

html body {
        margin: 0;
        padding: 0;
        background: #efefef;
        font-size: 12px;
        color: #1e1e1e;
}

/* Page displayed title area */
#titles {
        margin-left: 15px;
        padding: 10px;
        padding-left: 100px;
        background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
        color: #000000;
}
#titles h2 {
        color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
        background-color:#00ff00;
        width:100%;
}

/* Page displayed body content area */
#content {
        padding: 10px;
        background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP / Gopher directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
        margin: 0;
}

/* page displayed footer area */
#footer {
        font-size: 9px;
        padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_READ_TIMEOUT>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://10.239.121.26:8000/v1/chat/completions">http://10.239.121.26:8000/v1/chat/completions</a></p>

<blockquote id="error">
<p><b>Read Timeout</b></p>
</blockquote>

<p id="sysmsg">The system returned: <i>[No Error]</i></p>

<p>A Timeout occurred while waiting to read data from the network. The network or server may be down or congested. Please retry your request.</p>

<p>Your cache administrator is <a href="mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_READ_TIMEOUT&amp;body=CacheHost%3A%204368c20fb384%0D%0AErrPage%3A%20ERR_READ_TIMEOUT%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Mon,%2024%20Feb%202025%2005%3A38%3A58%20GMT%0D%0A%0D%0AClientIP%3A%2010.239.121.24%0D%0AServerIP%3A%2010.239.121.26%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Fv1%2Fchat%2Fcompletions%20HTTP%2F1.1%0AUser-Agent%3A%20python-requests%2F2.32.3%0D%0AAccept-Encoding%3A%20gzip,%20deflate%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20keep-alive%0D%0AAuthorization%3A%20Bearer%20EMPTY%0D%0AContent-Length%3A%20837%0D%0AContent-Type%3A%20application%2Fjson%0D%0AHost%3A%2010.239.121.26%3A8000%0D%0A%0D%0A%0D%0A">webmaster</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Mon, 24 Feb 2025 05:38:58 GMT by 4368c20fb384 (squid/5.9)</p>
<!-- ERR_READ_TIMEOUT -->
</div>
</body></html>

Predicting(gpqa_diamond):  25%|██████████████████████▎                                                                   | 49/198 [2:20:01<7:05:48, 171.46s/it]
Traceback (most recent call last):
  File "/data5/evalscope/evalscope/evaluator/evaluator.py", line 166, in get_answers
    answer_ds: List[dict] = future.result()
  File "/data0/miniconda3/envs/evalscope_1/lib/python3.10/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/data0/miniconda3/envs/evalscope_1/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/data0/miniconda3/envs/evalscope_1/lib/python3.10/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/data5/evalscope/evalscope/evaluator/evaluator.py", line 97, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/data5/evalscope/evalscope/models/server_adapter.py", line 47, in predict
    response = self.process_single_input(input_item, infer_cfg)
  File "/data5/evalscope/evalscope/models/server_adapter.py", line 64, in process_single_input
    response = self.send_request(request_json)
  File "/data5/evalscope/evalscope/models/server_adapter.py", line 108, in send_request
    raise RuntimeError(f'Failed to request to remote API after {max_retries} attempts: '
RuntimeError: Failed to request to remote API after 3 attempts: 504 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2020 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>
<style type="text/css"><!--
 /*
 * Copyright (C) 1996-2023 The Squid Software Foundation and contributors
 *
 * Squid software is distributed under GPLv2+ license and includes
 * contributions from numerous individuals and organizations.
 * Please see the COPYING and CONTRIBUTORS files for details.
 */

/*
 Stylesheet for Squid Error pages
 Adapted from design by Free CSS Templates
 http://www.freecsstemplates.org
 Released for free under a Creative Commons Attribution 2.5 License
*/

/* Page basics */
* {
        font-family: verdana, sans-serif;
}

html body {
        margin: 0;
        padding: 0;
        background: #efefef;
        font-size: 12px;
        color: #1e1e1e;
}

/* Page displayed title area */
#titles {
        margin-left: 15px;
        padding: 10px;
        padding-left: 100px;
        background: url('/squid-internal-static/icons/SN.png') no-repeat left;
}

/* initial title */
#titles h1 {
        color: #000000;
}
#titles h2 {
        color: #000000;
}

/* special event: FTP success page titles */
#titles ftpsuccess {
        background-color:#00ff00;
        width:100%;
}

/* Page displayed body content area */
#content {
        padding: 10px;
        background: #ffffff;
}

/* General text */
p {
}

/* error brief description */
#error p {
}

/* some data which may have caused the problem */
#data {
}

/* the error message received from the system or other software */
#sysmsg {
}

pre {
}

/* special event: FTP / Gopher directory listing */
#dirmsg {
    font-family: courier, monospace;
    color: black;
    font-size: 10pt;
}
#dirlisting {
    margin-left: 2%;
    margin-right: 2%;
}
#dirlisting tr.entry td.icon,td.filename,td.size,td.date {
    border-bottom: groove;
}
#dirlisting td.size {
    width: 50px;
    text-align: right;
    padding-right: 5px;
}

/* horizontal lines */
hr {
        margin: 0;
}

/* page displayed footer area */
#footer {
        font-size: 9px;
        padding-left: 10px;
}


body
:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }
:lang(he) { direction: rtl; }
 --></style>
</head><body id=ERR_READ_TIMEOUT>
<div id="titles">
<h1>ERROR</h1>
<h2>The requested URL could not be retrieved</h2>
</div>
<hr>

<div id="content">
<p>The following error was encountered while trying to retrieve the URL: <a href="http://10.239.121.26:8000/v1/chat/completions">http://10.239.121.26:8000/v1/chat/completions</a></p>

<blockquote id="error">
<p><b>Read Timeout</b></p>
</blockquote>

<p id="sysmsg">The system returned: <i>[No Error]</i></p>

<p>A Timeout occurred while waiting to read data from the network. The network or server may be down or congested. Please retry your request.</p>

<p>Your cache administrator is <a href="mailto:webmaster?subject=CacheErrorInfo%20-%20ERR_READ_TIMEOUT&amp;body=CacheHost%3A%204368c20fb384%0D%0AErrPage%3A%20ERR_READ_TIMEOUT%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Mon,%2024%20Feb%202025%2004%3A42%3A04%20GMT%0D%0A%0D%0AClientIP%3A%2010.239.121.24%0D%0AServerIP%3A%2010.239.121.26%0D%0A%0D%0AHTTP%20Request%3A%0D%0APOST%20%2Fv1%2Fchat%2Fcompletions%20HTTP%2F1.1%0AUser-Agent%3A%20python-requests%2F2.32.3%0D%0AAccept-Encoding%3A%20gzip,%20deflate%0D%0AAccept%3A%20*%2F*%0D%0AConnection%3A%20keep-alive%0D%0AAuthorization%3A%20Bearer%20EMPTY%0D%0AContent-Length%3A%20752%0D%0AContent-Type%3A%20application%2Fjson%0D%0AHost%3A%2010.239.121.26%3A8000%0D%0A%0D%0A%0D%0A">webmaster</a>.</p>
<br>
</div>

<hr>
<div id="footer">
<p>Generated Mon, 24 Feb 2025 04:42:04 GMT by 4368c20fb384 (squid/5.9)</p>
<!-- ERR_READ_TIMEOUT -->
</div>
</body></html>


模型服务是用什么部署的呢，感觉跟服务端有关

> 模型服务是用什么部署的呢，感觉跟服务端有关

使用vllm部署的。但是，本地跑脚本调用该服务接口，如果使用OpenAI接口请求，不设置timeout时间的话也会报504的错误，设置timeout时间后就没问题了。你们要不也改成openai请求的接口看看？可能是requests和openai包的区别
openai_api_key = "EMPTY"
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base, timeout=3600)
system_prompt = "Please reason step by step, and put your final answer within \\boxed{{}}."
chat_response = client.chat.completions.create(
            model=model,
            temperature=0.6,
            temperature=0,
            top_p=0.95,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": prompt},
            ]
        )
        response_content = chat_response.choices[0].message.content

R1类复杂的问题，reasoning_content的生成时间可能超过5min以上，有的可能达到10min的量级，类似场景建议使用stream模式。

> R1类复杂的问题，reasoning_content的生成时间可能超过5min以上，有的可能达到10min的量级，类似场景建议使用stream模式。

请问 service 类型的评测支持 stream 模式吗？翻了下代码好像没有。

> > R1类复杂的问题，reasoning_content的生成时间可能超过5min以上，有的可能达到10min的量级，类似场景建议使用stream模式。
> 
> 请问 service 类型的评测支持 stream 模式吗？翻了下代码好像没有。

目前正在修改这块的逻辑

代码已更新，可以使用main分支代码来使用`stream`参数，同时请求部分改为了直接使用 openai 库

> 代码已更新，可以使用main分支代码来使用`stream`参数，同时请求部分改为了直接使用 openai 库

请问最新的main分支代码为什么把stream 参数又给删了？我回退了3个commit才行😂

解决冲突有点问题，一会儿处理一下

> > 代码已更新，可以使用main分支代码来使用`stream`参数，同时请求部分改为了直接使用 openai 库
> 
> 请问最新的main分支代码为什么把stream 参数又给删了？我回退了3个commit才行😂

已修复

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！