Repository: modelscope/evalscope
Issue: bfcl_v3 数据集prompt长度超出模型的可输入长度
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

bfcl_v3 数据集prompt长度超出模型的可输入长度。

## EvalScope 版本（必填）
0.16.3

## 使用的工具
- [x] Native / 原生框架

## 执行的代码或指令
demo_1.py文件如下：

import os
from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='Qwen3-4B',
    api_url='base_url',
    api_key='api_key',
    eval_type='service',  # 使用API模型服务
    datasets=['bfcl_v3'],
    eval_batch_size=10,
    dataset_args={
        'bfcl_v3': {
            'extra_params':{
                # 模型在函数名称中拒绝使用点号（`.`）；设置此项，以便在评估期间自动将点号转换为下划线。
                'underscore_to_dot': True,
                # 模式是否为函数调用模型（Function Calling Model），如果是则会启用函数调用相关的配置；否则会使用prompt绕过函数调用。
                'is_fc_model': True,
            }
        }
    },
    generation_config={
        'temperature': 0,
        'max_tokens': 4096,
        'parallel_tool_calls': True,  # 启用并行函数调用
    },
)
run_task(task_cfg=task_cfg)
其中，'base_url'和'api_key'需要被替换，再运行python demo_1.py


## 错误日志

Traceback (most recent call last):
  File "/workspace/evalscope/demo_4.py", line 83, in <module>
    run_task(task_cfg=task_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    result = evaluate_model(task_cfg, outputs)
  File "/usr/local/lib/python3.10/site-packages/evalscope/run.py", line 131, in evaluate_model
    res_dict = evaluator.eval()
  File "/usr/local/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 455, in eval
    answers_list: list = self.get_answers(
  File "/usr/local/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 192, in get_answers
    answer_ds: List[dict] = future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 121, in _get_answer
    raise e
  File "/usr/local/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 113, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/adapters/bfcl_adapter.py", line 55, in predict
    response = self.generate_turn_with_tools(row, infer_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/adapters/bfcl_adapter.py", line 194, in generate_turn_with_tools
    responses = self.process_single_input(input_item, infer_cfg)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/adapters/server_adapter.py", line 66, in process_single_input
    response = self.send_request(request_json)
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/adapters/server_adapter.py", line 142, in send_request
    raise e
  File "/usr/local/lib/python3.10/site-packages/evalscope/models/adapters/server_adapter.py", line 134, in send_request
    response = self.client.chat.completions.create(**parsed_request)
  File "/usr/local/lib/python3.10/site-packages/openai/_utils/_utils.py", line 287, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/openai/resources/chat/completions/completions.py", line 925, in create
    return self._post(
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1249, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/usr/local/lib/python3.10/site-packages/openai/_base_client.py", line 1037, in request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 32768 tokens. However, you requested 38621 tokens (34525 in the messages, 4096 in the completion). Please reduce the length of the messages or completion. None", 'type': 'BadRequestError', 'param': None, 'code': 400}

## 运行环境

- 操作系统：鲲鹏cpu，arm64，linux
- Python版本：Python 3.10.16

## 其他信息
其他环境信息：
absl-py                       2.3.0
accelerate                    1.8.1
addict                        2.4.0
aiohappyeyeballs              2.6.1
aiohttp                       3.12.13
aiosignal                     1.3.2
annotated-types               0.7.0
anthropic                     0.49.0
antlr4-python3-runtime        4.13.2
anyio                         4.9.0
argcomplete                   3.6.2
async-timeout                 5.0.1
attrs                         25.3.0
bfcl_eval                     2025.6.23
black                         25.1.0
boto3                         1.38.44
botocore                      1.38.44
cachetools                    5.5.2
certifi                       2025.6.15
charset-normalizer            3.4.2
click                         8.2.1
cohere                        5.13.3
colorama                      0.4.6
contourpy                     1.3.2
cpm-kernels                   1.0.11
cycler                        0.12.1
dashscope                     1.23.6
datamodel-code-generator      0.25.7
datasets                      3.2.0
dill                          0.3.8
distro                        1.9.0
dnspython                     2.7.0
docstring_parser              0.16
einops                        0.8.1
email_validator               2.2.0
eval_type_backport            0.2.2
evalscope                     0.16.3
evaluate                      0.4.4
exceptiongroup                1.3.0
fastapi                       0.115.14
fastavro                      1.11.1
filelock                      3.18.0
fonttools                     4.58.4
frozenlist                    1.7.0
fsspec                        2024.9.0
func-timeout                  4.3.5
fuzzywuzzy                    0.18.0
genson                        1.3.0
google-api-core               2.25.1
google-auth                   2.40.3
google-cloud-aiplatform       1.93.1
google-cloud-bigquery         3.34.0
google-cloud-core             2.4.3
google-cloud-resource-manager 1.14.2
google-cloud-storage          2.19.0
google-crc32c                 1.7.1
google-genai                  1.20.0
google-resumable-media        2.7.2
googleapis-common-protos      1.70.0
gradio_client                 1.10.3
grpc-google-iam-v1            0.14.2
grpcio                        1.73.1
grpcio-status                 1.73.1
h11                           0.16.0
h5py                          3.14.0
hf-xet                        1.1.5
httpcore                      1.0.9
httpx                         0.28.1
httpx-sse                     0.4.0
huggingface-hub               0.33.0
idna                          3.10
immutabledict                 4.2.1
importlib_metadata            8.7.0
inflect                       5.6.2
isort                         5.13.2
jieba                         0.42.1
Jinja2                        3.1.6
jiter                         0.10.0
jmespath                      1.0.1
joblib                        1.5.1
json5                         0.12.0
jsonlines                     4.0.0
jsonschema                    4.24.0
jsonschema-specifications     2025.4.1
kiwisolver                    1.4.8
langdetect                    1.0.9
latex2sympy2_extended         1.10.1
Levenshtein                   0.27.1
lxml                          5.4.0
markdown-it-py                3.0.0
MarkupSafe                    3.0.2
matplotlib                    3.10.3
mdurl                         0.1.2
mistralai                     1.7.0
mmengine                      0.10.7
mmengine-lite                 0.10.7
modelscope                    1.27.1
mpmath                        1.3.0
multidict                     6.5.0
multiprocess                  0.70.16
mypy_extensions               1.1.0
networkx                      3.4.2
nltk                          3.9.1
numpy                         1.26.4
openai                        1.90.0
OpenCC                        1.1.9
opencompass                   0.4.2
opencv-python                 4.11.0.86
opencv-python-headless        4.11.0.86
overrides                     7.7.0
packaging                     25.0
pandas                        1.5.3
parameterized                 0.9.0
pathlib                       1.0.1
pathspec                      0.12.1
pillow                        11.2.1
pip                           23.0.1
platformdirs                  4.3.8
portalocker                   3.2.0
prettytable                   3.16.0
propcache                     0.3.2
proto-plus                    1.26.1
protobuf                      6.31.1
psutil                        7.0.0
pyarrow                       20.0.0
pyasn1                        0.6.1
pyasn1_modules                0.4.2
pydantic                      2.11.7
pydantic_core                 2.33.2
pyext                         0.7
Pygments                      2.19.2
pyparsing                     3.2.3
python-dateutil               2.9.0.post0
python-dotenv                 1.1.1
python-Levenshtein            0.27.1
pytz                          2025.2
PyYAML                        6.0.2
qwen-agent                    0.0.27
rank-bm25                     0.2.2
RapidFuzz                     3.13.0
referencing                   0.36.2
regex                         2024.11.6
requests                      2.32.4
retrying                      1.3.5
rich                          14.0.0
rouge                         1.0.1
rouge-chinese                 1.0.3
rouge-score                   0.1.2
rpds-py                       0.25.1
rsa                           4.9.1
s3transfer                    0.13.0
sacrebleu                     2.5.1
safetensors                   0.5.3
scikit-learn                  1.5.0
scipy                         1.15.3
seaborn                       0.13.2
sentence-transformers         4.1.0
setuptools                    65.5.1
shapely                       2.1.1
shellingham                   1.5.4
simplejson                    3.20.1
six                           1.17.0
sniffio                       1.3.1
sortedcontainers              2.4.0
sse-starlette                 2.3.6
starlette                     0.46.2
sympy                         1.14.0
tabulate                      0.9.0
tenacity                      9.0.0
termcolor                     3.1.0
threadpoolctl                 3.6.0
tiktoken                      0.9.0
timeout-decorator             0.5.0
tokenizers                    0.21.1
toml                          0.10.2
tomli                         2.2.1
torch                         2.7.1
tqdm                          4.67.1
transformers                  4.52.4
tree-sitter                   0.21.3
tree-sitter-java              0.21.0
tree-sitter-javascript        0.21.4
tree-sitter-languages         1.10.2
typer                         0.16.0
types-requests                2.32.4.20250611
typing_extensions             4.14.0
typing-inspection             0.4.1
tzdata                        2025.2
urllib3                       2.5.0
uvicorn                       0.34.3
wcwidth                       0.2.13
websocket-client              1.8.0
websockets                    15.0.1
wheel                         0.45.1
word2number                   1.1
writer-sdk                    2.2.0
xxhash                        3.5.0
yapf                          0.43.0
yarl                          1.20.1
zipp                          3.23.0


deepseek-r1-0528也有这个情况， generation_config={    'max_tokens': 160000,}正常，设置成 163840就会报错:
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 163840 tokens. However, you requested 163991 tokens (151 in the messages, 163840 in the completion). Please reduce the length of the messages or completion.", 'type': 'BadRequestError', 'param': None, 'code': 400}

vllm默认的max-model-len==163840

1. 可以配置模型服务，允许超过最大长度的输入
2. 也可以配置`ignore_errors`为True，可以跳过这些报错的样例

请问如何配置“允许超过最大长度的输入”？可以告诉我参数名吗？

在启动模型服务时，设置环境变量`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`

同样的问题，我想问设置generation_config={ 'max_tokens': 160000,}是直接透传给模型服务，还是会跳过评估数据集中超长的case

> 同样的问题，我想问设置generation_config={ 'max_tokens': 160000,}是直接透传给模型服务，还是会跳过评估数据集中超长的case

会放在请求参数中，传给模型服务

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！