Repository: modelscope/evalscope
Issue: multilingual_e5 embeddings模型压测失败！！！
Root cause: Validation Gap

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

使用multilingual_e5进行压测任务失败

## EvalScope 版本（必填）
v0.17.1

## 使用的工具
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

服务启动
``` 
python3 -m vllm.entrypoints.openai.api_server \
--trust-remote-code \
--served-model-name multilingual_e5 \
--api-key aicp_token \
--dtype auto \
--task embed \
--trust-remote-code \
--enable-prefix-caching \
--model /work/models/intfloat/multilingual-e5-base \
--tokenizer /work/models/intfloat/multilingual-e5-base \
--gpu-memory-utilization 0.95 \
--tensor-parallel-size 1 \
--chat_template "{%- if messages[0]['role'] == 'system' -%}    {%- set system_message = messages[0]['content'] -%}    {%- set messages = messages[1:] -%}{%- else -%}    {% set system_message = '' -%}{%- endif -%}{{ bos_token + system_message }}{%- for message in messages -%}    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}    {%- endif -%}    {%- if message['role'] == 'user' -%}        {{ 'USER: ' + message['content'] + '\n' }}    {%- elif message['role'] == 'assistant' -%}        {{ 'ASSISTANT: ' + message['content'] + eos_token + '\n' }}    {%- endif -%}{%- endfor -%}{%- if add_generation_prompt -%}    {{ 'ASSISTANT:' }} {% endif %}"
``` 
压测脚本
``` 
from evalscope import TaskConfig
task_cfg = TaskConfig(
    eval_backend='RAGEval',
    eval_config={
        'tool': 'MTEB',
        'model': [
            {
                'model_name': 'multilingual_e5',
                'api_base': 'http://0.0.0.0:8000/v1/',
                'api_key': 'aicp_token',
                'dimensions': None,
                'encode_kwargs': {
                    'batch_size': 10,
                    'truncation': True,
                    'max_length': 512,
                },
            }
        ],
        'eval': {
            'tasks': [
                'T2Retrieval',
            ],
            'verbosity': 2,
            'overwrite_results': True,
            'limits': 30,
        },
    },
)

from evalscope.run import run_task
run_task(task_cfg=task_cfg)
``` 

## 错误日志

``` 
2025-08-08 06:12:11,123 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
2025-08-08 06:12:17,006 - evalscope - INFO - Dump task config to ./outputs/20250808_061211/configs/task_config_a478aa.yaml
2025-08-08 06:12:17,008 - evalscope - INFO - {
    "model": "DummyCustomModel",
    "model_id": "custom_model",
    "model_args": {},
    "model_task": "text_generation",
    "template_type": null,
    "chat_template": null,
    "datasets": [],
    "dataset_args": {},
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "generation_config": {},
    "eval_type": "custom",
    "eval_backend": "RAGEval",
    "eval_config": {
        "tool": "MTEB",
        "model": [
            {
                "model_name": "multilingual_e5",
                "api_base": "http://0.0.0.0:8000/v1/",
                "api_key": "aicp_token",
                "dimensions": null,
                "encode_kwargs": {
                    "batch_size": 10,
                    "truncation": true,
                    "max_length": 512
                }
            }
        ],
        "eval": {
            "tasks": [
                "T2Retrieval"
            ],
            "verbosity": 2,
            "overwrite_results": true,
            "limits": 30,
            "output_folder": "./outputs/20250808_061211"
        }
    },
    "stage": "all",
    "limit": null,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs/20250808_061211",
    "outputs": null,
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
2025-08-08 06:12:17,008 - evalscope - INFO - Check `mteb` Installed
2025-08-08 06:12:17,782 - mteb.evaluation.MTEB - INFO -

## Evaluating 1 tasks:
────────────────────────────────────────────────────────── Selected tasks  ───────────────────────────────────────────────────────────
Retrieval
    - T2Retrieval, s2p


2025-08-08 06:12:17,796 - mteb.evaluation.MTEB - INFO -

********************** Evaluating T2Retrieval **********************
Using custom data configuration default-9f4aa3582522b311
2025-08-08 06:12:26,512 - datasets.builder - INFO - Using custom data configuration default-9f4aa3582522b311
Overwrite dataset info from restored data version if exists.
2025-08-08 06:12:26,513 - datasets.builder - INFO - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master
2025-08-08 06:12:26,513 - datasets.info - INFO - Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master
Found cached dataset t2_retrieval (/root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master)
2025-08-08 06:12:26,531 - datasets.builder - INFO - Found cached dataset t2_retrieval (/root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master)
Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master
2025-08-08 06:12:26,531 - datasets.info - INFO - Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval/default-9f4aa3582522b311/0.0.0/master
Using custom data configuration default-5f2523aab06dcfc9
2025-08-08 06:12:34,690 - datasets.builder - INFO - Using custom data configuration default-5f2523aab06dcfc9
Overwrite dataset info from restored data version if exists.
2025-08-08 06:12:34,690 - datasets.builder - INFO - Overwrite dataset info from restored data version if exists.
Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master
2025-08-08 06:12:34,690 - datasets.info - INFO - Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master
Found cached dataset t2_retrieval-qrels (/root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master)
2025-08-08 06:12:34,691 - datasets.builder - INFO - Found cached dataset t2_retrieval-qrels (/root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master)
Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master
2025-08-08 06:12:34,692 - datasets.info - INFO - Loading Dataset info from /root/.cache/modelscope/hub/datasets/C-MTEB___t2_retrieval-qrels/default-5f2523aab06dcfc9/0.0.0/master
2025-08-08 06:12:37,514 - mteb.abstasks.AbsTaskRetrieval - INFO - Subset: default
2025-08-08 06:12:37,514 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - Encoding Queries.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 43.15it/s]
2025-08-08 06:12:37,588 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - Sorting Corpus by document length (Longest first)...
2025-08-08 06:12:37,588 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - Encoding Corpus in batches... Warning: This might take a while!
2025-08-08 06:12:37,588 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - Encoding Batch 1/1...
  0%|                                                                                                           | 0/3 [00:00<?, ?it/s]
2025-08-08 06:12:37,597 - mteb.evaluation.MTEB - ERROR - Error while evaluating T2Retrieval: Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 512 tokens. However, you requested 2096 tokens in the input for embedding generation. Please reduce the length of the input.", 'type': 'BadRequestError', 'param': None, 'code': 400}
Traceback (most recent call last):
  File "/vllm-workspace/test.py", line 31, in <module>
    run_task(task_cfg=task_cfg)
  File "/opt/venv/lib/python3.12/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/evalscope/run.py", line 42, in run_single_task
    result = run_non_native_backend(task_cfg, outputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/evalscope/run.py", line 92, in run_non_native_backend
    backend_manager.run()
  File "/opt/venv/lib/python3.12/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 83, in run
    self.run_mteb(model_args, eval_args)
  File "/opt/venv/lib/python3.12/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 47, in run_mteb
    one_stage_eval(model_args_list[0], eval_args)
  File "/opt/venv/lib/python3.12/site-packages/evalscope/backend/rag_eval/cmteb/task_template.py", line 52, in one_stage_eval
    results = evaluation.run(model, **eval_args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/MTEB.py", line 672, in run
    raise e
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/MTEB.py", line 625, in run
    results, tick, tock = self._run_eval(
                          ^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/MTEB.py", line 307, in _run_eval
    results = task.evaluate(
              ^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 344, in evaluate
    scores[hf_subset] = self._evaluate_subset(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 353, in _evaluate_subset
    results = retriever(corpus, queries)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 485, in __call__
    return self.retriever.search(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 159, in search
    sub_corpus_embeddings = self.model.encode(
                            ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 420, in encode
    return self.encode_corpus(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 400, in encode_corpus
    corpus_embeddings = self.model.encode(
                        ^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/evalscope/backend/rag_eval/utils/embedding.py", line 232, in encode
    response = self.model.embed_documents(batch_texts, chunk_size=self.batch_size)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/langchain_openai/embeddings/base.py", line 579, in embed_documents
    response = self.client.create(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/openai/resources/embeddings.py", line 132, in create
    return self._post(
           ^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/openai/_base_client.py", line 1259, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/openai/_base_client.py", line 1047, in request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "This model's maximum context length is 512 tokens. However, you requested 2096 tokens in the input for embedding generation. Please reduce the length of the input.", 'type': 'BadRequestError', 'param': None, 'code': 400}
``` 

## 运行环境

- 操作系统：ubuntu22.04
- Python版本：3.10

## 其他信息

如果有其他相关信息，请在此处提供。


You're running into a known EvalScope limitation — it doesn’t truncate or segment long texts before calling multilingual_e5, which has a hard cap of 512 tokens. This causes the BadRequestError you're seeing.

We’ve solved this exact issue in our own evaluation pipeline by introducing a dynamic input segmenter and pre-embedding controller.

If you're interested in a reproducible patch or full integration guide, just let me know — happy to share the exact fix we've open-sourced.

使用main分支代码，已添加了check_embedding_ctx_length参数，设置为True来自动检查embedding的最大长度（会自动下载gpt使用的tokenizer，请确保网络链接正常）：
```python
"model": [
            {
                "model_name": "multilingual_e5",
                "api_base": "http://0.0.0.0:8000/v1/",
                "api_key": "aicp_token",
                "dimensions": null,
                "check_embedding_ctx_length": True
...
```

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！