Repository: modelscope/evalscope
Issue: 使用RAG评测中的MTEB的两阶段评测时，报错：mteb.evaluation.MTEB - ERROR - Error while evaluating T2Retrieval: FlashAttention only support fp16 and bf16 data type
Root cause: Configuration Error

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [x] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [x] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [x] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

请简要描述您遇到的问题。

## EvalScope 版本（必填）
v0.15.1

## 使用的工具
- [x] RAGEval backend


## 执行的代码或指令

from evalscope.run import run_task

one_stage_task_cfg = {
    "work_dir": "outputs",
    "eval_backend": "RAGEval",
    "eval_config": {
        "tool": "MTEB",
        "model": [
            {
                "model_name_or_path": "AI-ModelScope/m3e-base",
                "pooling_mode": None,
                "max_seq_length": 512,
                "prompt": "",
                "model_kwargs": {"torch_dtype": "auto"},
                "encode_kwargs": {
                    "batch_size": 128,
                },
            }
        ],
        "eval": {
            "tasks": [
                "TNews",
                "CLSClusteringS2S",
                "T2Reranking",
                "T2Retrieval",
                "ATEC",
            ],
            "verbosity": 2,
            "overwrite_results": True,
            "top_k": 10,
            "limits": 500,
        },
    },
}

two_stage_task_cfg = {
    "work_dir": "outputs",
    "eval_backend": "RAGEval",
    "eval_config": {
        "tool": "MTEB",
        "model": [
            {
                "model_name_or_path": "AI-ModelScope/m3e-base",
                "is_cross_encoder": False,
                "max_seq_length": 512,
                "model_kwargs": {"torch_dtype": "auto"},
                "encode_kwargs": {
                    "batch_size": 64,
                },
            },
            {
                "model_name_or_path": "OpenBMB/MiniCPM-Reranker",
                "is_cross_encoder": True,
                "max_seq_length": 512,
                "prompt": "为这个问题生成一个检索用的表示",
                "model_kwargs": {"torch_dtype": "auto"},
                "encode_kwargs": {
                    "batch_size": 32,
                },
            },
        ],
        "eval": {
            "tasks": ["T2Retrieval"],
            "verbosity": 2,
            "overwrite_results": True,
            "top_k": 5,
            "limits": 100,
        },
    },
}

run_task(task_cfg=two_stage_task_cfg)

## 错误日志

请粘贴完整的错误日志或控制台输出。
## Evaluating 1 tasks:
────────────────────────────────────────────────────────────────────── Selected tasks  ──────────────────────────────────────────────────────────────────────
Retrieval
    - T2Retrieval, s2p


2025-05-20 10:59:00,493 - mteb.evaluation.MTEB - INFO - 

********************** Evaluating T2Retrieval **********************
2025-05-20 10:59:00,494 - mteb.evaluation.MTEB - INFO - Loading dataset for T2Retrieval
2025-05-20 10:59:00,494 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - The custom predict function of the model will be used if not a SentenceTransformer CrossEncoder
2025-05-20 10:59:00,500 - mteb.abstasks.AbsTaskRetrieval - INFO - Subset: default
2025-05-20 10:59:00,503 - mteb.evaluation.evaluators.RetrievalEvaluator - INFO - Reranking the top 5 in batches... This might take a while!
  0%|                                                                                                                                  | 0/4 [00:00<?, ?it/sBe aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
2025-05-20 10:59:00,652 - transformers_modules.MiniCPM-Reranker.modeling_minicpm - WARNING - The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.float32.
Batches:   0%|                                                                                                                         | 0/4 [00:00<?, ?it/s]
2025-05-20 10:59:00,681 - mteb.evaluation.MTEB - ERROR - Error while evaluating T2Retrieval: FlashAttention only support fp16 and bf16 data type             
Traceback (most recent call last):
  File "//MTEB.py", line 75, in <module>
    run_task(task_cfg=two_stage_task_cfg)
  File "/usr/local/lib/python3.11/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/run.py", line 42, in run_single_task
    result = run_non_native_backend(task_cfg, outputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/run.py", line 89, in run_non_native_backend
    backend_manager.run()
  File "/usr/local/lib/python3.11/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 82, in run
    self.run_mteb(model_args, eval_args)
  File "/usr/local/lib/python3.11/site-packages/evalscope/backend/rag_eval/backend_manager.py", line 48, in run_mteb
    two_stage_eval(model_args_list[0], model_args_list[1], eval_args)
  File "/usr/local/lib/python3.11/site-packages/evalscope/backend/rag_eval/cmteb/task_template.py", line 71, in two_stage_eval
    results = evaluation.run(
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 477, in run
    raise e
  File "/usr/local/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 438, in run
    results, tick, tock = self._run_eval(
                          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 301, in _run_eval
    results = task.evaluate(
              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 312, in evaluate
    scores[hf_subset] = self._evaluate_subset(
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/abstasks/AbsTaskRetrieval.py", line 321, in _evaluate_subset
    results = retriever(corpus, queries)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 476, in __call__
    return self.retriever.search_cross_encoder(corpus, queries, self.top_k)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/evaluation/evaluators/RetrievalEvaluator.py", line 330, in search_cross_encoder
    scores = self.model.predict(  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mteb/models/sentence_transformer_wrapper.py", line 114, in predict
    return self.model.predict(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/evalscope/backend/rag_eval/utils/embedding.py", line 154, in predict
    embeddings = self.model.predict(sentences, **self.encode_kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/cross_encoder/util.py", line 68, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py", line 409, in predict
    model_predictions = self.model(**features, return_dict=True)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/MiniCPM-Reranker/modeling_minicpm.py", line 1420, in forward
    transformer_outputs = self.model(
                          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/MiniCPM-Reranker/modeling_minicpm.py", line 1097, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/MiniCPM-Reranker/modeling_minicpm.py", line 810, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/MiniCPM-Reranker/modeling_minicpm.py", line 563, in forward
    attn_output = self._flash_attention_forward(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/huggingface/modules/transformers_modules/MiniCPM-Reranker/modeling_minicpm.py", line 611, in _flash_attention_forward
    attn_output_unpad = flash_attn_varlen_func(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flash_attn/flash_attn_interface.py", line 1412, in flash_attn_varlen_func
    return FlashAttnVarlenFunc.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flash_attn/flash_attn_interface.py", line 901, in forward
    out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_ops.py", line 1123, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_library/custom_ops.py", line 305, in backend_impl
    result = self._backend_fns[device_type](*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
    return disable_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/_library/custom_ops.py", line 337, in wrapped_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/flash_attn/flash_attn_interface.py", line 169, in _flash_attn_varlen_forward
    out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.varlen_fwd(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: FlashAttention only support fp16 and bf16 data type
## 运行环境

- 操作系统：linux
- Python版本：3.11.11

## 其他信息

如果有其他相关信息，请在此处提供。


把model_kwargs参数去掉试试呢

![Image](https://github.com/user-attachments/assets/839d968e-b042-4ff1-b938-a9ed8a57fadf)
我把那个参数去掉以后，还是报同样的错误。

![Image](https://github.com/user-attachments/assets/1dc811c3-235b-45bb-8caa-630d816f802e)

> 把model_kwargs参数去掉试试呢

还是报同样的错误


该问题已经修复了，请拉取main分支代码来使用：
```python
two_stage_task_cfg = {
    'work_dir': 'outputs',
    'eval_backend': 'RAGEval',
    'eval_config': {
        'tool': 'MTEB',
        'model': [
            {
                'model_name_or_path': 'AI-ModelScope/m3e-base',
                'is_cross_encoder': False,
                'max_seq_length': 512,
                'prompt': '',
                'model_kwargs': {'torch_dtype': torch.float16},
                'encode_kwargs': {
                    'batch_size': 64,
                },
            },
            {
                'model_name_or_path': 'OpenBMB/MiniCPM-Reranker',
                'is_cross_encoder': True,
                'max_seq_length': 512,
                'prompt': '为这个问题生成一个检索用的表示',
                'model_kwargs': {'torch_dtype': torch.float16},
                'encode_kwargs': {
                    'batch_size': 32,
                },
            },
        ],
        'eval': {
            'tasks': ['T2Retrieval'],
            'verbosity': 2,
            'overwrite_results': True,
            'limits': 100,
        },
    },
}
```