Repository: modelscope/evalscope
Issue: TAU-Bench 测评结果全是0.
Root cause: Configuration Error

## 自查清单

在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题

## 问题描述

请简要描述您遇到的问题。

## EvalScope 版本（必填）
v0.xx.x

## 使用的工具
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式

## 执行的代码或指令

请提供您执行的主要代码或指令。

## 错误日志

请粘贴完整的错误日志或控制台输出。

## 运行环境

- 操作系统：
- Python版本：

## 其他信息

如果有其他相关信息，请在此处提供。

按照文档 https://evalscope.readthedocs.io/en/latest/third_party/tau_bench.html 的说明，在 NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4 模型上测评 TAU-Bench，使用的脚本如下：

```
import os
from evalscope import TaskConfig, run_task

os.environ['DASHSCOPE_API_KEY'] = "123456"
task_cfg = TaskConfig(
    model='NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',
    api_url='http://localhost:8000/v1',
    api_key=os.getenv('DASHSCOPE_API_KEY'),
    eval_type='openai_api',  # Use API model service
    datasets=['tau_bench'],
    dataset_args={
        'tau_bench': {
            'subset_list': ['airline', 'retail'],  # Select evaluation domains
            'extra_params': {
                'user_model': 'NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',  # User simulation model
                'api_key': os.getenv('DASHSCOPE_API_KEY'),
                'api_base': 'http://localhost:8000/v1',
                'generation_config': {
                    'temperature': 0.7,
                    'max_new_tokens': 1024
                }
            }
        }
    },
    eval_batch_size=1,
    # limit=5,  # Limit the number of evaluations for quick testing; remove for full evaluation
    generation_config={
        'temperature': 0.6,
        'n': 1,
        'max_tokens': 1024,
    },
)

run_task(task_cfg=task_cfg)
```

这是测评结果

<img width="837" height="141" alt="Image" src="https://github.com/user-attachments/assets/2ad26f63-f473-4a84-af6c-b18076d4f703" />

但是测评结果全是 0，不知道这是为什么。

两个模型的 max_tokens 都设置大一些，例如设置为12800：

```python
task_cfg = TaskConfig(
    model='NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',
    api_url='http://localhost:8000/v1',
    api_key=os.getenv('DASHSCOPE_API_KEY'),
    eval_type='openai_api',  # Use API model service
    datasets=['tau_bench'],
    dataset_args={
        'tau_bench': {
            'subset_list': ['airline', 'retail'],  # Select evaluation domains
            'extra_params': {
                'user_model': 'NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',  # User simulation model
                'api_key': os.getenv('DASHSCOPE_API_KEY'),
                'api_base': 'http://localhost:8000/v1',
                'generation_config': {
                    'temperature': 0.7,
                    'max_new_tokens': 12800
                }
            }
        }
    },
    eval_batch_size=1,
    # limit=5,  # Limit the number of evaluations for quick testing; remove for full evaluation
    generation_config={
        'temperature': 0.6,
        'n': 1,
        'max_tokens': 12800,
    },
)

run_task(task_cfg=task_cfg)
```

> 两个模型的 max_tokens 都设置大一些，例如设置为12800：
> 
> task_cfg = TaskConfig(
>     model='NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',
>     api_url='http://localhost:8000/v1',
>     api_key=os.getenv('DASHSCOPE_API_KEY'),
>     eval_type='openai_api',  # Use API model service
>     datasets=['tau_bench'],
>     dataset_args={
>         'tau_bench': {
>             'subset_list': ['airline', 'retail'],  # Select evaluation domains
>             'extra_params': {
>                 'user_model': 'NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4',  # User simulation model
>                 'api_key': os.getenv('DASHSCOPE_API_KEY'),
>                 'api_base': 'http://localhost:8000/v1',
>                 'generation_config': {
>                     'temperature': 0.7,
>                     'max_new_tokens': 12800
>                 }
>             }
>         }
>     },
>     eval_batch_size=1,
>     # limit=5,  # Limit the number of evaluations for quick testing; remove for full evaluation
>     generation_config={
>         'temperature': 0.6,
>         'n': 1,
>         'max_tokens': 12800,
>     },
> )
> 
> run_task(task_cfg=task_cfg)

<img width="1960" height="578" alt="Image" src="https://github.com/user-attachments/assets/12cc62bb-c340-412f-a712-7475ebe85910" />

请求长度太长了，超过了模型的最大上下文长度，然后就一直不停地重试，这个工具有办法筛选请求的长度吗？

1. 不支持筛选请求长度
2. 可以拉main分支代码再试试，多轮请求中把模型reasoning内容去掉了

I have same issue with tau:
```python
task_cfg = TaskConfig(
    model="qwen3",
    api_url="",
    api_key="xxxx",
    eval_type="openai_api",
    datasets=["tau_bench"],
    dataset_args={
        "tau_bench": {
            "extra_params": {
                "api_key": "xxx",
                "underscore_to_dot": True,
                "is_fc_model": False,
                "api_base": "",
                "user_model": "qwen3",
                'generation_config': {
                    'temperature': 0.7,
                    'max_new_tokens': 32000
                }
            }
        }
    },
    limit=10,
    eval_batch_size=10,
    generation_config={
        'temperature': 0.7,
        'n': 1,
        'max_tokens': 32000,
    },
)
```

Still get results 0:
```terminal
+------------+-----------+-------------+----------+-------+---------+---------+
| Model      | Dataset   | Metric      | Subset   |   Num |   Score | Cat.0   |
+============+===========+=============+==========+=======+=========+=========+
| qwen3 | tau_bench | mean_Pass^1 | airline  |    10 |       0 | default |
+------------+-----------+-------------+----------+-------+---------+---------+
| qwen3 | tau_bench | mean_Pass^1 | retail   |    10 |       0 | default |
+------------+-----------+-------------+----------+-------+---------+---------+
| qwen3 | tau_bench | mean_Pass^1 | OVERALL  |    20 |       0 | -       |
+------------+-----------+-------------+----------+-------+---------+---------+ 
```

> I have same issue with tau:
> 
> task_cfg = TaskConfig(
>     model="qwen3",
>     api_url="",
>     api_key="xxxx",
>     eval_type="openai_api",
>     datasets=["tau_bench"],
>     dataset_args={
>         "tau_bench": {
>             "extra_params": {
>                 "api_key": "xxx",
>                 "underscore_to_dot": True,
>                 "is_fc_model": False,
>                 "api_base": "",
>                 "user_model": "qwen3",
>                 'generation_config': {
>                     'temperature': 0.7,
>                     'max_new_tokens': 32000
>                 }
>             }
>         }
>     },
>     limit=10,
>     eval_batch_size=10,
>     generation_config={
>         'temperature': 0.7,
>         'n': 1,
>         'max_tokens': 32000,
>     },
> )
> Still get results 0:
> 
> ```
> +------------+-----------+-------------+----------+-------+---------+---------+
> | Model      | Dataset   | Metric      | Subset   |   Num |   Score | Cat.0   |
> +============+===========+=============+==========+=======+=========+=========+
> | qwen3 | tau_bench | mean_Pass^1 | airline  |    10 |       0 | default |
> +------------+-----------+-------------+----------+-------+---------+---------+
> | qwen3 | tau_bench | mean_Pass^1 | retail   |    10 |       0 | default |
> +------------+-----------+-------------+----------+-------+---------+---------+
> | qwen3 | tau_bench | mean_Pass^1 | OVERALL  |    20 |       0 | -       |
> +------------+-----------+-------------+----------+-------+---------+---------+ 
> ```

When I use vllm-0.10.0 as the inference engine, and evalscope-0.17.1 as the client, the score is not zero. This is my test script, you can try it. 

"""
```
import os
from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='qwen3-coder',
    api_url=os.getenv('API_URL'),
    api_key=os.getenv('API_KEY'),
    eval_type='service',  # 使用API模型服务
    datasets=['tau_bench'],
    dataset_args={
        'tau_bench': {
            'subset_list': ['airline', 'retail'],  # 选择评测领域
            'extra_params': {
                'user_model': 'qwen3-coder',  # 用户模拟模型
                'api_key': os.getenv('API_KEY'),
                'api_base': os.getenv('API_URL'),
                'generation_config': {
                    'temperature': 0.7,  # 降低随机性
                    'max_new_tokens': 32768
                }
            }
        }
    },
    eval_batch_size=5,
    # limit=5,  # Limit the number of evaluations for quick testing; remove for full evaluation
    generation_config={
        'temperature': 0.6,  # 降低随机性确保JSON格式正确
        'n': 1,
        'max_tokens': 32768
    },
    ignore_errors=True, 
)

run_task(task_cfg=task_cfg)
```
"""
and this is my result:
 ```
+------------+-----------+-------------+----------+-------+---------+---------+
| Model      | Dataset   | Metric      | Subset   |   Num |   Score | Cat.0   |
+============+===========+=============+==========+=======+=========+=========+
| qwen3 | tau_bench | mean_Pass^1 | airline  |    10 |       0.48 | default |
+------------+-----------+-------------+----------+-------+---------+---------+
| qwen3 | tau_bench | mean_Pass^1 | retail   |    10 |       0.6435 | default |
+------------+-----------+-------------+----------+-------+---------+---------+
| qwen3 | tau_bench | mean_Pass^1 | OVERALL  |    20 |       0.594 | -       |
+------------+-----------+-------------+----------+-------+---------+---------+ 
```


note : the test script is run on the host machine, but the docker container of inference engine.

I have tried out your method and I think that is because of the version issue, I downgraded to your evalscope version and is work.

The issue where evaluation results were returning zero has been resolved in the main branch of the code.