Repository: modelscope/evalscope
Issue: Critical Bug: Hardcoded agent='llm_agent_gt' in tau2bench Evaluation Causes Severely Inflated Scores
Root cause: Architectural Constraint

## Self-Check List

Before submitting an issue, please ensure you have completed the following steps:

- [x] I have carefully read the [[relevant user documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
- [x] I have reviewed the [[Frequently Asked Questions](https://evalscope.readthedocs.io/en/latest/get_started/faq.html)](https://evalscope.readthedocs.io/en/latest/get_started/faq.html)
- [x] I have searched and reviewed existing issues to confirm this is not a duplicate problem

## Problem Description

**Critical Bug Report: Default `llm_agent_gt` Mode in `tau2bench` Evaluation Causes Severely Inflated Results**

When evaluating the `tau2bench` dataset using evalscope packages installed from PyPI (v1.2.0 and v1.3.0), the `predict` function **hardcodes `agent='llm_agent_gt'` mode by default**, rather than selecting the appropriate agent model(llm_agent) based on configuration or task requirements.

This bug leads to the following critical issues:

1. **Severely Biased Evaluation Results**: Using `llm_agent_gt` (ground truth agent) mode artificially inflates evaluation scores because the agent has access to ground truth information, completely violating fairness principles in actual evaluations
2. **Misleading Users**: Users receive completely unreliable evaluation results without understanding the internal implementation
3. **Widespread Impact**: This bug existed in two officially released versions (1.2.0 and 1.3.0) and was only fixed on December 16th
4. **Code Quality Concerns**: This hardcoded default value selection lacks checks, revealing serious flaws in code quality control and testing processes

Code analysis shows the issue occurs in the `predict` function:

```python
def predict(model: Model, sample: Sample, adapter_instance) -> ModelOutput:
    build_model(agent_model=model, adapter_instance=adapter_instance)
    domain = sample.subset_key
    task = Task.model_validate(sample.metadata)
    res = run_task(
        domain=domain,
        task=task,
        agent='llm_agent_gt',  # ← Problem: Hardcoded ground truth agent
        user='user_simulator',
        llm_agent='agent',
        llm_user='user',
    )
    # ... subsequent processing
```

## EvalScope Version (Required)

- **Affected Versions**: v1.2.0, v1.3.0
- **Fixed Version**: Fixed in December 16th commit (v1.4.0)😮‍💨

## Tools Used

- [x] Native / Native framework

## Executed Code or Instructions

```bash
# Install affected versions
pip install evalscope==1.2.0  # or 1.3.0

# Run tau2bench evaluation (specific commands may vary)
evalscope eval tau2bench --model <model_name>
```

## Error Log

While there are no direct runtime errors, a **logic error** exists causing:

- Abnormally high evaluation results (unreasonably high scores)
- Reduced differentiation between different models
- Severe mismatch between actual performance and evaluation results

## Running Environment

- Operating System: Any (code logic issue independent of OS)
- Python Version: Python 3.8+

##

This is a known issue that was mentioned in issue #1038. We have fixed the problem in version v1.4.0. If you have any other issues, please feel free to raise them.