Repository: modelscope/evalscope
Issue: 近期会提供general_qa数据集吗
Root cause: Environment Incompatibility


## EvalScope 版本（必填）
v1.0.0

## 问题描述
感谢各位提供这么优秀的模型评估工具，并配套大量的可用数据集，使我们可以通过[文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#bbh)里数据集ID，和简单的命令拼接下载可用数据集。如：
```
modelscope download --dataset evalscope/bbh --local_dir ./data/bbh 
```

请问近期是否会在魔塔社区v1.0.0版本可用的general_qa数据集

general_qa是自定义的数据集，不提供下载，使用方法请参考：https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html#qa

那，如果需要做一个数据集供evalscope 1.0.0命令行评测使用的话，是需要做成parquet文件么，还是有其他什么要求

1. 如果使用general_qa，提供对应格式的jsonl即可。
2. 如果使用更复杂的数据集，可以参考：https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html 来添加一个benchmark，数据集能够被如下代码加载即可：
```python
from modelscope import MsDataset

dataset = MsDataset.load("/path/to/your/dataset")  # 替换为你的数据集
```


### 简介
我在使用自定义general_qa数据集做测试的时候，遇到了错误。
``` log
**********************************************************************
  0/2 [00:00<?, ?it/s]2025-09-23 07:50:33 - evalscope - ERROR: Error calculating metric BLEU:
**********************************************************************
  Resource punkt_tab not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt_tab')

  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt_tab/english/

  Searched in:
    - '/root/nltk_data'
    - '/app/venv/nltk_data'
    - '/app/venv/share/nltk_data'
    - '/app/venv/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
**********************************************************************
```
问题原因是缺少nltk的包，搜索资料后，手动下载去[nltk官网](https://www.nltk.org/nltk_data/)下载punkt_tab.zip，解压并放到`/usr/lib/nltk_data/tokenizers`下后，重新评估**解决**问题。似乎是答案出现英文的原因，也可能是模型问题，换qwen3以后就好了

### 数据集内容
数据集信息是从[七律](http://data.openkg.cn/dataset/7lore)下载并整理，其中两条如下：
```jsonl
{"query": "多面潘多拉的游戏的中文名是？", "response": "多面潘多拉的游戏"}
{"query": "五代纷争辽金英烈的出版社是？", "response": "辽宁少年儿童出版社"}
```
### 评测命令
在k8s pod里，使用vllm启动了一个简单的qwen2.5的0.5b的模型，然后执行以下命令评测上述数据集（数据集内容就上面两条）。
```shell
DS=general_qa && \
evalscope eval \
     --eval-type=openai_api \
     --model qwen2.5-0.5 \
     --model-id qwen2.5-0.5 \
     --datasets ${DS} \
     --dataset-args "{\"${DS}\": {\"local_path\": \"/app/data/${DS}\"}}" \
     --eval-batch-size 1 \
     --api-url http://127.0.0.1:8000/v1 \
     --api-key EMPTY
```
### 报错信息
执行时报错：
```
root@default-eval-judge-llm-job-0:/app/data/general_qa# DS=general_qa && \
evalscope eval \
     --eval-type=openai_api \
     --model qwen2.5-0.5 \
     --model-id qwen2.5-0.5 \
     --datasets ${DS} \
     --dataset-args "{\"${DS}\": {\"local_path\": \"/app/data/${DS}\"}}" \
     --eval-batch-size 1 \
     --api-url http://127.0.0.1:8000/v1 \
     --api-key EMPTY
2025-09-23 07:49:44 - evalscope - INFO: Args: Task config is provided with CommandLine type.
2025-09-23 07:49:44 - evalscope - INFO: Creating model qwen2.5-0.5 with eval_type=openai_api base_url=http://127.0.0.1:8000/v1, api_key=EMPTY, config=max_retries=None timeout=None batch_size=1 stream=None system_message=None max_tokens=2048 top_p=None temperature=0.0 stop_seqs=None best_of=None frequency_penalty=None presence_penalty=None logit_bias=None seed=None do_sample=None top_k=None n=None logprobs=None top_logprobs=None parallel_tool_calls=None internal_tools=None max_tool_output=None cache_prompt=None reasoning_effort=None reasoning_tokens=None reasoning_summary=None reasoning_history=None response_schema=None extra_body=None height=None width=None num_inference_steps=None guidance_scale=None, model_args={}
2025-09-23 07:49:45 - evalscope - INFO: Dump task config to ./outputs/20250923_074944/configs/task_config_8f5b32.yaml
2025-09-23 07:49:45 - evalscope - INFO: {
    "model": "qwen2.5-0.5",
    "model_id": "qwen2.5-0.5",
    "model_args": {},
    "model_task": "text_generation",
    "chat_template": null,
    "datasets": [
        "general_qa"
    ],
    "dataset_args": {
        "general_qa": {
            "name": "general_qa",
            "dataset_id": "/app/data/general_qa",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "default"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "请回答问题\n{question}",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "General-QA",
            "description": "A general question answering dataset for custom evaluation. For detailed instructions on how to use this benchmark, please refer to the [User Guide](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html#qa).",
            "tags": [
                "QA",
                "Custom"
            ],
            "filters": null,
            "metric_list": [
                "BLEU",
                "Rouge"
            ],
            "aggregation": "mean",
            "extra_params": {}
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "repeats": 1,
    "generation_config": {
        "batch_size": 1,
        "max_tokens": 2048,
        "temperature": 0.0
    },
    "eval_type": "openai_api",
    "eval_backend": "Native",
    "eval_config": null,
    "limit": null,
    "eval_batch_size": 1,
    "use_cache": null,
    "rerun_review": false,
    "work_dir": "./outputs/20250923_074944",
    "ignore_errors": false,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8000/v1",
    "timeout": null,
    "stream": null,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false
}
Processing records: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2699.91it/s]
Predicting[general_qa@default]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:46<00:00, 23.44s/it]
Reviewing[general_qa@default]:   0%|                                                                                                                                                  | 0/2 [00:00<?, ?it/s]2025-09-23 07:50:33 - evalscope - ERROR: Error calculating metric BLEU:
**********************************************************************
  Resource punkt_tab not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt_tab')

  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt_tab/english/

  Searched in:
    - '/root/nltk_data'
    - '/app/venv/nltk_data'
    - '/app/venv/share/nltk_data'
    - '/app/venv/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
**********************************************************************

2025-09-23 07:50:33 - evalscope - ERROR: Error when review sample 0: 1 validation error for SampleScore
score
  Input should be a valid dictionary or instance of Score [type=model_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/model_type
Reviewing[general_qa@default]:  50%|█████████████████████████████████████████████████████████████████████                                                                     | 1/2 [00:01<00:01,  1.65s/it]
2025-09-23 07:50:33 - evalscope - ERROR: Error calculating metric BLEU:
**********************************************************************
  Resource punkt_tab not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt_tab')

  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt_tab/english/

  Searched in:
    - '/root/nltk_data'
    - '/app/venv/nltk_data'
    - '/app/venv/share/nltk_data'
    - '/app/venv/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
**********************************************************************

Traceback (most recent call last):
  File "/app/venv/bin/evalscope", line 7, in <module>
    sys.exit(run_cmd())
             ^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/cli/cli.py", line 27, in run_cmd
    cmd.execute()
  File "/app/venv/lib/python3.11/site-packages/evalscope/cli/start_eval.py", line 32, in execute
    run_task(self.args)
  File "/app/venv/lib/python3.11/site-packages/evalscope/run.py", line 28, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/run.py", line 41, in run_single_task
    result = evaluate_model(task_cfg, outputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/run.py", line 143, in evaluate_model
    res_dict = evaluator.eval()
               ^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 100, in eval
    subset_score = self.evaluate_subset(subset, dataset)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 127, in evaluate_subset
    sample_scores = self.get_reviews(subset, task_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 271, in get_reviews
    raise exc
  File "/app/venv/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 254, in get_reviews
    sample_score = future.result()
                   ^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/python3.11.13/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 288, in _review_task_state
    sample_score = self.benchmark.calculate_metrics(task_state=task_state)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 593, in calculate_metrics
    sample_score = SampleScore(
                   ^^^^^^^^^^^^
  File "/app/venv/lib/python3.11/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for SampleScore
score
  Input should be a valid dictionary or instance of Score [type=model_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/model_type
```


## EvalScope 版本（必填）
v1.0.1

## 问题描述
使用qwen3-32B，评估general_qa数据集时，可能由于模型回答答案过于详细，导致bleu和rouge评分较低。
**请问对于general_qa数据集，evalscope会更新评估逻辑吗？或者说现在的版本需要调整prompt或其他参数使评分更真实吗**

<img width="1718" height="1138" alt="Image" src="https://github.com/user-attachments/assets/e36025a3-901f-4e1d-820f-55814c75b702" />

<img width="1680" height="1550" alt="Image" src="https://github.com/user-attachments/assets/9ce2ac3c-20cf-4c82-a8b9-441b372ba06c" />

见 https://github.com/modelscope/evalscope/issues/855