Repository: modelscope/evalscope
Issue: 测评自定义数据集好像有结果太短，导致BLEU score计算出错
Root cause: Environment Incompatibility

## 问题描述 / Issue Description

请简要描述您遇到的问题。 / Please briefly describe the issue you encountered.

测评自定义数据集好像有结果太短，导致BLEU score计算出错，
提示 Consider using lower n-gram order or use SmoothingFunction()，不过没找到控制n-gram或者使用SmoothingFunction的地方


## 使用的工具 / Tools Used
- [✔] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
task_cfg = TaskConfig(
    model='Qwen2.5-14B-Instruct',
    api_url='http://127.0.0.1:9002/v1/chat/completions',  # 推理服务地址
    api_key='EMPTY',
    eval_type=EvalType.SERVICE,   # 评测类型，SERVICE表示评测推理服务
    datasets=['general_qa'],  # 数据集名称(固定为data_collection表示使用混合数据集)
    limit=150,
    dataset_args={
        'general_qa': {
            'local_path': 'dataset/qa',  # 数据集ID 或 数据集本地路径
            'subset_list': ['test'],
            'metric_list': ['AverageROUGE'],
            'prompt_template': '请回答问题，并选出其中的正确答案  \n{query}'
            }
    },
    eval_batch_size=20,       # 发送请求的并发数
    generation_config={       # 模型推理配置
        'max_tokens': 20000,  # 最大生成token数，建议设置为较大值避免输出截断
        'temperature': 0.0,   # 采样温度 (deepseek 报告推荐值)
        'top_p': 0.95,        # top-p采样 (deepseek 报告推荐值)
        'n': 1                # 每个请求产生的回复数量 (注意 lmdeploy 目前只支持 n=1)
    },
)

run_task(task_cfg=task_cfg)
```

```bash
# 例如：在终端中执行的指令 / Terminal command executed
python script.py
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
2025-02-27 20:12:42,051 - evalscope - INFO - Use default settings: > few_shot_num: 0, > few_shot_split: None, > target_eval_split: test
Predicting(default): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [07:30<00:00,  3.00s/it]2025-02-27 20:20:12,506 - evalscope - INFO - Dump predictions to ./outputs/20250227_201240/predictions/Qwen2.5-14B-Instruct/general_qa_default.jsonl.
Reviewing(default):   0%|                                                                                                                                                              | 0/150 [00:00<?, ?it/s]Building prefix dict from the default dictionary ...
2025-02-27 20:20:12,540 - jieba - DEBUG - Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
2025-02-27 20:20:12,540 - jieba - DEBUG - Loading model from cache /tmp/jieba.cache
Loading model cost 0.943 seconds.
2025-02-27 20:20:13,483 - jieba - DEBUG - Loading model cost 0.943 seconds.
Prefix dict has been built successfully.
2025-02-27 20:20:13,483 - jieba - DEBUG - Prefix dict has been built successfully.
Reviewing(default):  20%|█████████████████████████████▊                                                                                                                       | 30/150 [00:01<00:03, 35.84it/s]/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 4-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
Reviewing(default):  68%|████████████████████████████████████████████████████████████████████████████████████████████████████▋                                               | 102/150 [00:02<00:00, 63.32it/s]/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 2-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/translate/bleu_score.py:577: UserWarning: 
The hypothesis contains 0 counts of 3-gram overlaps.
Therefore the BLEU score evaluates to 0, independently of
how many N-gram overlaps of lower order it contains.
Consider using lower n-gram order or use SmoothingFunction()
  warnings.warn(_msg)
Reviewing(default):  99%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████  | 148/150 [00:03<00:00, 41.73it/s]Traceback (most recent call last):
  File "/mnt/home/evalscope/EvalScopeQA.py", line 32, in <module>
    run_task(task_cfg=task_cfg)
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
               ^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 374, in eval
    reviews_list: list = self.get_reviews(subset_name=subset_name, answers_list=answers_list, **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 257, in get_reviews
    review_d = self._get_review(answer_d=answer_d, review_id=review_id, reviewer_spec=reviewer_spec)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/evaluator/evaluator.py", line 207, in _get_review
    review_result = self.data_adapter.match(gold_content, answer_content)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/benchmarks/general_qa/general_qa_adapter.py", line 105, in match
    bleu_dict = bleu_ngram_one_sample(pred, gold)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/evalscope/metrics/metrics.py", line 187, in bleu_ngram_one_sample
    predict = list(jieba.cut(predict)) if is_contains_chinese(predict) else word_tokenize(predict)
                                                                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/__init__.py", line 142, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/__init__.py", line 119, in sent_tokenize
    tokenizer = _get_punkt_tokenizer(language)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/__init__.py", line 105, in _get_punkt_tokenizer
    return PunktTokenizer(language)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/punkt.py", line 1744, in __init__
    self.load_lang(lang)
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/punkt.py", line 1750, in load_lang
    self._params = load_punkt_params(lang_dir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/vllm/lib/python3.11/site-packages/nltk/tokenize/punkt.py", line 1763, in load_punkt_params
    with open(f"{lang_dir}/collocations.tab", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: '/root/nltk_data/tokenizers/punkt_tab.zip/punkt_tab/english/collocations.tab'
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - [✔] Ubuntu

- Python版本 / Python Version:
  - [✔ ] 3.11
  - [ ] 3.10
  - [ ] 3.9


## 其他信息 / Additional Information

如果有其他相关信息，请在此处提供。 / If there is any other relevant information, please provide it here.


报错原因看起来是nltk包数据路径不对，手动解压一下这个 `/root/nltk_data/tokenizers/punkt_tab.zip`

> 报错原因看起来是nltk包数据路径不对，手动解压一下这个 `/root/nltk_data/tokenizers/punkt_tab.zip`

确实是这个原因，这是之前运行报错按提示手动拷过来的。不过我有点不明白了，做解压操作前限制每次测评100条是不会报错的。
另外，n-gram报错还是在的，所以说不影响结果吗？

n-gram 这个是warning，不是报错，意思是模型的输出打分是0，不影响结果

> 感谢你的反馈！我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！