Repository: modelscope/evalscope
Issue: [need help] failed to run benchmark with error

could you please help take a look at and shed some light?

```
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# evalscope perf  --url "http://127.0.0.1:8000/v1/chat/completions"  --parallel 1 --model mistralai/Mistral-7B-Instruct-v0.2 --number 500 --api openai --dataset longalpaca --read-timeout 600 --connect-timeout 600 --log-every-n-query 10
2025-06-25 15:22:13,771 - evalscope - INFO - Save the result to: outputs/20250625_152213/Mistral-7B-Instruct-v0.2

    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250625_152213/Mistral-7B-Instruct-v0.2",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "longalpaca",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-25 15:22:14,459 - evalscope - INFO - Test connection successful.

  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/msdatasets/ms_dataset.py", line 24, in <module>
    from modelscope.msdatasets.utils.hf_datasets_util import load_dataset_with_ctx
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 19, in <module>
    from datasets.data_files import (
ImportError: cannot import name 'get_metadata_patterns' from 'datasets.data_files' (/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/datasets/data_files.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 197, in benchmark
    async for request in get_requests(args):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 75, in get_requests
    async for request in generator:
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 46, in generate_requests_from_dataset
    for messages in message_generator:
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/plugin/datasets/longalpaca.py", line 19, in build_messages
    from modelscope import MsDataset
  File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/utils/import_utils.py", line 434, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/utils/import_utils.py", line 453, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import modelscope.msdatasets because of the following error (look up to see its traceback):
cannot import name 'get_metadata_patterns' from 'datasets.data_files' (/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/datasets/data_files.py)
2025-06-25 15:22:14,845 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<<async_generator_athrow without __name__>()>>
2025-06-25 15:22:14,845 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<statistic_benchmark_metric() running at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14>>
sys:1: RuntimeWarning: coroutine 'statistic_benchmark_metric' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# 

```

Check which version of the `datasets` library you are using, and try using version 3.2.0

`datasets` delete this function at 3.4.0 , check your version lower this 

@Yunnglin @wang-zhuo996 
```
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# pip show datasets
Name: datasets
Version: 3.6.0
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages
Requires: dill, filelock, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, pyyaml, requests, tqdm, xxhash
Required-by: evalscope
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# 
```

I reinstall with 3.20, but still suffer the same error
```
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# evalscope perf  --url "http://127.0.0.1:8000/v1/chat/completions"  --parallel 1 --model mistralai/Mistral-7B-Instruct-v0.2 --number 500 --api openai --dataset longalpaca --read-timeout 600 --connect-timeout 600 --log-every-n-query 10
2025-06-29 14:40:57,031 - evalscope - INFO - Save the result to: outputs/20250629_144057/Mistral-7B-Instruct-v0.2

    "debug": false,
    "wandb_api_key": null,
    "swanlab_api_key": null,
    "name": null,
    "outputs_dir": "outputs/20250629_144057/Mistral-7B-Instruct-v0.2",
    "max_prompt_length": 9223372036854775807,
    "min_prompt_length": 0,
    "prefix_length": 0,
    "prompt": null,
    "query_template": null,
    "apply_chat_template": true,
    "dataset": "longalpaca",
    "dataset_path": null,
    "frequency_penalty": null,
    "repetition_penalty": null,
    "logprobs": null,
    "max_tokens": 2048,
    "min_tokens": null,
    "n_choices": null,
    "seed": 0,
    "stop": null,
    "stop_token_ids": null,
    "stream": true,
    "temperature": 0.0,
    "top_p": null,
    "top_k": null,
    "extra_args": {}
}
2025-06-29 14:40:57,066 - evalscope - INFO - Test connection successful.

    from modelscope.msdatasets.utils.hf_datasets_util import load_dataset_with_ctx
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 44, in <module>
    from datasets.utils.file_utils import (OfflineModeIsEnabled,
ImportError: cannot import name 'OfflineModeIsEnabled' from 'datasets.utils.file_utils' (/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/datasets/utils/file_utils.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py", line 17, in async_wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 197, in benchmark
    async for request in get_requests(args):
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 75, in get_requests
    async for request in generator:
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/benchmark.py", line 46, in generate_requests_from_dataset
    for messages in message_generator:
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/plugin/datasets/longalpaca.py", line 19, in build_messages
    from modelscope import MsDataset
  File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/utils/import_utils.py", line 434, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/modelscope/utils/import_utils.py", line 453, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import modelscope.msdatasets because of the following error (look up to see its traceback):
cannot import name 'OfflineModeIsEnabled' from 'datasets.utils.file_utils' (/mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/datasets/utils/file_utils.py)
2025-06-29 14:40:57,352 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<<async_generator_athrow without __name__>()>>
2025-06-29 14:40:57,352 - asyncio - ERROR - Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<statistic_benchmark_metric() running at /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages/evalscope/perf/utils/handler.py:14>>
sys:1: RuntimeWarning: coroutine 'statistic_benchmark_metric' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# pip show datasets
Name: datasets
Version: 3.2.0
Summary: HuggingFace community-driven open-source library of datasets
Home-page: https://github.com/huggingface/datasets
Author: HuggingFace Inc.
Author-email: thomas@huggingface.co
License: Apache 2.0
Location: /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages
Requires: aiohttp, dill, filelock, fsspec, huggingface-hub, multiprocess, numpy, packaging, pandas, pyarrow, pyyaml, requests, tqdm, xxhash
Required-by: evalscope
(vast) root@salab-hpedl380g11-02:/mnt/nvme2n1/wayne/vast/perf/VUA# 
```

@Yunnglin @wang-zhuo996 
same error
```
evalscope==0.16.3
datasets==3.2.0
```
```
CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
evalscope eval \
    --model /root/code/outputs/checkpoint-100 \
    --generation-config '{"max_length":2048, "max_new_tokens":2048}' \
    --dataset-args '{"gsm8k": {"metric_list":["AverageAccuracy", "AveragePass@1", "Pass@1"]}}' \
    --datasets gsm8k \
    --limit 50 \
    --work-dir /root/code/outputs/eval-ckpt100
```
```
Traceback (most recent call last):
  File "/root/storage/miniconda3/envs/kkev/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/cli/start_eval.py", line 32, in execute
    run_task(self.args)
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    result = evaluate_model(task_cfg, outputs)
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/run.py", line 131, in evaluate_model
    res_dict = evaluator.eval()
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 452, in eval
    prompts = self.load_dataset()
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 72, in load_dataset
    dataset = self.data_adapter.load(
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/benchmarks/data_adapter.py", line 118, in load
    data_dict = self.load_from_hub(
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/evalscope/benchmarks/data_adapter.py", line 125, in load_from_hub
    from modelscope.msdatasets import MsDataset
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/modelscope/msdatasets/__init__.py", line 2, in <module>
    from modelscope.msdatasets.ms_dataset import MsDataset
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/modelscope/msdatasets/ms_dataset.py", line 24, in <module>
    from modelscope.msdatasets.utils.hf_datasets_util import load_dataset_with_ctx
  File "/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 44, in <module>
    from datasets.utils.file_utils import (OfflineModeIsEnabled,
ImportError: cannot import name 'OfflineModeIsEnabled' from 'datasets.utils.file_utils' (/root/storage/miniconda3/envs/kkev/lib/python3.10/site-packages/datasets/utils/file_utils.py)
```

@HT-NEKO What is the version of `modelscope` you are using?

> [@HT-NEKO](https://github.com/HT-NEKO) What is the version of `modelscope` you are using?

@Yunnglin  here is mine version.
```
# pip show evalscope
Name: evalscope
Version: 0.16.3
Summary: EvalScope: Lightweight LLMs Evaluation Framework
Home-page: https://github.com/modelscope/evalscope
Author: ModelScope team
Author-email: contact@modelscope.cn
License: 
Location: /mnt/nvme2n1/wayne/vast/lib/python3.12/site-packages
Requires: accelerate, datasets, immutabledict, jieba, jsonlines, langdetect, latex2sympy2-extended, matplotlib, modelscope, nltk, openai, pandas, pillow, pyarrow, pyyaml, requests, rouge-chinese, rouge-score, sacrebleu, scikit-learn, seaborn, sympy, tabulate, torch, tqdm, transformers, word2number
Required-by: 

```

when you install lower version `datasets` ，you shuold reinstall modelscope,like `pip install '.[all]'` . Then your command will using newly environmente.

My `datasets` version is 3.0.0 and welly run .

> [@HT-NEKO](https://github.com/HT-NEKO) What is the version of `modelscope` you are using?

sloved with `modelscope==1.27.1`
thanks!

> My `datasets` version is 3.0.0 and welly run .

sloved with `modelscope==1.27.1`
thanks!

> Thank you for your feedback! We will close this issue now. If you have any further questions, please feel free to reopen it. If EvalScope has been helpful to you, please consider giving us a STAR to show your support. Thank you!