Repository: modelscope/evalscope
Issue: evalscope 1.0版本 跑公域iquiz，抛出Couldn't find a dataset script on local side,Couldn't find 'AI-ModelScope/IQuiz' on the Hugging Face Hub either
Root cause: External Dependency Breakage

task_cfg = TaskConfig(
    # model='Qwen3-32B',
    # api_url='http://127.0.0.1:8996/v1/chat/completions',
    model=args.model,
    api_url='http://127.0.0.1:' + str(args.port) + '/v1/chat/completions',
    eval_type=args.eval_type,
    datasets=[
        f'{args.dataset}'
        # 'gsm8k','iquiz', 'aime24','mmlu','super_gpqa', 'bbh'
    ],
    dataset_args={
        'aime24': {
            'dataset_id': 'HuggingFaceH4/aime_2024',
            'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
            },
        'iquiz': {
            # 'local_path': "/usr/local/lib/python3.12/site-packages/evalscope/custom_eval/",
            # 'subset_list':['IQ', 'EQ'],
            'dataset_id': 'AI-ModelScope/IQuiz',
            'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
            },
        'mmlu': {
            # 'dataset_id': 'modelscope/mmlu',
            'dataset_id': 'cais/mmlu',   #1.0.2版本不需要dataset_id, 默认的是cais/mmlu
            'filters': {'remove_until': '</think>'},
            # 'subset_list':mmlu_subset_list
            },
        'super_gpqa': {
            'dataset_id': 'm-a-p/SuperGPQA',
            'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
            },
        'bbh': {
            'dataset_id': 'modelscope/bbh',
            'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
            }, 
        'gsm8k': {
            'dataset_id': 'modelscope/gsm8k',
            'filters': {'remove_until': '</think>'}  # Filter out the content of thinking
        },
    },
    eval_batch_size=args.batch_size,
    generation_config={
        'max_tokens': 30000,  # Max number of generated tokens, suggested to set a large value to avoid output truncation
        'temperature': args.temperature,  # Sampling temperature (recommended value per Qwen report)
        'top_p': args.top_p,  # top-p sampling (recommended value per Qwen report)
        'top_k': args.top_k,  # top-k sampling (recommended value per Qwen report)
        'n': 1,  # Number of replies generated per request
    },
    timeout=70000,  # Timeout
    stream=False,  # Use streaming output
    limit=args.sample_size,
    ignore_errors=True# Set to 100 samples for testing
)

run_task(task_cfg=task_cfg)

上面是评测任务的主要部分
执行命令是： python3 eval_public2.py --model Qwen3-30B-A3B --dataset iquiz  --sample_size 120 --eval_type server --port 8000

抛出的异常是：
19:23:26,540 - numexpr.utils - INFO: Note: NumExpr detected 44 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
2025-10-16 19:23:26,540 - numexpr.utils - INFO: NumExpr defaulting to 16 threads.
In evalscope Current temperature:  0.6
In evalscope Current top_p:  0.95
In evalscope Current top_k:  20
2025-10-16 19:23:26 - evalscope - WARNING: Deprecated: The `timeout` parameter is deprecated and will be removed in v1.1.0. Use `generation_config.timeout` instead.
2025-10-16 19:23:26 - evalscope - WARNING: Deprecated: The `stream` parameter is deprecated and will be removed in v1.1.0. Use `generation_config.stream` instead.
2025-10-16 19:23:26 - evalscope - INFO: Args: Task config is provided with TaskConfig type.
2025-10-16 19:23:27 - evalscope - INFO: Creating model Qwen3-30B-A3B with eval_type=server base_url=http://127.0.0.1:8000/v1/chat/completions, config={'timeout': 70000, 'batch_size': 10, 'stream': False, 'max_tokens': 30000, 'top_p': 0.95, 'temperature': 0.6, 'top_k': 20, 'n': 1}, model_args={}
2025-10-16 19:23:27 - evalscope - WARNING: server in model_apis.py has been deprecated since version 1.0.0. and will be removed in version 1.1.0. Use openai_api instead
2025-10-16 19:23:27 - evalscope - INFO: Dump task config to ./outputs/20251016_192326/configs/task_config_a30b61.yaml
2025-10-16 19:23:27 - evalscope - INFO: {
    "model": "Qwen3-30B-A3B",
    "model_id": "Qwen3-30B-A3B",
    "model_args": {},
    "model_task": "text_generation",
    "chat_template": null,
    "datasets": [
        "iquiz"
    ],
    "dataset_args": {
        "aime24": {
            "dataset_id": "HuggingFaceH4/aime_2024",
            "filters": {
                "remove_until": "</think>"
            }
        },
        "iquiz": {
            "name": "iquiz",
            "dataset_id": "AI-ModelScope/IQuiz",
            "output_types": [
                "generation"
            ],
            "subset_list": [
                "IQ",
                "EQ"
            ],
            "default_subset": "default",
            "few_shot_num": 0,
            "few_shot_random": false,
            "train_split": null,
            "eval_split": "test",
            "prompt_template": "回答下面的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"LETTER\"（不带引号），其中 LETTER 是 {letters} 中的一个。请在回答前进行一步步思考。\n\n问题：{question}\n选项：\n{choices}\n",
            "few_shot_prompt_template": null,
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "IQuiz",
            "description": "IQuiz is a benchmark for evaluating AI models on IQ and EQ questions. It consists of multiple-choice questions where the model must select the correct answer and provide an explanation.",
            "tags": [
                "Knowledge",
                "MCQ",
                "Chinese"
            ],
            "filters": {
                "remove_until": "</think>"
            },
            "metric_list": [
                "acc"
            ],
            "aggregation": "mean",
            "shuffle": false,
            "shuffle_choices": false,
            "review_timeout": null,
            "extra_params": {}
        },
        "mmlu": {
            "dataset_id": "cais/mmlu",
            "filters": {
                "remove_until": "</think>"
            }
        },
        "super_gpqa": {
            "dataset_id": "m-a-p/SuperGPQA",
            "filters": {
                "remove_until": "</think>"
            }
        },
        "bbh": {
            "dataset_id": "modelscope/bbh",
            "filters": {
                "remove_until": "</think>"
            }
        },
        "gsm8k": {
            "dataset_id": "modelscope/gsm8k",
            "filters": {
                "remove_until": "</think>"
            }
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "repeats": 1,
    "generation_config": {
        "timeout": 70000,
        "batch_size": 10,
        "stream": false,
        "max_tokens": 30000,
        "top_p": 0.95,
        "temperature": 0.6,
        "top_k": 20,
        "n": 1
    },
    "eval_type": "server",
    "eval_backend": "Native",
    "eval_config": null,
    "limit": 120,
    "eval_batch_size": 10,
    "use_cache": null,
    "rerun_review": false,
    "work_dir": "./outputs/20251016_192326",
    "ignore_errors": true,
    "debug": false,
    "seed": 42,
    "api_url": "http://127.0.0.1:8000/v1/chat/completions",
    "timeout": 70000,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 1,
    "judge_model_args": {},
    "analysis_report": false,
    "use_sandbox": false,
    "sandbox_type": "docker",
    "sandbox_manager_config": {},
    "sandbox_config": {}
}
2025-10-16 19:23:27 - evalscope - INFO: Loading dataset AI-ModelScope/IQuiz from modelscope > subset: IQ > split: test ...
Downloading [README.md]: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.33k/3.33k [00:00<00:00, 32.3MB/s]
2025-10-16 19:23:36,177 - modelscope - ERROR - >> Error loading AI-ModelScope/IQuiz: Unable to find 'hf://datasets/AI-ModelScope/IQuiz@master/IQ.jsonl' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.ndjson', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.xml', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.mkv', '.mp4', '.avi', '.mov', '.MKV', '.MP4', '.AVI', '.MOV', '.pdf', '.PDF', '.zip']
Traceback (most recent call last):
  File "/mnt/workspace/E2Eeval/eval_public2.py", line 76, in <module>
    run_task(task_cfg=task_cfg)
  File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 28, in run_task
    return run_single_task(task_cfg, run_time)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 41, in run_single_task
    result = evaluate_model(task_cfg, outputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/run.py", line 144, in evaluate_model
    res_dict = evaluator.eval()
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/evaluator/evaluator.py", line 94, in eval
    dataset_dict = self.benchmark.load_dataset()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 62, in load_dataset
    self.test_dataset, self.fewshot_dataset = self.load()
                                              ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 81, in load
    return self.load_from_remote()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 89, in load_from_remote
    test_dataset = self.load_subsets(test_load_func)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 215, in load_subsets
    subset_data = load_func(subset)
                  ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/benchmark/adapters/default_data_adapter.py", line 248, in load_subset
    dataset = loader.load()
              ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/evalscope/api/dataset/loader.py", line 88, in load
    dataset = MsDataset.load(
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/ms_dataset.py", line 308, in load
    with load_dataset_with_ctx(
  File "/usr/local/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1475, in load_dataset_with_ctx
    dataset_res = DatasetsWrapperHF.load_dataset(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 991, in load_dataset
    builder_instance = DatasetsWrapperHF.load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1125, in load_dataset_builder
    dataset_module = DatasetsWrapperHF.dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 1425, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /mnt/workspace/E2Eeval/AI-ModelScope/IQuiz/IQuiz.py or any data file in the same directory. Couldn't find 'AI-ModelScope/IQuiz' on the Hugging Face Hub either: FileNotFoundError: Unable to find 'hf://datasets/AI-ModelScope/IQuiz@master/IQ.jsonl' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.ndjson', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.xml', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.mkv', '.mp4', '.avi', '.mov', '.MKV', '.MP4', '.AVI', '.MOV', '.pdf', '.PDF', '.zip']