Repository: modelscope/evalscope
Issue: 选择winogrande测评报错
Root cause: Configuration Error

选择winogrande测评报错

Traceback (most recent call last):
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 462, in <module>
    inferencer.run()
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 114, in run
    self._score()
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\tasks\openicl_eval.py", line 117, in _score
    test_set = build_dataset_from_cfg(self.dataset_cfg).test
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\utils\build.py", line 13, in build_dataset_from_cfg
    return LOAD_DATASET.build(dataset_cfg)
  File "D:\Anaconda3\envs\swift\lib\site-packages\mmengine\registry\registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "D:\Anaconda3\envs\swift\lib\site-packages\mmengine\registry\build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\datasets\base.py", line 12, in __init__
    self.dataset = self.load(**kwargs)
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\datasets\winogrande.py", line 68, in load
    path = get_data_path(path)
  File "D:\Anaconda3\envs\swift\lib\site-packages\opencompass\utils\datasets.py", line 48, in get_data_path
    local_path = DATASETS_MAPPING[dataset_id]['local']
KeyError: './data/winogrande'

看起来是找不到数据集，参考opencompass[数据集准备部分](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/opencompass_backend.html#id3)

需要下载数据集到本地，放置在`data`目录下

数据集是有的，运行以下脚本可复现
 `swift eval --model_type qwen-1_8b-chat --name test2 --eval_dataset winogrande --model_id_or_path qwen/Qwen-1_8B-Chat`
在加载winogrande数据集报错，
```@LOAD_DATASET.register_module()
class WinograndeDataset(BaseDataset):
    """Disconnect from Huggingface, WinograndeDataset."""

    @staticmethod
    def load(path):
        path = get_data_path(path)
```
此处path 为` ./data/winogrande`,这个值应该是evalscope读取opencompass的配置传进来的。但是却没有处理好。
以下为get_data_path 方法

```

def get_data_path(dataset_id: str, local_mode: bool = False):

    # update the path with CACHE_DIR
    cache_dir = os.environ.get('COMPASS_DATA_CACHE', '')

    # For absolute path customized by the users
    if dataset_id.startswith('/'):
        return dataset_id

    # For relative path, with CACHE_DIR
    if local_mode:
        local_path = os.path.join(cache_dir, dataset_id)
        if not os.path.exists(local_path):
            return download_dataset(local_path, cache_dir)
        else:
            return local_path

    dataset_source = os.environ.get('DATASET_SOURCE', None)
    if dataset_source == 'ModelScope':
        ms_id = DATASETS_MAPPING[dataset_id]['ms_id']
        assert ms_id is not None, \
            f'{dataset_id} is not supported in ModelScope'
        return ms_id
    elif dataset_source == 'HF':
        # TODO: HuggingFace mode is currently not supported!
        hf_id = DATASETS_MAPPING[dataset_id]['hf_id']
        assert hf_id is not None, \
            f'{dataset_id} is not supported in HuggingFace'
        return hf_id
    else:
        # for the local path
        local_path = DATASETS_MAPPING[dataset_id]['local']
        local_path = os.path.join(cache_dir, local_path)
        
        if not os.path.exists(local_path):
            return download_dataset(local_path, cache_dir)
        else:
            return local_path
```
传入的是相对路径，但是没传local_mode参数报错

我们复现一下

目前已定位问题，可先尝试将 `opencompass/configs/datasets/winogrande/deprecated_winogrande_gen_a9ede5.py `中的`path`修改为 `opencompass/winogrande`来临时解决，后续会更新库来解决该问题。

尝试升级： 
pip install evalscope[opencompass] -U

同时check一下ms-opencompass版本是否>=0.1.1，如果不是，则升级： pip install ms-opencompass -U

另外，OpenCompass backend也支持按需加载数据集，无需预先下载data.zip压缩包，可设置： export DATASET_SOURCE=ModelScope

由于长时间未收到活动，我们将关闭此问题。如果您有任何疑问，请随时重新打开它。如果EvalScope对您有所帮助，欢迎给我们点个STAR以示支持，谢谢！