Repository: open-compass/opencompass
Issue: [Bug] Failed to prepare the AGIEval dataset for evaluation
Root cause: Validation Gap

### Describe the bug

Command:
```sh
python3 run.py configs/eval_internlm_7b.py
```

Dataprep steps:
```bash
cd ./data/AGIEval
git clone https://github.com/microsoft/AGIEval.git
cp -R AGIEval/data data
```

Logs:
```py
pyarrow.lib.ArrowTypeError: Expected bytes, got a 'list' object
```

### Environment

```python
{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-SXM4-80GB',
 'MMEngine': '0.8.1',
 'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0]',
 'TorchVision': '0.15.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.0+18efcb0',
 'sys.platform': 'linux'}
```

### Other information

_No response_

The steps for preparing the datasets including TheoremQA, TriviaQA and Xsum are also missing.

Sorry for the inconvenience caused. Due to licensing reasons, we were not able to upload the data files we used directly. We will soon complete the guide for creating these datasets. 

As for the `pyarrow` error you encountered in AGIEval, I suspect that the versions of `datasets` or other HuggingFace dependencies are incorrect. Please provide relevant infomation. Here is mine: 

```text
datasets==2.12.0
transformers==4.29.2
evaluate==0.4.0
pyarrow==11.0.0
```

> Sorry for the inconvenience caused. Due to licensing reasons, we were not able to upload the data files we used directly. We will soon complete the guide for creating these datasets.
> 
> As for the `pyarrow` error you encountered in AGIEval, I suspect that the versions of `datasets` or other HuggingFace dependencies are incorrect. Please provide relevant infomation. Here is mine:
> 
> ```
> datasets==2.12.0
> transformers==4.29.2
> evaluate==0.4.0
> pyarrow==11.0.0
> ```

Nope, I looked into it a little bit. It is because the some of the questions in the new AGIEval dataset have multiple answers. So for the variable `label` [here](https://github.com/InternLM/opencompass/blob/main/opencompass/datasets/agieval/agieval.py#L50), the type becomes inconsistent. It is `str` for single choice questions and `list` for multiple choice questions. You may take a look [here](https://github.com/microsoft/AGIEval/blob/main/data/v1/gaokao-physics.jsonl).

@tonysy Why is this closed? I don't think it has been resolved yet. There is no way to reproduce your results on the datasets I mentioned above, including AGIEval, TheoremQA, TriviaQA and Xsum.

Closed due to wrong operation and we re-open this issue. We are woking on the issue your mentioned(prepare the guideline for data preparation). If there are some more errorrs and questions in evaluation on dataset mentioned above, welcome to report more detailed information like environment and logs.

I am facing the same problem as the original poster. How can I resolve this issue?

We just uploaded a new `.zip` containing TheoremQA / TriviaQA / Xsum, please refer to

https://github.com/InternLM/opencompass/releases/tag/0.1.1

```
wget https://github.com/InternLM/opencompass/releases/download/0.1.1/OpenCompassData.zip
unzip OpenCompassData.zip
```

P.S. In this version, there is still bug lies in AGIEval subset gaokao-physics. We will fix it soon.

Feel free to re-open if needed.