Repository: open-compass/opencompass
Issue: [Bug] ARC-c: predictions and references have different length. len(predictions): 296, len(references): 1165
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```
{'CUDA available': True,
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'MMEngine': '0.10.6',
 'MUSA available': False,
 'OpenCV': '4.11.0',
 'PyTorch': '2.5.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 10.2\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: NO AVX\n'
                              '  - Build settings: BLAS_INFO=open, '
                              'BUILD_TYPE=Release, '
                              'CXX_COMPILER=/opt/rh/devtoolset-10/root/usr/bin/c++, '
                              'CXX_FLAGS=-ffunction-sections -fdata-sections '
                              '-D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=open, '
                              'TORCH_VERSION=2.5.1, USE_CUDA=OFF, '
                              'USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_GLOO=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, '
                              'USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.2 (main, Jan  8 2025, 06:13:36) [GCC 9.4.0]',
 'TorchVision': '0.20.1',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.3.9+',
 'sys.platform': 'linux',
 'transformers': '4.48.2'}
```

### Reproduces the problem - code/configuration sample

```
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import PPLInferencer
from opencompass.openicl.icl_evaluator import AccEvaluator
from opencompass.datasets import ARCDataset

ARC_c_reader_cfg = dict(
    input_columns=['question', 'textA', 'textB', 'textC', 'textD'],
    output_column='answerKey')

ARC_c_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template={
            opt: dict(
                round=[
                    dict(role='HUMAN', prompt=f'{{question}}\nA. {{textA}}\nB. {{textB}}\nC. {{textC}}\nD. {{textD}}'),
                    dict(role='BOT', prompt=f'Answer: {opt}'),
                ]
            ) for opt in ['A', 'B', 'C', 'D']
        },
    ),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=PPLInferencer))

ARC_c_eval_cfg = dict(evaluator=dict(type=AccEvaluator))

ARC_c_datasets = [
    dict(
        type=ARCDataset,
        abbr='ARC-c',
        path='/opencompass-main/data/ARC/ARC-c/ARC-Challenge-Test.jsonl',
        name='ARC-Challenge',
        reader_cfg=ARC_c_reader_cfg,
        infer_cfg=ARC_c_infer_cfg,
        eval_cfg=ARC_c_eval_cfg)
]
```

### Reproduces the problem - command or script

when i modify the path code(Dev.jsonl to Test.jsonl), it reports bug as titles.
Notably, it only occurs on 8 DP
```
python run.py --models hf_qwen2_5_7b --datasets ARC_c_ppl_2ef631 --max-num-workers 8
```
code failed, predictions and references have different length. len(predictions): 296, len(references): 1165
but when i run on --debug mode, code success
```
python run.py --models hf_qwen2_5_7b --datasets ARC_c_ppl_2ef631 --debug
```
This is so strange, cause when i modify path code from Test.jsonl to Dev.jsonl, both DP and Debug mode works.


### Reproduces the problem - error message

```
03/31 11:54:50 - OpenCompass - ERROR - /opencompass-main/opencompass/tasks/openicl_eval.py - _score - 258 - Task [qwen2.5-7b-hf/ARC-c]: predictions and references have different length. len(predictions): 296, len(references): 1165
```

### Other information

_No response_

please check the generated json file in your `predictions` dir

> please check the generated json file in your `predictions` dir

I checked the predictions dir and it was less than labels. but i don't know why, i ran at 8 DP mode and len(labels) = 1165, but each prediction json file only had 37(8 * 37 = 296) results.

can you share a screen shot about the predictions dir? Does there have some tmp file? for example tmp_arc_0.json

> can you share a screen shot about the predictions dir? Does there have some tmp file? for example tmp_arc_0.json

there's no tmp json file, but i re run the command, and the tmp dir under opencompass-main comes out 8 config.pys, each one of them has a test range parameter like this

![Image](https://github.com/user-attachments/assets/cd99c6b1-d781-4fab-8b10-b9a552cad0bd)
but my ARC dataset config file is just like above, and i don't know whether it's this test range that's affecting it.

It appears that your inference process is normal, resulting in only one inferred ARC JSON file. However, during the evaluation phase, the test range did not match due to the setting of the number of workers.

> It appears that your inference process is normal, resulting in only one inferred ARC JSON file. However, during the evaluation phase, the test range did not match due to the setting of the number of workers.

Sorry I didn't quite understand what you meant, so far it looks like the 8 tmp config.py files generated in the tmp directory under the opencompass-main directory each have a test_range parameter that results in only 37 pieces of data for each subset, but my arc dataset config file doesn't have any data in the reader_cfg to fill in the test_range parameter. I don't know exactly what the problem is and how to go about fixing it

![Image](https://github.com/user-attachments/assets/fa3d49ff-f95a-4bce-b39d-2f2956f220db)

> It appears that your inference process is normal, resulting in only one inferred ARC JSON file. However, during the evaluation phase, the test range did not match due to the setting of the number of workers.

My inference process seems to be out of whack as well, the total data is 1165 so each gpu should be infer 145 or 146 datas, but now it looks like the test_range parameter is causing only 37 datas to be infered per gpu, resulting in the length of the eval phase prediction not being the same as the length of the labels. Also, I printed the ARCDataset class to check its length and found that it is indeed 1165

The test_range parameter is automatically obtained by the sharding logic of opencompass and is not filled in by the user, so there is no need to fill in the test_range parameter. It is because you used --max-num-workers 8 to split the task that each subset only has 37 pieces of data.



> The test_range parameter is automatically obtained by the sharding logic of opencompass and is not filled in by the user, so there is no need to fill in the test_range parameter. It is because you used --max-num-workers 8 to split the task that each subset only has 37 pieces of data.

That's odd, because I simply changed the ARC-c config file from Dev.jsonl to Test.jsonl, i mean, Dev.jsonl with the same --max-num-workers 8 is able to infer and eval perfectly fine


> > The test_range parameter is automatically obtained by the sharding logic of opencompass and is not filled in by the user, so there is no need to fill in the test_range parameter. It is because you used --max-num-workers 8 to split the task that each subset only has 37 pieces of data.
> 
> That's odd, because I simply changed the ARC-c config file from Dev.jsonl to Test.jsonl, i mean, Dev.jsonl with the same --max-num-workers 8 is able to infer and eval perfectly fine

oh, I may know the problem, please check this file: opencompass/.cache/dataset_size.json, and find how many items in ARC-c 

> > > The test_range parameter is automatically obtained by the sharding logic of opencompass and is not filled in by the user, so there is no need to fill in the test_range parameter. It is because you used --max-num-workers 8 to split the task that each subset only has 37 pieces of data.
> > 
> > 
> > That's odd, because I simply changed the ARC-c config file from Dev.jsonl to Test.jsonl, i mean, Dev.jsonl with the same --max-num-workers 8 is able to infer and eval perfectly fine
> 
> oh, I may know the problem, please check this file: opencompass/.cache/dataset_size.json, and find how many items in ARC-c

it's 295, so should i change it to 1165? And I need to change it to 295 every time I infer dev.jsonl and 1165 when I infer test.jsonl?

Let me explain to you, the reason why opencompass introduces the mechanism of caching dataset size is to prevent certain dataset files from being too large, which can lead to excessive time spent loading this file when sharding. Therefore, when performing data sharding, it will directly read this cached size. When we test a dataset, we generally do not switch between the test set and the training set, so files with the same name might trigger this bug. You can directly replace this cache size, or if you find replacing it to be troublesome and your evaluation task does not contain files with particularly large sizes, you can simply delete this cache file before starting the script each time.

> Let me explain to you, the reason why opencompass introduces the mechanism of caching dataset size is to prevent certain dataset files from being too large, which can lead to excessive time spent loading this file when sharding. Therefore, when performing data sharding, it will directly read this cached size. When we test a dataset, we generally do not switch between the test set and the training set, so files with the same name might trigger this bug. You can directly replace this cache size, or if you find replacing it to be troublesome and your evaluation task does not contain files with particularly large sizes, you can simply delete this cache file before starting the script each time.

got it, thank you very much!!