Repository: open-compass/opencompass
Issue: [Bug] The error encountered while evaluating the model using the CMNLI dataset
Root cause: Configuration Error

### Describe the bug

The config of cmnli dataset is https://github.com/InternLM/opencompass/blob/af436f5951d25e0ed0f6ec66431f39ce6a02c801/configs/datasets/CLUE_cmnli/CLUE_cmnli_ppl_fdc6de.py

I use this dataset to evaluate chatglm2-6B. The details of the error are as follows:
OpenCompass - ERROR - /d1/pub/zhaohui/evaluation/opencompass/opencompass/tasks/openicl_eval.py - _score - 147 - Task [ChatGLM2-6B/cmnli]: predictions and references have different length. len(predictions): 12255, len(references): 12426

How to slove this error?

### Environment

python 

### Other information

_No response_

could you please provide the full config file?

ChatGLM2_6B=dict(
    abbr='ChatGLM2-6B',
    batch_size=64,
    max_out_len=100,
    max_seq_len=2048,
    model_kwargs=dict(
        device_map='auto',
        trust_remote_code=True),
    path='/d1/data/SFT/models/chatglm2-6b',
    run_cfg=dict(
        num_gpus=1),
    tokenizer_kwargs=dict(
        padding_side='left',
        proxies=None,
        truncation_side='left',
        trust_remote_code=True),
    tokenizer_path='/d1/data/SFT/models/chatglm2-6b',
    type='opencompass.models.HuggingFaceCausalLM')



cmnli_datasets=[
    dict(abbr='cmnli',
        data_files='./data/CLUE/cmnli/cmnli_public/dev.json',
        eval_cfg=dict(
            evaluator=dict(
                type='opencompass.openicl.icl_evaluator.AccEvaluator')),
        infer_cfg=dict(
            inferencer=dict(
                type='opencompass.openicl.icl_inferencer.PPLInferencer'),
            prompt_template=dict(
                template=dict(
                    contradiction=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='矛盾',
                                role='BOT'),
                            ]),
                    entailment=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='蕴含',
                                role='BOT'),
                            ]),
                    neutral=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='无关',
                                role='BOT'),
                            ])),
                type='opencompass.openicl.icl_prompt_template.PromptTemplate'),
            retriever=dict(
                type='opencompass.openicl.icl_retriever.ZeroRetriever')),
        path='json',
        reader_cfg=dict(
            input_columns=[
                'sentence1',
                'sentence2',
                ],
            output_column='label',
            test_split='train'),
        split='train',
        type='opencompass.datasets.HFDataset'),
    ]
datasets=[
    dict(abbr='cmnli',
        data_files='./data/CLUE/cmnli/cmnli_public/dev.json',
        eval_cfg=dict(
            evaluator=dict(
                type='opencompass.openicl.icl_evaluator.AccEvaluator')),
        infer_cfg=dict(
            inferencer=dict(
                type='opencompass.openicl.icl_inferencer.PPLInferencer'),
            prompt_template=dict(
                template=dict(
                    contradiction=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='矛盾',
                                role='BOT'),
                            ]),
                    entailment=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='蕴含',
                                role='BOT'),
                            ]),
                    neutral=dict(
                        round=[
                            dict(prompt='语句一：“{sentence1}”\n语句二：“{sentence2}”\n请问这两句话是什么关系？',
                                role='HUMAN'),
                            dict(prompt='无关',
                                role='BOT'),
                            ])),
                type='opencompass.openicl.icl_prompt_template.PromptTemplate'),
            retriever=dict(
                type='opencompass.openicl.icl_retriever.ZeroRetriever')),
        path='json',
        reader_cfg=dict(
            input_columns=[
                'sentence1',
                'sentence2',
                ],
            output_column='label',
            test_split='train'),
        split='train',
        type='opencompass.datasets.HFDataset'),
    ]

models=[
    dict(abbr='ChatGLM2-6B',
        batch_size=64,
        max_out_len=100,
        max_seq_len=2048,
        model_kwargs=dict(
            device_map='auto',
            trust_remote_code=True),
        path='/d1/data/SFT/models/chatglm2-6b',
        run_cfg=dict(
            num_gpus=1),
        tokenizer_kwargs=dict(
            padding_side='left',
            proxies=None,
            truncation_side='left',
            trust_remote_code=True),
        tokenizer_path='/d1/data/SFT/models/chatglm2-6b',
        type='opencompass.models.HuggingFaceCausalLM'),
    ]


Have you changed the `max-partition-size` in between different experiments, and resumed from the same experiment directory? If so, the prediction splits can have different sizes. The fastest way to fix it is to re-evaluate the model over CMNLI in a new experiment. 

> Have you changed the `max-partition-size` in between different experiments, and resumed from the same experiment directory? If so, the prediction splits can have different sizes. The fastest way to fix it is to re-evaluate the model over CMNLI in a new experiment.

No. All experiments were conducted using the same set of parameters and I don't change the max-partition-size 

Not quite sure about the reason, but I guess there was some resuming failure happening there.

Again, the solution is to reevaluate the model over CMNLI again. Let us know if the issue persists.

> could you please provide the full config file?



> Not quite sure about the reason, but I guess there was some resuming failure happening there.
> 
> Again, the solution is to reevaluate the model over CMNLI again. Let us know if the issue persists.

I reevaluate the model again but the error still occurs. I also attempted to evaluate other models, and I encountered the same error

@Leymore Have you run into a similar issue before?

Please check the content of `.cache/dataset_size.json`, is the value of key 'cmnli' 12426?  If not, just delete `.cache/dataset_size.json` and restart the script.

Feel free to re-open if needed.

> Please check the content of `.cache/dataset_size.json`, is the value of key 'cmnli' 12426? If not, just delete `.cache/dataset_size.json` and restart the script.

Same error and fixed by removing cache.