Repository: open-compass/opencompass
Issue: 如何使用如chatglm2-6B,Baichuan-13B-Chat,Baichuan-13B-Base在CHID数据集上复现出排行榜上的评估结果，复现的结果在CHID数据集上的结果与榜单上差距很多
Root cause: Documentation Deficiency

### Describe the bug

如何使用如chatglm2-6B,Baichuan-13B-Chat,Baichuan-13B-Base在CHID数据集上复现出排行榜上的评估结果，复现的结果在CHID数据集上的结果与榜单上差距很多

### Environment

如何使用如chatglm2-6B,Baichuan-13B-Chat,Baichuan-13B-Base在CHID数据集上复现出排行榜上的评估结果，复现的结果在CHID数据集上的结果与榜单上差距很多

### Other information

_No response_

Please provider more details:
1. Your reproduced results of chatglm2-6B, Baichuan-13B-Chat, Baichuan-13B-Base on CHID
2. Your configuration to run the experiment

The model is a public model provided by the authorities.The following is the configuration of the model and data set:

`#model configs
ChatGLM2_6B=dict(
    abbr='ChatGLM2-6B',
    batch_size=64,
    max_out_len=100,
    max_seq_len=2048,
    model_kwargs=dict(
        device_map='auto',
        trust_remote_code=True),
    path='/d2/pub/models/chatglm2-6b',
    run_cfg=dict(
        num_gpus=1),
    tokenizer_kwargs=dict(
        padding_side='left',
        proxies=None,
        truncation_side='left',
        trust_remote_code=True),
    tokenizer_path='/d2/pub/models/chatglm2-6b',
    type='opencompass.models.HuggingFaceCausalLM')
Baichuan_13B_Base=dict(
    abbr='Baichuan-13B-Base',
    batch_size=64,
    max_out_len=100,
    max_seq_len=2048,
    model_kwargs=dict(
        device_map='auto',
        trust_remote_code=True),
    path='/d1/data/SFT/models/Baichuan-13B-Base',
    run_cfg=dict(
        num_gpus=1),
    tokenizer_kwargs=dict(
        padding_side='left',
        proxies=None,
        truncation_side='left',
        trust_remote_code=True),
    tokenizer_path='/d1/data/SFT/models/Baichuan-13B-Base',
    type='opencompass.models.HuggingFaceCausalLM')

Baichuan_13B_Chat=dict(
    abbr='Baichuan-13B-Chat',
    batch_size=64,
    max_out_len=100,
    max_seq_len=2048,
    model_kwargs=dict(
        device_map='auto',
        trust_remote_code=True),
    path='/d1/data/SFT/models/Baichuan-13B-Chat',
    run_cfg=dict(
        num_gpus=1),
    tokenizer_kwargs=dict(
        padding_side='left',
        proxies=None,
        truncation_side='left',
        trust_remote_code=True),
    tokenizer_path='/d1/data/SFT/models/Baichuan-13B-Chat',
    type='opencompass.models.HuggingFaceCausalLM')


#dataset configs
datasets=[

    dict(abbr='chid-dev',
        eval_cfg=dict(
            evaluator=dict(
                type='opencompass.openicl.icl_evaluator.AccEvaluator'),
            pred_postprocessor=dict(
                type='opencompass.utils.text_postprocessors.first_capital_postprocess'),
            pred_role='BOT'),
        infer_cfg=dict(
            inferencer=dict(
                type='opencompass.openicl.icl_infer`

Here are the scores from my own run:
chatglm26b 24.75
baichuan-base-13b 32.18
baichuan-chat-13b 15.84

Please try this config:
https://github.com/InternLM/opencompass/blob/main/configs/datasets/FewCLUE_chid/FewCLUE_chid_ppl_8f2872.py

The config you are using require the model to follow instruction and output the choice option precisely, which is rather hard for LLM like chatglm6b etc. The ppl config, on the other hand, asks the model to judge which prompt is more preferable, is much easier and often results in higher accuracy.

Ok, I would like to ask which configuration file was used for the tnews and cmnli datasets

And test TNEWS and CMNLI data sets in gen mode or ppl mode

We found that some results from the model leaderboard cannot be reproduced. Will the official team consider providing scripts for testing the models on the leaderboard?

Most of the results in leader board are derived from the following list
https://github.com/InternLM/opencompass/blob/af436f5951d25e0ed0f6ec66431f39ce6a02c801/configs/datasets/collections/base_medium.py

However, models have preference on different prompts, some scores are retrieved from other configs. We are still trying to find out an elegant way to reveal more details during this process.

Well, thank you for your patience