Repository: open-compass/opencompass
Issue: About the inference results of the OpenCompass framework
Root cause: Configuration Error

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.

### Environment

`{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
 'GPU 0,1,2,3': 'Tesla P100-SXM2-16GB',
 'MMEngine': '0.8.2',
 'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.0+cu118',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2022.2-Product Build 20220804 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.10 (main, Jul 26 2023, 22:16:48) [GCC 9.4.0]',
 'TorchVision': '0.15.1+cu118',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.0+9fea89c',
 'sys.platform': 'linux'}`

### Reproduces the problem - code/configuration sample

1、To evaluate the qwen-7b-chat model using the OpenCompass framework, follow a similar approach as the models/hf_baichuan_13b_chat.py file. Create a new configuration file configs/models/hf_qwen_7b_chat.py to mimic the structure and settings.
code here：
`from opencompass.models import HuggingFaceCausalLM
models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='qwen-7b-chat-hf',
        path="/root/project/model/qwen-7b-chat",
        tokenizer_path='/root/project/model/qwen-7b-chat',
        tokenizer_kwargs=dict(padding_side='left',
                              truncation_side='left',
                              trust_remote_code=True,
                              use_fast=False,),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
        model_kwargs=dict(device_map='auto', trust_remote_code=True),
        run_cfg=dict(num_gpus=2, num_procs=1),
    )
]`
2、
To create a dataset named "my-ceval" that mimics the structure of the "ceval" dataset, you can follow a similar format and organization of the data.
After performing inference, when inspecting the files under the "prediction" section in the log, I noticed discrepancies between the results obtained from the framework's inference and directly using FastAPI to deploy the model and call the API. The differences are as follows:
opencompass：
`"origin_prompt": "\n下面是一道单项选择题，请选择正确答案，只需要回答A、B、C或者D，不要回答选项内容。\n问题：如果高兴对应悲伤，英勇对应什么？\nA.悲伤\nB.英勇\nC.勇敢\nD.怯懦\n答案：",
 "prediction": "C.勇敢\nENCHMARK是全球最大的独立第三方汽车安全评价机构之一，其测试结果具有很高的参考价值。在此次测试中，几何A以88.6%的综合得分获得五星评价，成为继帝豪GL之后，第二款获得五星评价的中国品牌车型。\n решил сделать несколько тестов на знание языка, вот один из них\n't\n решил сделать несколько тестов на знание язы"`

Results obtained by directly calling the API for inference:
`"prompt": "下面是一道单项选择题，请选择正确答案，只需要回答A、B、C或者D，不要回答选项内容。问题：如果高兴对应悲伤，英勇对应什么？\nA.悲伤\nB.英勇\nC.勇敢\nD.怯懦\n答案：",
"raw_prediction": "C. 勇敢"`

Is the excessive output of irrelevant information from OpenCompass due to setting max_out_len=100?
If this is the case, how can I set a suitable value for max_out_len?




### Reproduces the problem - command or script

python run.py configs/eval_collection.py -w outputs/eval_hf_qwen_7b_my_eval_gen

### Reproduces the problem - error message

config is here：
[20230811_160848.py.txt](https://github.com/InternLM/opencompass/files/12320185/20230811_160848.py.txt)


prediction result is here：
[my-ceval-algorithm.json.txt](https://github.com/InternLM/opencompass/files/12320181/my-ceval-algorithm.json.txt)


### Other information

None

No. The reason for the redundant output is the improper meta template for qwen-7b-chat. There is my config: [hf_qwen_7b_chat.py.txt](https://github.com/InternLM/opencompass/files/12320332/hf_qwen_7b_chat.py.txt)

The meta template is made according to [the official implement](https://huggingface.co/Qwen/Qwen-7B-Chat/blob/main/qwen_generation_utils.py#L131). The necessity of this meta template lies in the following of the same data schema during the SFT process.

Here is my ceval output: [ceval-education_science.json.txt](https://github.com/InternLM/opencompass/files/12320366/ceval-education_science.json.txt). Need not to worry about the redundant '
' as it will be truncated during the evaluation process [here](https://github.com/InternLM/opencompass/blob/10cbc2b1757522cf1cec168b2ae40347f58a52e1/opencompass/tasks/openicl_eval.py#L185-L190). 


> Here is my ceval output: [ceval-education_science.json.txt](https://github.com/InternLM/opencompass/files/12320366/ceval-education_science.json.txt). Need not to worry about the redundant '
' as it will be truncated during the evaluation process [here](https://github.com/InternLM/opencompass/blob/10cbc2b1757522cf1cec168b2ae40347f58a52e1/opencompass/tasks/openicl_eval.py#L185-L190).

Thanks for your reply.
Following your advice, I made changes to the meta template, and there was a slight improvement in the predictions made by OpenCompass. However, there are still differences between the results I obtained and the ones you provided. I'm not sure what could be causing this discrepancy.

ceval output is like the fllowing:
"prediction": "B\nuser\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"


configuration file is：
[20230811_211359.py.txt](https://github.com/InternLM/opencompass/files/12322000/20230811_211359.py.txt)

ceval output is：
[ceval-accountant.json.txt](https://github.com/InternLM/opencompass/files/12322014/ceval-accountant.json.txt)



> "origin_prompt": "\n下面是一道单项选择题，请选择正确答案，只需要回答A、B、C或者D，不要回答选项内容。\n问题：如果高兴对应悲伤，英勇对应什么？\nA.悲伤\nB.英勇\nC.勇敢\nD.怯懦\n答案：", "prediction": "C.勇敢\nENCHMARK是全球最大的独立第三方汽车安全评价机构之一，其测试结果具有很高的参考价值。在此次测试中，几何A以88.6%的综合得分获得五星评价，成为继帝豪GL之后，第二款获得五星评价的中国品牌车型。\n решил сделать несколько тестов на знание языка, вот один из них\n't\n решил сделать несколько тестов на знание язы"
> 
> Results obtained by directly calling the API for inference:
> "prompt": "下面是一道单项选择题，请选择正确答案，只需要回答A、B、C或者D，不要回答选项内容。问题：如果高兴对应悲伤，英勇对应什么？\nA.悲伤\nB.英勇\nC.勇敢\nD.怯懦\n答案：", "raw_prediction": "C. 勇敢"

You might know that language models are sensitive to the prompts. Even some extra "\n" characters can significantly affect the output.

From your provided data sample, the "origin_prompt" is not exactly the same as the one you used in API inference. That's the source of the discrepancy.