Repository: open-compass/opencompass
Issue: [Bug] Unexpected output when using "attn_implementation=eager"
Root cause: Configuration Error

### Prerequisite

- [x] I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
- [x] The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
 'GPU 0,1,2,3,4,5,6,7': 'NVIDIA GeForce RTX 3090',
 'MMEngine': '0.10.7',
 'MUSA available': False,
 'NVCC': 'Cuda compilation tools, release 12.3, V12.3.107',
 'OpenCV': '4.11.0',
 'PyTorch': '2.6.0+cu124',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2024.2-Product Build 20240605 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v3.5.3 (Git Hash '
                              '66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX512\n'
                              '  - CUDA Runtime 12.4\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
                              '  - CuDNN 90.1\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, '
                              'COMMIT_SHA=2236df1770800ffea5697b11b0bb0d910b2e59e1, '
                              'CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -fvisibility-inlines-hidden '
                              '-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
                              '-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
                              '-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
                              '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                              '-O2 -fPIC -Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wno-unused-parameter '
                              '-Wno-strict-overflow -Wno-strict-aliasing '
                              '-Wno-stringop-overflow -Wsuggest-override '
                              '-Wno-psabi -Wno-error=old-style-cast '
                              '-Wno-missing-braces -fdiagnostics-color=always '
                              '-faligned-new -Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
                              'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
                              'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
                              'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
                              'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
 'Python': '3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]',
 'lmdeploy': "not installed:No module named 'lmdeploy'",
 'numpy_random_seed': 2147483648,
 'opencompass': '0.4.1+cbf84fb',
 'sys.platform': 'linux',
 'transformers': '4.49.0'}

### Reproduces the problem - code/configuration sample

When I run:
```
opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 
```
the results are correct
but when I run:
```
\opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 --model-kwargs "attn_implementation=eager"
```
The test output shown in output/20250318/predictions/gsm8k.jsonl become !!!!!!!


### Reproduces the problem - command or script

When I run:
```
opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 
```
the results are correct
but when I run:
```
\opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 --model-kwargs "attn_implementation=eager"
```
The test output shown in output/20250318/predictions/gsm8k.jsonl become "predictions: !!!!!!!"


### Reproduces the problem - error message

When I run:
```
opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 
```
the results are correct
but when I run:
```
\opencompass --datasets gsm8k_gen_e9e91e --hf-path checkpoint/DeepSeek-R1-Distill-Qwen-7B/ --hf-num-gpus 1 --model-kwargs "attn_implementation=eager"
```
The test output shown in output/20250318/predictions/gsm8k.jsonl become "predictions: !!!!!!!"

### Other information

_No response_

You can add  `torch_dtype=bfloat16` to `--model-kwargs` to solve the problem.  
This is because OC will set` torch_dtype = torch.float16` as the default, and cumulative errors in eager patterns can be amplified, disrupting the probability distribution of attention weights. We will attempt to read this parameter from the model's default configuration file to avoid this issue.



> You can add `torch_dtype=bfloat16` to `--model-kwargs` to solve the problem. This is because OC will set` torch_dtype = torch.float16` as the default, and cumulative errors in eager patterns can be amplified, disrupting the probability distribution of attention weights. We will attempt to read this parameter from the model's default configuration file to avoid this issue.

Thanks! It works! Without torch_dtype = torch.float16 Qwen2.5 eager attention will overflow.

Closing this issue. Feel free to re-open if more help is needed, thanks!