Repository: open-compass/opencompass
Issue: [Bug] TypeError: cross_entropy_loss(): argument 'ignore_index' (position 5) must be int, not NoneType
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

{'CUDA available': True,
 'CUDA_HOME': '/usr/local/cuda',
 'GCC': 'gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)',
 'GPU 0': 'NVIDIA A40',
 'MMEngine': '0.8.4',
 'NVCC': 'Cuda compilation tools, release 11.1, V11.1.74',
 'OpenCV': '4.8.0',
 'PyTorch': '2.0.1',
 'PyTorch compiling details': 'PyTorch built with:\n'
                              '  - GCC 9.3\n'
                              '  - C++ Version: 201703\n'
                              '  - Intel(R) oneAPI Math Kernel Library Version '
                              '2023.1-Product Build 20230303 for Intel(R) 64 '
                              'architecture applications\n'
                              '  - Intel(R) MKL-DNN v2.7.3 (Git Hash '
                              '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n'
                              '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                              '  - LAPACK is enabled (usually provided by '
                              'MKL)\n'
                              '  - NNPACK is enabled\n'
                              '  - CPU capability usage: AVX2\n'
                              '  - CUDA Runtime 11.8\n'
                              '  - NVCC architecture flags: '
                              '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
                              '  - CuDNN 8.7\n'
                              '  - Magma 2.6.1\n'
                              '  - Build settings: BLAS_INFO=mkl, '
                              'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
                              'CUDNN_VERSION=8.7.0, '
                              'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                              'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
                              '-fabi-version=11 -Wno-deprecated '
                              '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                              '-DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER '
                              '-DUSE_FBGEMM -DUSE_QNNPACK '
                              '-DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK '
                              '-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC '
                              '-Wall -Wextra -Werror=return-type '
                              '-Werror=non-virtual-dtor -Werror=bool-operation '
                              '-Wnarrowing -Wno-missing-field-initializers '
                              '-Wno-type-limits -Wno-array-bounds '
                              '-Wno-unknown-pragmas -Wunused-local-typedefs '
                              '-Wno-unused-parameter -Wno-unused-function '
                              '-Wno-unused-result -Wno-strict-overflow '
                              '-Wno-strict-aliasing '
                              '-Wno-error=deprecated-declarations '
                              '-Wno-stringop-overflow -Wno-psabi '
                              '-Wno-error=pedantic -Wno-error=redundant-decls '
                              '-Wno-error=old-style-cast '
                              '-fdiagnostics-color=always -faligned-new '
                              '-Wno-unused-but-set-variable '
                              '-Wno-maybe-uninitialized -fno-math-errno '
                              '-fno-trapping-math -Werror=format '
                              '-Werror=cast-function-type '
                              '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                              'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                              'PERF_WITH_AVX512=1, '
                              'TORCH_DISABLE_GPU_ASSERTS=ON, '
                              'TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, '
                              'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
                              'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
                              'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
                              'USE_OPENMP=ON, USE_ROCM=OFF, \n',
 'Python': '3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0]',
 'TorchVision': '0.15.2',
 'numpy_random_seed': 2147483648,
 'opencompass': '0.1.2+1c7f952',
 'sys.platform': 'linux'}

### Reproduces the problem - code/configuration sample

from mmengine.config import read_base
with read_base():
    
    from .datasets.cmmlu.cmmlu_ppl import cmmlu_datasets
    from .models.megatron import models

datasets = []
datasets += cmmlu_datasets

### Reproduces the problem - command or script

python run.py \
    configs/eval_mmlu.py \
    -w outputs/mmlu \
    --debug \

### Reproduces the problem - error message

Traceback (most recent call last):
  File "/ssd/other/xxx/opencompass/run.py", line 378, in <module>
    main()
  File "/ssd/other/xxx/opencompass/run.py", line 231, in main
    exec_infer_runner(tasks, args, cfg)
  File "/ssd/other/xxx/opencompass/run.py", line 348, in exec_infer_runner
    runner(tasks)
  File "/ssd/other/xxx/opencompass/opencompass/runners/base.py", line 38, in __call__
    status = self.launch(tasks)
  File "/ssd/other/xxx/opencompass/opencompass/runners/local.py", line 62, in launch
    task.run()
  File "/ssd/other/xxx/opencompass/opencompass/tasks/openicl_infer.py", line 76, in run
    self._inference()
  File "/ssd/other/xxx/opencompass/opencompass/tasks/openicl_infer.py", line 124, in _inference
    inferencer.inference(retriever,
  File "/ssd/other/xxx/opencompass/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py", line 184, in inference
    sub_res = self.model.get_ppl_from_template(
  File "/ssd/other/xxx/opencompass/opencompass/models/base.py", line 106, in get_ppl_from_template
    return self.get_ppl(inputs, mask_length)
  File "/ssd/other/xxx/opencompass/opencompass/models/huggingface.py", line 271, in get_ppl
    return np.concatenate([
  File "/ssd/other/xxx/opencompass/opencompass/models/huggingface.py", line 272, in <listcomp>
    self._get_ppl(inputs=[text], mask_length=mask_length)
  File "/ssd/other/xxx/opencompass/opencompass/models/huggingface.py", line 300, in _get_ppl
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
  File "/home/xxx/.conda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xxx/.conda/envs/opencompass/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 1174, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/xxx/.conda/envs/opencompass/lib/python3.10/site-packages/torch/nn/functional.py", line 3029, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
TypeError: cross_entropy_loss(): argument 'ignore_index' (position 5) must be int, not NoneType

### Other information

_No response_

I'm not sure what is in the `from .models.megatron import models`, maybe a post of this file is needed.

Empirically, this error is caused by the missing of `pad_token_id` from `self.tokenizer`. You may set this token when dumping the tokenizer before running OpenCompass. Or, you may just remove all the occurrence of `self.tokenizer.pad_token_id` in the `get_ppl` function.

https://github.com/InternLM/opencompass/blob/4fc1701209adad832f7fca91640fac34331b8e29/opencompass/models/huggingface.py#L298-L299

https://github.com/InternLM/opencompass/blob/4fc1701209adad832f7fca91640fac34331b8e29/opencompass/models/huggingface.py#L310-L311


from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='megatron-step34000',
        path="./models/llama2/llama/llama-2-7b",
        tokenizer_path='./models/llama2/llama/llama-2-7b',
        tokenizer_kwargs=dict(padding_side='left',
                         truncation_side='left',
                         use_fast=False,
                         ),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
         model_kwargs=dict(device_map='auto'),
        batch_padding=False, # if false, inference with for-loop without batch padding
        run_cfg=dict(num_gpus=8, num_procs=1),
    )
]

This is the content in models.megatron

Where does the file in `./models/llama2/llama/llama-2-7b` comes from？It is downloaded directly from huggingface official or trained and converted by your team?
If latter is the case, set `pad_token_id` when dumping tokenizer.