Repository: open-compass/opencompass
Issue: [Bug] TypeError: cross_entropy_loss(): argument 'ignore_index' (position 5) must be int, not NoneType
Root cause: Validation Gap

### Prerequisite

- [X] I have searched [Issues](https://github.com/InternLM/opencompass/issues/) and [Discussions](https://github.com/InternLM/opencompass/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version](https://github.com/InternLM/opencompass).

### Type

I'm evaluating with the officially supported tasks/models/datasets.

### Environment

```python
python run.py configs/eval_qwen_7b_chat.py -w /mnt/home/00054055/project/code/opencompass/outputs/qwen_7b_chat_ceval_ppl_5_shot/
```

### Reproduces the problem - code/configuration sample

**eval_qwen_7b_chat.py**:
```python
from mmengine import read_base

with read_base():
    from .models.hf_qwen_7b_chat import models
    from .datasets.ceval.ceval_ppl import ceval_datasets as datasets
    from .summarizers.medium import summarizer 
```

**hf_qwen_7b_chat.py**:
```python
from opencompass.models import HuggingFaceCausalLM


models = [
    dict(
        type=HuggingFaceCausalLM,
        abbr='qwen-7b-chat-hf',
        path="/mnt/home/00054055/project/models/Qwen-7B-Chat",
        tokenizer_path='/mnt/home/00054055/project/models/Qwen-7B-Chat',
        tokenizer_kwargs=dict(padding_side='left',
                              truncation_side='left',
                              trust_remote_code=True,
                              use_fast=False,),
        max_out_len=100,
        max_seq_len=2048,
        batch_size=8,
        model_kwargs=dict(device_map='auto', trust_remote_code=True, revision='77d74f449c4b2882eac9d061b5a0c4b7c1936898'),
        run_cfg=dict(num_gpus=2, num_procs=1),
    )
]
```


### Reproduces the problem - command or script

```shell
/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/mmengine/utils/manager.py:113: UserWarning: <class 'mmengine.logging.logger.MMLogger'> instance named of OpenCompass has been created, the method `get_instance` should not accept any other arguments
  warnings.warn(
08/28 15:20:12 - OpenCompass - INFO - Partitioned into 3 tasks.

  0%|          | 0/3 [00:00<?, ?it/s]
                                     

  0%|          | 0/3 [00:02<?, ?it/s]
                                     

  0%|          | 0/3 [00:02<?, ?it/s]launch OpenICLInfer[qwen-7b-chat-hf/ceval-college_economics,qwen-7b-chat-hf/ceval-accountant,qwen-7b-chat-hf/ceval-tax_accountant,qwen-7b-chat-hf/ceval-physician,qwen-7b-chat-hf/ceval-civil_servant,qwen-7b-chat-hf/ceval-urban_and_rural_planner,qwen-7b-chat-hf/ceval-teacher_qualification,qwen-7b-chat-hf/ceval-college_programming,qwen-7b-chat-hf/ceval-electrical_engineer,qwen-7b-chat-hf/ceval-business_administration,qwen-7b-chat-hf/ceval-art_studies] on GPU 0,1
launch OpenICLInfer[qwen-7b-chat-hf/ceval-fire_engineer,qwen-7b-chat-hf/ceval-environmental_impact_assessment_engineer,qwen-7b-chat-hf/ceval-education_science,qwen-7b-chat-hf/ceval-professional_tour_guide,qwen-7b-chat-hf/ceval-college_chemistry,qwen-7b-chat-hf/ceval-metrology_engineer,qwen-7b-chat-hf/ceval-mao_zedong_thought,qwen-7b-chat-hf/ceval-law,qwen-7b-chat-hf/ceval-veterinary_medicine,qwen-7b-chat-hf/ceval-modern_chinese_history,qwen-7b-chat-hf/ceval-chinese_language_and_literature,qwen-7b-chat-hf/ceval-legal_professional,qwen-7b-chat-hf/ceval-logic,qwen-7b-chat-hf/ceval-middle_school_history,qwen-7b-chat-hf/ceval-plant_protection,qwen-7b-chat-hf/ceval-clinical_medicine,qwen-7b-chat-hf/ceval-computer_architecture,qwen-7b-chat-hf/ceval-middle_school_biology,qwen-7b-chat-hf/ceval-middle_school_politics,qwen-7b-chat-hf/ceval-middle_school_chemistry,qwen-7b-chat-hf/ceval-high_school_history] on GPU 2,3
08/28 15:21:52 - OpenCompass - WARNING - task OpenICLInfer[qwen-7b-chat-hf/ceval-fire_engineer,qwen-7b-chat-hf/ceval-environmental_impact_assessment_engineer,qwen-7b-chat-hf/ceval-education_science,qwen-7b-chat-hf/ceval-professional_tour_guide,qwen-7b-chat-hf/ceval-college_chemistry,qwen-7b-chat-hf/ceval-metrology_engineer,qwen-7b-chat-hf/ceval-mao_zedong_thought,qwen-7b-chat-hf/ceval-law,qwen-7b-chat-hf/ceval-veterinary_medicine,qwen-7b-chat-hf/ceval-modern_chinese_history,qwen-7b-chat-hf/ceval-chinese_language_and_literature,qwen-7b-chat-hf/ceval-legal_professional,qwen-7b-chat-hf/ceval-logic,qwen-7b-chat-hf/ceval-middle_school_history,qwen-7b-chat-hf/ceval-plant_protection,qwen-7b-chat-hf/ceval-clinical_medicine,qwen-7b-chat-hf/ceval-computer_architecture,qwen-7b-chat-hf/ceval-middle_school_biology,qwen-7b-chat-hf/ceval-middle_school_politics,qwen-7b-chat-hf/ceval-middle_school_chemistry,qwen-7b-chat-hf/ceval-high_school_history] fail, see
/mnt/home/00054055/project/code/opencompass/outputs/qwen_7b_chat_ceval_ppl_5_shot/20230828_152011/logs/infer/qwen-7b-chat-hf/ceval-fire_engineer.out

 33%|███▎      | 1/3 [01:40<03:20, 100.41s/it]
                                              

 33%|███▎      | 1/3 [01:40<03:20, 100.41s/it]launch OpenICLInfer[qwen-7b-chat-hf/ceval-computer_network,qwen-7b-chat-hf/ceval-operating_system,qwen-7b-chat-hf/ceval-college_physics,qwen-7b-chat-hf/ceval-advanced_mathematics,qwen-7b-chat-hf/ceval-high_school_physics,qwen-7b-chat-hf/ceval-high_school_chemistry,qwen-7b-chat-hf/ceval-high_school_biology,qwen-7b-chat-hf/ceval-middle_school_mathematics,qwen-7b-chat-hf/ceval-middle_school_physics,qwen-7b-chat-hf/ceval-marxism,qwen-7b-chat-hf/ceval-high_school_politics,qwen-7b-chat-hf/ceval-high_school_geography,qwen-7b-chat-hf/ceval-ideological_and_moral_cultivation,qwen-7b-chat-hf/ceval-high_school_chinese,qwen-7b-chat-hf/ceval-sports_science,qwen-7b-chat-hf/ceval-basic_medicine,qwen-7b-chat-hf/ceval-probability_and_statistics,qwen-7b-chat-hf/ceval-high_school_mathematics,qwen-7b-chat-hf/ceval-discrete_mathematics,qwen-7b-chat-hf/ceval-middle_school_geography] on GPU 2,3
08/28 15:21:57 - OpenCompass - WARNING - task OpenICLInfer[qwen-7b-chat-hf/ceval-college_economics,qwen-7b-chat-hf/ceval-accountant,qwen-7b-chat-hf/ceval-tax_accountant,qwen-7b-chat-hf/ceval-physician,qwen-7b-chat-hf/ceval-civil_servant,qwen-7b-chat-hf/ceval-urban_and_rural_planner,qwen-7b-chat-hf/ceval-teacher_qualification,qwen-7b-chat-hf/ceval-college_programming,qwen-7b-chat-hf/ceval-electrical_engineer,qwen-7b-chat-hf/ceval-business_administration,qwen-7b-chat-hf/ceval-art_studies] fail, see
/mnt/home/00054055/project/code/opencompass/outputs/qwen_7b_chat_ceval_ppl_5_shot/20230828_152011/logs/infer/qwen-7b-chat-hf/ceval-college_economics.out
```

### Reproduces the problem - error message

```shell
08/28 15:20:40 - OpenCompass - INFO - Task [qwen-7b-chat-hf/ceval-college_economics,qwen-7b-chat-hf/ceval-accountant,qwen-7b-chat-hf/ceval-tax_accountant,qwen-7b-chat-hf/ceval-physician,qwen-7b-chat-hf/ceval-civil_servant,qwen-7b-chat-hf/ceval-urban_and_rural_planner,qwen-7b-chat-hf/ceval-teacher_qualification,qwen-7b-chat-hf/ceval-college_programming,qwen-7b-chat-hf/ceval-electrical_engineer,qwen-7b-chat-hf/ceval-business_administration,qwen-7b-chat-hf/ceval-art_studies]
/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/mmengine/utils/manager.py:113: UserWarning: <class 'mmengine.logging.logger.MMLogger'> instance named of OpenCompass has been created, the method `get_instance` should not accept any other arguments
  warnings.warn(
08/28 15:20:44 - OpenCompass - WARNING - pad_token_id is not set for the tokenizer. Using eos_token_id as pad_token_id.
Using eos_token, but it is not set yet.

Loading checkpoint shards:   0%|          | 0/8 [00:00<?, ?it/s]
Loading checkpoint shards:  12%|█▎        | 1/8 [00:03<00:27,  3.89s/it]
Loading checkpoint shards:  25%|██▌       | 2/8 [00:08<00:25,  4.17s/it]
Loading checkpoint shards:  38%|███▊      | 3/8 [00:12<00:21,  4.31s/it]
Loading checkpoint shards:  50%|█████     | 4/8 [00:16<00:17,  4.26s/it]
Loading checkpoint shards:  62%|██████▎   | 5/8 [00:21<00:12,  4.21s/it]
Loading checkpoint shards:  75%|███████▌  | 6/8 [00:25<00:08,  4.23s/it]
Loading checkpoint shards:  88%|████████▊ | 7/8 [00:29<00:04,  4.18s/it]
Loading checkpoint shards: 100%|██████████| 8/8 [00:32<00:00,  3.80s/it]
Loading checkpoint shards: 100%|██████████| 8/8 [00:32<00:00,  4.04s/it]
/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/fuzzywuzzy/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
  warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
08/28 15:21:48 - OpenCompass - INFO - Start inferencing [qwen-7b-chat-hf/ceval-college_economics]

  0%|          | 0/55 [00:00<?, ?it/s]
100%|██████████| 55/55 [00:00<00:00, 517234.80it/s]
[2023-08-28 15:21:48,534] [opencompass.openicl.icl_inferencer.icl_ppl_inferencer] [INFO] Calculating PPL for prompts labeled 'A'

  0%|          | 0/7 [00:00<?, ?it/s]
  0%|          | 0/7 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/home/00054055/project/code/opencompass/opencompass/tasks/openicl_infer.py", line 147, in <module>
    inferencer.run()
  File "/mnt/home/00054055/project/code/opencompass/opencompass/tasks/openicl_infer.py", line 76, in run
    self._inference()
  File "/mnt/home/00054055/project/code/opencompass/opencompass/tasks/openicl_infer.py", line 124, in _inference
    inferencer.inference(retriever,
  File "/mnt/home/00054055/project/code/opencompass/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py", line 184, in inference
    sub_res = self.model.get_ppl_from_template(
  File "/mnt/home/00054055/project/code/opencompass/opencompass/models/base.py", line 106, in get_ppl_from_template
    return self.get_ppl(inputs, mask_length)
  File "/mnt/home/00054055/project/code/opencompass/opencompass/models/huggingface.py", line 278, in get_ppl
    return np.concatenate([
  File "/mnt/home/00054055/project/code/opencompass/opencompass/models/huggingface.py", line 279, in <listcomp>
    self._get_ppl(inputs=[text], mask_length=mask_length)
  File "/mnt/home/00054055/project/code/opencompass/opencompass/models/huggingface.py", line 307, in _get_ppl
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 1174, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 3029, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
TypeError: cross_entropy_loss(): argument 'ignore_index' (position 5) must be int, not NoneType
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1637419) of binary: /mnt/home/00054055/project/code/opencompass/.venv/bin/python
Traceback (most recent call last):
  File "/mnt/home/00054055/project/code/opencompass/.venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/mnt/home/00054055/project/code/opencompass/.venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/mnt/home/00054055/project/code/opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-08-28_15:21:56
  host      : ctmt230726184112jaq-dcb74d759-sptvj
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1637419)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

```

### Other information

Qwen-7b and Qwen-7b-chat run **ppl** eval meet this problem, dataset is ceval, gen eval works well.

I see #200 , same error, but I can't see how he solve the problem.
It looks like Missing pad_token_id is the reason leads that.
I downloaded model form huggingface. I found pad_token_id config in Qwen-7b-chat, but I don't know why not opencompass read it.

generation_config.json:
```json
{
  "chat_format": "chatml",
  "eos_token_id": 151643,
  "pad_token_id": 151643,
  "max_window_size": 6144,
  "max_new_tokens": 512,
  "do_sample": true,
  "top_k": 0,
  "top_p": 0.5,
  "transformers_version": "4.31.0"
}
```

Please try this pr, it will fix the `ignore_index` bug:

https://github.com/InternLM/opencompass/pull/287

Also, you may take this qwen / qwen-chat implement as a reference:

https://github.com/InternLM/opencompass/pull/286

> Please try this pr, it will fix the `ignore_index` bug:
> 
> #287
> 
> Also, you may take this qwen / qwen-chat implement as a reference:
> 
> #286

#287 works, thanks.