Model / Checkpoints mismatch
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When attempting to launch training using the specified configuration files and code snippet, a key mismatch error occurs between the model and the dictionary. How to proceed?

**Model:**
https://huggingface.co/nvidia/nemo-megatron-gpt-5B/resolve/main/nemo_gpt5B_fp16_tp1.nemo


**Steps/Code to reproduce bug**

1. Load the original configuration file:
    ```python
    ori_conf = OmegaConf.load('models/extracted_model/model_config.yaml')
    ```
2. Load the new configuration file:
    ```python
    conf = OmegaConf.load('config/megatron_gpt_config.yaml')
    ```
3. Update the model configuration with keys from the original configuration:
    ```python
    for key in ori_conf.keys():
        conf['model'][key] = ori_conf[key]
    ```
4. Set the global batch size based on the micro batch size:
    ```python
    conf['model']["global_batch_size"] = conf['model']["micro_batch_size"] * conf.get('data_model_parallel_size', 1) * conf.get('gradient_accumulation_steps', 1)
    ```
5. Update the data prefix:
    ```python
    conf['model']['data']['data_prefix'] = 'data_ready_for_training/validation'
    ```
6. Update tokenizer configuration:
    ```python
    NEW_TOKENIZER_PATH = "tokenizers/final_merged_arabic_english_gpt2base"
    conf['model']['tokenizer']['vocab_file'] = f"{NEW_TOKENIZER_PATH}/vocab.json"
    conf['model']['tokenizer']['merge_file'] = f"{NEW_TOKENIZER_PATH}/merges.txt"
    conf['model']['vocab_file'] = f"{NEW_TOKENIZER_PATH}/vocab.json"
    conf['model']['merges_file'] = f"{NEW_TOKENIZER_PATH}/merges.txt"
    ```
7. Save the modified configuration:
    ```python
    OmegaConf.save(config=conf, f='config/modifiedmegatron_gpt_config.yaml')
    ```
8. Launch the training process and observe the key mismatch error.

**Expected behavior**

The training process should launch successfully without any key mismatch errors. The configuration should be correctly applied, and the model should start training using the specified settings.

**Environment overview (please complete the following information)**

- **Environment location:** Bare-metal
- **Method of NeMo install:** pip install
    ```bash
    pip install nemo_toolkit[all]
    ```

**Environment details**

- **OS version:** Ubuntu 20.04
- **PyTorch version:** 1.11.0
- **Python version:** 3.8.10

**Additional context**

- **GPU model:** NVIDIA A100

The key mismatch error likely arises due to differences between the keys expected by the model and those present in the configuration files. Ensure that all necessary keys are correctly mapped and that there are no discrepancies between the configuration files.

**Log (Sample - not full)**

`Exception has occurred: RuntimeError
Error(s) in loading state_dict for MegatronGPTModel:
	Missing key(s) in state_dict: "model.embedding.word_embeddings.weight", "model.embedding.position_embeddings.weight", "model.decoder.layers.0.input_layernorm.weight", "model.decoder.layers.0.input_layernorm.bias", "model.decoder.layers.0.self_attention.linear_proj.weight", "model.decoder.layers.0.self_attention.linear_proj.bias", "model.decoder.layers.0.self_attention.linear_qkv.weight", "model.decoder.layers.0.self_attention.linear_qkv.bias", "model.decoder.layers.0.pre_mlp_layernorm.weight", "model.decoder.layers.0.pre_mlp_layernorm.bias", "model.decoder.layers.0.mlp.linear_fc1.weight", "model.decoder.layers.0.mlp.linear_fc1.bias", "model.decoder.layers.0.mlp.linear_fc2.weight", "model.decoder.layers.0.mlp.linear_fc2.bias", "model.decoder.layers.1.input_layernorm.weight", "model.decoder.layers.1.input_layernorm.bias", "model.decoder.layers.1.self_attention.linear_proj.weight", "model.decoder.layers.1.self_attention.linear_proj.bias", "model.decoder.layers.1.self_attention.linear_qkv.weight", 
	Unexpected key(s) in state_dict: "model.language_model.embedding.word_embeddings.weight", "model.language_model.embedding.position_embeddings.weight", "model.language_model.encoder.layers.0.input_layernorm.weight", "model.language_model.encoder.layers.0.input_layernorm.bias", "model.language_model.encoder.layers.0.self_attention.query_key_value.weight", "model.language_model.encoder.layers.0.self_attention.query_key_value.bias", "model.language_model.encoder.layers.0.self_attention.query_key_value._extra_state", "model.language_model.encoder.layers.0.self_attention.dense.weight", "model.language_model.encoder.layers.0.self_attention.dense.bias", "model.language_model.encoder.layers.0.self_attention.dense._extra_state", "model.language_model.encoder.layers.0.post_attention_layernorm.weight", 


**Config file used**
[megatron_gpt_config.yaml](https://github.com/NVIDIA/NeMo/blob/7cf703c576d7d8d1a55c602353c23ead88b92bb5/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L4)

Which version of nemo are you using? The model card: https://huggingface.co/nvidia/nemo-megatron-gpt-5B is using v1.11.0. 

I would recommend using a newer model if possible. This model is quite out of date and likely won't perform as well as more recent models.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.