Convert llama2-70b-hf to nemo file results in number of weights not in checkpoint but in model
State: closed | Pull request: no
Labels: name: bug

name: stale

I am following this tutorial (https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/llama2peft.html) to convert llama2-70b-hf to a nemo file. I am encountering an issue that a number of weights are not in checkpoint but in model. I wonder if this has anything to do with the `megatron_llama_config.yaml` file, though I double checked every config allies with that of a 7b model.

Config:
```yaml
nemo_config: {'mcore_gpt': True, 'micro_batch_size': 4, 'global_batch_size': 8, 'tensor_model_parallel_size': 1, 'pipeline_model_parallel_size': 1, 'virtual_pipeline_model_parallel_size': None, 'encoder_seq_length': 4096, 'max_position_embeddings': 4096, 'num_layers': 32, 'hidden_size': 4096, 'ffn_hidden_size': 11008, 'num_attention_heads': 32, 'init_method_std': 0.02, 'use_scaled_init_method': True, 'hidden_dropout': 0.0, 'attention_dropout': 0.0, 'ffn_dropout': 0.0, 'kv_channels': None, 'apply_query_key_layer_scaling': True, 'normalization': 'rmsnorm', 'layernorm_epsilon': 1e-05, 'do_layer_norm_weight_decay': False, 'make_vocab_size_divisible_by': 128, 'pre_process': True, 'post_process': True, 'persist_layer_norm': True, 'bias': False, 'activation': 'fast-swiglu', 'headscale': False, 'transformer_block_type': 'pre_ln', 'openai_gelu': False, 'normalize_attention_scores': True, 'position_embedding_type': 'rope', 'rotary_percentage': 1.0, 'attention_type': 'multihead', 'share_embeddings_and_output_weights': False, 'overlap_p2p_comm': False, 'batch_p2p_comm': True, 'num_query_groups': 32, 'tokenizer': {'library': 'sentencepiece', 'type': None, 'model': './llama2-7b-hf/tokenizer.model', 'vocab_file': None, 'merge_file': None, 'delimiter': None, 'sentencepiece_legacy': False}, 'native_amp_init_scale': 4294967296, 'native_amp_growth_interval': 1000, 'hysteresis': 2, 'fp32_residual_connection': False, 'fp16_lm_cross_entropy': False, 'megatron_amp_O2': False, 'grad_allreduce_chunk_size_mb': 125, 'grad_div_ar_fusion': True, 'gradient_accumulation_fusion': False, 'bias_activation_fusion': False, 'bias_dropout_add_fusion': False, 'masked_softmax_fusion': True, 'get_attention_mask_from_fusion': True, 'seed': 1234, 'resume_from_checkpoint': None, 'use_cpu_initialization': True, 'onnx_safe': False, 'apex_transformer_log_level': 30, 'gradient_as_bucket_view': True, 'sync_batch_comm': False, 'activations_checkpoint_granularity': None, 'activations_checkpoint_method': None, 'activations_checkpoint_num_layers': None, 'num_micro_batches_with_partial_activation_checkpoints': None, 'activations_checkpoint_layers_per_pipeline': None, 'sequence_parallel': False, 'transformer_engine': True, 'fp8': False, 'fp8_e4m3': False, 'fp8_hybrid': False, 'fp8_margin': 0, 'fp8_interval': 1, 'fp8_amax_history_len': 1, 'fp8_amax_compute_algo': 'most_recent', 'reduce_amax': True, 'use_emha': False, 'data': {'index_mapping_dir': None, 'data_impl': 'mmap', 'splits_string': '900,50,50', 'seq_length': '${model.encoder_seq_length}', 'skip_warmup': True, 'num_workers': 2, 'dataloader_type': 'single', 'reset_position_ids': False, 'reset_attention_mask': False, 'eod_mask_loss': False, 'validation_drop_last': True, 'no_seqlen_plus_one_input_tokens': False, 'pad_samples_to_global_batch_size': False, 'shuffle_documents': True}, 'nsys_profile': {'enabled': False, 'start_step': 10, 'end_step': 10, 'ranks': [0], 'gen_shape': False}, 'optim': {'name': 'fused_adam', 'lr': 0.0002, 'weight_decay': 0.01, 'betas': [0.9, 0.98], 'sched': {'name': 'CosineAnnealing', 'warmup_steps': 500, 'constant_steps': 50000, 'min_lr': 2e-05}}, 'precision': 32}
```

Some of the missing weights:
```txt
'model.embedding.word_embeddings.weight', 'model.rotary_pos_emb.inv_freq', 'model.decoder.layers.0.self_attention.linear_qkv.weight', 'model.decoder.layers.0.self_attention.linear_proj.weight', 'model.decoder.layers.0.mlp.linear_fc1.weight', 'model.decoder.layers.0.mlp.linear_fc2.weight', 'model.decoder.layers.0.self_attention.linear_qkv.layer_norm_weight', 'model.decoder.layers.0.mlp.linear_fc1.layer_norm_weight'
....
```

**Environment details**
I am using the NVIDIA docker.

Thank you!