Fail to merge lora with base model: "TypeError: ModelPT.restore_from() got an unexpected keyword argument 'mcore'"
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**
Encounter this error when execute scripts/nlp_language_modeling/merge_lora_weights/merge.py with lora and base gpt 126m model.

A clear and concise description of what the bug is.

**Steps/Code to reproduce bug**
execute this on A100 dgxa100.80g.2.norm
```bash
python3 /opt/NeMo/scripts/nlp_language_modeling/merge_lora_weights/merge.py \
                trainer.devices=1 \
                trainer.num_nodes=1 \
                trainer.precision=bf16 \
                tensor_model_parallel_size=1 \
                pipeline_model_parallel_size=1 \
                gpt_model_file=/mount_workspace/results/gpt3_126m/convert_nemo/results/megatron_gpt.nemo \
                lora_model_path=/mount_workspace/results/lora-tp1/checkpoints/megatron_gpt_peft_tuning-lora.nemo \
                merged_model_path=/mount_workspace/results/lora-merge/checkpoints/lora-gpt126m-merged.nemo"
```
failed with error message 
```
Error executing job with overrides: ['trainer.devices=1', 'trainer.num_nodes=1', 'trainer.precision=bf16', 'tensor_model_parallel_size=1', 'pipeline_model_parallel_size=1', 'gpt_model_file=/mount_workspace/results/gpt3_126m/convert_nemo/results/megatron_gpt.nemo', 'lora_model_path=/mount_workspace/results/lora-tp1/checkpoints/megatron_gpt_peft_tuning-lora.nemo', 'merged_model_path=/mount_workspace/results/lora-merge/checkpoints/lora-gpt126m-merged.nemo']
Traceback (most recent call last):
File "/opt/NeMo/scripts/nlp_language_modeling/merge_lora_weights/merge.py", line 238, in <module>
main() # noqa pylint: disable=no-value-for-parameter
File "/opt/NeMo/nemo/core/config/hydra_runner.py", line 126, in wrapper
_run_hydra(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 389, in _run_hydra
_run_app(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 452, in _run_app
run_and_report(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 216, in run_and_report
raise ex
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 213, in run_and_report
return func()
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 453, in <lambda>
lambda: hydra.run(
File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/opt/NeMo/scripts/nlp_language_modeling/merge_lora_weights/merge.py", line 198, in main
lora_model_cfg = MegatronGPTLoRAModel.restore_from(
TypeError: ModelPT.restore_from() got an unexpected keyword argument 'mcore'
```

Please list *minimal* steps or code snippet for us to be able to reproduce the bug.


**Expected behavior**
no error

**Environment overview (please complete the following information)**

 - Environment location: Nvidia DGX cloud 
 - Method of NeMo install:  container "nemofw-training:23.08.03-ucx-downgrade-no-redirect"

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version
- PyTorch version
- Python version

**Additional context**
failed at this [line](https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/merge_lora_weights/merge.py#L199) 
     
I tried manually remove the `mcore` param in script it seems passed the point. 



This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.