Creating a hf to nemo converter for phi 3 mini instruct
State: closed | Pull request: no

Hi ,

I am following the guidelines https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/checkpoints/dev_guide.html to create a converter. It looks like in addition to qkv reshaping and layer name mapping we need to load phi model into megatron_gpt_model. @cuichenx and @mayani-nv I noticed you have some PRs with phi 3 mini. Would you by any chance have a sample code. I've added code for "_create_rename_keys_" and "_adjust_tensor_shapes_" based on your code. I am struggling to load it into megatron_gpt_model. 

and I suspect its because I need something like https://github.com/NVIDIA/NeMo/blob/7f3ac6b88289048643d445ec752a4d21547f2788/nemo/collections/nlp/models/language_modeling/megatron/gemma2/gemma2_spec.py#L28

Is there someone I could work with to improve the documentation or to simplify the process for this conversion ?

thanks!

@LopezGG  the Phi3 mini has already been added [here](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/gpt/model/phi3mini.py). The steps to do the fine-tuning with Phi3 are also described in the recipe as [well](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/phi3_mini_4k_instruct.py). If you look at the first link, the qkv reshaping is performed to load Phi model into the megatron_gpt_model.

@LopezGG To add on to what @mayani-nv has said: Phi 3 mini is supported in NeMo 2.0, and we recommend you switch to 2.0 for new explorations with NeMo. You can find quick start instructions [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/quickstart.html).  

@LopezGG  a dummy example to use this Phi3 model can be run as following steps

```
# start the NeMo container interactively
docker run -it --rm --gpus all nvcr.io/nvidia/nemo:24.12

# download the ckpt 
python3 download_phi3ckpt.py

# set the NEMO_HOME path
export NEMO_HOME=/root/.cache/nemo

# run the inference to ensure parity with the HF format
python3 phi3_generate.py
```
The scripts used above are [download_phi3ckpkt.py](https://github.com/mayani-nv/NeMo/blob/phi3mini4k/scripts/llm/download_phi3ckpt.py) and [phi3_generate.py](https://github.com/mayani-nv/NeMo/blob/phi3mini4k/scripts/llm/phi3_generate.py)

amazing. Thank you @mayani-nv. I was just trying to get this done by toying around with HFPhi3Importer & HFPhi3Exporter. This is straightforward

@mayani-nv : I got my checkpoint but when I finetune I see that is missing some metadata

<img width="635" alt="Image" src="https://github.com/user-attachments/assets/584941fb-f1a3-4546-b2be-6f56401a0492" />

What would be a good way to debug this. I am attaching all the files except weights in case that might be helpful

[nemo_phi_instruct.zip](https://github.com/user-attachments/files/18411032/nemo_phi_instruct.zip)

Can you share the command how were you doing the finetuning? Are you using the [recipe](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/phi3_mini_4k_instruct.py)? The way I do it is using the following cli command inside the container

```
nemo llm finetune --factory phi3_mini_4k_instruct
[NeMo W 2025-01-14 06:01:30 nemo_logging:361] /usr/local/lib/python3.10/dist-packages/pyannote/core/notebook.py:134: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.
      cm = get_cmap("Set1")
.......
.......
Training epoch 0, iteration 110/999 | lr: 9.902e-05 | global_batch_size: 128 | global_step: 110 | reduced_train_loss: 0.1471 | consumed_samples: 14208 | val_loss: 0.2043
Training epoch 0, iteration 111/999 | lr: 9.899e-05 | global_batch_size: 128 | global_step: 111 | reduced_train_loss: 0.2019 | consumed_samples: 14336 | val_loss: 0.2043
Training epoch 0, iteration 112/999 | lr: 9.895e-05 | global_batch_size: 128 | global_step: 112 | reduced_train_loss: 0.1712 | consumed_samples: 14464 | val_loss: 0.2043
Training epoch 0, iteration 113/999 | lr: 9.892e-05 | global_batch_size: 128 | global_step: 113 | reduced_train_loss: 0.2207 | consumed_samples: 14592 | val_loss: 0.2043
Training epoch 0, iteration 114/999 | lr: 9.888e-05 | global_batch_size: 128 | global_step: 114 | reduced_train_loss: 0.2231 | consumed_samples: 14720 | val_loss: 0.2043
Training epoch 0, iteration 115/999 | lr: 9.885e-05 | global_batch_size: 128 | global_step: 115 | reduced_train_loss: 0.2529 | consumed_samples: 14848 | val_loss: 0.2043
Training epoch 0, iteration 116/999 | lr: 9.881e-05 | global_batch_size: 128 | global_step: 116 | reduced_train_loss: 0.209 | consumed_samples: 14976 | val_loss: 0.2043
Training epoch 0, iteration 117/999 | lr: 9.878e-05 | global_batch_size: 128 | global_step: 117 | reduced_train_loss: 0.2174 | consumed_samples: 15104 | val_loss: 0.2043
Training epoch 0, iteration 118/999 | lr: 9.874e-05 | global_batch_size: 128 | global_step: 118 | reduced_train_loss: 0.2147 | consumed_samples: 15232 | val_loss: 0.2043
Training epoch 0, iteration 119/999 | lr: 9.87e-05 | global_batch_size: 128 | global_step: 119 | reduced_train_loss: 0.167 | consumed_samples: 15360 | val_loss: 0.2043
[NeMo I 2025-01-14 07:18:56 model_checkpoint:497] Scheduled async checkpoint save for /workspace/nemo_experiments/default/2025-01-14_06-01-34/checkpoints/model_name=0--val_loss=0.20-step=119-consumed_samples=15360.0-last.ckpt
```
Also, I have only added support for the 4k context length so are you using the other variant?
@cuichenx  do you happen to know the error cause?

Extending to 8k could be an issue. I am calling on your recipe to increase the context length (for starters, I left cp=1)
[phi3_mini_4k_instruct_4k_8k_recipe](https://github.com/user-attachments/files/18414100/phi3_mini_4k_instruct_4k_8k.txt) (a variation of https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/llm/recipes/llama3_8b_16k.py)

<img width="
631" alt="Image" src="https://github.com/user-attachments/assets/25aebfcf-4488-4cf0-97d5-e4e129760559" />