Error in converting convert Huggingface LLaMA3.1 checkpoints into nemo checkpoint
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**
I'm trying to convert **LLama3.1-8b huggingface** checkpoint into **nemo** checkpoint using `convert_llama_hf_to_nemo.py` script. It's giving `ValueError: `rope_scaling` must be a dictionary with two fields, `type` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}`

Find the complete trace below :
```
 File "/opt/NeMo/scripts/checkpoint_converters/convert_llama_hf_to_nemo.py", line 327, in <module>
    convert(args)
  File "/opt/NeMo/scripts/checkpoint_converters/convert_llama_hf_to_nemo.py", line 126, in convert
    model = LlamaForCausalLM.from_pretrained(args.input_name_or_path)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3122, in from_pretrained
    config, model_kwargs = cls.config_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 609, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 761, in from_dict
    config = cls(**config_dict)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/configuration_llama.py", line 161, in __init__
    self._rope_scaling_validation()
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/configuration_llama.py", line 181, in _rope_scaling_validation
    raise ValueError(
ValueError: `rope_scaling` must be a dictionary with two fields, `type` and `factor`, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

```
**Steps/Code to reproduce bug**

```
python <path_to_NeMo>/scripts/checkpoint_converters/convert_llama_hf_to_nemo.py --input_name_or_path Meta-Llama-3.1-8B-Instruct/ --output_path Meta-Llama-3.1-8B-Instruct.nemo
```

**Expected behavior**

Successful conversion of llama3.1-8b hf checkpoints into nemo checkpoint.

**Environment overview (please complete the following information)**

 - I'm using `nvcr.io/nvidia/nemo:24.05.llama3.1`
 - `doker pull nvcr.io/nvidia/nemo:24.05.llama3.1` & `docker run -it --name nemo nvcr.io/nvidia/nemo:24.05.llama3.1`


**Additional context**

NeMo frameworks currently support only **linear** RoPE scaling for LLaMA models. I'm wondering if there are any ongoing efforts for **dynamic** or **yarn** RoPE scaling to support large context windows with LLaMA 3.1.


I have the same issue...

I have the same issue...

@XugZhao @Luxur @RadhaGulhane13 The issue is the current Llama 3.1 model on HF (which you fetch when running the conversion script) is built with the new version of the Transformers library, which was released after our `nvcr.io/nvidia/nemo:24.05.llama3.1`, and has some naming differences compared to the version of the Transformer library installed in the `nvcr.io/nvidia/nemo:24.05.llama3.1` container. To resolve the issue you can use the our dev container `nvcr.io/nvidia/nemo:dev`, and run the same command but add `--llama31 True`