Utilities to detect and drop deprecated arguments from NeMo 2.0 checkpoint context io.json
State: closed | Pull request: yes
Labels: name: Run CICD

name: r2.1.0

# What does this PR do ?

Detecting and dropping unknown (deprecated) arguments from NeMo 2.0 io.json context.

**Collection**: LLM

# Usage

Loading the context for a NeMo 2.0 checkpoint produced using [24.09 container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo/tags) and used with a more recent version results in the following error:
```
  File "/usr/local/lib/python3.10/dist-packages/fiddle/_src/daglish.py", line 786, in apply
    result = self.traversal_fn(value, state)
  File "/usr/local/lib/python3.10/dist-packages/fiddle/_src/building.py", line 180, in _build
    return call_buildable(value, arguments, current_path=state.current_path)
  File "/usr/local/lib/python3.10/dist-packages/fiddle/_src/building.py", line 120, in call_buildable
    return buildable.__build__(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/fiddle/_src/config.py", line 783, in __build__
    return self.__fn_or_cls__(*args, **kwargs)
TypeError: OptimizerConfig.__init__() got an unexpected keyword argument 'overlap_grad_reduce'
```

This is a consequence of a recent Megatron-Core  refactor that removed `overlap_grad_reduce` and `overlap_param_gather` from `OptimizerConfig`, see https://github.com/NVIDIA/Megatron-LM/commit/6ac4db023bf7eb0e825e09aa8532af4a2c55f359.

The context may be cleared from deprecated parameters and updated using the provided script to load model successfully for subsequent processing:
```
python scripts/llm/update_io_context.py \
    --model_path /opt/checkpoints/LLAMA3-8B-nemo2-24-09 \
    --build
```

Alternatively, `drop_unexpected_params(config)` can be directly called in `load` function from **nemo/lightning/io/mixin.py** in  [L754](https://github.com/NVIDIA/NeMo/blob/0133deb268812109eab5211f902d7c4b99b6ee95/nemo/lightning/io/mixin.py#L754).

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


[🤖]: Hi @janekl 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12411404781) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

> Looks great, I wonder if we can do this directly on the fly instead of via a script. What are your thoughts?

Yes we can definitely, I'll file a follow-up MR

[like]  Hemil Desai reacted to your message:
________________________________
From: Jan Lasek ***@***.***>
Sent: Friday, December 20, 2024 4:39:25 PM
To: NVIDIA/NeMo ***@***.***>
Cc: Hemil Desai ***@***.***>; Review requested ***@***.***>
Subject: Re: [NVIDIA/NeMo] Utilities to detect and drop deprecated arguments from NeMo 2.0 checkpoint context io.json (PR #11648)


Looks great, I wonder if we can do this directly on the fly instead of via a script. What are your thoughts?

Yes we can definitely, I'll file a follow-up MR

—
Reply to this email directly, view it on GitHub<https://github.com/NVIDIA/NeMo/pull/11648#issuecomment-2557357566>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB6Q25CX4XOU2IXPIJ5CIV32GRB33AVCNFSM6AAAAABT25MWBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNJXGM2TONJWGY>.
You are receiving this because your review was requested.Message ID: ***@***.***>