Adding a Garbage-collection callback for a synchronized garbage-collection across ALL processes
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Adding a Garbage-collection callback for a synchronized garbage-collection across ALL processes

**Collection**: [Note which collection this PR will affect]
Potentially affects all of them.

# Changelog 
- Garbage Collection callback for NeMo 2.0

# Usage
```
    trainer = nl.Trainer(
        strategy = strategy,
        devices=8,
		num_nodes=1,
		accelerator='gpu',
		logger=False,
		enable_checkpointing=False,
		use_distributed_sampler=False,
		#max_epochs=100,
		max_steps=200,
		log_every_n_steps=1,
		val_check_interval=10,
		limit_val_batches=10,
		limit_test_batches=200,
		accumulate_grad_batches=1,
		gradient_clip_val=1.0,
		benchmark=False,
		enable_model_summary=False,
		limit_train_batches=200,
		enable_progress_bar=False,
		num_sanity_val_steps=0,


        **callbacks = [GarbageCollectionCallback(gc_interval_train=20, gc_interval_val=40)],**


        plugins=nl.MegatronMixedPrecision(precision='bf16-mixed',)
    )
```


# Testing 

Log excerpt 
```
      self.communication_handle = torch.distributed._reduce_scatter_base(
    
[NeMo I 2024-08-26 14:07:29 garbage_collection:54] Running garbage collection at train global_step: 20


[NeMo I 2024-08-26 14:07:33 garbage_collection:54] Running garbage collection at train global_step: 40
[NeMo I 2024-08-26 14:07:34 garbage_collection:67] Running garbage collection at validation step: 40
[NeMo I 2024-08-26 14:07:37 garbage_collection:54] Running garbage collection at train global_step: 60
[NeMo I 2024-08-26 14:07:41 garbage_collection:54] Running garbage collection at train global_step: 80
[NeMo I 2024-08-26 14:07:42 garbage_collection:67] Running garbage collection at validation step: 80
[NeMo I 2024-08-26 14:07:45 garbage_collection:54] Running garbage collection at train global_step: 100
[NeMo I 2024-08-26 14:07:48 garbage_collection:54] Running garbage collection at train global_step: 120
[NeMo I 2024-08-26 14:07:49 garbage_collection:67] Running garbage collection at validation step: 120
[NeMo I 2024-08-26 14:07:52 garbage_collection:54] Running garbage collection at train global_step: 140
[NeMo I 2024-08-26 14:07:56 garbage_collection:54] Running garbage collection at train global_step: 160
[NeMo I 2024-08-26 14:07:56 garbage_collection:67] Running garbage collection at validation step: 160
[NeMo I 2024-08-26 14:07:59 garbage_collection:54] Running garbage collection at train global_step: 180
[NeMo I 2024-08-26 14:08:03 garbage_collection:54] Running garbage collection at train global_step: 200
[NeMo I 2024-08-26 14:08:04 garbage_collection:67] Running garbage collection at validation step: 200
`Trainer.fit` stopped: `max_steps=200` reached.
ENDING TIMING RUN AT 2024-08-26 02:08:12 PM
RESULT,large language model,1,79,root,2024-08-26 02:06:53 PM
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [x] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [x] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


How do we enable the collection callback?

> How do we enable the collection callback?

Refer to the usage in the PR description.