Abstract IOMixin and ability to mutate init parameters.
State: closed | Pull request: yes
Labels: name: Run CICD

# What does this PR do ?

Adds two helpful items for working with ABCs that eventually get IOMixin:
1. A parent `NeedsIOMixin` ABC that lets you inform a parent class that it will eventually be an IOMixin, and also catches cases where you use it before subclassing and inheriting from iomixin. This has been useful for making parent config types that children implement and extend.
2. A mutate_hparam function that allows you to change the init args that are saved in an iomixin object. This is useful for users who either want to initialize and lazily load configuration, or change configuration based on some other configuration in a way that you want to have saved for later as modified init args. This gives users a path to change a config object in code and save it, as well as have a config set some of it's parameters based on the contents of another config that it loads from a checkpoint, while also being savable itself into a new checkpoint with those modified parameters.

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)


Closing this for now. Sounds like this conflicts with another plan. To summarize that discussion with @marcromeyn:

1. if I can call a function to get access to old checkpoint settings `old_ckpt = io.load(self.ckpt_init_path).model.checkpoint` which does not impact global state for the current context/run (that's important)
2. Additionally if I can access global state for the checkpoint associated with self to mutate it (along with in parallel mutating self). Maybe something like `setattr(io.ckpt_ctx.trainer.model.checkpoint, key, var)` along with `setattr(self, key, var)` which would happen in parallel in the same way on every execution process.
3. Additionally if I don't touch any parallelism/cluster settings in the checkpoint which would mutate the global environment

These features are key in https://github.com/NVIDIA/bionemo-fw-ea/pull/97