Add support for Hydra multirun to NeMo
State: closed | Pull request: yes

Signed-off-by: smajumdar <titu1994@gmail.com>

# What does this PR do ?

Adds support to hydra multirun to NeMo core.

**Collection**: [ASR, Core]

# Changelog 

- Usage is similar to normal hydra script call, just the config should have some additional sections to setup multirun and you must use `-m` flag when calling the python script.

# Usage

At minimum (if all other arguments such as data loader and model related arguments are provided) - 

```python

python train_asr_adapter.py \
    -m \  # necessary
    trainer.devices=[0] \  # any additional args will get forwarded to all HP jobs. 
    exp_manager.exp_dir="<some directory>" \
    exp_manager.create_wandb_logger=true \  # highly recommended to use WandB for HP runs
    exp_manager.wandb_logger_kwargs.project="<Project Name>"  # Experiment name will be setup based on exp_manager.name

```

# Setting up configs to support multirun

## Setup Experiment manager

Inside experiment manager, the `name` argument **must** contain the concatination of the `${name}` and other hyper parameters. This is only needed if you use the two `resume` flags as True, but it is highly recommended nevertheless so you can launch and continue HP jobs until all are consumed.

``` yaml

exp_manager:
  exp_dir: null

  # Add a unique name for all hyperparameter arguments to allow continued training.
  # NOTE: It is necessary to add all hyperparameter arguments to the name !
  # This ensures successful restoration of model runs in case HP search crashes.
  name: ${name}-lr-${model.optim.lr}-adim-${model.adapter.dim}-sd-${model.adapter.adapter_strategy.stochastic_depth}
  ...
  checkpoint_callback_params:
    ...
    save_top_k: 1  # Dont save too many .ckpt files during HP search
    always_save_nemo: True # saves the checkpoints as nemo files for fast checking of results later
  ...

  # HP Search may crash due to various reasons, best to attempt continuation in order to
  # resume from where the last failure case occured.
  resume_if_exists: true
  resume_ignore_no_checkpoint: true
```

## Hydra Defaults

``` yaml

# Required for Hydra launch of hyperparameter search via multirun
defaults:
  - override hydra/launcher: nemo_launcher

# Hydra arguments necessary for hyperparameter optimization
hydra:
  sweep:
    dir: "."
    subdir: "."

  sweeper:
    params:  # place all the parameters you wish to search over here (corresponding to the rest of the config)
      model.optim.lr: 0.001,0.0001
      model.adapter.dim: 32,64,96,128
      model.adapter.adapter_strategy.stochastic_depth: 0.0,0.5,0.6,0.7,0.8,0.9

  # Arguments to the process launcher
  launcher:
    num_gpus: -1  # Number of gpus to use. Each run works on a single GPU.
    jobs_per_gpu: 1  # If each GPU has large memory, you can run multiple jobs on the same GPU for faster results (until OOM).

```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [x] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation