Correct way to use CyclicLR with config
State: closed | Pull request: no

Hi 👋 I'm trying to set up training with CyclicLR using config file and it fails every time. NeMo uses CyclicLR from PyTorch, so I guess it should support all the keyword arguments from `torch.optim.lr_scheduler.CyclicLR`, but I'm not sure how to pass them using config file. Here is the config I'm using:

```
  optim:
    name: adamw
    lr: ${model.lr}
    # optimizer arguments
    betas: [0.9, 0.98]
    weight_decay: 1e-3

   sched:
        name: CyclicLR
        iters_per_batch: null # computed at runtime; mandatory to have
        max_steps: null # computed at runtime or explicitly set here; mandatory to have
        
        # pytorch lightning args <mandatory>
        monitor: val_loss
        reduce_on_plateau: false
        
        base_lr: ${model.lr}
        max_lr: 1.0
        step_size_up: 5000
        step_size_down: 5000
        cycle_momentum: false
```

As far as I understand from docstring to this method `prepare_lr_scheduler` https://github.com/NVIDIA/NeMo/blob/a48be1f2679198905f650e05cc93104a9dda4ec6/nemo/core/optim/lr_scheduler.py#L524 
`iters_per_batch` and `max_steps` are mandatory, but every time I pass them I get this error:
```
Error executing job with overrides: []
Traceback (most recent call last):
  File "base_model.py", line 47, in main
    trainer.fit(asr_model)
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 873, in _run
    self.accelerator.setup(self, model)  # note: this sets up self.lightning_module
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/gpu.py", line 42, in setup
    return super().setup(trainer, model)
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 88, in setup
    self.setup_optimizers(trainer)
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 330, in setup_optimizers
    optimizers, lr_schedulers, optimizer_frequencies = self.training_type_plugin.init_optimizers(
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 223, in init_optimizers
    return trainer.init_optimizers(model)
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/pytorch_lightning/trainer/optimizers.py", line 34, in init_optimizers
    optim_conf = model.configure_optimizers()
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 558, in configure_optimizers
    self.setup_optimization()
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 549, in setup_optimization
    self._scheduler = prepare_lr_scheduler(
  File "/home/dev/anaconda3/envs/kenv/lib/python3.8/site-packages/nemo/core/optim/lr_scheduler.py", line 619, in prepare_lr_scheduler
    schedule = scheduler_cls(optimizer, **scheduler_args)
TypeError: __init__() got an unexpected keyword argument 'max_steps'
```

**But** even if I remove these keywords, I still get this error.
So, can anyone please help me to figure out what is wrong with my config file and/or provide link to documentation on how to correctly set PyTorch lr schedulers in NeMo config files?
Thanks.

Full config file and run script can be found here: https://github.com/zzomg/conformer_clr

Environment:

Packages:
- pyTorch_debug: True
- pyTorch_version: 1.9.0.
- pytorch-lightning: 1.4.5
- NeMo ver. 1.3.0