How do I get the scheduler going?
State: closed | Pull request: no

**Describe your question**

I am getting this message:

```
[NeMo W 2020-10-15 18:42:14 lr_scheduler:526] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
    Scheduler will not be instantiated !
```
when I do

`model.setup_optimization(optim_config=DictConfig(params['model']['optim']))`

The stock YAML file has `max_steps` as null and no setting for `iters_per_batch`:

```
trainer:
  gpus: 0 # number of gpus
  max_epochs: 5
  max_steps: null # computed at runtime if not set
  num_nodes: 1
  distributed_backend: ddp
  accumulate_grad_batches: 1
  checkpoint_callback: False  # Provided by exp_manager
  logger: False  # Provided by exp_manager
  row_log_interval: 100  # Interval of logging.
  val_check_interval: 1.0 # check once per epoch .25 for 4 times per epoch
```
How do I set this up to get a learning rate scheduler?

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 20.04, Anaconda Python 3.7
 - Method of NeMo install: pip install

**Environment details**

- OS version: Ubuntu 20.04LTS
- PyTorch version: 1.6.0
- Python version: 3.7.4
- GPU model: NVidia GeForce RTX 2080TI
- CUDA: 10.1.243
- cuDNN: 7.6.5



If max steps is not provided to the trainer, setup optimization will estimate effective max steps based on the training data loader. 

Therefore please call setup training data before calling setup optimization to allow estimation of maximum steps. 

I do call setup training data first:

```
from ruamel.yaml import YAML
from omegaconf import DictConfig
config_path = 'amharic_16000.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
train_manifest=f'{C.build_dir}/train_manifest.json'
test_manifest=f'{C.build_dir}/test_manifest.json'
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest
model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])
model.setup_optimization(optim_config=DictConfig(params['model']['optim']))
```
but I still get this message:

```
[NeMo I 2020-10-15 18:48:49 collections:173] Dataset loaded with 7440 files totalling 7.42 hours
[NeMo I 2020-10-15 18:48:49 collections:174] 24 files were filtered totalling 0.13 hours
[NeMo I 2020-10-15 18:48:50 collections:173] Dataset loaded with 1866 files totalling 1.98 hours
[NeMo I 2020-10-15 18:48:50 collections:174] 0 files were filtered totalling 0.00 hours
[NeMo I 2020-10-15 18:48:50 modelPT:572] Optimizer config = Novograd (
    Parameter Group 0
        amsgrad: False
        betas: [0.8, 0.5]
        eps: 1e-08
        grad_averaging: False
        lr: 0.0005
        weight_decay: 0.001
    )
[NeMo W 2020-10-15 18:48:50 lr_scheduler:526] Neither `max_steps` nor `iters_per_batch` were provided to `optim.sched`, cannot compute effective `max_steps` !
    Scheduler will not be instantiated !
(Novograd (
 Parameter Group 0
     amsgrad: False
     betas: [0.8, 0.5]
     eps: 1e-08
     grad_averaging: False
     lr: 0.0005
     weight_decay: 0.001
 ),
 None)
```

where the YAML file is:

```
name: &name "QuartzNet15x5"
sample_rate: &sample_rate 16000
repeat: &repeat 1
dropout: &dropout 0.0
separable: &separable true
labels: &labels [' ', ሀ, ሁ, ሂ, ሃ, ሄ, ህ, ሆ, ለ, ሉ,
                      ሊ, ላ, ሌ, ል, ሎ, ሏ, ሐ, ሑ, ሒ, ሔ, ሕ, መ, ሙ,
                      ሚ, ማ, ሜ, ም, ሞ, ሟ, ሠ, ሡ, ሣ, ሥ, ሦ, ረ, ሩ,
                      ሪ, ራ, ሬ, ር, ሮ, ሯ, ሰ, ሱ, ሲ, ሳ, ሴ, ስ, ሶ,
                      ሷ, ሸ, ሹ, ሺ, ሻ, ሼ, ሽ, ሾ, ሿ, ቀ, ቁ, ቂ, ቃ,
                      ቄ, ቅ, ቆ, ቋ, በ, ቡ, ቢ, ባ, ቤ, ብ, ቦ, ቧ, ቨ,
                      ቪ, ቫ, ቬ, ቭ, ተ, ቱ, ቲ, ታ, ቴ, ት, ቶ, ቷ, ቸ,
                      ቹ, ቺ, ቻ, ቼ, ች, ቾ, ቿ, ኋ, ነ, ኑ, ኒ, ና, ኔ,
                      ን, ኖ, ኗ, ኘ, ኙ, ኚ, ኛ, ኜ, ኝ, ኞ, አ, ኡ, ኢ,
                      ኤ, እ, ኦ, ኧ, ከ, ኩ, ኪ, ካ, ኬ, ክ, ኮ, ኳ, ኸ,
                      ኻ, ኼ, ኽ, ወ, ዉ, ዊ, ዋ, ዌ, ው, ዎ, ዐ, ዑ, ዓ,
                      ዕ, ዘ, ዙ, ዚ, ዛ, ዜ, ዝ, ዞ, ዟ, ዠ, ዢ, ዣ, ዤ,
                      ዥ, ዦ, የ, ዩ, ዪ, ያ, ዬ, ይ, ዮ, ደ, ዱ, ዲ, ዳ,
                      ዴ, ድ, ዶ, ዷ, ጀ, ጁ, ጂ, ጃ, ጄ, ጅ, ጆ, ጇ, ገ,
                      ጉ, ጊ, ጋ, ጌ, ግ, ጎ, ጓ, ጠ, ጡ, ጢ, ጣ, ጤ, ጥ,
                      ጦ, ጧ, ጨ, ጩ, ጪ, ጫ, ጬ, ጭ, ጮ, ጱ, ጲ, ጳ, ጴ,
                      ጵ, ጸ, ጹ, ጺ, ጻ, ጼ, ጽ, ጾ, ጿ, ፀ, ፁ, ፃ, ፄ,
                      ፅ, ፆ, ፈ, ፉ, ፊ, ፋ, ፌ, ፍ, ፎ, ፏ, ፐ, ፑ, ፒ,
                      ፓ, ፔ, ፕ, ፖ]

model:
  train_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: *labels
    batch_size: 24
    trim_silence: True
    max_duration: 16.5
    shuffle: True
    is_tarred: False
    tarred_audio_filepaths: null

  validation_ds:
    manifest_filepath: ???
    sample_rate: 16000
    labels: *labels
    batch_size: 24
    shuffle: False

  preprocessor:
    cls: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    params:
      normalize: "per_feature"
      window_size: 0.02
      sample_rate: *sample_rate
      window_stride: 0.01
      window: "hann"
      features: &n_mels 64
      n_fft: 512
      frame_splicing: 1
      dither: 0.00001
      stft_conv: false

  spec_augment:
    cls: nemo.collections.asr.modules.SpectrogramAugmentation
    params:
      rect_freq: 50
      rect_masks: 5
      rect_time: 120

  encoder:
    cls: nemo.collections.asr.modules.ConvASREncoder
    params:
      feat_in: *n_mels
      activation: relu
      conv_mask: true

      jasper:
        - filters: 128
          repeat: 1
          kernel: [11]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [13]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [15]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [17]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: *repeat
          kernel: [19]
          stride: [1]
          dilation: [1]
          dropout: *dropout
          residual: true
          separable: *separable
          se: true
          se_context_size: -1

        - filters: 256
          repeat: 1
          kernel: [21]
          stride: [1]
          dilation: [1]
          dropout: 0.0
          residual: false
          separable: *separable
          se: true
          se_context_size: -1

        - filters: &enc_feat_out 1024
          repeat: 1
          kernel: [1]
          stride: [1]
          dilation: [1]
          dropout: 0.0
          residual: false
          separable: *separable
          se: true
          se_context_size: -1

  decoder:
    cls: nemo.collections.asr.modules.ConvASRDecoder
    params:
      feat_in: 1024
      num_classes: 28
      vocabulary: *labels

  optim:
    name: novograd
    # cls: nemo.core.optim.optimizers.Novograd
    lr: .0005
    # optimizer arguments
    betas: [0.8, 0.5]
    weight_decay: 0.001

    # scheduler setup
    sched:
      name: CosineAnnealing

      # pytorch lightning args
      monitor: val_loss
      reduce_on_plateau: false

      # Scheduler params
      warmup_steps: null
      warmup_ratio: null
      min_lr: 0.0
      last_epoch: -1

trainer:
  gpus: 0 # number of gpus
  max_epochs: 5
  max_steps: null # computed at runtime if not set
  num_nodes: 1
  distributed_backend: ddp
  accumulate_grad_batches: 1
  checkpoint_callback: False  # Provided by exp_manager
  logger: False  # Provided by exp_manager
  row_log_interval: 100  # Interval of logging.
  val_check_interval: 1.0 # check once per epoch .25 for 4 times per epoch

exp_manager:
  exp_dir: null
  name: *name
  create_tensorboard_logger: True
  create_checkpoint_callback: True

hydra:
  run:
    dir: .
  job_logging:
    root:
      handlers: null

```

I see, it seems the model is being restored, not called from the Constructor. 

In such a case, the pytorch lightning Trainer object has not been attached, and therefore you will need to explicitly call `model.set_trainer(trainer)`. We require the trainer object in order to determine number of gpus, accumulated grad steps etc in order to compute the effective max steps. 

I'll explain that a little better in the primer notebooks. 

I do `trainer.fit(model)`.  By that construction I assumed that the trainer was "around" the model, not the other way around or intertwined.  So now I'm confused, but I will add the line.

For normal pytorch lightning models, one can build the model with assumptions around the dataset it will be trained and validates on, and therefore a person can easily explicitly set the scheduler with exact number of steps inside `configure_optimizer` method. 

That's not exactly the case for Nemo Models. Therefore some features from trainer are a requirement for model to construct itself independently of the domain. Experiment Manager and scheduler are a few such examples.