log_every_n_steps only applies to training, not validation/test
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Forgive me if this is a [pytorch lightning issue](https://github.com/PyTorchLightning/pytorch-lightning/issues/4527), and not a NeMo one. If that's the case let me know and I'll close here and continue there. Setting log_every_n_steps on the trainer limits the output of decoded examples in ASR, but not during validation. This causes massive output which in turn causes issues in Colab. As a result I have set `check_val_every_n_epochs` to prevent validation, but ideally I would like to validate and log the metrics to tensorboard, just without all the printed output. 

**Steps/Code to reproduce bug**

Run ASR notebook with `log_every_n_steps=250` and note that the validation outputs logs to terminal for every batch. 

**Expected behavior**

Both validation and training only log output every `log_every_n_steps` steps. 

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

Colab using NeMo 1.0.0b2


This is still an issue but note you can turn off the validation output entirely with 
`quartznet._wer.log_prediction=False` where quartznet is your model instance.

Validation frequency is handled by PTL and can be configured in NeMo via YAML or command line. You can use either `val_check_interval`: https://pytorch-lightning.readthedocs.io/en/stable/trainer.html#val-check-interval or `check-val-every-n-epoch`: https://pytorch-lightning.readthedocs.io/en/stable/trainer.html#check-val-every-n-epoch.

Note `val_check_interval` depends on whether it's a float (percentage of epoch) or int (number of steps).

Most NeMo example configs have `val_check_interval` already but using Hydra the flag can always be added if necessary with
`+trainer.val_check_interval=x`



Thank you @ericharper but I don't believe that will fix my problem. The issue isn't how often NeMo is validating, but how often it is printing out output logs (from WER.py). Please let me know if I'm misunderstanding something. 

The desired behavior is to validate every epoch, but instead of printing the reference and decoded strings as output every step, to do it every `log_every_n_steps` steps. The issue is that in Colab the excessive output causes lagging and sometimes an inability to save the output.  

I think the simplest solution would be to add a condition [here in WER.py](url) checking if `global step % log_every_n_steps == 0`. There may be a more elegant solution but if not I'd be happy to PR it. 

Sorry, closed issue by misclick. I've reopened.

None of the metrics can be dependent on Trainer - they are supposed to be independent from them by design. So the proposed solution would be problematic.  

While the colab issue is understandable, the easiest possible solution is to prevent string logging inside colab environment entirely by setting log_predictions to False, or to enable scrolling inside the cell (even that has some limits if the logs are extremely large).