fix validation_step_outputs initialization for multi-dataloader
State: closed | Pull request: yes
Labels: name: core

name: cherry-pick

name: ASR

# What does this PR do ?

Correctly handle initialization of validation_step_outputs and test_step_outputs (to log metrics) when multiple dataloaders are set after initialization of model.

**Collection**:
Core

# Changelog 

In modelPT.py - 
```python
  @property
  def validation_step_outputs(self):
      """
      Cached outputs of validation_step. It can be a list of items (for single data loader) or a list of lists
      (for multiple data loaders).
      Returns:
          List of outputs of validation_step.
      """
      if self._validation_step_outputs is not None:
          return self._validation_step_outputs

      # Initialize new output list
      self._validation_step_outputs = []
      # Check len(self._validation_dl) > 1 as sometimes single dataloader can be in a list: [<Dataloader obj>] when ds_item in
      # config has 1 item passed in a list
      if (
          self._validation_dl is not None
          and isinstance(self._validation_dl, (list, tuple))
          and len(self._validation_dl) > 1
      ):
          for _ in range(len(self._validation_dl)):
              self._validation_step_outputs.append([])

      return self._validation_step_outputs

  @validation_step_outputs.setter
  def validation_step_outputs(self, value):
      self._validation_step_outputs = value

  @property
  def test_step_outputs(self):
      """
      Cached outputs of test_step. It can be a list of items (for single data loader) or a list of lists (for multiple data loaders).
      Returns:
          List of outputs of test_step.
      """
      if self._test_step_outputs is not None:
          return self._test_step_outputs

      # Initialize new output list
      self._test_step_outputs = []
      # Check len(self._test_dl) > 1 as sometimes single dataloader can be in a list: [<Dataloader obj>] when ds_item in
      # config has 1 item passed in a list
      if self._test_dl is not None and isinstance(self._test_dl, (list, tuple)) and len(self._test_dl) > 1:
          for _ in range(len(self._test_dl)):
              self._test_step_outputs.append([])

      return self._test_step_outputs

    @test_step_outputs.setter
    def test_step_outputs(self, value):
        self._test_step_outputs = value

  def _cleanup_on_execution_end(self):
      """
      Utility function to clean up the module state at the end of execution.
      """

      # dynamic freezing cleanup
      if hasattr(self, '_freeze_cfg'):
          delattr(self, '_freeze_cfg')

      # Clear up the val and test output caches
      self._validation_step_outputs = None
      self._test_step_outputs = None

  def on_test_end(self):
      """ PyTorch Lightning hook:
          https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-test-end
      """

      self._cleanup_on_execution_end()

  def on_predict_end(self):
      """ PyTorch Lightning hook:
          https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html#on-test-end
      """

      self._cleanup_on_execution_end()
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation