RuntimeError: CUDA error: out of memory. Can't run the ASR_CTC_Language_Finetuning Tutorial while memory should be sufficient.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

"CUDA error: out of memory" was reported when training.
```bash
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File <timed eval>:1, in <module>

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:770, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    751 r"""
    752 Runs the full optimization routine.
    753 
   (...)
    767     datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`.
    768 """
    769 self.strategy.model = model
--> 770 self._call_and_handle_interrupt(
    771     self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    772 )

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:723, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    721         return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    722     else:
--> 723         return trainer_fn(*args, **kwargs)
    724 # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7
    725 except KeyboardInterrupt as exception:

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:811, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    807 ckpt_path = ckpt_path or self.resume_from_checkpoint
    808 self._ckpt_path = self.__set_ckpt_path(
    809     ckpt_path, model_provided=True, model_connected=self.lightning_module is not None
    810 )
--> 811 results = self._run(model, ckpt_path=self.ckpt_path)
    813 assert self.state.stopped
    814 self.training = False

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:1217, in Trainer._run(self, model, ckpt_path)
   1214 self._logger_connector.reset_metrics()
   1216 # strategy will configure model and move it to the device
-> 1217 self.strategy.setup(self)
   1219 # hook
   1220 if self.state.fn == TrainerFn.FITTING:

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/strategies/single_device.py:71, in SingleDeviceStrategy.setup(self, trainer)
     70 def setup(self, trainer: pl.Trainer) -> None:
---> 71     self.model_to_device()
     72     super().setup(trainer)

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/strategies/single_device.py:68, in SingleDeviceStrategy.model_to_device(self)
     67 def model_to_device(self) -> None:
---> 68     self.model.to(self.root_device)

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/pytorch_lightning/core/mixins/device_dtype_mixin.py:121, in DeviceDtypeModuleMixin.to(self, *args, **kwargs)
    119 out = torch._C._nn._parse_to(*args, **kwargs)
    120 self.__update_properties(device=out[0], dtype=out[1])
--> 121 return super().to(*args, **kwargs)

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/torch/nn/modules/module.py:927, in Module.to(self, *args, **kwargs)
    923         return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    924                     non_blocking, memory_format=convert_to_format)
    925     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
--> 927 return self._apply(convert)

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/torch/nn/modules/module.py:579, in Module._apply(self, fn)
    577 def _apply(self, fn):
    578     for module in self.children():
--> 579         module._apply(fn)
    581     def compute_should_use_set_data(tensor, tensor_applied):
    582         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    583             # If the new tensor has compatible tensor type as the existing tensor,
    584             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    589             # global flag to let the user control whether they want the future
    590             # behavior of overwriting the existing tensor or not.

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/torch/nn/modules/module.py:579, in Module._apply(self, fn)
    577 def _apply(self, fn):
    578     for module in self.children():
--> 579         module._apply(fn)
    581     def compute_should_use_set_data(tensor, tensor_applied):
    582         if torch._has_compatible_shallow_copy_type(tensor, tensor_applied):
    583             # If the new tensor has compatible tensor type as the existing tensor,
    584             # the current behavior is to change the tensor in-place using `.data =`,
   (...)
    589             # global flag to let the user control whether they want the future
    590             # behavior of overwriting the existing tensor or not.

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/torch/nn/modules/module.py:625, in Module._apply(self, fn)
    623 for key, buf in self._buffers.items():
    624     if buf is not None:
--> 625         self._buffers[key] = fn(buf)
    627 return self

File ~/miniconda3/envs/nemo/lib/python3.9/site-packages/torch/nn/modules/module.py:925, in Module.to.<locals>.convert(t)
    922 if convert_to_format is not None and t.dim() in (4, 5):
    923     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None,
    924                 non_blocking, memory_format=convert_to_format)
--> 925 return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

**Steps/Code to reproduce bug**
1. Copy the ipynb to a server with a GPU and install CUDA 11.6, CuDNN 8.4.1 , PyTorch 1.12.1
2. Install Nvidia Apex and run all the command listed in the first block
3. Run every block till `%%time
trainer.fit(char_model)`


**Expected behavior**

Start training as what I've seen when running in colab.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal **✓**, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install  **✓** or from source]. Please specify exact commands you used to install.
~~If method of install is [Docker], provide `docker pull` & `docker run` commands used~~

**Environment details**

~~If NVIDIA docker image is used you don't need to specify these.~~
Otherwise, please provide:
- OS version 5.4.0-100-generic Ubuntu SMP(Ubuntu 18.04.6 LTS)  x86_64 x86_64 x86_64 GNU/Linux
- PyTorch version 1.12.1
- Python version 3.9.13

**Additional context**

GPU model: 3090