Numba error while fine-tuning an ASR model
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When fine-tuning an ASR model, at some point throughout the first epoch of training, a Numba error occurs:

```
numba.cuda.cudadrv.driver.CudaAPIError: [1] Call to cuLaunchKernel results in CUDA_ERROR_INVALID_VALUE
```

**Steps/Code to reproduce bug**

Here is the code I run:

```python
import os
from argparse import ArgumentParser
from shutil import rmtree

import pytorch_lightning as pl
from pytorch_lightning.loggers import TensorBoardLogger
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning.strategies.ddp import DDPStrategy
from nemo.collections.asr.models.rnnt_bpe_models import EncDecRNNTBPEModel
from omegaconf import OmegaConf

def main(args):
    trainer = pl.Trainer(
        strategy=DDPStrategy(find_unused_parameters=False),
        accelerator=args.accelerator,
        num_nodes=args.devices,
        max_epochs=2,
    )

    config = OmegaConf.load(args.config)
    model = EncDecRNNTBPEModel.from_pretrained("stt_en_conformer_transducer_large")
    train_dl = model._setup_dataloader_from_config(config.train)
    test_dl = model._setup_dataloader_from_config(config.test)
    model.setup_optimization(config.optim)
    trainer.fit(model, train_dl, test_dl)


if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--config", default="config.yaml")
    parser.add_argument("--accelerator", default="gpu")
    parser.add_argument("--devices", default="1")
    args = parser.parse_args()
    main(args)
```

Here is the config:

```yaml
train:
  manifest_filepath: train.txt
  sample_rate: 16000
  batch_size: 1
  num_workers: 10
  pin_memory: true
  use_start_end_token: false
  trim_silence: false
  max_duration: 20.0
  min_duration: 0.1
  is_tarred: false
  tarred_audio_filepaths: null
  shuffle_n: 2048
  bucketing_strategy: "synced_randomized"
  bucketing_batch_size: null
  return_sample_id: false
  drop_last: false
  shuffle: false

test:
  manifest_filepath: test.txt
  sample_rate: 16000
  batch_size: 1
  num_workers: 10
  pin_memory: true
  use_start_end_token: false
  trim_silence: false
  max_duration: 20.0
  min_duration: 0.1
  is_tarred: false
  tarred_audio_filepaths: null
  shuffle_n: 2048
  bucketing_strategy: "synced_randomized"
  bucketing_batch_size: null
  return_sample_id: false
  drop_last: false
  shuffle: false

optim:
  name: adamw
  lr: 0.00001 # maximal learning rate
  betas: [0.9, 0.98]
  weight_decay: 0.001
```

Attached is a zip file that contains these files, as well as the data files. The data comes from English Common Voice 10.0. The audiofiles are MP3 files.

**Environment overview (please complete the following information)**

I have tried several Nemo images (1.15, 1.13, 1.10) from the Nvidia NGC cloud, and the error still persists. Our cluster runs on Enroot containers, so I import the Docker image with `enroot import`. The training starts as expected, and the predictions are always close to the references, until the Numba error occurs. 

**Additional context**

I had already discussed this issue with @titu1994 on issue #5929. He suggested using the NeMo images, but it didn't solve the problem. I would appreciate any further suggestions. Thank you for your time.

Here is the ZIP file containing everything needed to reproduce my issue:
[debug_asr.zip](https://github.com/NVIDIA/NeMo/files/11098389/debug_asr.zip)