libNVVM cannot be found when trying to train ASR transducer using GPU
State: closed | Pull request: no
Labels: name: bug

I am trying to train the stt_en_conformer_transducer_medium on another language. I am running the train script inside a docker environment that sees the gpus and I was able to run simple pytorch lightning script for training in the same docker using gpu(on the mnist dataset, for testing). However when I run the following script:

```python
mport pytorch_lightning as pl
from omegaconf import OmegaConf

from nemo.collections.asr.models import EncDecRNNTBPEModel
from nemo.core.config import hydra_runner
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager


@hydra_runner(config_path="config/", config_name="conformer_transducer_bpe_medium.yaml")
def main(cfg):
    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')

    trainer = pl.Trainer(**cfg.trainer)
    exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecRNNTBPEModel(cfg=cfg.model, trainer=trainer)
    print(asr_model)
    # Initialize the weights of the model from another model, if provided via config
    asr_model.maybe_init_from_pretrained_checkpoint(cfg)
    print(asr_model)
    trainer.fit(asr_model)

    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
        if asr_model.prepare_test(trainer):
            trainer.test(asr_model)


if __name__ == '__main__':
    main()  # noqa pylint: disable=no-value-for-parameter

```

I get the following error when the first epoch is being started:
```
numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend)
Internal error at <numba.core.typeinfer.CallConstraint object at 0x7f7e7aadeef0>.
libNVVM cannot be found. Do `conda install cudatoolkit`:
libnvvm.so: cannot open shared object file: No such file or directory
During: resolving callee type: type(CUDADispatcher(<function exponential at 0x7f7dfacbf1c0>))
During: typing of call at /mnt/zfs-mirror-02/homes/gpirlogeanu/git_repos/nemoasr/nemo/collections/asr/parts/numba/rnnt_loss/utils/cuda_utils/reduce.py (158)

Enable logging at debug level for details.

File "../../../nemo/collections/asr/parts/numba/rnnt_loss/utils/cuda_utils/reduce.py", line 158:
def _reduce_rows(I_opid: int, R_opid: int, acts, output, num_rows: int):
    <source elided>
        if I_opid == 0:
            curr = rnnt_helper.exponential(curr)
            ^
```

My gpus are seen inside the docker and by the trainer:
```
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
Added key: store_based_barrier_key:1 to store for rank: 1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
[NeMo I 2023-06-26 12:32:50 modelPT:721] Optimizer config = AdamW (
    Parameter Group 0
        amsgrad: False
        betas: [0.9, 0.98]
        capturable: False
        differentiable: False
        eps: 1e-08
        foreach: None
        fused: None
        lr: 5.0
        maximize: False
        weight_decay: 0.001
    )
[NeMo I 2023-06-26 12:32:50 lr_scheduler:910] Scheduler "<nemo.core.optim.lr_scheduler.NoamAnnealing object at 0x7f741c2202e0>" 
    will be used during training (effective maximum steps = 2510000) - 
    Parameters : 
    (d_model: 256
    warmup_steps: 10000
    warmup_ratio: null
    min_lr: 1.0e-06
    max_steps: 2510000
    )

  | Name              | Type                              | Params
------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0     
1 | encoder           | ConformerEncoder                  | 27.3 M
2 | decoder           | RNNTDecoder                       | 3.9 M 
3 | joint             | RNNTJoint                         | 1.2 M 
4 | loss              | RNNTLoss                          | 0     
5 | spec_augmentation | SpectrogramAugmentation           | 0     
6 | wer               | RNNTBPEWER                        | 0     
------------------------------------------------------------------------
32.5 M    Trainable params
0         Non-trainable params
32.5 M    Total params
64.994    Total estimated model params size (MB)
Epoch 0:   0%|                                                              | 0/6136 [00:00<?, ?it/s]Error executing job with overrides: []
Error executing job with overrides: []
```

I would greatly appreciate any help, as I have tried several scenarios, such as exporting the path to nvvm and libnvvm to NUMBAPRO_NVVM and NUMBAPRO_LIBDEVICE.

Thanks in advance!!