Error with ASR model exported to ONNX
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

After exporting an asr model to onnx "stt_es_citrinet_1024_gamma_0_25", I try to make an inference with onnxruntime and I get the following error:

2023-03-23 01:39:32.514839309 [E:onnxruntime:Default, cuda_call.cc:119 CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=w1 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data);
2023-03-23 01:39:32.514864313 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'/encoder/encoder/encoder.1/res.0.0/conv/Conv' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=w1 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data);

**Steps/Code to reproduce bug**

Export model: 

```
import nemo.collections.asr as nemo_asr

model = nemo_asr.models.EncDecCTCModelBPE.from_pretrained(model_name="stt_es_citrinet_1024_gamma_0_25", map_location='cuda')
model.export('/models/model.onnx', onnx_opset_version=18)
```

For legal reasons I cannot show the complete code, but I will try to put as much as I can:
```

import numpy as np
import soundfile as sf
import  onnxruntime as ort

def get_samples(audio_file, target_sr=16000):
    with sf.SoundFile(audio_file, 'r') as f:
        dtype = 'int16'
        sample_rate = f.samplerate
        samples = f.read(dtype=dtype)
        if sample_rate != target_sr:
            samples = librosa.core.resample(samples, sample_rate, target_sr)
        samples = samples.astype('float32')/32768
        samples = samples.transpose()
        return samples

class AudioChunkIterator():
    def __init__(self, samples, frame_len, sample_rate, chunk_len_in_secs):
        self._samples = samples
        self._chunk_len = chunk_len_in_secs*sample_rate
        self._start = 0
        self.output = True

    def __iter__(self):
        return self

    def __next__(self):
        if not self.output:
            raise StopIteration
        last = int(self._start + self._chunk_len)
        if last <= len(self._samples):
            chunk = self._samples[self._start: last]
            self._start = last
        else:
            chunk = np.zeros([int(self._chunk_len)], dtype='float32')
            samp_len = len(self._samples) - self._start
            chunk[0:samp_len] = self._samples[self._start:len(self._samples)]
            self.output = False         
        return chunk


ort_session = ort.InferenceSession('/models/model.onnx', providers=["CUDAExecutionProvider"])

**You can put the inference code by chunks for ctc models**

def _get_batch_preds(self, audio_signal, audio_signal_len, merge=True):
        processed_signal, processed_signal_len = self.preprocessor(
            input_signal=audio_signal, length=audio_signal_len
        )
        binding = ort_session.io_binding()

        processed_signal_tensor = processed_signal.contiguous()
        binding.bind_input(
            name=ort_session.get_inputs()[0].name, # audio_signal
            device_type='cuda',
            device_id=0,
            element_type=np.float32,
            shape=tuple(processed_signal_tensor.shape),
            buffer_ptr=processed_signal_tensor.data_ptr(),
        )
        
        
        # print(processed_signal_len, 'processed_signal_len')
        
        processed_signal_len_tensor = processed_signal_len.contiguous()
        binding.bind_input(
            name=ort_session.get_inputs()[1].name, #length
            device_type='cuda',
            device_id=0,
            element_type=np.int64,
            shape=tuple(processed_signal_len_tensor.shape),
            buffer_ptr=processed_signal_len_tensor.data_ptr(),
        )
                
        
        binding.bind_output('logprobs', 'cuda')

        ort_session.run_with_iobinding(binding)
        logprobs = binding.copy_outputs_to_cpu()
        print(logprobs)

```





**Expected behavior**

The expected behavior is to print the probabilities

**Environment overview (please complete the following information)**

 - Environment location: [Local]
 - Method of NeMo install: [pip install]. Please specify exact commands you used to install. : pip3 install nemo-toolkit[all]==1.16.0


**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04
- pytorch version: 2.0.0
- Python version: 3.8

**Additional context**

GPU model: Tesla T4
onnxruntime version: onnxruntime-gpu==1.14.1
onnx version: 1.13.1
Cuda version: I tried with different versions of cuda included
11.4, 11.6, 11.8, 12.0 
cudNN version: I tried with different versions of cudNN included
8.2.4, 8.4.0, 8.8.0