[Question] How to use exported ASR model?
State: closed | Pull request: no

I've successfully run this script (https://github.com/NVIDIA/NeMo/blob/v1.0.0b3/scripts/convasr_to_single_onnx.py) to convert QuartzNet15x5 to ONNX. But how do I then use that model for inference in python? I.e., what preprocessing steps do I need to do in order to pass inputs into the ONNX model?

To make it work for me, I had to do the following workaround, but what is the right way to do this?

```
import nemo.collections.asr as asr
import onnxruntime
import tempfile
import os
import json

asr_model = asr.models.EncDecCTCModel.from_pretrained(model_name='QuartzNet15x5Base-En', strict=True)
so = onnxruntime.SessionOptions()
so.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
so.intra_op_num_threads = 1
onnx_file = 'quartznet.onnx'
session = onnxruntime.InferenceSession(onnx_file, so)

wav_path = '/path/to/file.wav'
paths2audio_files = [wav_path]
with tempfile.TemporaryDirectory() as dataloader_tmpdir:
    with open(os.path.join(dataloader_tmpdir, 'manifest.json'), 'w') as fp:
        for audio_file in paths2audio_files:
            entry = {'audio_filepath': audio_file, 'duration': 100000, 'text': 'nothing'}
            fp.write(json.dumps(entry) + '\n')
    out_batch = []
    config = {'paths2audio_files': paths2audio_files, 'batch_size': 4, 'temp_dir': dataloader_tmpdir}
    temporary_datalayer = asr_model._setup_transcribe_dataloader(config)
    for test_batch in temporary_datalayer:
        out_batch.append(test_batch)

processed_signal, processed_signal_length = asr_model.preprocessor(
    input_signal=out_batch[0][0], length=out_batch[0][1],
)
processed_signal = processed_signal.cpu().numpy()
logits = session.run([output_name], {input_name: processed_signal})[0]
```

Is there any progress on this issue? I am currently trying to run inference for QuartzNet model using ONNX.  I used EncDecCTCModel.export to export my model in ONNX format and using onnxruntime to do inference as follows:

```
sess = onnxruntime.InferenceSession("model.onnx")
pred = sess.run(None, {input_name: input_.reshape(1, 64, -1)})[0]
```

However, input here is just an arbitrary tensor created with numpy. How should we process our audio files for this?

I'm also stuck on this. I can't find any information that describes how to process audio onnxruntime. Would really appreciate some guidance.

> I'm also stuck on this. I can't find any information that describes how to process audio onnxruntime. Would really appreciate some guidance.

Since onnx model includes the pytorch model only and not preprocessing or postprocessing parts, what we need to do is follow the NeMo code and see how it processes input and output of the model. I managed to succesfully use onnx exported QuartzNet model for ASR. I used default QuartzNet configs provided by Nvidia. What I did is as follows, you may need to adjust code structure a bit.

Note that this is partly same as code provided in the original question.


```
import onnxruntime
from nemo.collections.asr.models.ctc_models import EncDecCTCModel
# wer applied after getting logits from onnx model
from nemo.collections.asr.metrics import wer
# we will use these to replicate what NeMo does
from util import softmax, get_nemo_dataset


# create onnx session with model (assuming you already exported your model)
self.sess = onnxruntime.InferenceSession(model_to_load)
self.input_name = self.sess.get_inputs()[0].name
self.label_name = self.sess.get_outputs()[0].name

self.wer = wer.WER(
    vocabulary=self.params["labels"],  # your vocabulary here (characters in your alphabet)
    batch_dim_index=0,
    use_cer=False,  # I only tested with wer and not cer
    ctc_decode=True,
    dist_sync_on_step=True,
)

# your quartznet config
config_path = "./nemo_stt_engine/configs/quartznet5x5.yaml"
# your exported onnx model path
model_to_load = "./nemo_stt_engine/models/QuartzNet5x5/epoch=99.onnx"  

yaml = YAML(typ='safe')
with open(config_path) as f:
    self.params = yaml.load(f)

# create preprocessor (NeMo does this inside EncDecCTCModel, here we do it explicitly since we are going to use onnx and not EncDecCTCModel)
preprocessor_cfg = DictConfig(self.params['model']).preprocessor
self.preprocessor = EncDecCTCModel.from_config_dict(preprocessor_cfg)

# this part is just copy pasted from NeMo library code
with tempfile.TemporaryDirectory() as dataloader_tmpdir:
    with open(os.path.join(dataloader_tmpdir, 'manifest.json'), 'w') as fp:
        for audio_file in files:
            entry = {'audio_filepath': audio_file, 'duration': 100000, 'text': 'nothing'}
            fp.write(json.dumps(entry) + '\n')
    out_batch = []
    config = {'paths2audio_files': files, 'batch_size': 1, 'temp_dir': dataloader_tmpdir}
    temporary_datalayer = get_nemo_dataset(config, self.params["labels"], 16000)
    for test_batch in temporary_datalayer:
        out_batch.append(test_batch)

# preprocess audio just like NeMo does
processed_signal, processed_signal_length = self.preprocessor(input_signal=out_batch[0][0], length=out_batch[0][1], )
processed_signal = processed_signal.cpu().numpy()

# finally make an inference using onnx model
logits = self.sess.run([self.label_name], {self.input_name: processed_signal})

# if you want to decode your logits to sentence
probabilities = logits[0][0]
a = np.array([np.argmax(x) for x in probabilities])
a = np.expand_dims(a, 0)
a = torch.from_numpy(a)
prediction = self.wer.ctc_decoder_predictions_tensor(a)
```

@ulucsahin hi, where could I find `get_nemo_dataset` function?

My mistake, as far as I remember, it is a method that I wrote by using code parts from NeMo (I may copy pasted it completely from NeMo source code, but currently I can't recall it).

Here is my implementation for get_nemo_dataset.

```
def get_nemo_dataset(config, vocab, sample_rate=16000):
    augmentor = None

    config = {
        'manifest_filepath': os.path.join(config['temp_dir'], 'manifest.json'),
        'sample_rate': sample_rate,
        'labels': vocab,
        'batch_size': min(config['batch_size'], len(config['paths2audio_files'])),
        'trim_silence': True,
        'shuffle': False,
    }

    dataset = AudioToCharDataset(
        manifest_filepath=config['manifest_filepath'],
        labels=config['labels'],
        sample_rate=config['sample_rate'],
        int_values=config.get('int_values', False),
        augmentor=augmentor,
        max_duration=config.get('max_duration', None),
        min_duration=config.get('min_duration', None),
        max_utts=config.get('max_utts', 0),
        blank_index=config.get('blank_index', -1),
        unk_index=config.get('unk_index', -1),
        normalize=config.get('normalize_transcripts', False),
        trim=config.get('trim_silence', True),
        load_audio=config.get('load_audio', True),
        parser=config.get('parser', 'en'),
        add_misc=config.get('add_misc', False),
    )

    return torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=config['batch_size'],
        collate_fn=dataset.collate_fn,
        drop_last=config.get('drop_last', False),
        shuffle=True,
        num_workers=config.get('num_workers', 0),
        pin_memory=config.get('pin_memory', False),
    )
```

@ulucsahin thanks! 

well, it looks like the order of steps is the next and it should work for all models:

1. Export onnx model and config
```
import nemo.collections.asr as nemo_asr
from omegaconf import OmegaConf

modelReco = nemo_asr.models.ASRModel.restore_from(restore_path="stt_en_citrinet_256.nemo")
modelReco.export("stt_en_citrinet_256.onnx", onnx_opset_version=12)

modelReco = nemo_asr.models.ASRModel.restore_from(restore_path="stt_en_citrinet_256.nemo", return_config = True) # restore config is set to True
textfile = open("config_stt_en_citrinet_256.yaml", "w")
textfile.write(str(OmegaConf.to_yaml(modelReco)))
textfile.close()
```
now we have onnx and yaml files

2. Load onnx and yaml together with wav file:
```
import onnxruntime
import tempfile
from nemo.collections.asr.models.ctc_models import EncDecCTCModel
from nemo.collections.asr.data.audio_to_text import AudioToCharDataset
import os
import torch
import yaml
from omegaconf import DictConfig
import json
import numpy as np

# your quartznet config
config_path = "config_stt_en_citrinet_256.yaml"
# your exported onnx model path
model_to_load = "stt_en_citrinet_256.onnx"

with open(config_path) as f:
    params = yaml.safe_load(f)

# create onnx session with model (assuming you already exported your model)
sess = onnxruntime.InferenceSession(model_to_load)
input_name = sess.get_inputs()[0].name
label_name = sess.get_outputs()[0].name

# create preprocessor (NeMo does this inside EncDecCTCModel, here we do it explicitly since we are going to use onnx and not EncDecCTCModel)
preprocessor_cfg = DictConfig(params).preprocessor
preprocessor = EncDecCTCModel.from_config_dict(preprocessor_cfg)

audio_file = "test.wav"
labels = params['decoder']['vocabulary'] #"vocabulary" could be something else depending on model

# this part is just copy pasted from NeMo library code
with tempfile.TemporaryDirectory() as dataloader_tmpdir:
    with open(os.path.join(dataloader_tmpdir, 'manifest.json'), 'w') as fp:
        entry = {'audio_filepath': audio_file, 'duration': 100000, 'text': 'nothing'}
        fp.write(json.dumps(entry) + '\n')
    out_batch = []
    config = {'paths2audio_files': [audio_file], 'batch_size': 1, 'temp_dir': dataloader_tmpdir}
    temporary_datalayer = get_nemo_dataset(config, labels, 16000)
    for test_batch in temporary_datalayer:
        out_batch.append(test_batch)

# preprocess audio just like NeMo does
processed_signal, processed_signal_length = preprocessor(input_signal=out_batch[0][0], length=out_batch[0][1], )
processed_signal = processed_signal.cpu().numpy()

# finally make an inference using onnx model
logits = sess.run([label_name], {input_name: processed_signal})
probabilities = logits[0][0]

probs = softmax(probabilities)

#silence
labels.append("blank")

letters = get_letters(probs)
```

with several helper functions to get letters from logits:
```
def softmax(logits):
    e = np.exp(logits - np.max(logits))
    return e / e.sum(axis=-1).reshape([logits.shape[0], 1])

def get_nemo_dataset(config, vocab, sample_rate=16000):
    augmentor = None

    config = {
        'manifest_filepath': os.path.join(config['temp_dir'], 'manifest.json'),
        'sample_rate': sample_rate,
        'labels': vocab,
        'batch_size': min(config['batch_size'], len(config['paths2audio_files'])),
        'trim_silence': True,
        'shuffle': False,
    }

    dataset = AudioToCharDataset(
        manifest_filepath=config['manifest_filepath'],
        labels=config['labels'],
        sample_rate=config['sample_rate'],
        int_values=config.get('int_values', False),
        augmentor=augmentor,
        max_duration=config.get('max_duration', None),
        min_duration=config.get('min_duration', None),
        max_utts=config.get('max_utts', 0),
        blank_index=config.get('blank_index', -1),
        unk_index=config.get('unk_index', -1),
        normalize=config.get('normalize_transcripts', False),
        trim=config.get('trim_silence', True),
        load_audio=config.get('load_audio', True),
        parser=config.get('parser', 'en'),
        add_misc=config.get('add_misc', False),
    )

    return torch.utils.data.DataLoader(
        dataset=dataset,
        batch_size=config['batch_size'],
        collate_fn=dataset.collate_fn,
        drop_last=config.get('drop_last', False),
        shuffle=True,
        num_workers=config.get('num_workers', 0),
        pin_memory=config.get('pin_memory', False),
    )

def get_letters(probs):
    letters = []
    for idx in range(0, probs.shape[0]):
        current_char_idx = np.argmax(probs[idx])
        if labels[current_char_idx] != "blank":
            letters.append([labels[current_char_idx], idx])
    return letters
```

The main reason of using `.onnx`-model is the possibility to run the nemo-model for transcribing without `nemo`-classes.
Otherwise, why do I need to export the model to ONNX?

Is there a way to do the preprocessing and postprocessing without classes from `nemo`-library?


Can you provider onnx model dowload links? thank you!