Only silence is detected after converting vad_multilingual_marblenet.nemo to ONNX format.
State: closed | Pull request: no
Labels: name: bug

name: stale

Hello,

I tried to convert the vad_multilingual_marblenet.nemo model to ONNX format and perform the vad inference stage in the clustering_diar_infer.py script from the diarization example.

The following is the code I implemented:

```
        self._vad_model = EncDecClassificationModel.restore_from(model_path)
        self._vad_model.eval()
        self._vad_model = self._vad_model.to(self._device)

        self._vad_model.export('vad_onnx.onnx')
        ort_session = onnxruntime.InferenceSession('vad_onnx.onnx')        

        time_unit = int(self._vad_window_length_in_sec / self._vad_shift_length_in_sec)
        trunc = int(time_unit / 2)
        trunc_l = time_unit - trunc
        all_len = 0
        data = []

        for line in open(manifest_file, 'r', encoding='utf-8'):
            file = json.loads(line)['audio_filepath']
            data.append(get_uniqname_from_filepath(file))
        status = get_vad_stream_status(data)

        for i, test_batch in enumerate(tqdm(self._vad_model.test_dataloader(), desc='vad', leave=True)):
            test_batch = [x.to(self._device) for x in test_batch]
            with autocast():
                processed_signal, processed_signal_len = self._vad_model.preprocessor(input_signal=test_batch[0].to(self._vad_model.device),length=test_batch[1].to(self._vad_model.device))
                ort_inputs = {ort_session.get_inputs()[0].name: to_numpy(processed_signal)}			
                log_probs = torch.Tensor(ort_session.run([],ort_inputs))
                print(log_probs)
                probs = torch.softmax(log_probs, dim=-1)
```
The output of log_probs when running the original _run_vad() function from /nemo/collections/asr/models/clustering_diarizer.py#190 is the following:

```
tensor([[ 0.7704, -0.7704],
        [ 0.7360, -0.7361],
        [ 0.7352, -0.7353],
        ...,
        [ 1.3244, -1.3244],
        [ 1.3309, -1.3310],
        [ 1.2513, -1.2513]], grad_fn=<AddmmBackward0>)
vad:  20%|██████████▌                                          | 1/5 [00:03<00:12,  3.15s/it]tensor([[ 0.8877, -0.8878],
        [ 0.8913, -0.8913],
        [ 0.9222, -0.9222],
        ...,
        [ 0.5388, -0.5389],
        [ 0.5645, -0.5645],
        [ 0.4665, -0.4665]], grad_fn=<AddmmBackward0>)
vad:  40%|█████████████████████▏                               | 2/5 [00:06<00:09,  3.04s/it]tensor([[ 0.0150, -0.0150],
        [ 0.0219, -0.0220],
        [ 0.0102, -0.0102],
        ...,
        [-0.0134,  0.0133],
        [ 0.0912, -0.0912],
        [ 0.0313, -0.0313]], grad_fn=<AddmmBackward0>)
vad:  60%|███████████████████████████████▊                     | 3/5 [00:09<00:06,  3.04s/it]tensor([[-2.9656,  2.9656],
        [-3.0097,  3.0097],
        [-3.0016,  3.0015],
        ...,
        [ 0.1544, -0.1544],
        [ 0.2534, -0.2535],
        [ 0.2242, -0.2242]], grad_fn=<AddmmBackward0>)

```
When running the modfied function with onnx runtime, the output is:

```
tensor([[[ 0.7704, -0.7704],
         [ 0.7360, -0.7361],
         [ 0.7352, -0.7352],
         ...,
         [ 1.3244, -1.3244],
         [ 1.3309, -1.3310],
         [ 1.2513, -1.2513]]])
vad:  20%|██████████▌                                          | 1/5 [00:03<00:13,  3.31s/it]tensor([[[ 0.8877, -0.8878],
         [ 0.8913, -0.8913],
         [ 0.9222, -0.9222],
         ...,
         [ 0.5388, -0.5389],
         [ 0.5644, -0.5645],
         [ 0.4665, -0.4665]]])
vad:  40%|█████████████████████▏                               | 2/5 [00:06<00:08,  3.00s/it]tensor([[[ 0.0150, -0.0150],
         [ 0.0219, -0.0220],
         [ 0.0102, -0.0102],
         ...,
         [-0.0134,  0.0134],
         [ 0.0912, -0.0912],
         [ 0.0313, -0.0313]]])
vad:  60%|███████████████████████████████▊                     | 3/5 [00:09<00:05,  2.99s/it]tensor([[[-2.9656,  2.9656],
         [-3.0097,  3.0097],
         [-3.0016,  3.0015],
         ...,
         [ 0.1544, -0.1544],
         [ 0.2534, -0.2535],
         [ 0.2242, -0.2242]]])
```

The tensors look almost the same, except that when running my onnx implementation, there is no grad_fn=<AddmmBackward0>. Also, when running ONNX runtime, the script returns an error that says "There is silence detected in the entire audio file".

Could you please explain why there is only silence detected, despite the log_probs having the same output tensors in both cases, but the grad_fn missing? What are the correct steps for running inference using onnx?