Exporting Conformer CTC to TensorRT
State: closed | Pull request: no
Labels: name: stale

Hi,
I'm trying to export conformer CTC to tensorRT by using this script: `examples/asr/quantization/speech_to_text_quant_infer_trt.py`

OS: Linux 20.04
PyTorch 1.13.1
CUDA: 11.7
tensorrt: 8.6.1

I modified the min and max input shapes to fit the model num_features and added another input (because otherwise it fails):
```
profile.set_shape("length", min=(min_input_shape[0],), opt=(max_input_shape[0],), max=(max_input_shape[0],))
```

But now I get the error below. Any idea what is wrong?

```
[03/20/2024-08:46:08] [TRT] [W] /layers.0/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.1/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.2/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.3/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.4/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.5/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.6/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.7/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.8/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.9/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.10/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.11/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.12/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.13/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.14/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.15/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.16/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:08] [TRT] [W] /layers.17/self_attn/Reshape_7: IShuffleLayer with zeroIsPlaceHolder=true has reshape dimension at position 3 that might or might not be zero. TensorRT resolves it at runtime, but this may cause excessive memory consumption and is usually a sign of a bug in the network.
[03/20/2024-08:46:48] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[/pos_enc/Constant_output_0 + (Unnamed Layer* 57) [Shuffle].../layers.0/self_attn/MatMul_1]}.
[03/20/2024-08:46:48] [TRT] [E] 10: [optimizer.cpp::computeCosts::3869] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/pos_enc/Constant_output_0 + (Unnamed Layer* 57) [Shuffle].../layers.0/self_attn/MatMul_1]}.)
```

@Slyne (myself)  to check this issue.

@nirraviv89 Meanwhile could you help try tensorrt 10.0:
https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.0.0/tensorrt-10.0.0.6.linux.x86_64-gnu.cuda-12.4.tar.gz

The installation guide:
https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar


Thanks @Slyne this is the error I get with tensorrt 10.0:
```
[04/08/2024-07:16:08] [TRT] [W] Tactic Device request: 24316MB Available: 22723MB. Device memory is insufficient to use tactic.
[04/08/2024-07:16:08] [TRT] [W] UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested size of 24316 detected for tactic 0x0000000000000000.
[04/08/2024-07:16:08] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.
[04/08/2024-07:16:08] [TRT] [E] 10: [optimizer.cpp::computeCosts::4103] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.)
```

> Thanks @Slyne this is the error I get with tensorrt 10.0:
> 
> ```
> [04/08/2024-07:16:08] [TRT] [W] Tactic Device request: 24316MB Available: 22723MB. Device memory is insufficient to use tactic.
> [04/08/2024-07:16:08] [TRT] [W] UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested size of 24316 detected for tactic 0x0000000000000000.
> [04/08/2024-07:16:08] [TRT] [E] 10: Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.
> [04/08/2024-07:16:08] [TRT] [E] 10: [optimizer.cpp::computeCosts::4103] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[ONNXTRT_squeezeTensor_643 + /layers.0/conv/Transpose_1.../layers.1/self_attn/MatMul_1]}.)
> ```

If you only want to export conformer to Tensorrt, you may check this file to export nemo conformer to onnx: 
```
https://github.com/Slyne/NeMo/blob/dgalvez/triton-inference-server/examples/asr/triton-inference-server/scripts/export_asr_ctc_onnx.py
```
Convert onnx model to tensorrt model by using trtexec:
```
trtexec --onnx=onnx_model/model.onnx \
        --minShapes=audio_signal:1x80x100,length:1 \
        --optShapes=audio_signal:16x80x1000,length:16 \
        --maxShapes=audio_signal:32x80x6000,length:32 \
        --fp16 \
        --saveEngine=encoder.trt
```
For this quantization script issue,  I haven't get a chance to start it.  I think I can check this issue this week. (Based on my previous experience, I didn't see perf gain by using trt + quantization.)

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.