A quantized and deployable version of the model on Nvidia Orins with TensorRT
#44
by quantshah - opened
Hi,
We got the model quantized and deployable on edge devices such as NVIDIA Orin.
https://huggingface.co/embedl/parakeet-tdt-0.6b-v3-quantized-tensorrt
NVIDIA L4 (TensorRT 10.16)
| Precision | Mean Latency | p95 | Throughput | Engine Size | Peak GPU Memory | Speedup vs FP32 |
|---|---|---|---|---|---|---|
| FP32 | 32.82 ms | 33.62 ms | 30.5 qps | 2364 MiB | 2570 MiB | 1.00× |
| FP16 | 16.83 ms | 17.10 ms | 59.4 qps | 1177 MiB | 1280 MiB | 1.95× |
| INT8 + FP16 | 14.57 ms | 14.74 ms | 68.6 qps | 758 MiB | 862 MiB | 2.25× |
NVIDIA Jetson AGX Orin (TensorRT 10.3, JetPack 6)
| Precision | Mean Latency | p95 | Throughput | Engine Size | Peak GPU Memory | Speedup vs FP32 |
|---|---|---|---|---|---|---|
| FP32 | 62.40 ms | 62.46 ms | 16.0 qps | 2331 MiB | 2526 MiB | 1.00× |
| FP16 | 35.62 ms | 35.66 ms | 28.1 qps | 1174 MiB | 1274 MiB | 1.75× |
| INT8 + FP16 | 34.32 ms | 34.35 ms | 29.1 qps | 706 MiB | 806 MiB | 1.82× |
You can try out getting this model quantized yourself with Embedl Deploy, similar to https://docs.embedl.com/embedl-deploy/latest/auto_tutorials/sam3.html. We will publish a tutorial with the full workflow soon.