logeshsenthil13/MobileNet_V3_Small

MobileNetV3-Small — PyTorch → ONNX → TensorRT benchmark

PyTorch baseline → ONNX (FP32/FP16) → TensorRT engines → INT8/INT4 PTQ with NVIDIA ModelOpt, with accuracy and numerical parity checked against the PyTorch FP32 reference at every hop.

source env.sh          # ALWAYS do this first (see "Environment gotchas")
bash run_all.sh        # ~20 min end to end

---

Read this first — three things that are not what you'd expect

1. This machine is not a Jetson Orin Nano. It is x86_64 with an NVIDIA RTX 6000 Ada Generation (48 GB, SM 8.9), CUDA 13.0, driver 580. The pipeline is identical either way, but every latency number below is an Ada desktop GPU number — do not quote them as Orin numbers. On Orin (SM 8.7, far less memory bandwidth) the ranking of precisions will also differ, because the INT8 result here is dominated by reformat overhead that a bandwidth-starved part would weigh differently.

2. TensorRT 11 deleted --fp16, --int8, --best and --calib from trtexec. Strongly-typed networks are now the only mode, and passing --fp16 is an error rather than a no-op. Engine precision comes entirely from the ONNX file:

Engine	Comes from	Produced by
FP32	FP32 ONNX	2_export_onnx.py
FP16	FP16 ONNX (fp16 weights, fp32 IO)	2_export_onnx.py
INT8	Q/DQ ONNX	5_quantize.sh (ModelOpt)
INT4	blocked Q/DQ ONNX	5b_quantize_int4.py (ModelOpt)

There is no implicit-calibration path left in trtexec. Explicit Q/DQ via ModelOpt is the only way to get an INT8 engine.

3. The default ModelOpt INT8 recipe produces a 0.1% top-1 model. Not a typo, and not a calibration-data bug. See "Why INT8 collapses" below — it took a working recipe to get from 0.10% to 66.64%.

---

Results

5000 held-out ImageNet val images (5/class). Calibration used a disjoint 1000 images (1/class). Latency is batch 1.

Backend	Prec	Top-1 %	Top-5 %	ΔTop-1	Mean ms	FPS	Speedup	Size MB	CosSim	Top-1 agree %
pytorch	fp32	67.880	87.840	—	1.3416	745	1.00×	—	—	—
onnxruntime	fp32	67.860	87.860	−0.020	0.5120	1953	2.62×	9.71	1.000000	99.78
onnxruntime	fp16	68.000	87.900	+0.120	0.5261	1901	2.55×	4.88	0.999970	99.30
onnxruntime	int8	65.360	85.640	−2.520	1.1514	869	1.17×	5.05	0.977230	84.10
onnxruntime	int8_hp32	66.640	86.880	−1.240	0.8102	1234	1.66×	9.82	0.983000	87.26
onnxruntime	int4	67.320	87.500	−0.560	0.5431	1841	2.47×	4.43	0.997760	94.14
tensorrt	fp32	67.900	87.880	+0.020	0.2909	3437	4.61×	10.64	1.000000	99.80
tensorrt	fp16	68.080	87.900	+0.200	0.2388	4188	5.62×	5.88	0.999970	99.24
tensorrt	int8	66.060	86.820	−1.820	0.3353	2983	4.00×	3.99	0.981850	87.00
tensorrt	int8_hp32	66.520	86.900	−1.360	0.3140	3185	4.27×	3.92	0.982980	87.68
tensorrt	int4	67.340	87.480	−0.540	0.3012	3320	4.45×	5.42	0.997750	94.16

ORT latency includes H2D/D2H copies; TRT latency is GPU compute only (matches trtexec's "GPU Compute Time"). CosSim / agreement are against results/ref_logits.npy, the PyTorch FP32 logits.

Sanity check on the harness: PyTorch scores 67.880% on this 5000-image subset against torchvision's published 67.668% on the full 50k — the preprocessing and the WNID→class-index mapping are right.

What the numbers say

FP16 wins outright — fastest and most accurate (+0.20 pp, which is noise on 5000 images). On this model and this GPU it is the only precision worth shipping.

INT8 is slower than FP16 and slower than FP32. That is not a bug. The engine inspector explains it:

Engine	Reformat layers	Output dtypes
fp32	1	Float ×131
fp16	1	Half ×121
int8	39	Half ×99, Int8 ×48
int8_hp32	30	Float ×76, Int8 ×47
int4	1	Float ×134

MobileNetV3-Small is ~60 MFLOPs — at batch 1 it is latency-bound by kernel launch and memory traffic, not by arithmetic. The working INT8 recipe deliberately keeps the 11 depthwise convs and the SE blocks in float, so the engine ping-pongs between INT8 and FP16 regions 39 times, and the conversion costs more than the INT8 math saves. INT8 still wins on size (3.99 MB vs 5.88 MB) — take it if you are memory-constrained, not if you want speed.

INT4 gives no speedup, by design. ModelOpt's ONNX INT4 path quantizes Gemm/MatMul only — it is an LLM weight-only path. In MobileNetV3-Small that is exactly the two classifier layers: 1.6M of 2.5M parameters but under 3% of the FLOPs. Note the int4 engine has all-Float outputs and 1 reformat layer — TRT dequantizes the INT4 weights to float and does the matmul in float. You get a smaller ONNX (4.43 MB) and −0.54 pp; you do not get faster inference. The engine is 5.42 MB, larger than INT8's, because the backbone stays FP16 while INT8 shrinks all 52 convs.

---

Why INT8 collapses (and the recipe that fixes it)

Measured, not guessed. Same calibration data, same 5000 eval images:

Recipe	Top-1
plain --quantize_mode int8 (entropy)	0.10 %
--calibration_method max	0.24 %
--high_precision_dtype fp32	1.06 %
activation scales replaced with p99.9 percentile clipping	6.96 %
exclude the stem conv	0.96 %
exclude depthwise convs only	1.70 %
--op_types_to_quantize Conv Gemm only	0.92 %
both: Conv Gemm + exclude 11 depthwise convs	66.64 %

Ruled out first, so the diagnosis is not folklore:

• Calibration data is correct — min −2.1179 / max 2.6400 are exactly a black and a white pixel after ImageNet normalization, and all 1000 classes are covered.
• Weights are quantized correctly — all 52 convs are per-channel with scale = max|w|/127 exactly, median relative weight error 0.0039.
• The graph is structurally correct — weights, biases and Conv attributes all match the FP32 graph node-for-node; Q/DQ chains are well-formed; zero-points are int8 zeros (symmetric).
• It is not a ModelOpt bug. A pure PyTorch fake-quant simulation with no ONNX involved reproduces it: per-tensor INT8 activations + per-channel INT8 weights give 33% top-1 at best, and skipping the depthwise conv inputs moves it to 72%.

Two independent things break the model, and you must fix both:

(a) Depthwise conv inputs. Every channel of a depthwise conv is its own filter, and the incoming activation ranges differ by orders of magnitude across channels. TensorRT supports exactly one per-tensor activation scale, so the small channels round to zero. Layer-wise divergence isolates the first failure to node #2 — the first depthwise conv — where cosine similarity against FP32 drops to 0.656 while the two layers before it are still at 0.997.

(b) The SE blocks and residual adds. --op_types_to_quantize Conv Gemm is what stops ModelOpt quantizing GlobalAveragePool / HardSigmoid / Mul / Add. The SE Mul outputs are the worst tensors in the network: max/p99.9 ratios up to 13×, so a per-tensor scale leaves the real data in ~13 of 127 levels.

For context, the stem is the textbook example of the problem — p99 of |activation| is 7.6 but the max is 105.5, so entropy calibration picked scale 0.683 (range ±86.8) and the bulk of the data ended up using about 7 of 127 levels.

--high_precision_dtype fp16 costs a further ~1.3 pp on top (66.64 → 65.36), because the Q/DQ scales are then stored as fp16 and the smallest weight scales (down to 5e-11) underflow — ModelOpt warns "values will be replaced with 6.0e-08". Both variants are built so you can pick; int8_hp32 is the more accurate one and, on this GPU, also the faster one.

The 11 depthwise convs are detected from the graph (group > 1), never hand-listed — see qdq_tools.py list-depthwise.

---

Environment gotchas

source env.sh fixes all three. Do it in the current shell — source env.sh | tail runs in a subshell and the exports are lost.

1. ONNX Runtime silently falls back to CPU. onnxruntime-gpu here is built against CUDA 12 while the system is CUDA 13, so it fails with libcublasLt.so.12: cannot open shared object file and quietly drops to CPUExecutionProvider (~4× slower). The CUDA-12 libs are installed as nvidia-*-cu12 wheels, just not on the loader path. Verified safe: torch (cu13) and ORT (cu12) coexist in one process.

2. trtexec and the Python runtime are different TensorRT versions. /usr/bin/trtexec links the system TensorRT 11.0.0; the venv's Python tensorrt is 11.1.0.106. A plan built by 11.0.0 will not deserialize:

The engine plan file is not compatible with this version of TensorRT,
expecting library version 11.1.0.106 got .

Putting the venv's tensorrt_libs first on LD_LIBRARY_PATH makes the same trtexec binary link 11.1.0. It still prints "TensorRT version: 11.0.0" — that string is baked into the binary; what matters is the libnvinfer it loaded. 6_build_engines.sh sets this itself and verifies every plan with --getPlanVersionOnly, so a mismatch is caught at build time rather than in step 7.

3. INT4 needs a block size TensorRT can use. ModelOpt defaults to 128 and the CLI has no --block_size. classifier.0 weight is (1024, 576) and 576/128 = 4.5, so TRT refuses to parse:

Assertion failed: inputSize % scaleSize == 0: Inferred block size is not an
integer. Input volume = 589824, scale volume = 5120.

5b_quantize_int4.py calls the same ModelOpt quantize() with block_size=64 (576/64 = 9, 1024/64 = 16 — both exact).

4. ModelOpt 0.45.0 writes the wrong axis for blocked INT4. It emits axis=0 while laying the scale out for axis=1:

weight (1024, 576)  block_size 64  scale (1024, 9)   <- 9 = ceil(576/64), i.e. axis 1

Both ONNX Runtime and TensorRT reject this. qdq_tools.py fix-int4-axis patches it; 5b_quantize_int4.py calls it automatically.

Also: ORT's CUDA EP cannot run blocked INT4 DequantizeLinear at all even when the axis is right, so INT4 ONNX eval runs on CPU EP.

---

Files

env.sh	source this first — venv, loader paths, TRTEXEC
common.py	dataset, preprocessing cache, metrics, latency, parity
0_prepare_data.py	disjoint eval/calib splits + uint8 preprocessing cache
1_pytorch_baseline.py	PyTorch FP32 accuracy, latency, reference logits
2_export_onnx.py	FP32 + FP16 ONNX export, with a torch-vs-ONNX check
3_onnx_eval.py	ONNX accuracy + latency + parity (any precision)
4_make_calib.py	calibration .npz, keyed to the real ONNX input name
5_quantize.sh	ModelOpt INT8 CLI (both high_precision_dtype variants)
5b_quantize_int4.py	INT4 with a TRT-compatible block size
qdq_tools.py	depthwise-conv detection; INT4 blocked-axis fix
6_build_engines.sh	trtexec engine builds + version verification
7_trt_eval.py	TRT accuracy + latency + parity + layer precisions
8_report.py	collects results/*.json into the table above
run_all.sh	the whole pipeline

Useful variations

# Larger eval set (49/class = 49000 images; calib stays disjoint)
python 0_prepare_data.py --eval-per-class 49
# ...then re-run steps 1,3,7 to refresh accuracy at the new sample size

# Reproduce the INT8 collapse
bash 5_quantize.sh baseline
python 3_onnx_eval.py --onnx artifacts/mnv3s_int8_naive.onnx --tag int8_naive

# Reproduce the INT4 TRT parse failure
bash 5_quantize.sh int4_cli_broken
bash 6_build_engines.sh int4_cli

# Force CPU calibration if the CUDA EP misbehaves
EPS=cpu bash 5_quantize.sh int8

# Throughput instead of latency
python 7_trt_eval.py --engine engines/mnv3s_fp16.plan --tag fp16 --latency-batch 64

Reproducing on an actual Jetson Orin Nano

The scripts are portable; three things change.

• env.sh — the loader-path fixes are specific to this venv. On JetPack, ORT and TensorRT come from the system, so most of it can go; keep VAL_ROOT/TRTEXEC.
• 6_build_engines.sh — drop --memPoolSize=workspace:4096 (Orin Nano has 8 GB shared) and expect much longer build times. Add --useDLACore only if you also add --allowGPUFallback; MobileNetV3's HardSwish/HardSigmoid are not DLA ops.
• Expect the precision ranking to change. The 39 reformat layers that make INT8 slower than FP16 here are a fixed per-layer cost; on a much slower GPU the INT8 math saves proportionally more, so INT8 may come out ahead. Re-measure — do not carry these numbers over.