nxp/midas-v2-imx

MiDaS v2.1 Small

Introduction

MiDaS [1] is a family of deep learning models for monocular depth estimation. Multiple versions have been created over the years, ranging from CNN-based to Transformer-based architectures. This repository uses the v2.1 Small variant, based on EfficientNet-Lite [2], which is the lightest and fastest member of the family and well suited for edge deployment.

Key Features of MiDaS v2.1 Small

• Monocular depth estimation: Predicts a relative inverse-depth (disparity) map from a single RGB image.
• Zero-shot cross-dataset transfer: Trained on 10 diverse datasets, giving strong generalization across scenes.
• Lightweight backbone: EfficientNet-Lite provides a good accuracy / latency trade-off on embedded NPUs.

Model Description

This repository contains pre-compiled model files optimized for NXP i.MX processors.

• Base Model: intel/lite-model/midas/v2_1_small/1
• Original Model Authors: Intel ISL
• Original License: MIT
• Modified by: NXP

Modifications

This model is a derivative work with the following changes from the original:

• Quantization: Post-training integer quantization (int8) using commercially useable representative ImageNetV2 images
• Compilation: Compiled for i.MX 93 Ethos-U65 NPU using NXP Vela tool (lf-6.18.20_2.0.0)
• Compilation: Compiled for i.MX 95 / i.MX 952 Neutron NPU using eIQ Neutron SDK 3.1.3

Model Information

Information	Value
Input shape	RGB image (256, 256, 3), float32 normalized to [0, 1]
Input example	(Public domain image)
Output shape	Depth map tensor (1, 256, 256, 1) — relative inverse depth
Output example
FLOPS	9.214 GOPs
File size (int8)	~17 MB
Source framework	PyTorch / TensorFlow Lite

Version and changelog

Initial release of quantized int8 model for i.MX 8M Plus, i.MX 93, i.MX 95, and i.MX 952.

Tested configurations

The int8 model has been tested on i.MX 8M Plus, i.MX 93, i.MX 95 and i.MX 952 using benchmark-model (see i.MX Machine Learning User Guide).

Training and evaluation

The model was trained on 10 diverse datasets including ReDWeb, DIML, Movies, MegaDepth, WADA, ApolloScape, IRS, HRWSI, TartanAir, and BlendedMVS. The training procedure is described in the original repository and associated papers [1, 3].

Conversion/Quantization

The original float32 TFLite model is downloaded from TFHub. It is then post-training quantized to int8 using the tflite2tensorflow tool with representative calibration images sampled from a commercially-usable ImageNetV2 subset. The quantization procedure can be reproduced by running recipe.sh.

Download and run

To reproduce the quantized and compiled model files from scratch, run:

bash recipe.sh

This will produce:

• original_model/midas_v2_1_small_quant.tflite — int8 model for i.MX 8M Plus (CPU/NPU)
• imx93/lf-6.18.20_2.0.0/midas_v2_1_small_quant_vela.tflite — Vela-compiled for i.MX 93 Ethos-U65
• imx95/3.1.3/midas_v2_1_small_quant_converted.tflite — Neutron-compiled for i.MX 95
• imx952/3.1.3/midas_v2_1_small_quant_converted.tflite — Neutron-compiled for i.MX 952

An example of how to run inference is provided in example.py:

pip install opencv-python numpy tensorflow matplotlib
python example.py --model original_model/midas_v2_1_small_quant.tflite --input example_input.jpg

Please refer to the i.MX Machine Learning User Guide for details on running models on each target platform.

Full on-device examples leveraging GStreamer/NNStreamer and this model can be found in nxp-nnstreamer-examples on GitHub

Origin

Model implementation: https://github.com/isl-org/MiDaS

Model weights downloaded from TFHub.

[1] Ranftl, Rene, et al. "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.3 (2020): 1623-1637.

[2] https://blog.tensorflow.org/2020/03/higher-accuracy-on-vision-models-with-efficientnet-lite.html

[3] Ranftl, Rene, Alexey Bochkovskiy, and Vladlen Koltun. "Vision transformers for dense prediction." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.