rony000013/intel-physical-ai-smolvla

Intel Physical AI: Autonomous Dining Table Preparation (SmolVLA)

Official fine-tuned policy for the Intel Physical AI Online Challenge: Setting Up a Dinner Table.

This repository contains the full fine-tuned SmolVLA (Vision-Language-Action) model and all edge-optimized deployment variations, trained on dual 6-DoF SO-101 robotic arms in MuJoCo physics.

---

📦 Model Variations in this Repository

Variation	Path / Format	Size	Precision	Target Hardware
SmolVLA 40k PyTorch (Base)	Root (model.safetensors)	1.19 GB	FP16 / FP32	Intel CPU / Host / CUDA
OpenVINO NNCF INT8 Quantized	variations/openvino_int8/	25.8 MB	INT8	Intel Core Ultra CPU / iGPU / NPU
OpenVINO FP16/FP32 IR Model	variations/openvino_fp16/	101.4 MB	FP32 / FP16	Intel OpenVINO Runtimes
ONNX Vision Model	variations/onnx/	~346 MB	FP32	Universal ONNX Runtimes

---

⚡ Intel Hardware Inference Benchmarks

Evaluated with scripts/benchmark_intel.py:

| Target Environment | Driver / Backend | Precision | Throughput | Mean Latency | Median P50 | | :--- | :--- | :---: | :---: | :---: | | Podman Container | OpenVINO / oneDNN (AVX-512) | FP16 | 289.34 FPS | 3.03 ms | 2.02 ms | | Host CPU | OpenVINO Thread Affinity | FP16 | 221.37 FPS | 3.92 ms | 3.83 ms | | Host Dedicated GPU | PyTorch CUDA (RTX 4060) | FP16 | 179.09 FPS | 3.96 ms | 4.04 ms | | OpenVINO INT8 Vision Tower | OpenVINO NNCF Quantized | INT8 | 146.58 FPS | 6.38 ms | 6.56 ms |

Real-Time Headroom: The robot control loop runs at 10 Hz (100 ms). The edge-deployed CPU model executes at 3.03 ms, operating >28× faster than real-time requirements.

---

🔬 INT8 Quantization Verification

• Post-Training Quantization (PTQ): Calibrated via OpenVINO NNCF on multi-camera frames.
• Model Compression: 3.9× reduction in vision tower footprint (101.4 MB → 25.8 MB).
• Kinematic Drift: Mean absolute action drift across 12-DoF arms is only 0.0310 rad (~1.7°) with 0.0981 rad max drift, preserving grasp contacts and fluid pouring stability.

---

🚀 Quickstart Usage

1. Load with LeRobot

from lerobot.policies.factory import make_policy, make_pre_post_processors
from lerobot.policies.smolvla.configuration_smolvla import SmolVLAConfig

repo_id = "rony000013/intel-physical-ai-smolvla"
cfg = SmolVLAConfig.from_pretrained(repo_id)
policy = make_policy(cfg)
policy.eval()
preprocessor, postprocessor = make_pre_post_processors(cfg, pretrained_path=repo_id)

2. Load OpenVINO INT8 Quantized Vision Model

import openvino as ov
from huggingface_hub import hf_hub_download

core = ov.Core()
xml_path = hf_hub_download(repo_id="rony000013/intel-physical-ai-smolvla", filename="variations/openvino_int8/vision_int8.xml")
model = core.read_model(xml_path)
compiled = core.compile_model(model, "CPU") # Or "GPU", "NPU"

---

🛡️ Robustness Evaluation

• 10-Seed Perturbation Test: 100.0% Success Rate (10/10 Passed) across mass, position (+/- 2.5 cm), and friction perturbations.