mlx-community/sam-3d-objects-bf16

SAM 3D Objects — MLX BF16

BF16 weights of Meta SAM 3D Objects, including the upstream MoGe-v1 depth estimator, converted for the MLX runtime in mlx-vlm (mlx_vlm/models/sam3d_objects). This repository holds weights and metadata only; the inference code lives in mlx-vlm. This is an unofficial derivative. SAM components retain the SAM License; MoGe retains MIT; FlexiCubes retains Apache-2.0.

The 3,709 BF16 tensors across seven safetensors files cover the structure and pose flow model, the sparse-latent flow model, image and point-map conditioners, the occupancy decoder, both Gaussian decoders (32 or 4 Gaussians per voxel), the mesh decoder, and MoGe-v1. The model generates geometry and pose; it has no text or tool-calling interface.

Usage

Install an mlx-vlm version that includes sam3d_objects, download this repository, and run the CLI or the Python API:

pip install mlx-vlm
hf download mlx-community/sam-3d-objects-bf16 --local-dir sam-3d-objects-bf16
python -m mlx_vlm.models.sam3d_objects.generate \
  --model sam-3d-objects-bf16 --image image.png --mask mask.png \
  --output output --formats gaussian mesh

from mlx_vlm.models.sam3d_objects.io import read_mask, read_png
from mlx_vlm.models.sam3d_objects.pipeline import Pipeline

pipeline = Pipeline.from_pretrained("sam-3d-objects-bf16")
image, mask = read_png("image.png")[..., :3], read_mask("mask.png")
result = pipeline.generate(image, mask, seed=42)

pipeline.astream(requests) streams ordered events for an iterable or async iterable of Request values, one request at a time with backpressure; the same weights also load through mlx_vlm.utils.load_model. Inputs are HWC RGB/RGBA MLX arrays and HW masks. Automatic MoGe depth is included; external HWC point maps use +X left, +Y up, +Z forward with NaN for invalid points. Results include local-space Gaussians, mesh vertices/faces/colors, latent features, occupied voxels, and a separate pose (rotation, translation, scale). The mlx-vlm model README documents the full API, CLI, JSONL streaming, and the inference optimizations.

Conversion and validation

SAM weights were converted directly from the official tensor-only PyTorch ZIP checkpoints through a restricted stdlib reader. Tensor bytes are first preserved in safetensors, then mapped to MLX layouts and cast to BF16. The duplicate occupancy decoder is removed. MoGe uses its original safetensors export. No PyTorch or NumPy is imported by conversion or inference.

All parameter names/shapes were checked with strict loading, every floating tensor is BF16, and the loaded model has zero trainable parameters. Component checks against the original reference in FP32 MLX CPU cover attention, both flow models, occupancy decoding, interpolation, pretrained MoGe, and FlexiCubes. Maximum absolute error was below 0.000002 on these fixtures; mesh face indices matched exactly. This is not a claim of bitwise BF16 end-to-end parity or a reconstruction-quality evaluation.

See validation.json and conversion.json for details. The original sources are SAM 3D Objects and MoGe.

Local performance

Hardware: Apple M5 Max, 128 GiB unified memory. Input: 448×672 RGB plus object mask; automatic MoGe depth, 25 structure steps and 25 latent steps, both Gaussian heads and mesh. One warmup was excluded; the same seed produced identical latent outputs.

Metric	Result
Median latency (3 runs)	15.68 s
Maximum peak memory	10.98 GB

Two additional async requests matched the synchronous latent output while the event-loop heartbeat continued. These measurements describe one input.