MoGe-2 ViT-B for Core ML
A Core ML conversion of MoGe-2 ViT-B, the monocular geometry model from Microsoft Research (code). The model and its weights are the work of its authors, Ruicheng Wang and colleagues. What is here is only the conversion, made by Outmake so the model runs on a Mac with no Python.
Outmake uses it for Blur Background: it reads how far away each part of a picture is, and the blur is laid on by that distance.
Files
Core ML fixes a model's input size, so there is one package for each picture shape. Each is 195 MB, 16-bit floats.
| Package | Input (height x width) | For |
|---|---|---|
moge2_vitb_574x854.mlpackage |
574 x 854 | wide pictures |
moge2_vitb_854x574.mlpackage |
854 x 574 | tall pictures |
moge2_vitb_700x700.mlpackage |
700 x 700 | square pictures |
SHA256SUMS lists every file with its size. convert.py is the script that made them.
Input and output
- Input
image:1 x 3 x H x W, RGB, 0 to 1. - Output
points:1 x H x W x 3, a 3D point for each pixel, right up to one shift along z. - Output
mask:1 x H x W, 0 to 1. Low where the model gives no answer, which includes sky and plain studio backdrops.
The shift along z and the focal length are not in the model's output. MoGe finds them afterwards with a small least-squares fit, and anything using these files has to do the same. Normals and metric scale, which the original model also gives, are left out of this conversion.
How close it is to the original
Against the authors' ONNX export on the same pictures, depth from these packages correlates at 0.99 or better.
On an M-series MacBook Pro with computeUnits = .cpuAndGPU, one picture takes about 0.35 s. With the Neural Engine allowed, the model took 41 s to load and ran three times slower, so the Neural Engine is best left off.
Licence
MIT, the same as MoGe. The original copyright notice is in LICENSE-MoGe.
Citation
Cite the original work: see the MoGe repository for the papers.
- Downloads last month
- -