mlx-community/Ming-Image-0.1-Design-Layer-bf16

MLX (bf16) snapshot of inclusionAI/Ming-Image-0.1-Design-Layer (MIT) for Apple Silicon, loaded by the Swift/MLX port ming-image-swift.

Ming-Image-0.1-Design-Layer decomposes a flat design (poster, signage, card, UI) into editable RGBA layers:

• text, logos, characters and objects each come back on their own straight-alpha layer, front-most first;
• the background layer has the covered regions filled in;
• the model also returns its own recomposited frame.

Quantized tiers: 8-bit (28.3 GB, within 0.1 dB of bf16, 48 GB Macs) and 4-bit (20.5 GB, within 0.2 dB, 36 GB Macs).

It shares the architecture of Ming-Image-0.1-Design, with the DiT trained for multi-frame output: one frame per layer plus the composite. The model conditions on the input twice: as a VAE-encoded reference frame, and through the MLLM's vision tower.

What's in this repo

The upstream diffusers-style tree, with each component stored at the precision the port runs it:

Component	Stored	vs upstream
mllm/: Ling-mini-2.0 MoE MLLM, Qwen2.5 ViT, tokenizer	bf16, 34.0 GB	byte-identical (and identical to Design's)
connector/: Qwen2-1.5B non-causal connector	bf16, 3.1 GB	cast from f32
mlp/: query tokens and projection heads	f32, 0.12 GB	byte-identical
transformer/: multi-frame S3-DiT with learned padding tokens	bf16, 12.3 GB	cast from f32 (upstream ships this DiT at f32)
vae/: RGBA VAE	bf16, 0.25 GB	byte-identical

Total: 49.8 GB. The casts are MLX's own f32 → bf16 conversion, the same one the port applies when it loads the upstream checkpoint, so the port runs bit-identical parameters from either repo (checked tensor by tensor).

Parity (Swift port vs the PyTorch reference)

Components on the fp32 CPU parity lane, against reference goldens:

Component	Result
Input preprocessing (Pillow-exact resizes, image tokens, 3-D positions)	bit-exact
ViT features	relL2 3.5e-5
Conditioning, with the reference's expert routing	6.5e-6 / 3.8e-5
Reference latents	1.4e-6
Layer DiT, 5 frames + reference, padded and full-size	≤ 1.5e-5

End to end:

• 2 steps at CFG 2.0: final latents relL2 1.8e-5, and the composite and all 4 layers are within 1 LSB of the reference PNGs.
• Production bf16 on five real signage stills at the 512 bucket: the layers recomposite to 24.4–31.3 dB against the input, matching the PyTorch reference within 0.1 dB.

Spec format

The model reads a per-layer specification, front-most first:

Decompose this image into 4 layers with the following specifications:

Number of layers: 4
Layer 1: Centered dark-navy bold sans-serif headline text in the upper middle.
Layer 2: Bright blue cloud-shaped logo badge at the top center.
Layer 3: Cartoon squirrel character standing in the lower left.
Layer 4: Light sky-blue background with clouds, green hills and flowers.

Without a spec, "Decompose this image into N layers." also works, but a precise spec isolates text and objects better.

Memory and speed (M5 Max)

Measured as process phys_footprint, with MLX's buffer cache capped at 2 GB (MLXEngine's default). The reference profile is 12 steps at CFG 2.0.

• Post-load resident: 13.2 GB.
• Peak: 50.1–50.4 GB for 4 to 12 layers at the 1024 bucket, in the conditioning stage (the 34 GB MLLM loads, conditions, and is released).
• Under MLXEngine: it declares 13.5 GB resident plus 44.7 GB activation, which needs a 96 GB Mac. Requests are capped at 12 layers, the measured envelope. On 48 GB use the 8-bit tier, and on 36 GB the 4-bit tier.
• Speed: 131 s for 4 layers at the 512 bucket from a 2160×3840 still, and about 11 minutes at the 1024 bucket. The output keeps the input's aspect ratio at the bucket's size.

Use (Swift / MLXEngine)

import MLXMingImage
import MLXToolKit

let package = MingImageLayerPackage(configuration: MingImageLayerConfiguration(snapshotPath: "<this repo, downloaded>"))
try await package.load()
let response = try await package.run(LayerDecomposeRequest(
    image: Image(format: .png, data: try Data(contentsOf: designURL)),
    spec: spec,                       // as above; or pass layerCount alone
    resolution: 1024)) as! LayerDecomposeResponse
// response.layers[0] is the front-most layer; response.composite is the model's recomposition

Under MLXEngine (the layerDecompose capability, contract 1.48.0), register MingImageLayerPackage.registration with MingImageLayerConfiguration(repo: "mlx-community/Ming-Image-0.1-Design-Layer"), and the engine downloads this repo on first use.

License

MIT, as the upstream weights. The upstream LICENSE is included.