mnmly/vjepa2.1-vit-giant-384

V-JEPA 2.1 ViT-g/16 (384px) encoder, safetensors for MLX

The V-JEPA 2.1 ViT-g/16 video encoder from Meta FAIR (facebookresearch/vjepa2), repackaged as plain safetensors for the Swift / MLX port mlx-swift-vjepa2.

Unofficial. This is a format conversion of Meta's released checkpoint, not a Meta release. The tensors are exactly the encoder that torch.hub.load('facebookresearch/vjepa2', 'vjepa2_1_vit_giant_384') loads.

Source checkpoint	vjepa2_1_vitg_384.pt → target_encoder
Upstream code	facebookresearch/vjepa2 @ 204698b (app/vjepa_2_1/models)
Parameters	1,013,267,968
Architecture	width 1408, depth 40, 22 heads (64-dim), MLP 6144, patch 16, tubelet 2, 3-axis RoPE
Input	384×384 crops, ImageNet normalization; 1 frame → image path, 2+ frames → video path
Output	[N, 1408] patch features (N = T/2 · 24 · 24), plus 4-level hierarchical features [N, 5632]

Files

File	dtype	Size	SHA-256
model.safetensors	float32 (bit-identical to the source checkpoint)	4.05 GB	b57c5806ae0e34f1f5f0e5c4c72f0bc84e51cc06bf0df5cdaaedea671154573d
model-fp16.safetensors	float16 (round-to-nearest from fp32)	2.03 GB	64f8d5bedb801f35429f643c5eeb093d2f3115bb3e089f8011901c292952c583
predictor.safetensors	float32 predictor (bit-identical to the checkpoint's predictor)	222 MB	32bf5a7ccb70755cccfe4879d50c33ac0e548c0f677cbbb030ab8b751cc98869
predictor-fp16.safetensors	float16 predictor	111 MB	6798f7f20e9f3f96d6cb63694b46514bb04ea662e5821c29146ee04c05e2cdc5
config.json	architecture, preprocessing, provenance

The predictor (55,400,704 parameters) is the one Meta released in the same checkpoint. It predicts the features of hidden tokens from the encoder's features of the visible ones; it outputs 5632-wide features (all four hierarchical levels; it reads the 4-level encoder output).

Keys and tensor layouts follow upstream's state dicts after _clean_backbone_key. The Conv3d patch embeddings stay in PyTorch layout (O, I, kT, kH, kW); the Swift loader transposes them at load time. That means you can also load the files into upstream's PyTorch VisionTransformer / VisionTransformerPredictor with load_state_dict(..., strict=True).

Usage

Swift (mlx-swift-vjepa2)

import VJEPA2

// Downloads model-fp16.safetensors (or model.safetensors for .float32) from this repo and caches it.
let session = try await VJEPA2Session.load(pretrained: .giant, dtype: .float16)
let frames = try await VJEPA2VideoReader.frames(url: clipURL, count: 16, stride: 2)
let encoding = session.encode(frames)        // encoding.features: N × 1408 floats

Predict hidden patches (the JEPA path) by also loading the predictor:

let session = try await VJEPA2Session.load(pretrained: .giant, dtype: .float16, predictor: true)
let mask = VJEPA2TokenMask.block(in: session.tokenGrid(frameCount: frames.count), rows: 6 ..< 18, columns: 6 ..< 18)
let prediction = try session.predict(frames, mask: mask)   // mask.target.count × 5632 floats

CLI: vjepa2-cli embed --model giant --video clip.mp4, or vjepa2-cli predict --model giant --video clip.mp4. Or pick the model in the V-JEPA 2 Viewer app.

PyTorch (upstream code)

import torch
from safetensors.torch import load_file
from app.vjepa_2_1.models import vision_transformer as vit  # from facebookresearch/vjepa2

encoder = vit.vit_giant_xformers(
    img_size=(384, 384), patch_size=16, num_frames=64, tubelet_size=2, use_sdpa=True, use_SiLU=False,
    wide_SiLU=True, uniform_power=False, use_rope=True, img_temporal_dim_size=1, interpolate_rope=True)
encoder.load_state_dict(load_file("model.safetensors"), strict=True)

Parity

The weights are checked against upstream's own PyTorch code. The upstream reference runs in fp32 on CPU, on a real 16-frame clip, a 4-frame clip, and a single image. Values are cosine similarity:

Exact path (MLX CPU stream), every saved block	GPU fp32 per stage (min)	End-to-end fp32: video / hierarchical / image	End-to-end fp16
1.000000	0.999996	0.999997 / 0.999998 / 0.999997	0.999985

The predictor is checked the same way, on masks from upstream's own mask generator: every predictor block matches at cosine ≥ 0.9999997 on the exact path, and pixels → masked encoder → predictor matches at ≥ 0.99998 (fp32) / ≥ 0.99997 (fp16) on the GPU.

Preprocessing (an OpenCV-exact bilinear resize, center crop, and normalization) reproduces upstream's tensor bit for bit. The fixture generator and the test suite ship with mlx-swift-vjepa2.

License

The weights and code are released by Meta under the MIT License. That license is reproduced verbatim in LICENSE, with the copyright notice "Copyright (c) Meta Platforms, Inc. and affiliates.". This repackaging adds no restrictions. If you redistribute these files, keep the license and copyright notice with them.

Citation

@article{murlabadia2026vjepa2_1,
  title={V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning},
  author={Mur-Labadia, Lorenzo and Muckley, Matthew and Bar, Amir and Assran, Mahmoud and
Sinha, Koustuv and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas and Bardes, Adrien},
  journal={arXiv preprint arXiv:2603.14482},
  year={2026}
}

@article{assran2025vjepa2,
  title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
  author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
Rabbat, Michael and Ballas, Nicolas},
  journal={arXiv preprint arXiv:2506.09985},
  year={2025}
}