gatilin/vila_u_256_visual_codec

VILA-U 256 complete visual codec, depth predictor and projector

The complete original VILA-U-256 visual package: 1107 visual entries plus the separate four-entry projector. Both official safetensors files are unchanged and match their original LFS SHA-256 values. The trained SigLIP image/text branches, residual quantizer, RGB decoder, six-layer depth predictor and language-space projector are retained. The separate Llama language model and its tokenizer are excluded.

from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download('gatilin/vila_u_256_visual_codec')
sys.path.insert(0, folder)
from component import load_model, load_projector
visual = load_model(folder)
projector = load_projector(folder)
rgb = torch.zeros(1, 3, 256, 256)  # native RGB NCHW, [-1,1]
with torch.inference_mode():
    ids, quantized = visual.rqvaesiglip.encode_image(rgb)
    # ids: [B,16,16,4]. Four residual depths use one shared codebook.
    vectors = visual.rqvaesiglip.quantizer.embed_code(ids)
    reconstruction = visual.rqvaesiglip.decode(vectors)
    language_features = projector(vectors.flatten(1,2))  # [B,256,4096]

Install requirements.txt in a dedicated environment. The native visual architecture is pinned to mit-han-lab/vila-u revision 4ea42ea. The SigLIP architecture config is bundled at its recorded fixed HF revision, avoiding runtime metadata or redundant pretrained-weight downloads. The native depth predictor is available as visual.rqtransformer; its body input must come from the correct full-model language-state interface for actual generation.

There are 16384 valid IDs, from 0 through 16383. A 16385th stored embedding row is padding. Four depths share one codebook; they are not four independently trained vocabularies. The native straight-through arithmetic can differ slightly from direct lookup; its bound and exact canonical depth-sum/decode paths are recorded in validation.json.

Executed CPU float32 validation covers official file hashes, strict loading, all 1111 restored state values, shared-codebook/padding rules, fixed 256px encoding and reconstruction, language-space projection, short depth teacher-forcing and native four-depth generation/lookup, and the retained trained text tower. Synthetic fixtures do not establish paper image quality, VQA or complete language-conditioned generation. Re-run with python validate.py.

Portability changes are documented in code_provenance.json: Python 3.11+ dataclass default-factory compatibility, bundled SigLIP architecture metadata, and preservation of the input dtype after interpolation rather than hardcoded bfloat16. The original bfloat16 upsampling operation is tested exactly; a complete CUDA bfloat16 pipeline was not tested. Weight bytes are unchanged.

The official weight card and project use MIT (LICENSE). Apache-licensed Hugging Face/Google source notices and the original KakaoBrain RQ-VAE terms remain applicable; see APACHE_LICENSE, RQ_VAE_LICENSE and the unchanged file headers. The source model card, original configurations, extraction provenance and validation report are included.