gatilin/unitok_large_visual_codec

UniTok Large unified visual codec

The complete trained trainer.unitok mapping from the official tokenizer checkpoint: ViTamin-L encoder, attention projections, eight normalized codebooks, RGB decoder, semantic image projection and trained CLIP text tower. All 1335 tensors preserve their original names, dtypes and bytes. Discriminator, optimizer and trainer state are excluded.

Install requirements.txt before loading. In particular, the native hybrid embedder requires the official timm==1.0.8; newer timm constructors use incompatible arguments. The exact text-transformer implementation vendored by the original UniTok project is bundled here; no external OpenCLIP installation is required.

from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download('gatilin/unitok_large_visual_codec')
sys.path.insert(0, folder)
from component import load_model
codec = load_model(folder)
x = torch.zeros(1, 3, 256, 256)  # RGB, NCHW, [-1, 1]
with torch.inference_mode():
    ids = codec.img_to_idx(x)  # [B, 8, 256]
    rgb = codec.idx_to_img(ids)
    semantic_features = codec.encode_image(x, normalize=True)  # [B, 768]

Each spatial position uses eight local IDs in [0, 4095], one per 4096-by-8 codebook. The checkpoint's vocab_size=32768 is the sum of the eight codebook sizes, not the upper bound of every local ID. Code lookup normalizes each sub-codebook before concatenating 64-dimensional vectors. The native decoder assumes a square token grid; this release validates its trained 256-by-256 image path.

preprocessor_config.json records the deterministic validation transform in the pinned project's utils/data.py: resize the shorter edge to 256 with Lanczos, center-crop 256-by-256, convert to RGB, convert to a [0,1] tensor and apply 2*x-1. Downstream MLLM prompt processors have additional conventions.

Native source is pinned to FoundationVision/UniTok revision bb8012e, with imports relocated into the bundled native package. The text tower directly instantiates the same configured default branch of the original vendored builder, using its unchanged TextTransformer and utilities; native/open_clip/source_model.py retains the original constructor as a reference. The wrapper disables training checkpointing and strictly restores the complete trained mapping. Text features use codec.encode_text; CLIP BPE tokenization should follow the original project. No BPE assets or language generator are included.

validation.json records source tensor payload hashes, strict loading, codebook geometry, encode/ID/decode behavior, direct lookup parity, the native quantizer's small floating point difference, and normalized image/text features. These synthetic checks do not reproduce paper quality or downstream accuracy benchmarks. Re-run with python validate.py.

Weight terms follow the official model-card Apache-2.0 license (LICENSE). Bundled project code follows its MIT license (CODE_LICENSE); OPENCLIP_LICENSE and the unchanged reference source retain the original OpenAI attribution. Original third-party notices in individual files are retained. Source container integrity, mapping, per-tensor hashes and code provenance are included.