gatilin/pixelumm_pixel_io

PixelUMM native pixel interfaces (S8-F18-R01)

Image UND/GEN raw-pixel linear embedders, GEN video-tube embedder, image/video output heads and retained timestep MLP. This checkpoint has no separate video UND tube embedder; video understanding uses image frames. The timestep MLP is present in state but the release disables additive timestep/noise embeddings. Shared MoT backbone is excluded. Research interface bundle, not a standalone VAE or semantic encoder.

原生图像/视频像素接口研究包：保留 UND/GEN 图像投影、GEN 视频 tube 投影与输出头；共享 MoT 主干未包含。时间 MLP 虽在权重中，发布配置不启用其加性注入。

Provenance

Extracted from nvidia/PixelUMM at 4bb3ce9a112e7e1c7d375b1c5bd64f54d259ecc8. Native code revision and attribution are recorded in code_provenance.json and NOTICE. No language model is instantiated by the loader. This repository is an unofficial component export, not an official release from the original authors.

• Tensors: 16; stored parameters/elements: 52,461,312.
• Tensor payload: 104,922,624 bytes; dtype: ['torch.bfloat16'].
• Main checkpoint SHA256: b615edadb2f7880bc7da684819a2f7421700831147c282f237be64e45ec1873e.
• The interface-only source definition is preserved as source_config.json (upstream supplies no JSON model configuration); the custom root config documents this package only. This is an explicit Python loader, not a registered AutoModel or timm architecture.

Use

Install requirements.txt in a dedicated environment. Source files are vendored at a pinned upstream revision. The loader strictly checks every state key.

import sys, torch
from huggingface_hub import snapshot_download
path = snapshot_download("gatilin/pixelumm_pixel_io")
sys.path.insert(0, path)
from component import load_model
model = load_model(path)  # CPU; BF16 for UniAR, FP32 for other components
patches = torch.zeros(16, 16 * 16 * 3)  # native HWC order per patch
with torch.inference_mode():
    und = model.pixel_patch_embed(patches)
    gen = model.pixel_patch_embed_mot_gen(patches)
    video_gen = model.pixel_video_patch_embed_mot_gen(torch.zeros(4, 4 * 16 * 16 * 3))
    rgb_patches = model.unpatchify_head(torch.zeros(16, 4096))
# These projections require the omitted backbone for semantic behavior/generation.
# PixelUMM's noise-time convention: z_t=(1-t)*clean + t*noise.

Validation

validation.json records all checks: original source tensor bytes match, strict loading passes, 3 native equivalence cases pass, plus the useful-resolution smoke case 256_image_and_video_tube_io. Tests ran with FP32 on CPU. Where supported, native eager reference attention was compared to SDPA in the exported loader. Discrete token/bit outputs are checked exactly; floating outputs use tolerance documented in the validation script.

The reference loads original source ranges directly; the exported loader independently reads the assembled Safetensors checkpoint. Tests cover only the component APIs. They do not establish downstream understanding accuracy, text-to-image quality, or GPU/FlashAttention equivalence. Synthetic input images/tensors are used; reconstruction is lossy and no pixel-fidelity benchmark is claimed.

Reproduce extraction with python extract_pixelumm_io.py --output PATH from this snapshot; then run python validate_unified_components.py PATH. The byte comparison requires .download source chunks, which extraction creates but this repository intentionally does not upload.

License and attribution

The original weight and code licenses remain applicable. Preserve upstream author notices. See LICENSE, NOTICE, SOURCE_README.md, and source-file headers.

Weights: NVIDIA One-Way Noncommercial License, restricted to noncommercial research/evaluation; complete license in LICENSE.weights.pdf. Vendored modeling_utils.py: CC BY-NC 4.0, full text in LICENSE.code-cc-by-nc.txt. Other upstream code: Apache 2.0 where its header states so. Do not relabel this bundle Apache-only. DCP extraction is reproduced with extract_pixelumm_io.py and its restricted metadata reader; source_metadata.bin preserves the exact original metadata bytes.