mickyhimself/LeVJEPA-Qwen3-8B-Native-CLS

LeVJEPA-Qwen3-8B Native CLS

Final checkpoint of the completed levjepa_qwen3_cls16fps_seed42_20261024_v4 image/video question-answering experiment. This is a fine-tuned Qwen3-8B language model plus its trained visual connector. The final video SFT checkpoint is step 2084.

Each sampled frame is independently encoded by the frozen LeVJEPA-VideoMix-Large vision backbone with temporal input T=1. Its native CLS feature (1024 dimensions) passes through a LayerNorm and two-layer GELU MLP to produce one 4096-dimensional Qwen3 input embedding per frame. Frames are concatenated in time order.

This release is the earlier one native CLS token per frame model. It is separate from the later matched native_cls, patch_mean, and four-token patch_2x2 experiments. It is not an official Qwen3-VL model or the original untuned Qwen3-8B.

Files and loading

• model-*.safetensors, configuration and tokenizer: the full fine-tuned Qwen3-8B language model, not a LoRA adapter.
• connector.safetensors: the trained connector. Tensor values are exactly equal to the final training export; only redundant shared storage was removed.
• qwen3_cls_vqa/, cls_vqa/, controlled/: model, prompt and preprocessing code copied from the verified training snapshot.
• inference.py: a portable inference entry point for this release.
• training_provenance.json, SHA256SUMS.json, release_checks.json: source revisions, stage summaries, checksums and release validation.

The frozen vision backbone is fetched separately from its pinned upstream revision e831a0347737fcaa660b39c57d41c109de399845. Its code uses trust_remote_code=True; inspect that upstream code before use. Loading the root with AutoModelForCausalLM alone loads only the language component. Use the included entry point to insert visual features and perform image/video inference.

pip install -r requirements.txt
python inference.py --video /path/to/video.mp4 --question "Describe what happens in the video."
python inference.py --image /path/to/image.jpg --question "What is happening in this image?"

Download this entire repository before running the commands. A local copy of the pinned vision model can be supplied with --vision-dir /path/to/vision-model. The entry point uses CUDA, BF16 and greedy decoding with thinking disabled. It samples videos at 16 FPS, preserves the full clip, and raises an error if the 512-frame bound is exceeded. It never silently truncates. --fps and --max-frames are explicit inference overrides; changes depart from the training recipe. Preprocessing uses aspect-ratio-preserving letterboxing to 224×224 and ImageNet normalization. The default text plus generation budget is 1028 tokens.

Training

Base language model: Qwen/Qwen3-8B, revision b968826d9c46dd6066d109eabc6255188de91218.

Stage	Steps	Training examples	Learning rate	Trainable components
Image alignment	2159	552551	1e-4	Connector
Image SFT	1408	360417	1e-5	Connector and Qwen3
Video SFT	2084	533423	5e-6	Connector and Qwen3

Each stage uses one epoch, global batch size 256 and seed 42. The vision model is frozen throughout. Video training uses 16 FPS, variable full-clip lengths with a 512-frame maximum, and one visual token per frame. Qwen3 thinking is disabled during both training and inference. Runtime: PyTorch 2.7.1, Transformers 4.57.1; DeepSpeed 0.17.5 was used for SFT training and is not needed for inference.

Validation and limitations

The release preparation verifies completion of all three training stages, checks all four final language-model shards against the training export receipts, and checks exact equality of every connector tensor after conversion. CPU checks cover connector loading/forward, the non-thinking prompt and Python syntax. The portable entry point has not received a new end-to-end GPU generation test at publication.

This repository makes no new benchmark or streaming-performance claim. Independent per-frame encoding does not itself establish streaming ability, effective visual grounding, spatial precision or long-video performance. This checkpoint has no audio input or persistent streaming-memory mechanism. Generated answers can be incorrect.

Licenses and attribution

The fine-tuned Qwen3 component derives from the Apache-2.0-licensed Qwen3-8B model; the upstream license is included as LICENSE. The connector and accompanying release code are distributed under Apache-2.0.

The separately downloaded LeVJEPA-VideoMix-Large backbone is distributed by its authors under CC BY-NC 4.0. That component is not included in this repository, and its own license continues to apply when using the full vision-language system. See the upstream model card for attribution and terms.