LLaVA-OneVision-2-4B-p16m2-Instruct

This is a repackaging, not a new model. The weights are microsoft/Mage-VL (Apache-2.0), byte-for-byte unmodified. What changed is the packaging: module names, class names and config schema were realigned to lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct, and the StreamMind streaming gate was removed.

The point is to have a 4B model that drops into the LLaVA-OneVision-2 train/eval stack, where the smallest official release is 8B.

Why this is only a rename

Mage-VL and LLaVA-OneVision-2 ship the same modeling code. After renaming identifiers, modeling_mage_vl.py and modeling_llava_onevision2.py differ by nothing but class-name strings, docstrings, one import, and the ~100 lines Mage adds for its streaming gate. The two checkpoints have identical state_dict key sets — 696 keys, zero difference. Mage's own source even refers to the base as "the OV model".

Mage-VL differs from LOV-2-8B only in scale and preprocessing knobs:

	LOV-2-8B	this model
LLM	Qwen3-8B — 4096 / 12288, rope 8e6	Qwen3-4B-Instruct-2507 — 2560 / 9728, rope 5e6
ViT	24L · 1024 · 4096 · 16 heads, patch 14	same, patch 16
projector	2-layer MLP → 4096	2-layer MLP → 2560
streaming gate	none	removed here

That shape is already registered upstream as llava-onevision2-4b-p16m2, and tools/convert_checkpoint/config/llava-onevision2-4b-p16m2/*.json maps exactly the HF weight names this checkpoint ships.

What was verified

Loaded with transformers 5.12.1 using the official LOV-2 modules:

model class      : LlavaOnevision2ForConditionalGeneration
total params     : 4.742B   (visual 0.330B + language 4.022B)
ckpt keys 696 | model keys 696 | missing: NONE | unexpected: NONE
sampled tensors  : 4/4 byte-identical to microsoft/Mage-VL

Real image→text inference (224×224 synthetic image, white background, red circle):

image_grid_thw : [[1, 14, 14]]   image tokens: 49
top-5 next tok : [('A', 18.12), ('Red', 17.75), ('Circle', 17.0), ...]
OUTPUT >>> A red circle.

Streaming removal, checked on the shipped files:

files with 'streammind' in name : NONE
'streammind' in shipped .py     : NONE
'streammind' in shipped .json   : NONE
gate-ish attributes / submodules: NONE

streammind_gate.py and streammind_gate.safetensors (1.07 GB) are not included. The gate lived in a separate safetensors file that was never referenced by model.safetensors.index.json, so removing it does not touch the main weights.

Usage

Requires transformers>=5.7.0.

from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "momo321654/LLaVA-OneVision-2-4B-p16m2-Instruct"
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype="bfloat16", device_map="cuda")
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=128)
print(processor.tokenizer.decode(
    out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training

ms-swift — auto-detected from config.json's architectures, no custom registration needed:

model_type : llava_onevision2
template   : llava_onevision2
model_arch : language_model=[model.language_model, lm_head]
             aligner=model.visual.merger
             vision_tower=model.visual

Megatron-Core (in the LLaVA-OneVision-2 repo):

bash examples/llava_onevision2/convert/convert_4b_p16m2_hf_to_mcore.sh \
     <this_repo> <mcore_out> 1 1
# then
--model-name llava-onevision2-4b-p16m2
--hf-tokenizer-path <this_repo>
--load <mcore_out>

Note: upstream ships convert_4b_p16m2_hf_to_mcore.sh but no matching mcore_to_hf script — you have to adapt the p16m3 one. Also, config/llava-onevision2-4b-p16m2/qwen3.json has rope_theta: 1000000 and tie_word_embeddings: true in its huggingface block, while the correct values are 5000000 and false. Training is unaffected (the Megatron side reads rotary_base=5000000 from the model registry), but a mcore→HF export needs those two fields fixed by hand.

Caveats — please read

• The vision tower is Mage-ViT, trained from scratch by Microsoft at patch
  16. It is not the lmms-lab OneVision-Encoder. Only the architecture is shared.
• This is not an official LLaVA-OneVision-2 release. Do not compare it to the LOV-2 paper's numbers — different weights, different training data, different recipe. For Mage-VL's actual benchmark results see the Mage-VL model card.
• The codec-video path is not the one Mage-VL shipped. This repo carries the official LOV-2 codec_video_processing_llava_onevision2.py, which merges canvases into 4-frame attention windows (Mage does not) and has no DCVC-RT neural-codec backend. Image and frame-sampled video are unaffected. If you need Mage's exact codec behaviour, use microsoft/Mage-VL directly.
• No benchmark evaluation was run on this repackaging. Verification covered weight identity, clean loading and a single correct inference — not accuracy.

Credits

• Weights: microsoft/Mage-VL (Apache-2.0)
• Architecture & code: LLaVA-OneVision-2 by EvolvingLMMs-Lab / Glint Lab
• LLM backbone: Qwen/Qwen3-4B-Instruct-2507