embodi00-vlm-stage1

a compact vision-language model checkpoint from the first stage of a four-stage training pipeline for a robotics-focused vlm. this stage aligns a frozen siglip2 encoder with a frozen smollm2 decoder through a trainable projector and a set of new token embeddings.

architecture

• vision encoder: google/siglip2-base-patch16-384
• language model: HuggingFaceTB/SmolLM2-135M
• input resolution: 384 x 384
• visual tokens: 144, after 2x2 spatial compression
• parameters: 232m

training

this is stage 1 of 4 in the embodi vlm pipeline. the goal of this stage is to teach the projector to map siglip2 features into an embedding distribution smollm2 can consume, before any general multimodal or spatial training.

trainable:

• multimodal projector
• image boundary embeddings
• task-token embeddings
• coordinate-token embeddings
• structural-token embeddings
• new output rows

frozen:

• siglip2
• original smollm2 parameters

data:

• 70% pixmo-cap
• 30% pixmo-points

steps: 2500

learning rates:

• projector: 1e-3
• new token embeddings: 3e-4

this repository holds the best stage 1 checkpoint, selected at step 1000 by validation loss.

results

metric	value
validation loss	2.675
caption token accuracy	42.1%
point token accuracy	46.8%
unconstrained point syntax validity	98%
point accuracy within normalized radius 0.05	4%
mean point distance	0.287

usage

the checkpoint uses the custom embodi_vlm architecture.

from pathlib import Path

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

from embodi_vlm import EmbodiVLM, EmbodiVLMConfig

checkpoint = Path(snapshot_download("embodi/embodi00-vlm-stage1"))

config = EmbodiVLMConfig.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint / "tokenizer")

model = EmbodiVLM.from_backbones(config, tokenizer)
model.load_weights(checkpoint)
model.eval()