vls-speech-head

A causal transformer that reads cached vision-language hidden states and emits Mimi codec tokens, so speech is generated from an image without writing a caption first. The vision backbone stays frozen; only this head is trained.

Several sizes and seeds are published. Seeds matter here: single-seed numbers on this task move by more than the effects people usually compare.

file	params	seed	epoch	val loss	AR token acc
430m-seed42/best.pt	430.1M	42	63	2.177	8.27%
430m-seed43/best.pt	430.1M	43	69	2.1731	7.19%
173m-seed42/best.pt	173.5M	42	46	2.1749	6.3%
173m-seed43/best.pt	173.5M	43	36	2.175	7.5%

Token accuracy reads low by design. The head is judged on whether the decoded speech carries the same meaning as the target, not on whether it reproduces the reference token string, and past the first few epochs those two move in opposite directions.

Loading

Each checkpoint carries its own config, so no companion file is needed:

from huggingface_hub import hf_hub_download
import torch

p = hf_hub_download("seonglae/vls-speech-head", "<name>/best.pt")
ck = torch.load(p, map_location="cpu", weights_only=False)
ck["config"]            # model / training / data settings
ck["model_state_dict"]  # weights

Conditioning is the frozen backbone's last-layer hidden states under a constant prompt. config/ mirrors each run's settings as standalone JSON.