Halcyon-4B

Early-fusion autoregressive text-to-image model from Text-to-Image Generative Modeling without Language Pretraining. Trained from scratch on GPIC (100M image-text pairs, 256x256) with a joint text-image vocabulary and no language pretraining. The paper calls this size Halcyon-4B.

Variants

Branch	Training configuration	Step	CFG	gFD	GenEval	DPG
main	T2I without caption dropout + 10% I2T (paper Table 9, eta 0.1), plus a partial extra anneal	98,000	4.6	29.30	55.33	78.61
baseline	Baseline T2I, caption dropout 0.1 (paper Table 4)	97,632	4.5	28.43	56.39	77.90

gFD, GenEval and DPG-Bench are the preprint's numbers for that configuration on GPIC Test-50K at 256x256 (lower gFD is better), at the CFG scale shown.

main continues the mixed T2I + I2T run past the one-epoch endpoint (step 97,632) with a linear anneal from 7.5e-5 to 0 over 500 steps; this checkpoint is step 98,000, 368 steps into that anneal. The paper numbers are for the configuration, not necessarily this step.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("radicalnumerics/Halcyon-4B", dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained("radicalnumerics/Halcyon-4B", revision="baseline", dtype="bfloat16")

Model

Architecture	Qwen3 decoder-only (Qwen3ForCausalLM), random init, tied embeddings
Parameters	3.84B (36 layers, hidden 2560, FFN 9728, 32/8 query/KV heads, head dim 128)
Vocabulary	80,640 = 16,384 text + 64,000 image + 256 special
Context	1,280: up to 254 caption tokens, [SOV], 1,024 image tokens (32x32 grid), [EOV], [EOS]
Training	1 epoch of GPIC = 97,632 steps at global batch 1,024; peak LR 7.5e-4, warmup 1,000, constant to step 49,595, cooldown to peak/10; AdamW (0.9, 0.95), weight decay 0.1, clip 1.0, image loss weight 0.5, bf16
Sampling	caption ids + [SOV], then 1,024 image ids with CFG against the bare [SOV] prefix; temperature 1.0, no top-k or top-p

Token ids

Text ids 0 to 16383; image id = 16384 + FSQ code (0 to 63999); [SOV] 80384, [EOV] 80385, [PAD] 80386, [EOS] 80387. Both tokenizers, the full layout and a machine-readable vocab_layout.json are in radicalnumerics/Halcyon-Tokenizers.

Citation

@article{chandrasegaran2026halcyon,
  title  = {Text-to-Image Generative Modeling without Language Pretraining},
  author = {Chandrasegaran, Keshigeyan and Zhao, Yue and Thomas, Armin W. and Adeli, Ehsan and Wu, Jiajun and Niebles, Juan Carlos and Fei-Fei, Li and Poli, Michael},
  year   = {2026},
  note   = {Preprint}
}