Halcyon-0.6B

Early-fusion autoregressive text-to-image model from Text-to-Image Generative Modeling without Language Pretraining. Trained from scratch on GPIC (100M image-text pairs, 256x256) with a joint text-image vocabulary and no language pretraining. The paper calls this size Halcyon-0.52B.

Variants

Branch	Training configuration	Step	CFG	gFD	GenEval	DPG
main	Cooldown from the baseline run at step 49,595 with 10% I2T mixing, caption dropout 0.1 kept; not a paper configuration	97,632
baseline	Baseline T2I, caption dropout 0.1 (paper Table 4)	97,632	6.0	48.02	44.91	76.84

gFD, GenEval and DPG-Bench are the preprint's numbers for that configuration on GPIC Test-50K at 256x256 (lower gFD is better), at the CFG scale shown.

main mixes 10% I2T only during the cooldown half and keeps caption dropout at 0.1, so it is neither the paper's baseline nor its zero-waste arm; no paper numbers are attached.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("radicalnumerics/Halcyon-0.6B", dtype="bfloat16")
model = AutoModelForCausalLM.from_pretrained("radicalnumerics/Halcyon-0.6B", revision="baseline", dtype="bfloat16")

Model

Architecture	Qwen3 decoder-only (Qwen3ForCausalLM), random init, tied embeddings
Parameters	523M (28 layers, hidden 1024, FFN 3072, 16/8 query/KV heads, head dim 128)
Vocabulary	80,640 = 16,384 text + 64,000 image + 256 special
Context	1,280: up to 254 caption tokens, [SOV], 1,024 image tokens (32x32 grid), [EOV], [EOS]
Training	1 epoch of GPIC = 97,632 steps at global batch 1,024; peak LR 7.5e-4, warmup 1,000, constant to step 49,595, cooldown to peak/10; AdamW (0.9, 0.95), weight decay 0.1, clip 1.0, image loss weight 0.5, bf16
Sampling	caption ids + [SOV], then 1,024 image ids with CFG against the bare [SOV] prefix; temperature 1.0, no top-k or top-p

Token ids

Text ids 0 to 16383; image id = 16384 + FSQ code (0 to 63999); [SOV] 80384, [EOV] 80385, [PAD] 80386, [EOS] 80387. Both tokenizers, the full layout and a machine-readable vocab_layout.json are in radicalnumerics/Halcyon-Tokenizers.

Citation

@article{chandrasegaran2026halcyon,
  title  = {Text-to-Image Generative Modeling without Language Pretraining},
  author = {Chandrasegaran, Keshigeyan and Zhao, Yue and Thomas, Armin W. and Adeli, Ehsan and Wu, Jiajun and Niebles, Juan Carlos and Fei-Fei, Li and Poli, Michael},
  year   = {2026},
  note   = {Preprint}
}