Halcyon-Tokenizers

Text and image tokenizers for the Halcyon models in this collection, from Text-to-Image Generative Modeling without Language Pretraining. Both were trained from scratch on GPIC.

Path	What
tokenizer.json, tokenizer_config.json	text byte-level BPE, 16,384 ids, loads with AutoTokenizer
image_tokenizer/di8x8_242172_ema.pt	image tokenizer weights (PyTorch state dict, 521 MiB)
vocab_layout.json	joint vocabulary layout and special ids of the Halcyon LMs
bpe16k_final.report.json	training report of the text BPE

Text tokenizer

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("radicalnumerics/Halcyon-Tokenizers")
ids = tok("A woman carries a child on her back.").input_ids  # ids < 16,384

Byte-level BPE with 16,384 ids and no special tokens, trained on GPIC captions (21.3 GB) plus OpenWebText (11.9 GB). On GPIC test captions: 4.774 bytes per token, mean 44.5 tokens per caption (p50 36, p99 182). Captions are capped at 254 tokens, which is the model_max_length.

Image tokenizer

Cosmos-style encoder-decoder with finite scalar quantization, trained on GPIC at 256x256: 8x spatial downsampling, 32x32 = 1,024 codes per image in raster order, codebook 64,000 (FSQ levels 8, 8, 8, 5, 5, 5), 137M parameters, decoder width 192. Trained for one epoch with reconstruction and LPIPS losses, then half an epoch with a PatchGAN adversarial loss; 242,172 steps, EMA weights. Reconstruction on GPIC Test-50K: rFD 18.86, PSNR 25.61, SSIM 0.792, LPIPS 0.062.

The file is a plain state dict (205 tensors, standard torch storages only); the tokenizer code is not part of this repo. Input: RGB resized to 256x256 (bicubic), scaled to [-1, 1]. Output: 1,024 codes in [0, 63999].

Token ids and sequence format

Ids	Meaning
0 to 16383	text BPE ids
16384 to 80383	image codes, id = 16384 + FSQ code
80384	[SOV], begin of image
80385	[EOV], end of image
80386	[PAD]
80387	[EOS]
80388 to 80639	reserved, unused

A text-to-image sequence is caption ids, [SOV], 1,024 image ids, [EOV], [EOS], then [PAD] to the 1,280-token context. At inference the prompt is caption ids followed by [SOV]; the model emits the 1,024 image ids. The same layout is in vocab_layout.json.

Citation

@article{chandrasegaran2026halcyon,
  title  = {Text-to-Image Generative Modeling without Language Pretraining},
  author = {Chandrasegaran, Keshigeyan and Zhao, Yue and Thomas, Armin W. and Adeli, Ehsan and Wu, Jiajun and Niebles, Juan Carlos and Fei-Fei, Li and Poli, Michael},
  year   = {2026},
  note   = {Preprint}
}