U32k: 32k byte-level Unigram tokenizer

A 32,768-id byte-level Unigram tokenizer (HuggingFace tokenizers format, ByteLevel pre-tokenizer and decoder) with 256 control tokens and 32,512 learned pieces. English-first, with code and 45 additional languages. Trained with an exact GPU Unigram EM trainer (1.57M seed candidates, shrink 0.999 per prune round, 2,223 rounds, 2 final EM passes).

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("xTimeCrystal/U32k")

ids = tok("The quick brown fox")["input_ids"]
text = tok.decode(ids)                                  # exact roundtrip (byte-level)
batch = tok(["a", "longer text"], padding=True, return_tensors="pt")   # pad id 0

Without transformers:

from tokenizers import Tokenizer
raw = Tokenizer.from_file("tokenizer.json")
raw.encode("text").ids

BOS/EOS are not inserted automatically (no post-processor). Add them explicitly:

ids = [tok.bos_token_id] + tok(text)["input_ids"] + [tok.eos_token_id]

Special tokens (ids 0-255)

id	token	id	token
0	<|pad|>	9	<|tool_call|>
1	<|begin_of_text|> (bos)	10	<|tool_result|>
2	<|end_of_text|> (eos)	11	<|think|>
3	<|user|>	12	<|end_think|>
4	<|assistant|>	13	<|mask|>
5	<|system|>	14	<|sep|>
6	<|start_header|>	15	<|unk|>
7	<|end_header|>	16-18	<|fim_prefix|> <|fim_middle|> <|fim_suffix|>
8	<|end_turn|>	19-20	<|repo_name|> <|file_sep|>

Ids 21-255 are <|reserved_N|>.

Tokenization conventions

• Byte-level: any input roundtrips exactly; no unknown tokens in practice.
• Digits are always single tokens (" 7" or "7", never "12").
• Symbol runs take at most one leading space and never a trailing one; pieces never end in a space.
• Multi-word pieces are allowed (no word-splitting rule); a piece containing a space after its first symbol always begins with a space.
• Pieces never cross Unicode character boundaries.

Training data

region	size	EM influence	contents
English	980 MB	97.2%	cleaned Nemotron-CC web text
Code	346 MB	1.9%	14 languages, uniform
Latin-script	75 MB	0.2%	de fr es it pt nl pl tr vi id cs ro hu da (FineWeb-2-HQ)
Chinese + Russian	47 MB	0.25%	FineWeb-2-HQ
Japanese + Korean	45 MB	0.2%	FineWeb-2-HQ / FineWeb-2
Greek, Hindi, Arabic, Hebrew	43 MB	0.1%	FineWeb-2-HQ / FineWeb-2
23-language pool	39 MB	0.15%	th bn ta km ka hy my kn si te gu ml pa lo bo am uk sr or dv mr ne mn (FineWeb-2 snippets)

Influence is the region's share of the EM expected counts (per-byte weighting), independent of raw size. Cleaning: length band, cross-document line dedup, HTML-density and navigation filters, repeated-fragment stripping, per-language p99 length cut, database-dump removal.

Evaluation (bytes per token, higher is better)

text	this tokenizer	Mistral v0.3 (32k)
FineWeb English held-out	4.82 (15.4% fewer tokens)	4.08
Code held-out	2.90 (3.2% fewer tokens)	2.81
Chinese	2.83	3.00
Japanese	2.71	2.76
Korean	2.70	2.52
Russian	3.22	5.38
German / French	2.21 / 2.53	3.09 / 3.69
Greek / Hebrew / Arabic / Hindi	2.31 / 2.44 / 2.33 / 2.92	1.83 / 1.79 / 1.98 / 2.54

Notes

• model.vocab scores are log-probs. The 256 control tokens and 21 never-observed byte singles (bytes 0xF5-0xFF, impossible in valid UTF-8) carry a -1e30 score; harmless for encoding, floor them if you read scores as a prior.
• convert_ids_to_tokens shows ByteLevel forms (Ġ = space, Ċ = newline); use decode for text.