Code and tool-calling tokenizer
The default is structured-code-v1: 32,768 entries, fitted on approximately 64 MiB on Modal. It was explicitly selected for code and tool-calling workloads. This repository contains tokenizer assets, not language-model weights.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("jfrick/scholarly-code-tokenizer")
text = '{"name":"read_file","arguments":{"path":"src/main.py"}}'
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
# The previous balanced default remains available:
previous = AutoTokenizer.from_pretrained(
"jfrick/scholarly-code-tokenizer", subfolder="variants/2048mib-32768"
)
The default is also available in variants/structured-code-v1-32768. All 18 original
learning-curve variants remain available under their existing names. Pin a commit
revision when training models; changing a tokenizer requires retokenizing data.
Old tokenized shards and model checkpoints must not be relabeled with the new IDs.
Fitting and measurements
The focused fitting mixture is 40% real Python excerpts, 20% JSON, 20% Git diffs and 20% tool-call formats by UTF-8 bytes. JSON, diffs and tool formats are generated controlled fixtures derived from Python excerpts. Fitting and evaluation source parents are disjoint, but generator templates are shared. They are not production traces. Source Python comes from the pinned, license-filtered CodeParrot inventory in the original source recipe; raw source text is not distributed. Source datasets retain their respective licenses and provenance.
Compared with the previous 2 GiB / 32K balanced default, this tokenizer uses 2.81% fewer tokens on Python, 8.21% fewer on JSON, 11.83% fewer on Git diffs, and 13.97% fewer on tool-format fixtures. It uses 13.99% more tokens on the broader scholarly mixture. This is an explicit specialization trade-off, not a claim of better language-model quality or valid tool execution.
Structured-output report · PDF · Original learning-curve report
The reports preserve their original experimental conclusions; this model card records the subsequent user decision to promote the focused tokenizer.
Contract and reproducibility
Byte-level BPE; full 256-byte alphabet; no normalization, prefix insertion or automatic BOS/EOS. PAD=0, BOS=1, EOS=2, UNK=3. Add document boundaries explicitly. No model context length is implied. All held-out examples pass exact round-trip and zero-UNK checks. JSON validity and tool-call success require model evaluation.
Standard loading is verified with transformers==4.57.6, tokenizers==0.22.2;
no custom remote code is required. Decode with skip_special_tokens=False to
preserve special literals. The Rust tokenizers.Tokenizer.from_file API also works.
Publication manifest pins all tokenizer hashes. The default variant includes its immutable freeze manifest, fitting corpus metadata and evaluation metrics. Implementation PR.