contacts-and-crops-v1 tokenizer
Byte-level tokenizer (3848-token vocabulary) for the MarinFold contacts-and-crops-v1 protein document format (exp130 / issue #130; corpus exp132 / issue #132). Used to train the exp137 1.5B model (issue #137).
The vocabulary is a strict superset of timodonnell/contacts-v1-tokenizer
(ids 2–2845 are byte-identical), extended with the <xyz-000..999> coordinate
tokens (ids 2846–3846, identical to the contacts-and-coordinates-v1 tokenizer)
and a single <crop> token (id 3847). Consequence: contacts-v1 documents
tokenize identically under this tokenizer, so a contacts-v1 validation loss
computed here is directly comparable to contacts-v1 models.
Files are a byte-for-byte copy of the tokenizer/ directory published with the
corpus at
hf://buckets/open-athena/MarinFold/data/document_structures/contacts_and_crops_v1/tokenizer/.