Add root fast tokenizer serialization
#27
by michaelfeil - opened
Add tokenizer.json at the repository root for consumers that load the fast tokenizer serialization directly, including Infinity v3.
Converted from the original vocabulary and tokenizer configuration using AutoTokenizer with use_fast=True. The resulting JSON is identical to the existing onnx/tokenizer.json at this revision. No weights or tokenizer configuration are changed.
Validation: 156 comparisons of token IDs, offsets, token type IDs and attention masks covering casing, Unicode, sentence pairs, padding and left/right truncation. Infinity 3.0.0 GPU NER inference and all aggregation modes passed using this tokenizer.