poolside

/

Laguna-XS-2.1

like 95

Follow

Poolside 496

On transformers 5.x (verified on 5.12.1), loading this tokenizer from a local download (snapshot_download / git clone) warns on every load:

The tokenizer you are loading from '…/Laguna-XS-2.1' with an incorrect regex pattern:
https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e.
This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag
when loading this tokenizer to fix this issue.

There are three intertwined problems here worth untangling:

1. The published regex is the known-bad Mistral conversion pattern

The word-level Split stage of tokenizer.json's pre-tokenizer (stage [1] in the Sequence below) is

(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

— exactly the pattern the linked mistralai thread identified as an incorrect conversion of Tekken's real, case-aware regex:

[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+

If Laguna was trained with Tekken-style pre-tokenization, the shipped tokenizer.json tokenizes differently from training (case boundaries in particular) and should be regenerated with the correct regex. If instead the model was trained with tokenizer.json exactly as published, the file is self-consistent and the warning is a false positive — but users have no way to tell which, and the warning explicitly tells them their tokenization is wrong.

2. The remedy transformers suggests actively corrupts this tokenizer

The published pre-tokenizer is a 3-stage Sequence:

[0] Split      (?:\r?\n)+(?!\r?\n)      behavior=MergedWithNext   ← custom newline stage
[1] Split      <the regex above>        behavior=Isolated
[2] ByteLevel

transformers' fix (in tokenization_utils_tokenizers.py) blindly replaces pre_tokenizer[0] of a Sequence, so on this repo fix_mistral_regex=True overwrites the newline stage with the corrected Tekken regex and leaves the incorrect regex in place at [1]:

[0] Split      <corrected Tekken regex>   ← newline stage silently gone
[1] Split      <incorrect regex, still there>
[2] ByteLevel

The result differs from the published tokenizer:

tok = AutoTokenizer.from_pretrained(path)
tok.encode("HelloWorld, l'École!")   # [2, 49208, 81, 388, 76, 34889, 88458, 70]

tok = AutoTokenizer.from_pretrained(path, fix_mistral_regex=True)
tok.encode("HelloWorld, l'École!")   # [2, 6352, 10571, 81, 388, 76, 34889, 88458, 70]

but it isn't a clean correction either — so a user who follows the warning's advice ends up with a pipeline that matches neither the published tokenizer nor a properly regenerated one.

3. Why the warning fires at all (and why only locally)

On 5.12.1 the detection treats any local model whose config.json has no transformers_version field as a suspect Mistral tokenizer — and this repo's config.json omits that field. Loads by hub id do not warn, because the remote path only checks for base_model:…mistralai repo tags. So hub users silently get the flagged regex while local users are warned on every load.

Suggested fix

• If the shipped regex does not match training: regenerate and republish tokenizer.json with the intended pre-tokenizer regex (keeping the newline stage if it's intentional).
• If it does match training: add a transformers_version field to config.json (which silences the false positive on transformers 5.x) and consider noting in the model card that fix_mistral_regex must not be set for this model, since it mangles the pre-tokenizer as shown above.

(Found while fine-tuning locally. The trust_remote_code loading bug is reported separately in #3.)