fix: Remediate 1830 glitch tokens in tokenizer.json

#95

Glitch Token Remediation — Tokenizer Patch

Summary

This PR patches tokenizer.json to remediate 1830 glitch tokens identified
by embedding vector analysis. Glitch tokens are vocabulary entries whose embeddings
have collapsed to near-identical representations, causing unpredictable model behavior
when they appear in input.

No model weights are modified — this is a tokenizer-only fix.

Technique: Hybrid (merge pruning + placeholder rename)

The patch uses two complementary strategies:

1. Merge pruning (328 tokens): Removes the BPE merge rules that
produce multi-character glitch tokens. The vocab entry stays (ID preserved) but
becomes unreachable — text that would have matched is now tokenized as trained
constituent subwords instead.

2. Placeholder rename (1502 tokens): For single-character glitch tokens
(rare Unicode characters) that have no producing merge rule, the vocab string is renamed to
<glitch_pruned_ID>. The original character now falls through to byte_fallback,
encoding as well-trained UTF-8 byte tokens.

Safety: Placeholder strings cannot be triggered by user input. BPE builds tokens
bottom-up from characters via merge rules only — since no merge rule produces these strings,
they are permanently unreachable. Gemma's split-digits tokenization policy provides an
additional layer of safety.

543 merges were removed for 328 merge-pruned tokens because some glitch tokens have more than one BPE merge path, and every path must be removed.

16 flagged IDs were left unchanged because they are special tokens, byte-fallback tokens, or single ASCII characters, which the tokenizer must keep.

Before / after examples: google/gemma-2b

Merge prune example: ſammen, token ID 134910

Prompt: Please repeat the following string exactly, with nothing else: "ſammen"

Tokens for the string Model output
Before ['ſammen'] → [134910] WebElementEntity ❌
After ['ſ', 'ammen'] → [235433, 12493] ſammen ✅

Placeholder rename example: 𑄣 (U+11123), token ID 253904

Prompt: Please repeat the following string exactly, with nothing else: "𑄣"

Tokens for the string Model output
Before ['𑄣'] → [253904] Scénario ❌
After ['<0xF0>', '<0x91>', '<0x84>', '<0xA3>'] → [457, 362, 349, 380] 𑄣 ✅

⚠️ Note for Agentic Systems

Many remediated glitch tokens are shared across Gemma model families (Gemma 1, 2, 3, 4).
In agentic pipelines, care should be taken that these token strings are not inadvertently
injected into prompts for unpatched models. We recommend applying this patch consistently
across all Gemma models used in a pipeline.

What is preserved

  • ✅ len(tokenizer) — unchanged (256,000)
  • ✅ All token IDs — stable, no re-indexing
  • ✅ Chat template — identical to original
  • ✅ tokenizer_config.json — identical to original
  • ✅ All special tokens and added_tokens — unchanged
  • ✅ Normal text tokenization — verified via benchmarks
  • ✅ Zero model weight changes

Diff summary

Original Patched Delta
Vocab size 256,000 256,000 0
Merges 580,604 580,061 −543
Vocab renamed — 1502 +1502

Vocab renames (1502 entries)

Click to expand the first 200 renamed vocab entries

Showing the first 200 of 1502 renamed entries (renamed_vocab.tsv). Full list: https://huggingface.co/gkielian/gemma-glitch-staging/tree/main/google_gemma-2b

ID Original Patched
238396 (U+E653) <glitch_pruned_238396>
240483 拶 (U+62F6) <glitch_pruned_240483>
240610 (U+E70D) <glitch_pruned_240610>
241690 稣 (U+7A23) <glitch_pruned_241690>
242265 蝙 (U+8759) <glitch_pruned_242265>
242301 赁 (U+8D41) <glitch_pruned_242301>
242409 饪 (U+996A) <glitch_pruned_242409>
242439 󰀄 (U+F0004) <glitch_pruned_242439>
242588 螃 (U+8783) <glitch_pruned_242588>
244450 ܃ (U+0703) <glitch_pruned_244450>
244537 Ꮓ (U+13C3) <glitch_pruned_244537>
244997 (U+F25C) <glitch_pruned_244997>
245126 (U+F25D) <glitch_pruned_245126>
245474 ༔ (U+0F14) <glitch_pruned_245474>
245810 ౧ (U+0C67) <glitch_pruned_245810>
245817 񱜆 (U+71706) <glitch_pruned_245817>
245837 (U+E632) <glitch_pruned_245837>
245844 (U+E63B) <glitch_pruned_245844>
246059 ཿ (U+0F7F) <glitch_pruned_246059>
246547 ཋ (U+0F4B) <glitch_pruned_246547>
246588 ྜ (U+0F9C) <glitch_pruned_246588>
246712 ୮ (U+0B6E) <glitch_pruned_246712>
246760 ୫ (U+0B6B) <glitch_pruned_246760>
246876 ༈ (U+0F08) <glitch_pruned_246876>
246884 ୪ (U+0B6A) <glitch_pruned_246884>
247025 (U+F6DC) <glitch_pruned_247025>
247215 (U+E901) <glitch_pruned_247215>
247313 Ꮵ (U+13E5) <glitch_pruned_247313>
247377 ཥ (U+0F65) <glitch_pruned_247377>
247409 ༑ (U+0F11) <glitch_pruned_247409>
247445 ꩻ (U+AA7B) <glitch_pruned_247445>
247458 曏 (U+66CF) <glitch_pruned_247458>
247635 ܆ (U+0706) <glitch_pruned_247635>
247641 ܇ (U+0707) <glitch_pruned_247641>
247739 Ꮫ (U+13DB) <glitch_pruned_247739>
247780 ྞ (U+0F9E) <glitch_pruned_247780>
247798 ῎ (U+1FCE) <glitch_pruned_247798>
248178 Ꮴ (U+13E4) <glitch_pruned_248178>
248221 ꪳ (U+AAB3) <glitch_pruned_248221>
248309 〬 (U+302C) <glitch_pruned_248309>
248337 (U+F21D) <glitch_pruned_248337>
248384 ྛ (U+0F9B) <glitch_pruned_248384>
248403 ⤥ (U+2925) <glitch_pruned_248403>
248440 炝 (U+709D) <glitch_pruned_248440>
248449 ⤦ (U+2926) <glitch_pruned_248449>
248471 ⮑ (U+2B91) <glitch_pruned_248471>
248517 (U+F63A) <glitch_pruned_248517>
248619 ྶ (U+0FB6) <glitch_pruned_248619>
248660 (U+E140) <glitch_pruned_248660>
248691 ྰ (U+0FB0) <glitch_pruned_248691>
248709 婠 (U+5A60) <glitch_pruned_248709>
248798 (U+E3D0) <glitch_pruned_248798>
248896 眎 (U+770E) <glitch_pruned_248896>
248911 (U+E5F1) <glitch_pruned_248911>
248926 ྃ (U+0F83) <glitch_pruned_248926>
248927 Ꮄ (U+13B4) <glitch_pruned_248927>
248959 ㈵ (U+3235) <glitch_pruned_248959>
248977 ౮ (U+0C6E) <glitch_pruned_248977>
249050 ౨ (U+0C68) <glitch_pruned_249050>
249141 ୯ (U+0B6F) <glitch_pruned_249141>
249228 (U+E902) <glitch_pruned_249228>
249260 (U+F639) <glitch_pruned_249260>
249293 Ꮟ (U+13CF) <glitch_pruned_249293>
249322 ۺ (U+06FA) <glitch_pruned_249322>
249361 ྪ (U+0FAA) <glitch_pruned_249361>
249389 (U+E603) <glitch_pruned_249389>
249390 (U+F64B) <glitch_pruned_249390>
249401 鲮 (U+9CAE) <glitch_pruned_249401>
249409 ᅚ (U+115A) <glitch_pruned_249409>
249482 (U+E073) <glitch_pruned_249482>
249483 (U+E6E7) <glitch_pruned_249483>
249546 ౫ (U+0C6B) <glitch_pruned_249546>
249570 儅 (U+5105) <glitch_pruned_249570>
249585 ၞ (U+105E) <glitch_pruned_249585>
249589 嗞 (U+55DE) <glitch_pruned_249589>
249621 (U+E771) <glitch_pruned_249621>
249642 ၟ (U+105F) <glitch_pruned_249642>
249647 嗵 (U+55F5) <glitch_pruned_249647>
249660 ၠ (U+1060) <glitch_pruned_249660>
249663 Ꮽ (U+13ED) <glitch_pruned_249663>
249714 𞤢 (U+1E922) <glitch_pruned_249714>
249717 ༞ (U+0F1E) <glitch_pruned_249717>
249751 䊉 (U+4289) <glitch_pruned_249751>
249784 ܞ (U+071E) <glitch_pruned_249784>
249803 (U+F106) <glitch_pruned_249803>
249810 Ꭸ (U+13A8) <glitch_pruned_249810>
249828 ౬ (U+0C6C) <glitch_pruned_249828>
249845 (U+F139) <glitch_pruned_249845>
249865 (U+F63B) <glitch_pruned_249865>
249906 ݈ (U+0748) <glitch_pruned_249906>
249928 ܔ (U+0714) <glitch_pruned_249928>
249954 ⫘ (U+2AD8) <glitch_pruned_249954>
249984 刋 (U+520B) <glitch_pruned_249984>
249991 (U+F649) <glitch_pruned_249991>
249993 ﮯ (U+FBAF) <glitch_pruned_249993>
250021 🗪 (U+1F5EA) <glitch_pruned_250021>
250120 (U+F63C) <glitch_pruned_250120>
250128 妧 (U+59A7) <glitch_pruned_250128>
250163 ஬ (U+0BAC) <glitch_pruned_250163>
250185 ྠ (U+0FA0) <glitch_pruned_250185>
250199 𑄴 (U+11134) <glitch_pruned_250199>
250221 䊏 (U+428F) <glitch_pruned_250221>
250229 隂 (U+9682) <glitch_pruned_250229>
250243 䉴 (U+4274) <glitch_pruned_250243>
250245 儈 (U+5108) <glitch_pruned_250245>
250282 ܊ (U+070A) <glitch_pruned_250282>
250295 (U+E065) <glitch_pruned_250295>
250304 Ꮞ (U+13CE) <glitch_pruned_250304>
250338 (U+E049) <glitch_pruned_250338>
250392 Ꮷ (U+13E7) <glitch_pruned_250392>
250433 㜵 (U+3735) <glitch_pruned_250433>
250464 耑 (U+8011) <glitch_pruned_250464>
250482 ⇹ (U+21F9) <glitch_pruned_250482>
250538 ῞ (U+1FDE) <glitch_pruned_250538>
250547 (U+F641) <glitch_pruned_250547>
250549 𞤭 (U+1E92D) <glitch_pruned_250549>
250622 ౭ (U+0C6D) <glitch_pruned_250622>
250630 咘 (U+5498) <glitch_pruned_250630>
250634 熘 (U+7198) <glitch_pruned_250634>
250639 (U+E977) <glitch_pruned_250639>
250640 (U+F63D) <glitch_pruned_250640>
250667 (U+E245) <glitch_pruned_250667>
250668 (U+E692) <glitch_pruned_250668>
250705 ୬ (U+0B6C) <glitch_pruned_250705>
250721 犼 (U+72BC) <glitch_pruned_250721>
250729 (U+F10B) <glitch_pruned_250729>
250738 ༴ (U+0F34) <glitch_pruned_250738>
250770 毉 (U+6BC9) <glitch_pruned_250770>
250782 𝅥 (U+1D165) <glitch_pruned_250782>
250800 ஡ (U+0BA1) <glitch_pruned_250800>
250824 𞤮 (U+1E92E) <glitch_pruned_250824>
250853 鼕 (U+9F15) <glitch_pruned_250853>
250874 磲 (U+78F2) <glitch_pruned_250874>
250887 ﭜ (U+FB5C) <glitch_pruned_250887>
250891 𞤫 (U+1E92B) <glitch_pruned_250891>
250905 魠 (U+9B60) <glitch_pruned_250905>
250918 䊳 (U+42B3) <glitch_pruned_250918>
250931 (U+E040) <glitch_pruned_250931>
250934 (U+F63E) <glitch_pruned_250934>
250943 ౩ (U+0C69) <glitch_pruned_250943>
250969 ⌵ (U+2335) <glitch_pruned_250969>
250971 㤵 (U+3935) <glitch_pruned_250971>
250973 婼 (U+5A7C) <glitch_pruned_250973>
250990 (U+E0A9) <glitch_pruned_250990>
251013 髌 (U+9ACC) <glitch_pruned_251013>
251033 푥 (U+D465) <glitch_pruned_251033>
251054 繝 (U+7E5D) <glitch_pruned_251054>
251067 𐄙 (U+10119) <glitch_pruned_251067>
251073 ྥ (U+0FA5) <glitch_pruned_251073>
251124 𐄁 (U+10101) <glitch_pruned_251124>
251129 ୩ (U+0B69) <glitch_pruned_251129>
251135 哙 (U+54D9) <glitch_pruned_251135>
251179 ໊ (U+0ECA) <glitch_pruned_251179>
251214 ㄰ (U+3130) <glitch_pruned_251214>
251218 峮 (U+5CEE) <glitch_pruned_251218>
251240 ỽ (U+1EFD) <glitch_pruned_251240>
251247 猹 (U+7339) <glitch_pruned_251247>
251258 (U+F640) <glitch_pruned_251258>
251278 禩 (U+79A9) <glitch_pruned_251278>
251279 蜮 (U+872E) <glitch_pruned_251279>
251285 (U+E60B) <glitch_pruned_251285>
251320 ޒ (U+0792) <glitch_pruned_251320>
251373 ༛ (U+0F1B) <glitch_pruned_251373>
251374 ཪ (U+0F6A) <glitch_pruned_251374>
251381 珵 (U+73F5) <glitch_pruned_251381>
251389 (U+F63F) <glitch_pruned_251389>
251427 𞤪 (U+1E92A) <glitch_pruned_251427>
251432 ഀ (U+0D00) <glitch_pruned_251432>
251453 鱻 (U+9C7B) <glitch_pruned_251453>
251460 Ꮈ (U+13B8) <glitch_pruned_251460>
251466 ▖ (U+2596) <glitch_pruned_251466>
251477 苁 (U+82C1) <glitch_pruned_251477>
251496 ஥ (U+0BA5) <glitch_pruned_251496>
251497 ஦ (U+0BA6) <glitch_pruned_251497>
251499 ྑ (U+0F91) <glitch_pruned_251499>
251509 ⟝ (U+27DD) <glitch_pruned_251509>
251516 蒌 (U+848C) <glitch_pruned_251516>
251522 왘 (U+C658) <glitch_pruned_251522>
251524 (U+E80D) <glitch_pruned_251524>
251525 (U+EAE4) <glitch_pruned_251525>
251554 鼋 (U+9F0B) <glitch_pruned_251554>
251558 (U+E619) <glitch_pruned_251558>
251559 (U+E801) <glitch_pruned_251559>
251560 (U+E978) <glitch_pruned_251560>
251586 鹮 (U+9E6E) <glitch_pruned_251586>
251599 𐐀 (U+10400) <glitch_pruned_251599>
251614 垍 (U+578D) <glitch_pruned_251614>
251628 푖 (U+D456) <glitch_pruned_251628>
251629 (U+E050) <glitch_pruned_251629>
251630 (U+F8FC) <glitch_pruned_251630>
251631 ﮨ (U+FBA8) <glitch_pruned_251631>
251632 𑄨 (U+11128) <glitch_pruned_251632>
251646 ܖ (U+0716) <glitch_pruned_251646>
251670 ྾ (U+0FBE) <glitch_pruned_251670>
251672 ᪥ (U+1AA5) <glitch_pruned_251672>
251676 ⟞ (U+27DE) <glitch_pruned_251676>
251683 焿 (U+713F) <glitch_pruned_251683>
251691 푡 (U+D461) <glitch_pruned_251691>
251703 ࿙ (U+0FD9) <glitch_pruned_251703>
251717 扨 (U+6268) <glitch_pruned_251717>

Removed merges (543)

Click to expand the first 200 of 543 removed merges

Showing the first 200 of 543 removed merges (removed_merges.txt). Full list: https://huggingface.co/gkielian/gemma-glitch-staging/tree/main/google_gemma-2b

["">", "😂"]  →  ">😂
["$_", "?"]  →  $_?
[")$", "_,"]  →  )$_,
[")", "$_,"]  →  )$_,
[")$_", ","]  →  )$_,
[")$_", "."]  →  )$_.
[")$", "_."]  →  )$_.
[")", "$_."]  →  )$_.
[")", ")$_"]  →  ))$_
["))", "$_"]  →  ))$_
["))$", "_"]  →  ))$_
["=", " "]  →  = 
[">", "\<^"]  →  >\<^
[">\", "<^"]  →  >\<^
[">\<", "^"]  →  >\<^
["Adv", "Ex"]  →  AdvEx
["Answer", "Step"]  →  AnswerStep
["At", "bildēt"]  →  Atbildēt
["Di", "berdayakan"]  →  Diberdayakan
["Di", "wed"]  →  Diwed
["Diwedd", "ar"]  →  Diweddar
["Diwed", "dar"]  →  Diweddar
["English", "Choose"]  →  EnglishChoose
["Gilla", "Gilla"]  →  GillaGilla
["ICTO", "GRAM"]  →  ICTOGRAM
["Line", "VCR"]  →  LineVCR
["Nd", "Ex"]  →  NdEx
["OUT", "UBE"]  →  OUTUBE
["ParameterLimits", "V"]  →  ParameterLimitsV
["Window", "LineVCR"]  →  WindowLineVCR
["\<", "^"]  →  \<^
["\", "<^"]  →  \<^
["^(@)", "$_"]  →  ^(@)$_
["^(@", ")$_"]  →  ^(@)$_
["acter", "ísticas"]  →  acterísticas
["actéris", "ti"]  →  actéristi
["acté", "risti"]  →  actéristi
["alak", "ip"]  →  alakip
["ala", "kip"]  →  alakip
["bildschirm", "foto"]  →  bildschirmfoto
["bild", "titel"]  →  bildtitel
["bl", "uza"]  →  bluza
["blu", "za"]  →  bluza
["bra", "kk"]  →  brakk
["brak", "k"]  →  brakk
["br", "akk"]  →  brakk
["cit", "roën"]  →  citroën
["col", "gante"]  →  colgante
["dá", "ms"]  →  dáms
["dáms", "ké"]  →  dámské
["emen", "tara"]  →  ementara
["ement", "ara"]  →  ementara
["enab", "log"]  →  enablog
["ena", "blog"]  →  enablog
["fel", "pa"]  →  felpa
["geno", "digd"]  →  genodigd
["g", "eſ"]  →  geſ
["ge", "ſ"]  →  geſ
["ha", "ikusbot"]  →  haikusbot
["idler", "tid"]  →  idlertid
["ielle", "icht"]  →  ielleicht
["iel", "leicht"]  →  ielleicht
["ient", "ras"]  →  ientras
["ien", "tras"]  →  ientras
["i", "entras"]  →  ientras
["iffa", "nce"]  →  iffance
["iff", "ance"]  →  iffance
["ikus", "bot"]  →  ikusbot
["isGrid", "AdvEx"]  →  isGridAdvEx
["is", "Ora"]  →  isOra
["isOra", "ColElement"]  →  isOraColElement
["i", "ſche"]  →  iſche
["iſ", "che"]  →  iſche
["iſ", "chen"]  →  iſchen
["i", "ſchen"]  →  iſchen
["iſche", "n"]  →  iſchen
["i", "ſen"]  →  iſen
["iſe", "n"]  →  iſen
["iſ", "en"]  →  iſen
["iſ", "ten"]  →  iſten
["i", "ſten"]  →  iſten
["iſt", "en"]  →  iſten
["ja", "queta"]  →  jaqueta
["kos", "zulka"]  →  koszulka
["l", "brakk"]  →  lbrakk
["lina", "wan"]  →  linawan
["lin", "awan"]  →  linawan
["lx", "task"]  →  lxtask
["lxt", "ask"]  →  lxtask
["maca", "cão"]  →  macacão
["mac", "acão"]  →  macacão
["maj", "ánló"]  →  majánló
["mak", "eat"]  →  makeat
["make", "at"]  →  makeat
["mb", "gg"]  →  mbgg
["mbg", "g"]  →  mbgg
["ni", "ſſe"]  →  niſſe
["pe", "cabe"]  →  pecabe
["pec", "abe"]  →  pecabe
["pend", "entif"]  →  pendentif
["pendenti", "f"]  →  pendentif
["ping", "ente"]  →  pingente
["pin", "gente"]  →  pingente
["puls", "eira"]  →  pulseira
["pulse", "ira"]  →  pulseira
["r", "brakk"]  →  rbrakk
["rugu", "ay"]  →  ruguay
["sand", "alia"]  →  sandalia
["sandal", "ia"]  →  sandalia
["sand", "ália"]  →  sandália
["sprzed", "am"]  →  sprzedam
["suki", "enka"]  →  sukienka
["suk", "ienka"]  →  sukienka
["tama", "ris"]  →  tamaris
["tam", "aris"]  →  tamaris
["ta", "maris"]  →  tamaris
["ver", "wijs"]  →  verwijs
["vid", "axl"]  →  vidaxl
["vida", "xl"]  →  vidaxl
["vide", "obot"]  →  videobot
["video", "bot"]  →  videobot
["vors", "chaubild"]  →  vorschaubild
["vorschau", "bild"]  →  vorschaubild
["za", "Sy"]  →  zaSy
["zi", "yaretçi"]  →  ziyaretçi
["ésult", "ats"]  →  ésultats
["ésultat", "s"]  →  ésultats
["ô", "grafia"]  →  ôgrafia
["üş", "onlu"]  →  üşonlu
["ſ", "am"]  →  ſam
["ſa", "m"]  →  ſam
["ſ", "ammen"]  →  ſammen
["ſam", "men"]  →  ſammen
["ſ", "ch"]  →  ſch
["ſch", "aft"]  →  ſchaft
["ſ", "chaft"]  →  ſchaft
["ſch", "e"]  →  ſche
["ſ", "che"]  →  ſche
["ſche", "n"]  →  ſchen
["ſch", "en"]  →  ſchen
["ſ", "chen"]  →  ſchen
["ſ", "cher"]  →  ſcher
["ſche", "r"]  →  ſcher
["ſch", "er"]  →  ſcher
["ſe", "hen"]  →  ſehen
["ſel", "ben"]  →  ſelben
["ſel", "f"]  →  ſelf
["ſ", "elf"]  →  ſelf
["ſe", "lf"]  →  ſelf
["ſel", "ves"]  →  ſelves
["ſ", "em"]  →  ſem
["ſe", "m"]  →  ſem
["ſi", "cht"]  →  ſicht
["ſ", "icht"]  →  ſicht
["ſ", "tra"]  →  ſtra
["ſt", "ra"]  →  ſtra
["ſ", "u"]  →  ſu
["ſſ", "el"]  →  ſſel
["ſ", "ſel"]  →  ſſel
["ſſe", "l"]  →  ſſel
["ſ", "ſen"]  →  ſſen
["ſſ", "en"]  →  ſſen
["ſſe", "n"]  →  ſſen
["ſſ", "o"]  →  ſſo
["ſ", "ſo"]  →  ſſo
["ſſ", "ung"]  →  ſſung
["Бил", "галда"]  →  Билгалда
["Дере", "кк"]  →  Дерекк
["Ч", "ыгана"]  →  Чыгана
["вге", "ний"]  →  вгений
["нг", "ред"]  →  нгред
["орга", "ш"]  →  оргаш
["хар", "ш"]  →  харш
["بار", "دا"]  →  باردا
["ख", "रीदारी"]  →  खरीदारी
["हि", "ंदीखरीदारी"]  →  हिंदीखरीदारी
["คร", "ื่อง"]  →  ครื่อง
["ง", "ิน"]  →  งิน
["་", "་"]  →  ་་
["ოლი", "ო"]  →  ოლიო
["‌آم", "باردا"]  →  ‌آمباردا
["↑↑↑", "</"]  →  ↑↑↑</
["▁$", "_-"]  →  ▁$_-
["▁$_", "-"]  →  ▁$_-
["▁,", "‬"]  →  ▁,‬
["▁Accepted", "Loading"]  →  ▁AcceptedLoading
["▁An", "ſ"]  →  ▁Anſ
["▁Be", "ſ"]  →  ▁Beſ
["▁B", "eſ"]  →  ▁Beſ
["▁Con", "ſ"]  →  ▁Conſ
["▁Die", "ſe"]  →  ▁Dieſe
["▁D", "iſ"]  →  ▁Diſ
["▁Di", "ſ"]  →  ▁Diſ
["▁ERS", "ITY"]  →  ▁ERSITY
["▁Facebook", "Sign"]  →  ▁FacebookSign
["▁GO", "tt"]  →  ▁GOtt
["▁G", "Ott"]  →  ▁GOtt
["▁Ge", "iſt"]  →  ▁Geiſt
["▁G", "eſ"]  →  ▁Geſ
["▁Ge", "ſ"]  →  ▁Geſ

Files changed

  • tokenizer.json — BPE merge rules pruned, 1502 vocab entries renamed

Algorithm Details

For full details on the glitch token collection algorithm and remediation techniques,
reach out to gkielian.

Related

This is part of a series of tokenizer fixes across all Gemma model repositories
(Gemma 1, 2, 3, 4, and MedGemma — 18 models total).

cc @dougreid @ssmoot

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment