tau-vision
/

llama-3-tokenizer-fix

cwaud commited on May 16

Commit

ea70931

•

1 Parent(s): 374ecbe

Update tokenizer_config.json

Files changed (1) hide show

tokenizer_config.json CHANGED Viewed

@@ -9,7 +9,7 @@
       "special": true
     },
     "128001": {
-      "content": "<|eot_id|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,

       "special": true
     },
     "128001": {
+      "content": "<|end_of_text|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,