migtissera
/

Tess-v2.5-Phi-3-medium-128k-14B

Text Generation

Generated from Trainer

Inference Endpoints

Model card Files Files and versions Community

migtissera commited on 8 days ago

Commit

3a4dbce

•

1 Parent(s): fb84573

Update tokenizer_config.json

Files changed (1) hide show

tokenizer_config.json +2 -2

tokenizer_config.json CHANGED Viewed

@@ -25,7 +25,7 @@
       "normalized": false,
       "rstrip": true,
       "single_word": false,
-      "special": false
     },
     "32000": {
       "content": "<|endoftext|>",
@@ -121,7 +121,7 @@
       "normalized": false,
       "rstrip": false,
       "single_word": false,
-      "special": false
     },
     "32012": {
       "content": "<|end_of_text|>",

       "normalized": false,
       "rstrip": true,
       "single_word": false,
+      "special": true
     },
     "32000": {
       "content": "<|endoftext|>",
       "normalized": false,
       "rstrip": false,
       "single_word": false,
+      "special": true
     },
     "32012": {
       "content": "<|end_of_text|>",