jaswanthrk
/

tokenizer_BPE_32k_vocab_all_data

Model card Files Files and versions Community

jaswanthrk commited on Jan 23

Commit

d6656b9

•

1 Parent(s): 641a0ba

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +3 -3

tokenizer.json CHANGED Viewed

@@ -135,9 +135,9 @@
         "fuse_unk": false,
         "byte_fallback": false,
         "vocab": {
-            "[UNK]": 0,
-            "[CLS]": 1,
-            "[SEP]": 2,
             "[PAD]": 3,
             "[MASK]": 4,
             "\u0000": 5,

         "fuse_unk": false,
         "byte_fallback": false,
         "vocab": {
+            "<unk>": 0,
+            "<s>": 1,
+            "</s>": 2,
             "[PAD]": 3,
             "[MASK]": 4,
             "\u0000": 5,