add tokenizer

Files changed (4) hide show

merges.txt CHANGED Viewed

The diff for this file is too large to render. See raw diff

special_tokens_map.json CHANGED Viewed

@@ -1,6 +1,5 @@
 {
   "bos_token": "<|endoftext|>",
   "eos_token": "<|endoftext|>",
-  "pad_token": "<|endoftext|>",
   "unk_token": "<|endoftext|>"
 }

 {
   "bos_token": "<|endoftext|>",
   "eos_token": "<|endoftext|>",
   "unk_token": "<|endoftext|>"
 }

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

vocab.json CHANGED Viewed

The diff for this file is too large to render. See raw diff