Hugging Face's logo

loubnabnl
/
tokenizer-pile

Model card Files Community
loubnabnl HF staff commited on
Commit
df3a55b
•
1 Parent(s): ca4c7d1

Upload tokenizer

Browse files
Files changed (4) hide show
  1. merges.txt +0 -0
  2. tokenizer.json +0 -0
  3. tokenizer_config.json +0 -1
  4. vocab.json +0 -0
merges.txt CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -4,7 +4,6 @@
4
  "<|endoftext|>"
5
  ],
6
  "bos_token": "<|endoftext|>",
7
- "clean_up_tokenization_spaces": true,
8
  "eos_token": "<|endoftext|>",
9
  "model_max_length": 1000000000000000019884624838656,
10
  "tokenizer_class": "GPT2Tokenizer",
 
4
  "<|endoftext|>"
5
  ],
6
  "bos_token": "<|endoftext|>",
 
7
  "eos_token": "<|endoftext|>",
8
  "model_max_length": 1000000000000000019884624838656,
9
  "tokenizer_class": "GPT2Tokenizer",
vocab.json CHANGED
The diff for this file is too large to render. See raw diff