patrickvonplaten commited on
Commit
0aecafc
1 Parent(s): d139d98

remove added tokens

Browse files
tokenizer/added_tokens.json DELETED
@@ -1,4 +0,0 @@
1
- {
2
- "<|endoftext|>": 49407,
3
- "<|startoftext|>": 49406
4
- }
 
 
 
 
tokenizer_2/added_tokens.json DELETED
@@ -1,5 +0,0 @@
1
- {
2
- "!": 0,
3
- "<|endoftext|>": 49407,
4
- "<|startoftext|>": 49406
5
- }