Commit
•
07fc755
1
Parent(s):
eb84e5f
change endoftext value to len of tokenizer vocab size
Browse files- added_tokens.json +1 -1
added_tokens.json
CHANGED
@@ -1 +1 @@
|
|
1 |
-
{"<|endoftext|>":
|
|
|
1 |
+
{"<|endoftext|>": 50257}
|