Commit
•
be6ca24
1
Parent(s):
ce1cc1f
Update README.md
Browse files
README.md
CHANGED
@@ -1,6 +1,8 @@
|
|
1 |
# Pile of Law Tokenizer
|
2 |
|
3 |
-
This tokenizer should be a drop-in replacement for the GPT2Tokenizer. It has the same
|
|
|
|
|
4 |
|
5 |
Usage:
|
6 |
|
|
|
1 |
# Pile of Law Tokenizer
|
2 |
|
3 |
+
This tokenizer should be a drop-in replacement for the GPT2Tokenizer. It has the same special tokens, but was trained on a random 1M samples from [the pile of law](https://huggingface.co/datasets/pile-of-law/pile-of-law) train split.
|
4 |
+
|
5 |
+
It has exactly 52,000 tokens, which is not identical to GPT2.
|
6 |
|
7 |
Usage:
|
8 |
|