Commit
•
be6ca24
1
Parent(s):
ce1cc1f
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
# Pile of Law Tokenizer
|
| 2 |
|
| 3 |
-
This tokenizer should be a drop-in replacement for the GPT2Tokenizer. It has the same
|
|
|
|
|
|
|
| 4 |
|
| 5 |
Usage:
|
| 6 |
|
|
|
|
| 1 |
# Pile of Law Tokenizer
|
| 2 |
|
| 3 |
+
This tokenizer should be a drop-in replacement for the GPT2Tokenizer. It has the same special tokens, but was trained on a random 1M samples from [the pile of law](https://huggingface.co/datasets/pile-of-law/pile-of-law) train split.
|
| 4 |
+
|
| 5 |
+
It has exactly 52,000 tokens, which is not identical to GPT2.
|
| 6 |
|
| 7 |
Usage:
|
| 8 |
|