Commit ·
ea0f998
1
Parent(s): e474a77
Add tokenizer from openai/gpt-oss-20b
Browse files- Copy tokenizer files from openai/gpt-oss-20b
- Vocab size: 199,998
- Files tracked with Git LFS
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- special_tokens_map.json +3 -0
- tokenizer.json +3 -0
- tokenizer_config.json +3 -0
special_tokens_map.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8464cabd6eda239fe46ebf8ae63b46c417721784a961a022f6b59174a2cda0e2
|
| 3 |
+
size 440
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3
|
| 3 |
+
size 27868174
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9279e942392b742d633c7adbb89ebe002c98399db8926a7af5125c726f404070
|
| 3 |
+
size 4200
|