Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
Spaces:
yhavinga
/
dutch-tokenizer-arena
like
5
Running
App
Files
Files
Community
main
dutch-tokenizer-arena
2 contributors
History:
79 commits
yhavinga
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
c78da21
8 days ago
css
add compression leaderboard
13 days ago
images
update
8 months ago
js
add grok mixtral
21 days ago
patcher
add compression leaderboard
13 days ago
stats
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
utils
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
vocab
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
.gitattributes
2.32 kB
add grok mixtral
21 days ago
.gitignore
171 Bytes
update
9 months ago
README.md
25.5 kB
Hack some Dutch tokenizers into it
4 months ago
app.py
565 Bytes
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
app_compression.py
4.75 kB
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
app_playground.py
9.81 kB
update
13 days ago
config.py
1.72 kB
Add Llama tokenizer creation for Dutch, English, Code, Markdown and TeX.
8 days ago
examples.py
3.16 kB
add compression leaderboard
13 days ago
requirements.txt
96 Bytes
Pin protobuf dep
2 months ago
util.py
8.15 kB
Hack some Dutch tokenizers into it
4 months ago