Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
Spaces:
eson
/
tokenizer-arena
like
53
Running
App
Files
Files
Community
1
367a536
tokenizer-arena
/
vocab
/
baichuan
/
demo.py
eson
add more tokenizers
f4973d4
9 months ago
raw
Copy download link
history
blame
No virus
106 Bytes
from
vocab.baichuan
import
tokenizer
id1 = tokenizer.encode(
"<pad>"
)
token1 = tokenizer.decode(
125696
)