Hugging Face
Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up
versae
/
scandeng-tokenizer
like
0
Text Generation
Transformers
mistral
License:
apache-2.0
Model card
Files
Files and versions
Community
Train
Deploy
Use this model
main
scandeng-tokenizer
Ctrl+K
Ctrl+K
1 contributor
History:
4 commits
versae
Add HF tokenizer converted from SentencePiece
04b8476
12 months ago
.gitattributes
Safe
1.52 kB
initial commit
12 months ago
README.md
Safe
28 Bytes
initial commit
12 months ago
config.json
Safe
595 Bytes
Create config.json
12 months ago
convert.sh
Safe
1.1 kB
Add HF tokenizer converted from SentencePiece
12 months ago
merges.txt
Safe
568 kB
Add HF tokenizer converted from SentencePiece
12 months ago
sentencepiece.model
Safe
770 kB
LFS
SentencePiece byte fallback 32k
12 months ago
sentencepiece.vocab.bak
Safe
486 kB
Add HF tokenizer converted from SentencePiece
12 months ago
special_tokens_map.json
Safe
72 Bytes
Add HF tokenizer converted from SentencePiece
12 months ago
tokenizer.json
Safe
1.85 MB
Add HF tokenizer converted from SentencePiece
12 months ago
tokenizer_config.json
Safe
826 Bytes
Add HF tokenizer converted from SentencePiece
12 months ago
vocab.json
Safe
687 kB
Add HF tokenizer converted from SentencePiece
12 months ago
