YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
HPT v2 — Tokenizer
Brazilian AI Tokenizer — SentencePiece BPE 32K
About
Bilingual (Portuguese + English) + Code tokenizer for HPT v2 (118M params).
Trained on:
- 143 MB Portuguese Wikipedia (20K articles)
- 191 MB English Wikipedia (35K articles)
- Code samples (Python, JavaScript, SQL, Bash)
Special Tokens
| Token | ID | Usage |
|---|---|---|
<pad> |
0 | Padding |
<s> |
1 | BOS |
</s> |
2 | EOS |
<unk> |
3 | Unknown |
| `< | user | >` |
| `< | assistant | >` |
| `< | system | >` |
| `< | thinking | >` |
| `< | /thinking | >` |
Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("tokenizer/hpt_tokenizer.model")
# Encode
ids = sp.EncodeAsIds("Oi, tudo bem?")
# Decode
text = sp.DecodeIds(ids)
Stats
- Vocab size: 32,000
- Training corpus: 334 MB (PT + EN + Code)
- Character coverage: 99.95%
- Byte fallback: enabled (no UNK ever)
NeuralSpark Labs 🇧🇷
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support