🏥 medical-bpe-pubmed
A byte-level BPE tokenizer trained on 45,000 PubMed abstracts.
Vocabulary size: 16,000 tokens.
Built as part of a teaching lab to demonstrate that domain training, not vocab size, drives tokenization efficiency on in-domain text.
📊 Held-out fertility results
Evaluated on 5,000 PubMed abstracts never seen during training (lower = better):
| Tokenizer | Vocab | Medical fertility | General fertility |
|---|---|---|---|
| medical-bpe-pubmed (this model) | 16k | 1.375 ✅ | 1.575 ❌ |
| o200k_base (GPT-4o) | ~200k | 1.430 | 1.163 ✅ |
| cl100k_base (GPT-4) | ~100k | 1.460 | 1.173 |
| general-bpe (wikitext) | 16k | 1.747 ❌ | 1.208 |
general-bpe and this tokenizer have identical algorithm and identical vocab size.
The only variable is training domain → domain is the cause of the 0.372 fertility gap.
⚡ Usage
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("paulvision/medical-bpe-pubmed")
text = "Acetylcholinesterase inhibitors are used in Alzheimer's disease treatment."
tokens = tok.tokenize(text)
print(tokens)
# custom-med: ~fewer pieces for clinical terms vs cl100k
🏋️ Training details
| Detail | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Training corpus | 45,000 PubMed abstracts (slinusc/PubMedAbstractsSubset) |
| Vocab size | 16,000 |
| Held-out eval | 5,000 disjoint PubMed abstracts (seed-42 split) |
| Pre-tokenizer | ByteLevel (no unknown tokens possible) |
| Special tokens | <pad>, <|endoftext|> |
⚠️ Important — pretrained model trap
This tokenizer cannot be swapped onto a pretrained LLM (LLaMA, Qwen, etc.) without retraining from scratch.
Token IDs must match embedding rows. Swapping produces a scrambled model.
Use this tokenizer for:
- ✅ Training a new medical LM from scratch
- ✅ Corpus analysis and fertility measurement
- ✅ Teaching / research on domain tokenization
- ❌ LoRA on top of a pretrained LLM (use the original tokenizer +
add_tokens()instead)
🔗 Lab repository
tokenization-explainer — full lab with notebook, theory docs, and Mermaid diagrams.