sayakpaul's picture
sayakpaul HF staff
Create README.md (#1)
2ea9793
|
raw
history blame contribute delete
No virus
245 Bytes

This is a Unigram tokenizer trained on the Wikitext dataset. Refer to the train_unigram.py script within this repository to know how it was trained.