stevenmaschan/imdb-tokenizers

IMDb tokenisers

Word-level and BPE tokenisers for English movie reviews, trained with the Hugging Face tokenizers library on the 50,000 unlabelled reviews (the unsupervised split) of stanfordnlp/imdb. They were made for a university NLP project comparing sentiment classifiers on IMDb.

file	model	vocabulary
word_vs_32000_mf_2.json	whole words; other words become [UNK]	32,000
bpe_vs_8000_mf_2.json	BPE (pieces inside a word start with ##)	8,000
bpe_vs_16000_mf_2.json	BPE	16,000
bpe_vs_32000_mf_2.json	BPE	32,000
bpe_vs_64000_mf_2.json	BPE	64,000

mf_2: tokens (word-level) or merges (BPE) seen fewer than 2 times were not added to the vocabulary.

Usage

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file(hf_hub_download("stevenmaschan/imdb-tokenizers", "bpe_vs_32000_mf_2.json"))
tokenizer.encode("I didn't like it!!!!!").tokens
# ['i', 'did', "n't", 'like', 'it', '!', '!']

Text cleaning

The cleaning is part of each tokeniser, so raw reviews can be passed in directly:

1. HTML line breaks (<br />) become spaces.
2. Characters broken by a Windows-1252 decoding error (\x96, \x97, \x85, \x91) become -, ... and '; runs of two or more hyphens become -; curly (’) and acute (´) apostrophes become '.
3. Unicode NFKC normalisation; runs of !, ? and . are capped at !!, ?? and ....
4. Lowercasing.
5. Splitting into words and punctuation, with clitics split off: didn't → did n't, kubrick's → kubrick 's.