Try using a KenLM model trie binary version, if not found try using the hash table binary version

Sync with data tooling repo, using edugp/kenlm models, updating viz to use quantiles for coloring and ad-hoc viz for the registry dataset

Run tokenizer before computing perplexity and format

Replicate default cc_net preprocessing at inference time on KenlmModel.get_perplexity

Check if file exists before attempting to remove

Remove also the '{language}.sp.model' file on failure

Remove corrupt KenLM model files.

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.