Run tokenizer before computing perplexity and format

Replicate default cc_net preprocessing at inference time on KenlmModel.get_perplexity

Format file

Add distiluse-base-multilingual-cased-v2

Add tests and fix issue when splitting into sentences, to grab the minimum number between total sentences and sample size, rather than total original documents and sample size

Check if file exists before attempting to remove

Remove also the '{language}.sp.model' file on failure

Add CLI and refactor

Remove corrupt KenLM model files.

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.