Commit History

Sync with data tooling repo, using edugp/kenlm models, updating viz to use quantiles for coloring and ad-hoc viz for the registry dataset
3c30fa3

edugp commited on

Run tokenizer before computing perplexity and format
7b62017

edugp commited on

Add tests and fix issue when splitting into sentences, to grab the minimum number between total sentences and sample size, rather than total original documents and sample size
d131aa3

edugp commited on

Add CLI and refactor
86e673e

edugp commited on

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.
a86046b

edugp commited on