Commit History
delete unused models
34eca0f
Merge branch 'main' of https://huggingface.co/spaces/huggingface/text-data-filtering
f6058aa
back to before portuguese
091dbe4
Update app.py
c4be503
Update app.py
72d8e96
update visu for Portuguese
2b811ac
7 languages supported
ea01f38
new kenlm models
b37a555
add register information
061d2e4
new filter on word repetition ratio
4809033
visualization: small step for the slider on flagged words ratio
fa81556
visualization: choose between several languages
0610f9d
fix bug
0319ee2
distributions for the filters on words and discarded words by filter
da13b29
visualization: upload our own stop words and flagged words list
5d56c36
quick fix
1bc0c1e
everything in expanders
2c2527f
display distributions in sidebar and filtering parameters in expanders
5d485e5
rename badwords to flagged words + new flagged words list of 68 words
f217a73
button to download parameters
bfbcd60
add warning message
649ea6a
better visualization
8f0da78
fix division by 0 in compute_special_characters_ratio
b607b76
new tool to analyse our own doc
6f25c5c
fix requirements
d463071
fix packages
924da6e
test
d1e3e7b
correction of bug
22701ae
delete app_2
c340078
merge
6ddbf7d
filter on repetition removal
693f997
Update app.py
189d6aa
chinese visu
611e98e
Delete en_examples_with_stats_no_small_docs.json
58d483d
Delete en_examples_with_stats_ldnoob.json
b190ef8
Delete en_examples_with_stats.json
0376199
remove zipf's law and update of the doc
3fd19c1
visu with discarded documents by filter
14574d7
new visu
6303415
faster visu (less documents)
07c617e
new visu
0add2d4
test
a547ccb
Create LICENSE
eabb5f9
Teven Le Scao
commited on
