File size: 843 Bytes
69f376e 0ffc363 69f376e cd8b714 69f376e cd8b714 |
1 2 3 4 5 6 7 8 9 10 11 |
---
license: mit
---
Fasttext model used for filtering in [DataComp-LM](https://arxiv.org/abs/2406.11794) to produce [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0).
The model classifies between `__label__hq` and `__label__cc` which correspond to "high-quality" (i.e., OH2.5 and Reddit ELI5 data) and "low-quality" (i.e., web-crawled data from Common Crawl) respectively. We use the score given to `__label__hq` to filter our documents via a percentile-based threshold.
See our [dclm](https://github.com/mlfoundations/dclm/tree/main/baselines#fasttext-filtering) repo for documentation about how we applied to to filter data in our experiments.
See [fasttext documentation](https://fasttext.cc/docs/en/python-module.html) for general documentation on fasttext classifiers and how to use them with python.
|