ontocord/riverbed

These are basic classifiers and a BM25 index of Wikipedia used for data tooling research. Using kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1's classifier (MIT) and TurkuNLP's register classifiers.

import fasttext
if not os.path.exists("expert_classify.ftz"):
    os.system("wget http://dl.turkunlp.org/register-labeling-model/fasttext_model.bin")
    os.system("wget https://huggingface.co/ontocord/riverbed/resolve/main/rj_model.bin")
    os.system("wget https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifer-v1/resolve/main/model_textbook_quality.bin"
    os.system("wget https://huggingface.co/ontocord/riverbed/resolve/main/expert_classify.ftz")

### red pajama filter. pred_label "__label__wiki" is data we do not wish to keep.
red_pajama_model = fasttext.load_model("rj_model.bin")
(pred_label, pred_prob) = red_pajama_model.predict(text)
if pred_label == "__label__cc":
     pred_prob = 1 - pred_prob


### turkunlp registry labeler: https://github.com/TurkuNLP/register-labeling
domain_model = fasttext.load_model("fasttext_model.bin")
(pred_label, pred_prob) = domain_model.predict(text)

### Pile domain such as github, arxiv, etc.
pile_model = fasttext.load_model("expert_classify.ftz")
(pred_label, pred_prob) = pile_model.predict(text)

### Textbook quality - e.g., textbooks are all you need
textbook_model = fasttext.load_model("model_textbook_quality.bin")
(pred_label, pred_prob) = pile_model.predict(text)

See the files here: https://huggingface.co/ontocord/riverbed/tree/main