Super-squash branch 'main' using huggingface_hub

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+language:
+- en
+- ru
+- zh
+- de
+- ja
+- es
+- fr
+- it
+- pt
+- pl
+- nl
+- id
+- tr
+- cs
+- vi
+- sv
+- fa
+- ar
+- el
+- da
+- hu
+---
+
+# FineWeb-HQ-Classifiers
+
+This repository contains the model weights of the trained deep learning classifiers used to identify structured and knowledge-rich samples for the [FineWeb-HQ](https://huggingface.co/datasets/epfml/FineWeb-HQ) and [FineWeb2-HQ](https://huggingface.co/datasets/epfml/FineWeb2-HQ) datasets.
+
+FineWeb-HQ and FineWeb2-HQ are high-quality, model-filtered, multilingual pretraining datasets derived as subsets of [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2). The datasets were created by selecting the top 10% of FineWeb and FineWeb-2 documents. The classifiers use [XLM-RoBERTa](https://huggingface.co/FacebookAI/xlm-roberta-base) embeddings to score the documents.
+
+For more details, see our paper [Enhancing Multilingual LLM Pretraining with Model-Based Data Selection](https://arxiv.org/abs/2502.10361).
+
+## Quickstart
+
+Classifiers use a simple architecture that takes mean-pooled XLM-RoBERTa embeddings as input and outputs the score logit.
+
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoModel, AutoTokenizer
+import huggingface_hub
+
+
+class BinaryClassifier(torch.nn.Module):
+    def __init__(self, embedding_dim=768, hidden_dim=256):
+        super(BinaryClassifier, self).__init__()
+        self.classifier = torch.nn.Sequential(
+            torch.nn.Linear(embedding_dim, hidden_dim),
+            torch.nn.ReLU(),
+            torch.nn.Dropout(0.2),
+            torch.nn.Linear(hidden_dim, 1),
+        )
+
+    def forward(self, X):
+        return self.classifier(X)
+
+    def to_pt(self, file_name):
+        torch.save(self.state_dict(), file_name)
+
+    def from_pt(file_name, embedding_dim=768, hidden_dim=256):
+        state_dict = torch.load(
+            file_name,
+            weights_only=True,
+            map_location=torch.device("cpu"),
+        )
+        classifier = BinaryClassifier(
+            embedding_dim=embedding_dim,
+            hidden_dim=hidden_dim,
+        )
+        classifier.load_state_dict(state_dict)
+        classifier.eval()
+        return classifier
+
+
+if __name__ == "__main__":
+    embedding_model_name = "FacebookAI/xlm-roberta-base"
+    tokenizer = AutoTokenizer.from_pretrained(embedding_model_name)
+    embedding_model = AutoModel.from_pretrained(
+        embedding_model_name,
+        dtype=torch.bfloat16,
+    )
+
+    classifiers_dir = huggingface_hub.snapshot_download("epfml/FineWeb-HQ-Classifiers")
+    classifier_model_en = BinaryClassifier.from_pt(f"{classifiers_dir}/eng_Latn.pt")
+
+    def score_sample(text, classifier_model):
+        inputs = tokenizer([text], return_tensors="pt")
+        embeddings = embedding_model(**inputs).last_hidden_state.float().mean(1)
+        score = F.sigmoid(classifier_model(embeddings)).item()
+        return score
+
+    text = "Question: How is bipolar disorder different from unipolar depression or 'regular' depression?\nAnswer: Both bipolar disorder and major depression are typically associated with depressive episodes. So both illnesses are accompanied by depressions. The difference is that in bipolar disorder people also have periods of elevation -- or severe irritability. We call these manic or hypomanic episodes."
+    score = score_sample(text, classifier_model_en)
+    print(f"{score:0.4f}")  # 0.9338
+
+    text = "Custom Wedding Gifts\nPersonalized photo frames, albums & keepsakes. Heirloom quality!\nCustom Engraved Journals\nHandmade in Florence Italy. Dozens of sizes and paper styles!"
+    score = score_sample(text, classifier_model_en)
+    print(f"{score:0.4f}")  # 0.0001
+```
+
+For a more efficient implementation, see our [Github repository](https://github.com/epfml/fineweb2-hq). 
+
+## Citation information
+```
+@article
+{messmer2025multilingdatacomp,
+title={Enhancing Multilingual LLM Pretraining with Model-Based Data Selection},
+author={Bettina Messmer and Vinko Sabolčec and Martin Jaggi},
+journal={arXiv},
+year={2025},
+url={https://arxiv.org/abs/2502.10361},
+}
+```

arb_Arab.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:587a82928fa3681cdd3e12db7617aad566af6ecccf540bb8d945d784593b2ea5
+size 790528

ces_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f8d2fcf9e38b86e965de907a83c0aafae80d1f272890b4b3f3f84a0078c5568
+size 790528

cmn_Hani.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2cec7c131c0fc3a4a8140a19e9cc0b8623254ac5a4678fd04ccaeff11a28f05
+size 790528

dan_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:00852caa2c2f4fff806827124b3b1a492d22108886b51f76ba2c78a20d95eaf3
+size 790528

deu_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c46fe8847d6608e95989d8dd45004d6be4ad0b504476bc8624042ebef473fc75
+size 790528

ell_Grek.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40f3117d4a6a0c133a24d18837560de22427e4db915c864f580c47bc2402f66a
+size 790528

eng_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b4cd4c8edf0dde329d8856ee4d0c0704767c44120613adabc00b106d8f7c5f0
+size 790528

fas_Arab.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53c4db06fe17bbdd80bbae6ebcd71d401e54453636f9c3535f627a9a60a77ce7
+size 790528

fra_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f426bf679283af98c907758069557961c1b5b54d254d34f06899ac88d2ca888
+size 790528

hun_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c15195f8edc341cb2caf1d0147b895a789609306c7c6024862ae4e34b2bf6d0
+size 790528

ind_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fedf1f41af5ef5a7855501d07c0d6f62ec2b999af32506250b3ff209cb72be46
+size 790528

ita_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a844d93000de757aafea035b8586ea7e5b5e009aae9c6ac532b4dec756a990d3
+size 790528

jpn_Jpan.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7396d86ae1a55544baf66babd5a2eebdab20e543d9d0886f8e77a093d3173161
+size 790528

nld_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62278bf608fa2a47ecd8f5f308d3af0343919595d2ceef15f64c6ee8e9e9ff03
+size 790528

pol_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ee5e285a1b7d5b21ac2485d3d0fbfcb02f5e368b44f0f056cfcb276d205f746
+size 790528

por_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e9d2e6cf1a3a12961caec2c8b209df13f5d354a63d3c4b9f958c6ede7dca6ef
+size 790528

rus_Cyrl.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0463c126ce77198352f6cb1129c36c9782cad82011189c36bc6364cacb96e6d
+size 790528

spa_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70551bb9135847f2c2ee60e778ff7c8bba5928207a811bc8b50651ec0ee40e00
+size 790528

swe_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e3b21ea9d86176a16e4860906fb79b89ee7f41e602b687d48af249e5d2d6bde
+size 790528

tur_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3472898cc521a25eaac304e4bec14f7bae77b692f9b388cb97517ba80d9b2f62
+size 790528

vie_Latn.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7c4edbc9505b8d2f86a7297541e50701f3ba48fc930fdc4c6e0870ed9930235
+size 790528