Update web.py

web.py

@@ -432,7 +432,7 @@ def web_data():
         P("We directly read WARC files instead of WET files and extracted text using Trafilatura. Similar to RefinedWeb, we avoid using Machine Learning (ML)-based metrics for filtering documents to prevent bias introduced by ML models. Importantly, we apply global deduplication across the entire dataset, whereas previous works only use local deduplication. Note that although The Pile also employed global deduplication on its web data (Pile-CC), this accounted for just 0.6\% of 74 snapshots."),
 
         Details(
-                Summary("Open Me - WARC TEST"),
+                Summary("Text Extraction Examples"),
                 DV2("data/sample_wet.json", "data/sample_warc.json", 3),
             ),
         #DV2("data/sample_wet.json", "data/sample_warc.json", 3),
@@ -442,16 +442,16 @@ def web_data():
         After text extraction, the non-English texts are then filtered out by fastText language identifier with a threshold of 0.65.
         This step removes over 60% of the whole data.
         """),
-        DV(
-            "data/sample_non_en.json",
-            3,
-            "Sample documents that are classified as non-English",
-        ),
-        DV(
-            "data/sample_en_low.json",
-            3,
-            "Sample documents that are classified as English but with score less than 0.65",
-        ),
+        Details(
+                Summary("Sample documents that are classified as non-English"),
+                DV("data/sample_non_en.json", 3),
+            ),
+        
+        Details(
+                Summary("Sample documents that are classified as English but with score less than 0.65"),
+                DV("data/sample_en_low.json",3),
+            ),  
+        
         H4("1.3 URL Filtering"),
         P("""
         Following RefinedWeb [3], we use a manually inspected URL blocklist to filter fraudulent and/or adult websites.
@@ -463,6 +463,7 @@ def web_data():
         articles, sex education, technical blogs, etc. Specifically, we randomly took 903M URLs and matched them with
         4.6M domain names in the UT1 blocklist. 24 URL domains were detected with more than 4k matches, which are shown below.
         """),
+        
         DVS(urls_high_matches, "24 URL domains with more than 4k matches"),
         P("""
         We manually removed the following 6 domains from the UT1 blocklist so that they will not be removed from our dataset.
@@ -481,11 +482,11 @@ def web_data():
             non_web_urls,
             "curated url domains that are excluded from our dataset",
         ),
-        DV(
-            "data/sample_url_exclusion.json",
-            0,
-            "Sample documents whose urls are in our curated url domain list",
-        ),
+        
+        Details(
+                Summary("Sample documents whose urls are in our curated url domain list"),
+                DV("data/sample_url_exclusion.json", 0,),  
+            ),
         H3("2. Line-Level Removal"),
         P("""
         Before computing the quality signals that can be used for filtering low-quality documents, we perform the line-level