Cleaner Pretraining Corpus Curation with Neural Web Scraping (2024)

Abstract: The web contains large-scale, diverse, and abundant information to satisfy the information-seeking needs of humans. Through meticulous data collection, preprocessing, and curation, webpages can be used as a fundamental data resource for language model pretraining. However, when confronted with the progressively revolutionized and intricate nature of webpages, rule-based/feature-based web scrapers are becoming increasingly inadequate. This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement, demonstrating its potential in extracting higher-quality data to facilitate the language model pretraining. All of the code is available at https://github.com/OpenMatch/NeuScraper.

## introduction large language models (llms) have shown impressive performance in various nlp tasks as the size of models scaling up @xcite @xcite . however, recent findings in scaling laws indicate that both model size and training data should be scaled proportionally @xcite , posing a significant challenge in acquiring sufficiently large pretraining datasets or even raising concerns about data scarcity @xcite . to curate more data for pretraining, researchers pay more attention to collecting more valuable data from the web. the web-crawled datasets, such as commoncrawl, have been widely used for pretraining, facilitating the development of language models @xcite @xcite * indicates corresponding author. @xcite . nevertheless, prior research has demonstrated that, even after aggressive cleaning, the quality of pre-extracted text provided by commoncrawl still fails to reach the expected @xcite @xcite . the reason lies in that advertisements, banners, hyperlinks, and other harmful content are usually mixed within the primary content of the page, thereby only extracting these primary contents brings lots of noise to pretraining @xcite . web scrapers provide opportunities to extract valuable content from the raw html pages @xcite . however, rule-based and heuristic scrapers have notable limitations. on the one hand, web pages are becoming increasingly sophisticated, requiring more intricate underlying code to deal with the page layout @xcite . in this case, maintaining the scraper rules is timeconsuming and requires much human effort. on the other hand, html functions as a markup language, enabling web designers to customize web pages according to individual preferences. consequently, web pages frequently lack complete standardization, which may mislead the rule-based web scrapers @xcite . in this paper, we present a simple, fast, and effective neural web scraper (neuscraper) designed to extract primary content from webpages. neuscraper employs a shallow neural architecture and integrates layout information for efficient scraping. our experiments demonstrate that neuscraper surpasses baseline scrapers, achieving a 20% improvement in performance and generating a higher-quality corpus for language model pretraining. notably, neuscraper shows the potential of high processing speeds when utilized on gpu. the easy-to-use and open-source tool, neuscraper, can facilitate the creation of largescale corpora for pretraining. ## related work leveraging web scrapers for extraction provides a promising way to extract high-quality content from webpages. such a web scraping task is usually defined as text extraction, boilerplate removal, template removal, or generic web extraction in different webpage processing pipelines @xcite @xcite , which is distinguished from the web information extraction task that extracts the entities from webpages @xcite . the web scrapers can be divided into rule-based and feature-based methods. rule-based web scrapers start from web wrappers, which often need manual designs or a wrapper induction system for producing @xcite . the web wrappers usually need to be tailored for each webpage, which is not feasible to process large-scale webpages @xcite . a more conventional approach is to create a document object model (dom) tree, which assists in building a rule-based scraper @xcite or help the comparison of webpages @xcite . additionally, the work also incorporates tag cumulative distributions @xcite ), text density @xcite , and tag ratios @xcite to benefit the content extraction from webpages. except for these rule-based methods, some scrapers use feature-based approaches to better extract the primary contents from webpages. specifically, they divide the webpage into several blocks using rules built based on the html tags or dom tree. then they extract dozens to hundreds of handcrafted features from these blocks, such as markup, text/document features @xcite , linguistic, structural & visual features @xcite and dom tree-based features @xcite . these features can be fed into svm @xcite , conditional random fields @xcite , logistic regressions (peters and lecocq, 2013) or convolutional neural network @xcite to classify whether the texts in the block are the primary content of the webpages. ## neural web scraper this section introduces our neural web scraper (neuscraper) to extract primary contents from webpages. we first introduce the sequence modeling method of webpages (sec. 3.1) and then describe our neural-based web scraper (sec. 3.2). ## experimental methodology in this section, we describe the datasets, baselines, evaluation metrics and implementation details. dataset. we use clueweb22 @xcite baseline. the scraping baselines consist of nine open-sourced web scrapers, including basic html manipulators (html2text and inscriptis @xcite ), generic webpage parsers (beautifulsoup4, lxml and htmlparser), rule-based scrapers (justext and readability) and machine learning-based scraper (boilerpipe @xcite ). trafilatura @xcite is our main baseline, which combines different rules and heuristic methods. evaluation metrics. the accuracy, precision, recall, and f1 score, are used to evaluate the effectiveness in extracting primary contents. furthermore, we use different scrapers to produce the web corpus and pretrain language models. the quality of scraping can be demonstrated by the results of standard downstream tasks. ## evaluation result in this section, we first show the effectiveness of different scrapers in extracting primary contents from the raw webpages. subsequently, we evaluate the quality of the extracted data and utilize it to pretrain language models of varying scales. ## conlusion this paper proposes neuscraper, which employs a shallow neural architecture to clean the webpages. the experimental results show the effectiveness of neuscraper. the open-sourced and easy-used web scraper may facilitate the research on language model pretraining.