Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions (2024)

Abstract: This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work.The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.

## introduction and motivation online news articles are a favored data source for a wide-ranging set of nlp applications including social/political analysis @xcite @xcite , market prediction @xcite , and are used as training data for language models @xcite . in such projects, it is often the first step to compile a corpus of news articles to analyze. this requires (1) identifying the urls of news articles belonging to a particular set of online newspapers for download, and (2) extracting the article content from the surrounding html so that only the full article text remains. oppenheimer wins oscars by max mustermann oppenheimer wins 7 oscars in 2024, including best picture. christopher nolan's blockbuster "oppenheimer" swept the academy awards in 2024, winning awards such as best picture, best actor and best director. nolan's first oscar this marks director christopher nolan's first [..] tags: entertainment, oscars 2024, ... plaintext title authors body.summary body.paragraph body.subheadline body.paragraph topics figure 1 : an example article scraped by fundus. next to the plain text of the article, attributes such as title, authors, paragraphs, subheadlines and topics are directly accessible. in particular the second task of content extraction -also referred to as web scraping or boilerplate removal @xcite -is known to be challenging since each online newspaper uses different html and text formatting guidelines. this makes it non-trivial to distinguish between article content and other elements such as adverts, unrelated asides, captions, etc. to address these issues, several libraries have been developed that streamline the crawling and content extraction of online newspapers @xcite @xcite . limitations. however, existing libraries rely on generic methods for content extraction, based either on heuristics or trained machine learning models. this allows them to be applied across an arbitrary number of online newspapers, but comes at a cost of extraction accuracy: the quality of the news article texts varies depending on how well the heuristics or learned rules capture the html formatting of a particular newspaper. for instance, the evaluation presented in this paper shows that existing frameworks encounter difficulties with at least one newspaper, resulting in f1-scores below 60% for all articles retrieved from this source. this means that, due to their generic nature, existing libraries provide no guarantee and no means to ensure that scraped articles are textually complete and without artifacts. put more plainly, it may be argued that existing libraries prioritize quantity (i.e. scaling across many newspapers) over quality (i.e. high-quality extraction of complete article texts and metaattributes). this may cause problems in use cases in which the overall quality of a news corpus is more important than its quantity @xcite . contributions. with this paper, we present fun-dus, a news crawling library in which we pursue an orthogonal approach to prior work. rather than aiming for a set of general rules applicable to all newspapers, our library uses separate, manually created html content extractors -referred to as parsers within the library -for each online newspaper. this allows us to match extraction methodologies specifically to a newspaper and thus manually optimize the accuracy of text extraction. further, as figure 1 illustrates, this enables us to write more complex content extractors compared to prior work to preserve a news article's structure (distinguishing between paragraphs, subheadlines, and the article summary), and extract meta-attributes such as topics. in more detail, our contributions are: we find that fundus outperforms all other libraries in terms of yielding complete and artifactfree text (see ## related work ## fundus we introduce fundus with a usage example, discuss our article and publisher-based logic, and illustrate how we distinguish between forward and backward crawling. ## evaluation we comparatively evaluate fundus against prominent scraping libraries. our goals are to (1) determine the quality of our bespoke content extraction approach compared to the generic approaches of prior work, and to ( 2 ) better understand the data potential of fundus, i.e. to estimate the size of news corpora that fundus can create. ## conclusion and outlook we presented fundus, an easy-to-use news scraper built on the idea of bespoke content extractors for supported online newspapers. our evaluation shows that our approach successfully optimizes for quality, indicating that fundus is a viable option for use cases in which data quality is a priority. further, we combine both crawling and scraping functionality in a single pipeline, and support access to the static web archive cc-news. with fundus' open-source approach, we invite the community to contribute support for additional online newspapers. to assist in this process, we plan to investigate semi-automatic methods to suggest extraction rules in future work.