Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction (2021)

Abstract: An essential operation in web corpus construction consists in retaining the desired content while discarding the rest. Another challenge finding one’s way through websites. This article introduces a text discovery and extraction tool published under open-source license. Its installation and use is straightforward, notably from Python and on the command-line. The software allows for main text, comments and metadata extraction, while also providing building blocks for web crawling tasks. A comparative evaluation on real-world data also shows its interest as well as the performance of other available solutions. The contributions of this paper are threefold: it references the software, features a benchmark, and provides a meaningful baseline for similar tasks. The tool performs significantly better than other open-source solutions in this evaluation and in external benchmarks.

## introduction 1.gathering texts from the web as useful monolingual text corpora across languages are highly relevant for the nlp community @xcite , web corpora seem to be a natural way to gather language data. corpus construction usually involves "crawling, downloading, 'cleaning' and de-duplicating the data, then linguistically annotating it and loading it into a corpus query tool" @xcite . however, although text is ubiquitous on the web, drawing accurate information from web pages can be difficult. in addition, the vastly increasing variety of corpora, text types and use cases makes it more and more difficult to assess the usefulness and appropriateness of certain web texts for given research objectives. as a result, content adequacy, focus and quality need to be evaluated after the downloads @xcite . a significant challenge lies in the ability to extract and pre-process web data to meet scientific expectations with respect to text quality. an essential operation in corpus construction consists in retaining the desired content while discarding the rest, a task carrying various names referring to specific subtasks or to pre-processing as a whole: web scraping, boilerplate removal, web page segmentation, web page cleaning, template extraction, or content extraction. this step is sometimes overlooked although it involves a series of design decisions and turning points in data processing. depending on the purpose of data collection, adequate filtering and quality assessment can be crucial. it has a significant impact on a wide range of downstream applications like text analysis, information retrieval, link analysis, page adaptation to other terminals and screens, and especially natural language processing pipelines. another challenge is how to find one's way through the web, notably as linguistic data are gathered by running targeted web crawlers @xcite . as web crawling involves discarding much of the downloaded content @xcite , especially link filtering and prioritization can prove to be tricky for contexts in which data collection is just the first step of a project, so that time resources for this task are scarce. data collection approaches using the commoncrawl foot_0 have flourished as they allow for faster download and processing by skipping (or more precisely outsourcing) the crawling phase. barring the fact that finding one's "own" way through the web can be preferable, such data should not be used without forethought and exhaustive filtering. beside the discovery of relevant websites, a major issue consists in selecting appropriate content after download and processing @xcite , which can be com-123 plex due to unexpected machine-generated flaws and biases. finally, depending on the project's jurisdiction, legal aspects of retrieving and granting access to web documents can be unclear or restrictive. boundaries of copyright law are not clear when it comes to corpus building @xcite so that some corpus infrastructure projects leave it to users to decide what to do from a copyright standpoint @xcite . copyright and intellectual property rights usually do not apply to resources such as language models or n-grams @xcite , so are shuffled sentences @xcite . web corpora focusing on manually selected sources under creative commons licenses have been built @xcite , although only a very small proportion of websites use them @xcite . corpora based on machine-checked licenses have also been developed @xcite , as well as systems to merge annotation with web parts from the commoncrawl @xcite . considering the progresses of annotation tools, is can be easier to retrieve documents directly from the web or from archives and to process them to one's taste. ## conclusions and outlook the variety of contexts and text genres leads to important design decisions impacting web corpora: could and should the tooling be adapted to particular sources that are targeted or should the extraction be as generic as possible to provide opportunistic ways of gathering information? due to corpus size or limited resources, the second option is often best. the software package introduced here can help facilitate text data collection and enhance corpus quality. it can answer two research questions related to web corpus construction: how can an accessible generic extraction be run on web pages? and how can text content be found given a list of websites? in the evaluation, trafilatura performs significantly better than other open-source solutions, which is corroborated by external benchmarks. the article also provided a fast and meaningful baseline which can be used in similar extraction tasks. most scraping tools are developed considering particular page styles, whereas linguistic and geographic factors are most probably reflected in html structure diversity. in addition, different eras of web development result in diverging "htm-lects". these discrepancies deeply affect extraction processes and can lead to diverging performances. trafilatura tries to mitigate these biases but cannot bridge all potential gaps. while some large-scale natural language processing and language modeling algorithms can be expected to smooth out irregularities to a certain extent, uses requiring a low margin of error and close reading approaches can greatly benefit from refinements during construction and processing of corpora. as this tool has been released under an open-source license and field-tested by users, feedback loops and collaborative work will hopefully be carried on and foster further improvements. although the extraction parameters are configurable, recall-and precision-oriented settings will be made available to make major extraction settings more convenient. presets corresponding to different usage scenarios could be developed. comment extraction still has to be evaluated although most libraries do not offer this functionality. forthcoming additions include refinements of navigation functions, notably further work on a spider in order to be able to derive links from websites which do not provide sitemaps or web feeds.