A uto S craper: A Progressive Understanding Web Agent for Web Scraper Generation (2024)

Abstract: Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website, while language agents, empowered by large language models (LLMs), exhibit poor reusability in diverse web environments. In this work, we introduce the paradigm of generating web scrapers with LLMs and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently. AutoScraper leverages the hierarchical structure of HTML and similarity across different web pages for generating web scrapers. Besides, we propose a new executability metric for better measuring the performance of web scraper generation tasks. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Our work is now open-source.

## introduction web scraping is a process where software automates the extraction of data from websites, typically using bots or web scrapers to gather specific information @xcite . it is important because it allows for efficient data collection and aggregation, which can be crucial for market research, competitive analysis, and real-time data monitoring. due to the diversity of sources and information on the internet, the construction of a web scraper requires substantial human effort. consequently, two types of methods for automatic web information acquisition have been proposed, categorized as wrapper-based and language-agent-based (sarkhel † corresponding authors. 1 resources of this paper can be found at https:// github.com/ez-hwh/autoscraper q1: what's the age of stephen curry? et al., 2023). the wrapper-based method entails complex sequences of operations within customized rule-based functions, which are designed to efficiently access and retrieve desired data from websites, which is especially beneficial for structured websites with stable layouts @xcite @xcite . conversely, the language-agent-based method leverages powerful natural language processing capabilities of large language models (llms) to interpret free-text queries and directly extract data within websites to meet the demand, effectively handling both structured and dynamic web content @xcite . although both types of methods facilitate web scraping to varying degrees, as shown in figure 1 , they exhibit significant shortcomings in terms of scalability. wrapper-based method, while reusable, struggles with entirely new website structures, which necessitates extensive human effort to develop additional customized functions @xcite . conversely, although language-agent-based methods demonstrate superior performance in adapting to new content, their reliance on a limited number of superpowerful api-based llms for web scraping incurs considerable time and financial costs. together, these challenges impede the broader adoption and scalability of current web scraping technologies, limiting their practicality in dynamic and diverse web environments. to address the shortcomings of the aforementioned two paradigms, the paradigm of generating web scrapers with llms would be the optimal solution. on one hand, compared to wrapper-based methods, it fully leverages the reasoning and reflection capacities of llms, reducing manual design on new tasks and enhancing scalability. on the other hand, compared to language-agent-based methods, it introduces repeatable extraction procedures, reducing the dependency on llms when dealing with similar tasks, and thereby improving efficiency when handling a large number of web tasks. however, there are several challenges associated with using llms to generate web scrapers: ## related work wrapper-based methods for web scraping utilize the hierarchical structure of the webpage. method of this category includes rule-based @xcite , learning wrappers (i.e a dom-specific parser that can extract content) @xcite @xcite , heuristic algorithm @xcite and deep learning neural network @xcite @xcite . these methods demand substantial human involvement, including creating wrapper annotations, applying heuristic scoring rules (such as visual proximity), crafting features for neural network input, and using prior knowledge for verification. therefore, it is difficult for wrapper-based methods to automatically scale up when facing web scraping tasks across a large number of different websites. with the emergence of powerful llms (ope-nai, 2023; @xcite , language agents (sumers et al., 2023) now operate in interactive environments, leveraging llm-based reasoning, grounding, learning, and decisionmaking. general language agents, such as chainof-thought @xcite , reflexion @xcite , self-refine @xcite , and self-debug (chen et al., 2023), capitalize on llms' self-reflection capabilities for iterative planning optimization. however, these agents do not effectively utilize web structural features and fail to simplify the web environment after unsuccessful planning attempts, limiting the optimization of subsequent planning. current language agents primarily aim to streamline the web environment @xcite @xcite and develop strategies for planning and interacting with the web @xcite . nevertheless, these frameworks mainly focus on the concept of open-world web simulation environments @xcite @xcite , which encompass a broad spectrum of tasks found in real-life scenarios, such as online shopping, flight booking, and software development. these task scenarios are oriented towards individuals and have significantly different requirements for accuracy and efficiency compared to web scraping. as a result, current language-agent-based methods cannot effectively exploit the html structural similarities across multiple web pages, reducing their dependency on llms when performing repetitive operations and leading to inefficiencies. ## preliminaries in this section, we first define the scraper generation task and then present the dataset collection process and its corresponding evaluation metrics. ## autoscraper in this section, we describe our framework au-toscraper for generating a scraper to extract specific information from semi-structured html. our approach is divided into two phases: first, we adopt a progressive generation module that utilizes the hierarchical structure of web pages; second, we employ a synthesis module based on results from multiple web pages. the overall framework is presented in figure 2 . ## experiment intending to put autoscraper to practical use, we investigate the following research questions: 1) can autoscraper outperform the state-of-theart scraper generation methods? 2) how does au-toscraper framework improve the performance of the scraper generation task? 3) does auto-scraper meet the requirements for web scraping tasks, specifically being accurate and efficient? ## discussion in this section, we will discuss other aspects of autoscraper, including its comparison with existing website information extraction methods, efficiency analysis of autoscraper, and the limitations of the current approach. ## conclusion in this paper, we introduce the scraper generation task and the paradigm that combines llms and scrapers to improve the reusability of the current language-agent-based framework. we then propose autoscraper , a two-phase framework including progressive generation and synthesis module to generate a more stable and executable action sequence. our comprehensive experiments demonstrate that autoscraper can outperform the state-of-the-art baseline in the scraper generation task.