LLM -Based Web Data Collection for Research Dataset Creation (2025)

Abstract: Researchers across many fields rely on web data to gain new insights and validate methods. However, assembling accurate and comprehensive datasets typically requires manual review of numerous web pages to identify and record only those data points relevant to specific research objectives. The vast and scattered nature of online information makes this process time-consuming and prone to human error. To address these challenges, we present a human-in-the-loop framework that automates web-scale data collection end-to-end using large language models (LLMs). Given a textual description of a target dataset, our framework (1) automatically formulates search engine queries, (2) navigates the web to identify relevant web pages, (3) extracts the data points of interest, and (4) performs quality control to produce a structured, research-ready dataset. Importantly, users remain in the loop throughout the process and can inspect and adjust the framework’s decisions to ensure alignment with their needs. We introduce techniques to mitigate both search engine bias and LLM hallucinations during data extraction. Experiments across three diverse data collection tasks show that our framework greatly outperforms existing methods, while a user evaluation demonstrates its practical utility. We release our code at https://github.com/tberkane/web-data-collection to help other researchers create custom datasets more efficiently.

## introduction web data underpin research across many fields-including political science, economics, and public health-by enabling both method validation and new discoveries. researchers typically depend on datasets previously collected by others, made publicly available by institutions, or assembled manually through web searches and information extraction. for example, news articles are valuable resources for reconstructing event timelines, tracking public attention, and compiling summary statistics. the data points contained in such articles become particularly useful when aggregated into datasets and validated prior to release. however, manual collection is time-consuming and risks missing relevant web pages or data points. for instance, the johns hopkins university covid-19 dashboard @xcite , which reported global case and death counts, illustrates this challenge: data were initially collected manually from news and social media sources, but the process soon proved unsustainable and required automation. in recent years, major social media platforms have restricted data access for both the public and for researchers. previously, social media data were often openly available through application programming interfaces (apis), providing invaluable resources for studying socio-behavioral phenomena. notable recent examples of access limitations include the shutdown of meta's crowdtangle platform in august 2024 @xcite and severe re-strictions to twitter's (coalition for independent technology research, 2023) and reddit's (reddit @xcite apis in january and april 2023, respectively. these limitations have increased the need for researchers to independently collect data, often under significant time and budget constraints. to address these challenges, we present a framework for end-to-end, web-scale data collection that leverages recent advances in large language models (llms). our method begins with a user-provided textual description of a target dataset, searches for relevant web pages and pdf documents, selectively extracts data, and produces a structured dataset. the framework is designed for human-in-the-loop interaction, allowing users to continuously monitor and adjust the data collection process to ensure alignment with their objectives. it maintains transparency by making each extracted data point traceable to its source via direct quotation. further, we implement corrections for recency and geographical biases introduced by search engines and mitigate llm hallucinations through source grounding. a final quality control step automatically flags potentially anomalous data points, such as outliers and duplicates, for manual review. we evaluate the effectiveness of our framework on three diverse data collection tasks. for each, we construct a reference dataset via manual search and aggregation of results from multiple methods. we benchmark our framework against state-of-theart tools, including @xcite and perplexity deep research @xcite , and observe substantial performance gains: for example, on the first data collection task, our framework improves f1-score by 71.3 and 37.6 percentage points compared to perplexity deep research and chatgpt deep research, respectively. further, we conduct an ablation study demonstrating the effectiveness of our quality control step as well as our bias and hallucination mitigation techniques. finally, a user evaluation illustrates the practical value of our framework, showcasing both its strengths and real-world limitations. ## methodology figure 1 shows the steps of our framework. ## results and analysis the relatively poor performance of chatgpt search-especially in terms of recall and f1score-partly reflects the llm's limited context window. because only a small set of search results (e.g., a dozen web pages) can be handled, the llm cannot aggregate information across the many sources needed for web-scale data collection. in the climate events task, our framework's precision is comparatively lower because several earthquake events were incorrectly attributed to cameroon. these errors stemmed from a source website that erroneously listed earthquakes as occurring in cameroon, which our framework did not flag. yet manual review later identified these as misclassifications, highlighting our method's sensitivity to misleading sources and reinforcing the need for human validation of extracted data. unlike the fully autonomous deep research baselines, our framework is designed for human oversight throughout the data collection process-including the removal of duplicates and the resolution of flagged issues. this additional user input affords greater control and typically yields higher-quality datasets that are better aligned with 12615 research objectives. because our system casts a wider net, it also retrieves more duplicate data points, especially for major events. although these duplicates can facilitate cross-source corroboration, they also increase the need for post-processing. in contrast, the deep research baselines retrieve fewer duplicates, thus reducing manual curation. yet user review is still required to identify occasional errors or hallucinations. the main cost of our framework owes to the use of the anthropic api for data extraction; other steps-such as query and schema generation, pdfto-markdown conversion, and quality control-add negligible expense. on average, extracting data from a single web page costs approximately @xmath00.14 per data point (@xmath10.13 per data point (@xmath20.14 per data point ($6.12 in total). ## conclusion this paper presents a framework to automate the creation of high-quality research datasets via largescale web data collection. by combining llms with a human-in-the-loop, our method enables efficient extraction of structured data while mitigating various biases that can affect data collection. our experimental results and user evaluation highlight several key strengths of the framework. the template-based search query system, along with efficient re-ranking and filtering methods, make it feasible to collect data at scale across diverse domains. grounding, bias mitigation, and quality control steps help ensure the integrity and transparency of the resulting datasets-attributes that are crucial for scientific research.