The Future of Web Data Mining: Insights from Multimodal and Code-based Extraction Methods (2024)

Abstract: The extraction of structured data from websites is critical for numerous Artificial Intelligence applications, but modern web design increasingly stores information visually in images rather than in text. This shift calls into question the optimal technique, as language-only models fail without textual cues while new multimodal models like GPT-4 promise image understanding abilities. We conduct the first rigorous comparison between text-based and vision-based models for extracting event metadata harvested from comic convention websites. Surprisingly, our results between GPT-4 Vision and GPT-4 Text uncover a significant accuracy advantage for vision-based methods in an applies-to-apples setting, indicating that vision models may be outpacing language-alone techniques in the task of information extraction from websites. We release our dataset and provide a qualitative analysis to guide further research in multi-modal models for web information extraction.

## introduction the extraction of structured information from websites represents a critical challenge in the field of artificial intelligence (ai), especially in the context of rapidly evolving web technologies. as the virtual world becomes increasingly central to diverse aspects of society, the ability to efficiently and accurately mine web data is of high importance. this task, commonly known as web scraping, entails navigating the complexities of varied website architectures to extract useful information. the ubiquity of dynamic, visually-rich, and interactive content in modern web design further complicates this landscape, presenting a formidable challenge for automated data extraction technologies. historically, web scraping has been dominated by rule-based systems @xcite ) @xcite , meticulously designed to accommodate the specific structures of individual websites. the inherent diversity in web design necessitates a tailored approach for each site, significantly limiting the scalability of these systems. moreover, the dynamic nature of web content, where a single page may present different types of data based on user interaction or other factors such as location or time, adds another layer of complexity. because of the bespoke nature, rulebased systems often struggle to adapt to dynamic elements, often requiring manual intervention for maintenance and updates. in the realm of machine learning (ml), the application to web scraping presents unique challenges. the vast differences between websites render the tuning of existing ml systems a daunting task. in most cases, ml-based scraping methods must operate in a zero-shot or few-shot setting, where the model has little to no prior exposure to the specific website from which data is to be extracted. this scenario places a heavy reliance on the innate capabilities of the model to generalize across highly varied environments, a task that has traditionally proven to be challenging for ml systems. as a result, these methods have often been less effective than their rule-based counterparts. the advent of advanced multimodal ai models has signaled a potential paradigm shift in web scraping methodologies. pioneering models such as @xcite and llava @xcite have demonstrated remarkable capabilities in dealing with complex, multimodal data. these models are equipped to understand and interpret information that spans across text, images, and other web elements, offering a more holistic approach to data extraction. their prowess in zero-shot performance, where the model can generate useful responses without prior specific training on a task, suggests a significant potential for application in web scraping. despite these advancements, the field lacks a comprehensive and rigorous analysis of such multimodel ai models in the context of extracting practical web data. this gap in research motivates our current study, where we aim to critically evaluate and compare the effectiveness of these cutting-edge techniques in web scraping. our contributions are as follows: ## related works information extraction from websites has traditionally relied on processing raw html code and other text-based structures. @xcite presents a dataset of html code with well-defined tasks. for example, on a webpage that describes a book, the dataset asks a system to retrieve the title, author, isbn-13, publisher, and publish-date using the html. both @xcite and dom-lm @xcite aim to simplify the dom tree and feed simplified text embeddings to dense models, achieving state-of-theart results on benchmarks. more recently, large language models (llms) have been used to either directly extract information from website html, or to generate a python program to extract the information from the html @xcite . they found this method, @xcite , outpaces methods directly using roberta @xcite to answer questions, a zero-shot relation extraction method @xcite and dom-lm @xcite . however, these languageonly approaches are intrinsically limited when data is stored visually. research on pairing vision and language capabilities together in a single model has made rapid progress in interpreting images with text, with models like gpt-4 demonstrating excellent text extraction capabilities from structured documents (ope-nai, 2023), even establishing a new state-of-the-art on the text visual question answering (textvqa) dataset @xcite , a dataset designed to challenge model's ability to reason with images. research is rapid and prolific in multimodal modeling, including the recent work of the multilingual pali @xcite ) and the modular system of mplug-2 @xcite for multimodal question answering (qa). the dataset by @xcite features hand-labeled news articles in raw html format, focusing on identifying critical article components like titles and publication dates. similarly, the klarna product page dataset @xcite ) contains 51,701 annotated product sale pages for locating key web elements such as buy buttons and prices. additionally, the boilerplate detection using shallow text features dataset @xcite includes html files labeled to distinguish main content from extraneous elements like advertisements, thus aiding in refining web scraping accuracy. none of the aforementioned datasets provide the ability to compare purely text based and multimodal models on event information extraction. ## conclusion and future work in this work, we carry out the first rigorous comparison on practical website data showing strengths of emerging visual approaches versus enduring precision of code for harvesting event details. our evaluations reveal superior performance in visualbased methods with unparalleled adaptability on designs with heavy imagery. as visual richness accelerates across the web, combining modalities will likely further outpace language-only methods and overcome the shortcomings from unimodal methodologies by blending state-of-the-art coding reasoning with cross-format graphical resilience. furthermore, we release our dataset to facilitate additional development in event information extraction. more broadly, our findings posit the understanding of complex webpage images as an important frontier with tangible value for structured data min-ing from online resources. gpt-4 vision proves supreme through an average exact match of 85% and fuzzy matching rates of 95% and 95% for name and location data, respectively. we provide strong evidence that rather than competing, effectively integrating textual and visual cues can pave the way for next-generation techniques to achieve new levels of reliability in real-world information extraction across the full diversity of modern web experiences -establishing multimodal web comprehension as a critical area for cross-disciplinary ai development moving forward. our future work includes expanding this analysis to a wide range of other datasets, including swde and the klarna product pages.