MUSTIE : Multimodal Structural Transformer for Web Information Extraction (2023)

Abstract: The task of web information extraction is to extract target fields of an object from web pages, such as extracting the name, genre and actor from a movie page. Recent sequential modeling approaches have achieved state-of-the-art results on web information extraction. However, most of these methods only focus on extracting information from textual sources while ignoring the rich information from other modalities such as image and web layout. In this work, we propose a novel MUltimodal Structural Transformer (MUST) that incorporates multiple modalities for web information extraction. Concretely, we develop a structural encoder that jointly encodes the multimodal information based on the HTML structure of the web layout, where high-level DOM nodes, and low-level text and image tokens are introduced to represent the entire page. Structural attention patterns are designed to learn effective cross-modal embeddings for all DOM nodes and low-level tokens. An extensive set of experiments are conducted on WebSRC and Common Crawl benchmarks. Experimental results demonstrate the superior performance of MUST over several state-of-the-art baselines.

## introduction the world wide web has grown explosively in the past decades, with millions of new web pages being created everyday. web pages and documents have been widely used and become a powerful resource for humans to obtain information. for example, figure 1 shows a movie page from the imdb website, which contains structured movie information including movie name, description, genre, etc. this information is essential to facilitate new experiences in applications like web search and retrieval @xcite . there has been an enduring demand for automatic information extraction from unstructured figure 1 : an example of a movie page from the imdb website. the extractions of movie name, description, genre, duration, director, actor and release date are highlighted with colored bounding boxes on the web page. or semi-structured web pages to create structured knowledge bases @xcite . therefore, it is an important research problem to extract structured information from web pages @xcite . web information extraction @xcite poses a lot of challenges to researchers in both academia and industry, due to the unstructured nature and the diverse layout patterns of the web documents @xcite . moreover, web data often contains multiple modalities such as texts, tables, and images. a substantial amount of research @xcite has been proposed for automatic web information extraction, including early works of template-based extraction @xcite . however, these methods clearly do not scale up to billions of websites. deep learning models @xcite attempt to use supervisions from markup pages @xcite to build different extractors for different fields. with the recent development of natural language processing @xcite , language models have been successfully applied to web informa-tion extraction. these methods first convert the web document to a text sequence by concatenating all the text nodes @xcite or to a connected graph by using the rendered page @xcite , and then adopt sequential modeling such as lstm @xcite or attention networks @xcite to extract the target fields from the web. more recently, several multimodal language models @xcite have been proposed to extract web information from both textual and visual signals. despite achieving promising results on web information extraction, there are several major limitations for existing natural language models. first, they encode each modality of the web document independently with an individual encoder, which fails to capture the connections among different modalities, resulting in a less effective web representation. second, they do not fully encode the semi-structure html layout, which carries important knowledge about the correlations between different fields. for example, in figure 1 , the dom nodes corresponding to the movie 'name' usually appear directly after the image node in the html, while the 'release date' and 'duration' nodes are often siblings. therefore, encoding the structural html would benefit the information extraction. third, the texts and images from individual modalities are simply concatenated, making existing transformer models incapable of handling large web documents. to address these challenges, in this work, we propose a novel multimodal structural transformer (namely must), which incorporates multiple modalities for web information extraction. in particular, we design a multimodal encoder with a structural attention mechanism to jointly encode all the dom nodes from multiple modalities, and learn the cross-modal embeddings for them. intuitively, must leverages the web layout structure that naturally connects dom nodes from all modalities for more effective attention weight computation. the information of the target fields is then extracted from the learned node embeddings. we conduct evaluations of our model on websrc and common crawl benchmarks, and show the superior performance of must over several state-of-the-art methods. the experimental results also demonstrate the effectiveness of the structural attention in modeling web documents with multimodal data. the main contributions are summarized as follows: web information extraction early works in web information extraction are wrapper induction methods @xcite , which construct templates by learning the desired patterns from the web documents. several deep learning methods @xcite are proposed to extract or classify a text node to a set of fields using its textual and visual features, e.g., classify whether a text node is the 'name' field. with the recent advancement in natural language processing (nlp) @xcite , an increasing number of language models @xcite @xcite have been developed for web information extraction. these methods can be further divided into three main groups. the first group contains the sequential modeling approaches @xcite , which construct a text sequence by concatenating all the text nodes from the web and performing the extraction. form2seq @xcite designs a seq-toseq model with an rnn. webformer @xcite ) merges all the text nodes from the html and trains a model with hierarchical attention. the second group includes the graph learning models @xcite , which treat the web document as a graph connecting multiple rendered components and directly learn the web representation on the graph. formnet @xcite generates a structure-aware graph from the rendered web document and uses the graph convolutional network (gcn) for obtaining the node embeddings. the third group consists of the multimodal methods @xcite @xcite , which learn to extract field information from both textual and visual clues on the web. layoutlmv2 @xcite adopts a two-stream multimodal transformer encoder to model the interaction among text and image. structure and efficient transformers our work is also related to those transformer models @xcite @xcite ) that focus on efficiently encoding structure and large sequences. etc @xcite and longformer @xcite describe a method to use a global memory with a relative attention pattern @xcite to represent the structure text input. transformer xl @xcite develops an approach to encode long text sequences beyond a fixed size. hibert @xcite uses hierarchical attention on the equally divided input blocks. random sparse attention is utilized in bigbird @xcite to reduce the quadratic computations to linear time. these methods achieve promising results in dealing with structure and large input. however, they cannot be directly applied to encode html layout with multiple modalities.