C o VA : Context-aware Visual Attention for Webpage Information Extraction (2022)

Abstract: Webpage information extraction (WIE) is an important step to create knowledge bases. For this, classical WIE methods leverage the Document Object Model (DOM) tree of a website. However, use of the DOM tree poses significant challenges as context and appearance are encoded in an abstract manner. To address this challenge we propose to reformulate WIE as a context-aware Webpage Object Detection task. Specifically, we develop a Context-aware Visual Attention-based (CoVA) detection pipeline which combines appearance features with syntactical structure from the DOM tree. To study the approach we collect a new large-scale datase of e-commerce websites for which we manually annotate every web element with four labels: product price, product title, product image and others. On this dataset we show that the proposed CoVA approach is a new challenging baseline which improves upon prior state-of-the-art methods.

## introduction webpage information extraction (wie) is an important step when creating a large-scale knowledge base @xcite which has many downstream applications such as knowledge-aware question answering @xcite and recommendation systems @xcite . classical methods for wie, like wrapper induction @xcite @xcite , rely on the publicly available source code of websites. the code is commonly parsed into a document object model (dom) tree. the dom tree is a programming language independent tree representation of any website, which contains all its elements. it can be obtained using various libraries like puppeteer. these elements contain information about their location in the rendered webpage, styling like font size, etc., and text if it is a leaf node. state of the art method in wie @xcite ) uses text and markup information and employ cnn-bilstm encoder @xcite on the sequence of html nodes obtained from dom to learn the embedding of each node. however, using only the dom tree for wie is increasingly challenging for a variety of reasons: 1) webpages are programmed to be aesthetically pleasing; 2) oftentimes content and style is separated in website code and hence the dom tree; 3) the same visual result can be obtained in a plethora of ways; 4) branding banners and advertisements are interspersed with information of interest. for this reason, recently, wie applied optical character recognition (ocr) on rendered websites followed by word embedding-based natural language extraction @xcite . however, as mentioned before, recent webpages are highly enriched with visual content, and classical word embeddings don't capture this contextual information. for instance, text in advertising banners may be interpreted as valuable information. for this reason, a simple ocr detection followed by natural language processing techniques is a suboptimal for wie @xcite . in response to these challenges we develop wie based on a visual representation of a web element and its context. this permits to address the aforementioned four challenges. moreover, visual features are independent of the programming language (e.g., html for webpages, dart for android or ios apps) and partially also the website language (e.g., arabic, chinese, english). intuitively, we aim to mimic the ability of humans to detect the location of target elements like product price, product title and product image on a webpage in a foreign language like the one shown in fig. 1 . for this, we develop a context-aware webpage object detection (wod), which we refer to as context-aware visual attention-based detection (cova), where entities like prices are objects. somewhat differently from an object in natural images which can be detected largely based on its appearance, objects on a webpage are strongly defined by contextual information. e.g., a cat's appearance is largely independent of its nearby objects, whereas a product price is a highly ambiguous object (fig. 2 ). it refers to the price of a product only when it is contextually related to a product title and a product image. the developed wod uses a graph attention based architecture, which leverages the underlying syntactic dom tree @xcite to focus on important context @xcite while classifying an element on a webpage. once these web elements are identified, the relevant information e.g. price and title can be obtained from the corresponding dom nodes. these information can then be indexed and used for applications like product search and price comparison across online retailers. to facilitate this task we create a dataset of 7.7k english product webpage screenshots along with dom information spanning 408 different websites (domains). we compare the results of cova with existing and newly created baselines that take visual features into account. we show that cova leads to substantial improvements while yielding interpretable contextual representations. in summary, we make the following contributions: english product webpage screenshots from 408 online retailers for object detection from product webpages. our dataset is ∼ 10× larger than existing datasets. 4. we show the interpretability of cova using attention visualizations (sec. 6.5) 5. we claim and validate that visual features (without textual content) along with dom information are sufficient for many tasks while allowing cross-domain and cross-language generalizability. cova trained on english webpages perform well on chinese webpages (sec. 6.4). ## related work webpage information extraction (wie) has been mainly addressed with wrapper induction (wi). wi aims to learn a set of extraction rules from html code or text, using manually labeled examples and counter-examples @xcite @xcite . these often require human intervention which is timeconsuming, error-prone @xcite , and does not generalize to new templates. supervised learning, which treats wie as a classification task has also garnered significant attention. traditionally, natural language processing techniques are employed over html or dom information. structural and semantic features @xcite are obtained for each part of a webpage to predict categories like title, author, etc. @xcite figure 2 : example webpage showing multiple possible prices (red), but relatively fewer possible title (green) or image (purple) casts wie as a html node selection problem using features such as positions, areas, fonts, text, tags, and links. @xcite proposes a neural network to learn representation of a dom node by combining text and markup information. a cnn-bilstm encoder is employed to learn the embeddings for html node. @xcite develops a transformer architecture to learn spatial dependency between dom nodes. unlike these work which depends on text information, we aim to learn representation of a dom node using only visual cues. joshi and @xcite develop a semantic similarity between blocks of webpages using textual and dom features to extract the key article on a webpage. visual features have been extensively employed to generate visual wrappers for pattern extraction. mostly, these utilize hand-crafted visual features from a webpage, e.g., area size, font size, and type. @xcite develop a visual block tree of a webpage using visual and layout features along with the dom tree information. subsequent works use this tree for tasks like webpage segmentation, visual wrapper generation, and web record extraction @xcite @xcite . @xcite aims to develop domain-specific wrappers which generalize across unseen templates and don't need manual intervention. they develop a unified model that encodes visual, textual, and positional features using a single cnn. object detection (od) techniques in computer vision, which aims to detect and classify all objects, has been extensively studied for natural images. deep learning methods such as yolo @xcite , r-cnn variants @xcite @xcite , etc. yielded state-of-the-art results in od. od methods that can capture contextual information are of particular interest here. @xcite learn local and global context by object presence and localization and use a product of experts model @xcite to combine them. @xcite proposes a short path context module which transforms the integrated feature maps by considering local feature affinities. graph convolutional networks (gcn) (kipf and welling, 2016) was proposed to learn a node representation while taking neighbors of a node into account. using it, @xcite represent a visually rich document as a complete graph of text content obtained by passing ocr @xcite . they employ gcn to learn node representations for each web element. recently, attention mechanisms have also shown remarkable ability in capturing contextual information @xcite . @xcite propose a transformer architecture for language modeling. @xcite use attention over a bilstm-crf layer for named entity recognition (ner) on biomedical data. word vectors learned on bert @xcite , which use self-attention, have yielded state-of-the-art results on 11 nlp tasks. separately, attention has been used for contextual learning in od @xcite @xcite and image captioning @xcite . attention mechanisms have also been employed over graphs to learn an optimal representation of nodes while taking graph structure into account @xcite . moreover, attention permits to interpret result, which is often desired in many applications. we show our visualizations depicting this advantage below (sec. 6.5). ## problem formulation the dom tree captures the syntactical structure of a webpage similar to a parse tree of a natural language. our goal is to extract semantic information exploiting this syntactic structure. we view a leaf web element as a word and the webpage as a document with the dom tree as its underlying parse tree. formally, we represent a webpage w as the set @xmath0, v 2 , . . . , v i , . . . , v n , d} where v i denotes the visual representation of the i-th web element, n denotes number of web elements, and d refers to the dom tree which contains the relations between the web elements. our goal is to learn a parametric function f θ (y i |w, i) which extracts a visual representation v i of the i-th web element from website w so as to accurately predict label y i of the web element. in the following we consider four labels for a product, i.e., y @xmath1, title, image, others}. the parameters θ are obtained by minimizing the following supervised classification loss information of a webpage is present in the leaves of the dom tree, i.e., the web elements i. web elements are an atomic entity which is characterized by a rectangular bounding box. we can extract the target information y i from the dom tree if we know the exact leaf bounding boxes of the desired element. therefore, we can view wie as an object detection (od) task where objects are leaf elements and might contain the desired entity (target). however, identity y i of a web element is heavily dependent on its context, e.g., price, title, and image of a product are most likely to be in same or nearby sub-tree in comparison to unrelated web elements such as advertisements. similarly, there can be multiple instances of price-like elements. however, the correct price would be contextually positioned with product title and image (fig. 2 ). therefore, we formulate wie as a context-aware od. we use the dom tree to identify context for a web element. we represent the syntactic closeness between web elements through edges in the graph (discussed in next section). we then employ a graph attention mechanism @xcite to attend to the most important contexts. ## proposed end-to-end pipeline -cova in this section, we present our context-aware visual attention-based end-to-end pipeline for webpage object detection (cova) which aims to learn function f to predict labels @xmath2, y 2 , . . . , y n ] for a webpage. the input to cova consists of 1. a screenshot of a webpage, 2. list of bounding boxes [x, y, w, h] of the web elements, and 3. neighborhood information for each element obtained from dom. it should be noted that bounding boxes of the web elements are relatively accurate and doesn't pose challenges similar to od for natural images. the graph representation extraction computes for every web element i its set of neighboring web elements n i . the rn consists of a convolutional neural net (cnn) and a positional encoder aimed to learn a visual representation v i for each web element @xmath3, . . . , n }. the gat combines the visual representation v i of the web element i to be classified and those of its neighbors, i.e., v k ∀@xmath4. finally, the visual and contextual representations of the web element are concatenated and passed through the fc layer to obtain the classification output. we describe each of the components next. ## dataset generation to the best of our knowledge there is no large-scale dataset for wie with visual annotations for object detection. so far, the structured web data extraction (swde) dataset @xcite is the only known large dataset that can be used for training deep neural networks for wie @xcite . swde dataset contains webpage html codes which is not sufficient to render it into a screenshot (since it contains links to old and non-existent urls). because of this we create a new large-scale labeled dataset for object detection on english product webpage screenshots along with dom information. we chose e-commerce websites since those have been a de-facto standard for wie @xcite . our dataset generation consists of two steps: 1. search the web with 'shopping' keywords to aggregate diverse webpages and employ heuristics to automate labeling of product price, title, and image, 2. manual correction of incorrect labels. we discuss both steps next. ## ablation studies importance of positional features: ## conclusion & future work in this paper, we reformulated the problem of webpage ie (wie) as a context-aware webpage object detection. we created a large-scale dataset for this task and is available publicly. we proposed cova