W eb SRC : A Dataset for Web-Based Structural Reading Comprehension (2021)

Abstract: Web search is an essential way for humans to obtain information, but it’s still a great challenge for machines to understand the contents of web pages. In this paper, we introduce the task of web-based structural reading comprehension. Given a web page and a question about it, the task is to find an answer from the web page. This task requires a system not only to understand the semantics of texts but also the structure of the web page. Moreover, we proposed WebSRC, a novel Web-based Structural Reading Comprehension dataset. WebSRC consists of 400K question-answer pairs, which are collected from 6.4K web pages with corresponding HTML source code, screenshots, and metadata. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no. We evaluate various strong baselines on our dataset to show the difficulty of our task. We also investigate the usefulness of structural information and visual features. Our dataset and baselines have been publicly available.

## introduction web pages are the most common source of human knowledge and daily information. with the help of modern search engines, people can easily locate web pages and find information by simply typing some keywords. however, traditional search engines only retrieve web pages related to the query and highlight the possible answers @xcite , they can't understand the web pages and answer the query based on contents. the rapid development of question answering systems and knowledge graphs enables search engines to answer simple questions directly @xcite , but they still each html tag begins with a starting tag and ends with a closing tag (with a slash in tag). <@xmath0. (c) shows the text extracted from the web page. (d) contains some sample questions for the web page. there are two kinds of structures for each web page: spatial structure and logical structure. the spatial structure is how the information is visually organized, and the logical structure is how the information is organized by semantics. figure 1(a) shows the spatial structure of the web page, e.g., how the texts are arranged and what are their relative positions. the logical structure can be deduced by the spatial structure and the semantics of the texts. for example, this image introduces the infor-datasets #domain #website task #query with image weir @xcite 4 40 closedie 32 no swde @xcite 8 80 closedie 32 no expanded swde @xcite mation about two cars, with car names at the top followed by the detailed specifications. a human can easily answer the questions in figure 1(d ) by referring to the relevant section in the logical structure. but for computers, it's hard to understand the logical structure by just taking the spatial structure (the image) as input due to the lack of common sense. computers need to infer the answers from the font size, the color and the spatial relations between texts, let alone they need to extract texts from the image and understand them. an alternative way is to utilize the text from web page. figure 1(c ) shows the texts extracted from figure 1 (a). as we can see, the layout structure is lost in the plain text, and the text is just a concatenation of short phrases without a meaningful context. it would be difficult to answer questions only based on such texts. besides texts, we can also parse the html (hypertext markup language) document, i.e. the source code of the web page. it describes the structure of the webs page and uses html elements (tags) to display the contents. we will use the term tag and element in this paper interchangeably. figure 1(b) shows the html code corresponding to the part of the web page highlighted in the red box. html is a kind of semi-structured document @xcite , where tags with different structural semantics serve as separators. it's also called the "self-describing" structure. an html document can be parsed into a tree-like structure called dom 2 (document object model), where the tree nodes are elements in the html, and texts are all leaf nodes in the tree. an html dom tree can serve as a structural representation of the web page, where visually similar items on the web page would be sub-trees with similar structures. for example in figure 1 , the html structure is identical for the segment in the blue and red boxes. they are only different in the text. however, due to the complexity of rendering html code into a web page, a single html would not be enough to rep-2 https://en.wikipedia.org/wiki/document_object_model resent the full logic structure of the web page. for example, in figure 1(b) , the @xmath1, but they play different semantic roles in the web page, i.e. the first span indicates an attribute and the second contains the corresponding value. we need to leverage both the visual and structural information to gain a comprehensive understanding. to promote researches in question answering on web pages, we introduce websrc, a dataset for reading comprehension on structural web pages. the task is to answer questions about web pages, which requires a system to have a comprehensive understanding of the spatial structure and logical structure. websrc consists of 6.4k web pages and 400k question-answer pairs about web pages. for each web page, we manually chose one segment from it and saved the corresponding html code, screenshot, and metadata like positions and sizes. questions in websrc were created for each segment. answers are either text spans from web pages or yes/no. taking the html code, screenshot, metadata as well as question as input, a model is to predict the answer from the web page. the comparison of websrc with other datasets with html documents is illustrated in to summarize, our contributions are as follows: ## related work machine reading comprehension (mrc) models have achieved excellent performance on plain text corpus @xcite in recent years. traditional datasets for machine reading comprehension @xcite @xcite @xcite @xcite contain plain text passages and qas about them. however, html code in the form of semi-structured documents is different from the ordinary textual corpus. recently, multi-modal mrc has gained the interest of researchers. multi-modal mrc datasets with both images and texts are proposed, such as movieqa @xcite , tqa @xcite , comics @xcite and recipeqa @xcite . images in these datasets provide different information from texts, and texts are supplementary descriptions for images. text vqa @xcite @xcite is a kind of vqa (visual question answering) task @xcite , whose task is to answer questions about a realworld image, and questions in this task are about the texts in the image. however, there is no existing text or layout description available in the image, but we can access them easily on web pages. information extraction for web pages has been investigated intensively @xcite . previous studies mainly focus on building templates for html dom tree, called wrapper induction @xcite @xcite , or using well designed visual features like font sizes, element sizes, and positions @xcite . these methods require abundant human labor to label templates and analyze features, which makes it hard to generalize to unseen websites. @xcite proposed a program synthesis based technique to extract web information. some studies focused on recognizing tables from web pages @xcite and tried to model the physical and logical structure of tables in html, @xcite proposed to use a graph to represent the table structure. some web qa datasets are proposed @xcite @xcite , but they only contain text snippets extracted from web pages. @xcite proposed a dataset called weir, consisting of 40 websites from 4 domains. @xcite proposed swde, which contains 124,291 web pages from 80 websites, and @xcite expanded swde for openie. all these datasets only contain html code for extraction, and the task is to extract pre-defined attributes of entities in web pages, e.g. the author of a book. layout analysis @xcite is the task to analyze document images like contracts, bills, and business emails. iit-cdip @xcite and rvl-cdip @xcite are two datasets collected for document classification. @xcite proposed funsd for form understanding and @xcite organized sroie competition for receipt understanding. publaynet @xcite and docbank @xcite are proposed to benchmark the task of layout recognition in academic papers. however, compared to the images in the layout analysis task, web pages are much more complex in organizing information. the terms to be recognized are relatively stable in layout analysis, while web pages may contain various information that is hard to be pre-defined. ## data collection the construction of our dataset consists of five stages: § 3.3 web page selection, § 3.4 web page collection, § 3.5 question labeling, § 3.6 data augmentation and § 3.7 final review. we will describe each stage in detail below. ## dataset analysis in this section, we conduct throughout analysis of websrc. we only show some major results here and for more statistics please refer to appendix a. the statistics of different types of websites are shown in ## baseline models we propose three baseline models for websrc. they take different kinds of context into consideration. we describe these models in detail below. ## experiments 6.1 dataset splits we manually divide our dataset into train/dev/test sets at the website level, where the training set contains 50 websites, dev and test contain 10 websites respectively. both the dev set and the test set have all three types of websites and have a similar distribution of website types. the detailed statistics of each set are shown in ## conclusion in this paper, we introduce websrc, a multi-modal dataset for web-based structural reading comprehension with both html documents and screenshots. the task is to answer questions about the web pages. we evaluate several baselines on our dataset, and the results showed that incorporating layout features with textual contents is crucial to web understanding, but how to utilize such structural information requires further investigation. we hope this work can push the research on web-based structural reading comprehension forward. in the future, we will go beyond web pages to explore more structural reading comprehension tasks.