A Graph Representation of Semi-structured Data for Web Question Answering (2020)

Abstract: The abundant semi-structured data on the Web, such as HTML-based tables and lists, provide commercial search engines a rich information source for question answering (QA). Different from plain text passages in Web documents, Web tables and lists have inherent structures, which carry semantic correlations among various elements in tables and lists. Many existing studies treat tables and lists as flat documents with pieces of text and do not make good use of semantic information hidden in structures. In this paper, we propose a novel graph representation of Web tables and lists based on a systematic categorization of the components in semi-structured data as well as their relations. We also develop pre-training and reasoning techniques on the graph model for the QA task. Extensive experiments on several real datasets collected from a commercial engine verify the effectiveness of our approach. Our method improves F1 score by 3.90 points over the state-of-the-art baselines.

## introduction question answering (qa) has become an important feature in most search engines as it delivers information to users in an effective and easy-to-understand manner. answers to questions are often extracted from web tables and lists. for example, figure 1 shows the search result page (serp) of questions q 1 "cities with the highest gdp in the world" and q 2 "the best skydiving locations in the world", where the answer to q 1 is from a web table, while that to q 2 is from a web list. comparing to unstructured plain text, semi-structured web data, such as web tables and lists, are more effective to represent rich relational information. relations among various elements in a web table or a list may be useful in answering user questions. according to the statistics from a global commercial search engine, there are hundreds of millions of semi-structured data pieces, including tables and lists on the web, and the intents of 30% of user queries can be answered by semi-structured data. previous attempts towards question answering (qa) using semi-structured data on the web are mainly ir-based approaches @xcite . typically, those methods convert semi-structured data into documents by sequentially rearranging text cells to adapt to language models @xcite @xcite . those studies do not make use of inherent structural relationships among components of web tables or lists. for example, the rearrangement does not consider the vertical relations among cells locating in the same columns, such as the relation among "new york", "tokyo" and "los angeles" in figure 1(a) . some recent studies leverage tabular structure implicitly. for example, @xcite cast tables as matrices of text and apply convolutional neural networks for table embedding. @xcite cut tables into smaller fragments. however, the structural information in web tables and lists is more complex than the simple adjacency relation of matrices. how to take the best advantage of both the text † equal contribution. work done when the first author was an intern at microsoft stca. ‡ jian pei's research is supported in part by the nserc discovery grant program. all opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies. in this paper, we tackle the problem of web qa over semi-structured data, and make the following contributions. first, in section 2, we systematically categorize different components in semi-structured web data, including captions, headers, subject columns, attribute columns, and cells, as well as their relations, including cell-cell relation, header-cell relation, subject-attribute relation, and caption-content relation. we propose grasslm, a graph model to jointly represent both text and structural information in semi-structured data for web qa in section 3. our grasslm model explicitly represents different types of components as nodes in a graph and their relations as edges. our model integrates heterogeneous information effectively, including text and structures, and reveals hidden semantic correlations across various components naturally. second, in section 3, we apply two pre-training techniques for graph models. in particular, we design a novel node prediction objective (npo) to leverage graph structure in node embedding. this pre-training task requires a model to predict the entire content for a masked node from the unmasked neighbor nodes, which guides the model to learn attention over the context. the attention is used in the graph reasoning stage, where the representation of each node is updated by the aggregated information from the neighbor nodes. in this way, the inherent semantic correlations among neighbor nodes are propagated via the structural connections in the graph. comparing to the previous methods, this graph pre-training and reasoning mechanism better exploits structural information in tables and lists. last, to compare our model with the state-of-the-art methods, we create new datasets which contain real-world queries collected from a global commercial search engine paired with table and list data mined from the web. each pair query, table/list is further labeled by crowdsourcing annotators with consensus on relevance. the experimental results on the test sets, reported in section 4, show that grasslm outperforms the best state-of-the-art baselines up to 1.77 in f1 score on average. ## problem statement let s be a semi-structured data example, either a web table t or a web list l. there are different types of tables and lists, for example, relational tables, entity tables, matrix tables, enumerate lists and group lists @xcite . our method is generally applicable to all those types, therefore, we do not distinguish them in this paper. 2.1 components of web semi-structured data following crestan and pantel (2011) and eberius et al. (2015), we divide a web semi-structured data example into various components, as illustrated in figure 2 . a caption c is a direct description that is usually adjunct to the content body of the semi-structured data. for example, in figure 1 "top 10 cities by projected gdp" and "30 jaw dropping places for skydiving in the world!" are captions for the table and the list, respectively. data content refers to the body of semi-structured data, which consists of multiple rows and columns. a special row, the header, often locates at the top of the table. the elements in a header often describe the classes that the content of the table belongs to. for example, in figure 2 , the first row consisting of "rank", "city", and "country" is the header of the table. the elements in the remaining rows of the table are cells. vertically, cells are grouped into columns, where we identify subject columns and attribute columns. subject columns refer to one or more key subjects or entities of the table, while attribute columns list the attribute information of the corresponding subjects or entities. in figure 2 , the column of "city" is a subject column, while the columns of 'rank" and "country" are both attribute columns. to recognize subject columns, we adopt a heuristic method @xcite , which calculates the distinct string ratio as seeds for subject classification. our empirical study finds that this simple method achieves an accuracy over 95%. besides, a schema classification method @xcite is applied to detect and transpose vertical web tables into horizontal ones. a list can be regarded as a special type of table, which has only one single column, and has no header. ## method in this section, we proposed grasslm, which is a graph model of semi-structured data on the web for qa. figure 3 shows the overall structure of grasslm. grasslm is composed of three components. first, a pre-trained language model generates token-level contextual embedding for the concatenation of an input query q and a semi-structured data example s. second, a graph construction module converts the initial plain text embedding into graphs. last, a graph reasoning and classification module predicts matching results. ## experiments we evaluate the grasslm model and other baselines on three datasets, including one table qa dataset, one list qa dataset and one small dataset of complex query-table pairs. in addition, we leverage two other large-scale datasets for pre-training. we describe the three datasets as follows. for pre-training of grasslm, we leverage the following semi-structured datasets. table pre-training dataset (table-pretrain) is a large-scale unlabelled web table dataset, which is extracted from 10 million web pages sampled from the index of one commercial search engine. the sampling is conducted from the set of pages seen by united state users in the period from feb 2019 to dec 2019. after filtering out incomplete or empty web tables, we extract 7m web tables. we compare grasslm with several strong baselines. single-field document retrieval (sdr) @xcite and multi-field document retrieval (mdr) @xcite are two representative methods that treat a semi-structured data example as a single document or multi-fielded document. they apply an ir approach for qa @xcite . semantic table retrieval (str) @xcite introduces a semantic representation for web tables. the representation includes sets of extracted concepts and entities. bert @xcite is a powerful transformer-based model, which has demonstrated impressive performance in the semantic matching task. we apply this model to a concatenation of the query and the sequential tokens in a semi-structured data example, and then use a multi-layer perception for classification. all the previous methods do not consider the structure information in tables or lists. in this paper, we applied bert-base as our backbone model and baseline. the last baseline is tapas @xcite , a recent state-of-theart approach of qa over tables. this method encodes rows and columns to embed structural information of tables. to measure the accuracy of matching, we use the average f1 as our metric. precision, recall, and f1 score are computed on the number of true positives (tp), false positives (fp), and false negatives (fn). f1 score is the harmonic mean of precision and recall. since the matching prediction task is casted as a binary classification task, we consider f1 score as the metric and calculate average f1 based on that. all methods are implemented in pytorch @xcite and trained on an ubuntu 16.04 with 64gb memory and eight gtx 1080 ti gpu. for all data-sets, we randomly select 80% of the records as training set, 10% as validation set and the remaining 10% as test set. we train the model using training data, and fix model parameters based on the best model performance on validation set. we then test the model on test set. we perform three random runs and report both mean and standard deviation for testing performance. we use stochastic gradient descent (sgd) with a learning rate of 2e-5. we use mini-batches of size 64, with batch size 8 for each of 8 gpus, we use with 1 hidden-layer of 768 hidden units. we use dropout with a rate of 0.5, which is applied to all feedforward neural networks. for the pre-trainng process, we use a batch size of 64 and fine-tune for 4 epochs over the large-scale data-set for two unsupervised task. for each task, we selected the fine-tuning learning rate of 2e-5. for the graph convolutional network, we applied a bi-lstm with hidden-layer with 768 hidden units on the top of transformer output. the gcn contains two convolutional layers with the hidden size of 1,536. after node-level convolution, we adapted mean-pooling for graph representation. as to the positional embedding, we created a fixed sinusoidal embedding with 768 hidden units. for all baseline models, we use pre-trained corresponding transformer models as word embedding and using the output of token [cls] as sentence embedding. out-of-vocabulary (oov) words are hashed to one of 100 random embedding each initialized to mean 0 and standard deviation 1. all other hidden layer weights were initialized from random gaussian distribution with mean 0 and standard deviation 0.01. each hyperparameter setting was run on a same machine as the grasslm, using adagrad for optimization with initial accumulator value of 0.1. ## related work early studies on query-table matching adopt ir approaches. for example, @xcite and @xcite convert web tables into multi-field documents and apply document retrieval pipelines proposed in @xcite . @xcite propose to create semantic features at text level, concept level and entity level. these methods mainly consider the textual information in tables, but largely ignore the inherent structural information in tables. in recent years, learning representations for semi-structured data has received increasing interest. @xcite propose to apply convolutional models to web table. the rationale is to consider a web table as a matrix of text, analogous to an image of pixels. however, their model does not show strong performance, partly because the semantic relationship among neighbor cells in tables may be far more complex than the simple adjacency relation among neighbor pixels in an image. @xcite propose tapas, a weakly supervised table parsing method. tapas models the structure information of tables by explicitly encoding rows and columns. similarly, @xcite propose tabert, which focuses on pre-training methods for the table qa task. the authors design a pipeline for learning row-level and column-level representations. müller et al. (2019) our work is also generally related to the broad areas of graph neural networks and pre-training techniques, we refer interested readers to @xcite and @xcite for comprehensive surveys. ## conclusion semi-structured data on the web, including tables and lists, present a rich source for web qa. most of the previous methods do not take full advantage of structural information in semi-structured data. in this paper, we propose a novel approach to model both textual and structural information in semi-structured data. extensive experimental results verify the effectiveness of our approach.