Does Structure Matter? Encoding Documents for Machine Reading Comprehension (2021)

Abstract: Machine reading comprehension is a challenging task especially for querying documents with deep and interconnected contexts. Transformer-based methods have shown advanced performances on this task; however, most of them still treat documents as a flat sequence of tokens. This work proposes a new Transformer-based method that reads a document as tree slices. It contains two modules for identifying more relevant text passage and the best answer span respectively, which are not only jointly trained but also jointly consulted at inference time. Our evaluation results show that our proposed method outperforms several competitive baseline approaches on two datasets from varied domains.

## introduction machine reading comprehension (mrc) is the task of reading a given text and answering questions about it @xcite . some mrc tasks such as squad @xcite ,and sharc @xcite provide a short text snippets as the context documents; while others such as triviaqa @xcite , natural questions @xcite and doc2dial @xcite use full articles as documents. most top performing models on mrc tasks use different variants of transformers @xcite . transformer-based models typically only consider a certain number of tokens, utilize a sliding window approach @xcite or segment the document into passages @xcite due to the constraint on the size of input sequence. more recent works explore how to scale up input length @xcite @xcite @xcite but still mainly focus on flat sequences. in addition to scaling up input length, etc @xcite also propose to deal with encoding structured inputs by mail the conviction occurred within the last 30 days at a tvb office online you will need tree view ... ... ... ... <@xmath0... </@xmath1... </@xmath2... </@xmath3... <@xmath4, date of birth, and @xmath5... sequence view based on relative position encoding @xcite through the global-local mechanism. a series of recent work explores incorporating structured knowledge embedded in text into mrc @xcite . however, such kind of linking information for creating triples is not necessarily prominent in documents other than wikipedia. some works segment the document content based on its semantic structures and rank them based on their relevance to the query @xcite @xcite @xcite . another thread of works, on hierarchical document encoding @xcite @xcite , first obtain sentence level representations then encode document based on the sentence vectors. those works do not directly apply on fine-grained answer extraction across sentences. in many online documents, certain important information unfolds through the semantic relations of hierarchical structures such as parent-child and siblings between different parts of the document. figure 1 illustrates the difference when using a doc-ument with and without the structure information for a mrc task. for query u1, it is crucial to keep in mind we are in the context of "how to appeal a tvb ticket" and "online" while reading the passage of "you will need" to find the answer to the user query. however, conventional transformers fail to capture such contextual information when the text is too long to fit in the maximum sequence length allowed. in this work, we explore the utilization of document structure for the focused task of fine-grained machine reading comprehension on document. we propose a transformer-based method that reads a document as tree slices; it jointly learns the relevance of paragraphs and spans, and then performs a cascaded inference to find the best answer span. our work is intuitively inspired by how people read through documents @xcite based on structural cues such as titles and subtitles, and then focus on the relevant parts to search for an answer. we utilize the structural information naturally available in online documents for identifying tree slices. each slice corresponds to nodes along a path from a root node to a lower level child node as illustrated by the right part of figure 1 . thus, we are able to capture the essential structural information for the inference that could be outside of a conventional sliding window or text segment. compared to approaches such as longformer @xcite or etc @xcite , our approach can be directly applied to many existing pretrained models, and has a small gpu memory footprint. rikinet @xcite employs a dynamic paragraph dual-attention reader and a multi-level cascaded answer predictor, while our tree slices consider hierarchical structures above paragraphs, and our cascaded inference is in beam search style rather than greedy decoding style in rikinet. we evaluate on two datasets with structured documents: one obtained from natural questions @xcite , which is based on wikipedia articles, and one from doc2dial @xcite , which is based on web pages of several domains. our proposed method is compared with several baselines to see performance gain on both datasets. for example, our method achieves 4% gain of f1 on doc2dial, which shows its superiority on small-scaled dataset across multiple domains. our contributions can be summarized as follows: ## approach we adopt a transformer-based document-tree-slice encoder with joint learning and cascaded inference. our approach is influenced by the pattern of human behavior during reading @xcite , which is to focus on a smaller portion at a time and favor the more relevant parts while looking for answer. this approach can also overcome the constraint on fixed-length input allowed by the common transformer architecture @xcite . more importantly, this enables us to always include important structural context information during encoding. ## data our focused task is utilizing document structure in contextual representation for fine-grained mrc. since there is very few prior mrc datasets that provides document structure information, we identified two public datasets where html markup tags are available in the document data together with qa pairs, and extract tree structure out of the html documents for mrc. data script could be found at http://html2struct.github.io . extract tree structure to obtain the tree representation of documents from the two datasets, we first parse html files to get markup tags of the textual content elements, which corresponds to the titles, lists, tables and paragraphs. we consider the different levels of title tags as the main indicators of the hierarchical structures such as parent-child and siblings. thus, the stem nodes are inherently section or subsection titles of the article and leaf nodes are typically paragraphs, list content or table content. we assign the article title as the tree root. please refer to appendix a for more details about the data statistics for the experiment. nqstruct natural question @xcite provides qa pairs that are grounded in wikipedia articles. the original task provides answers in two formats: long answer, typically a paragraph or table; short answer, typically one or more entities. in our task, we focus on identifying the short answer given the whole document as the input, and do not use the long answers data. we observe the bias on answers appearing in first paragraph, which is significant enough to serve as a baseline @xcite . thus, we follow @xcite to alleviate such bias by only considering the questions where the short answer does not come from the first paragraph. as a result, we derive a subset of 48k examples from about 100k examples with short answers from training and dev sets. d2dstruct doc2dial @xcite provides document-grounded dialogues with annotations of dialogue scenes, which allow us to identify question-answer pairs that are most related to our target task. specifically, we combine each turn of the agent responding to a user query, together with the previous dialogue context, as a question. the public dataset contains over 4.1k documentgrounded dialogues based on about 450 documents from different domains, and we derive 9.3k qa pairs out of it. ## experiments and results we compare our proposed method (treejc in short) with several baseline methods. next we describe the baselines, the experiment settings, and present the evaluation results. ## conclusion we introduce a new transformer-based method with joint learning and cascaded inference inspired by the tree structures of documents for machine reading comprehension. it outperforms several competitive baselines on two datasets from multiple domains. in particular, our study demonstrates that the proposed model is effective to encode longer documents with deep contexts for mrc tasks.