Relying on Discourse Analysis to Answer Complex Questions by Neural Machine Reading Comprehension (2021)

Abstract: Machine reading comprehension (MRC) is one of the most challenging tasks in natural language processing domain. Recent state-of-the-art results for MRC have been achieved with the pre-trained language models, such as BERT and its modifications. Despite the high performance of these models, they still suffer from the inability to retrieve correct answers from the detailed and lengthy passages. In this work, we introduce a novel scheme for incorporating the discourse structure of the text into a self-attention network, and, thus, enrich the embedding obtained from the standard BERT encoder with the additional linguistic knowledge. We also investigate the influence of different types of linguistic information on the model’s ability to answer complex questions that require deep understanding of the whole text. Experiments performed on the SQuAD benchmark and more complex question answering datasets have shown that linguistic enhancing boosts the performance of the standard BERT model significantly.

## introduction machine reading comprehension (mrc) reflects the ability to read and understand an unstructured text and answer questions regarding it. aiming to find the relevant answer to a question in the form of a text span, the mrc models should demonstrate deep understanding of the language and text organization. transformer models that achieve state-of-theart results on multiple natural language processing (nlp) tasks have been successfully applied to the mrc. however, while the ideal mrc model should read most words superficially and pay attention only to the essential ones @xcite , the attention mechanism in the standard transformers attends to all words without explicit constraint which results in inaccurate concentration on some less important text spans. lately, the researchers have actively examined the ability of the deep learning (dl) models to understand language and build accurate linguistic-enhanced internal representation. recent works have revealed that traditional dl models that ignore additional linguistic knowledge, such as syntax or semantic, achieves lower accuracy on such complex tasks as natural language understanding (nlu) or mrc @xcite @xcite . it has been shown that incorporating explicit syntactic @xcite and semantic @xcite relations into the attention mechanism leads to better linguistically motivated word representations beneficial for the mrc task. moreover, providing exact, concise answers frequently requires not just syntactic/meaning similarity but an overall structure of thoughts expressed by an author @xcite , i.e., some claims introduced by an author and logical connections existing among them. this information is encoded by discourse structure of a text that, as long as syntax and semantic, is believed to provide valuable information that could help the model to capture all the hidden dependencies existing in the text and to pay attention only to the relevant words while answering the corresponding question. in this paper, we explore if and how discourselevel features (discourse relations connecting the text spans), fed to a neural mrc model on top of syntactic and semantic features or independently, can help to answer complex, long, multi-sentence questions. we intend to develop a neural method that selects relevant words by only considering the related subset of words, w.r.t. syntactic, semantic, and discourse-level importance. to provide feature encoding we use a self-attention network (san) enriched with the discourse features (such as explanation, condition, etc.) retrieved from a text and combine it with the classical transformer encoder to build linguistically-enhanced text representation. overall, the contribution of this paper is threefold: first, we introduce a novel discourse-aware transformer-based model to construct the enriched internal representation of the text. second, we develop an ensemble mrc model that combines syntax, semantic, and discourse mrc components. third, we conduct experiments on various questionanswering (qa) datasets to assess the ability of the linguistically enriched model to answer complex questions and estimate the influence of each source of linguistic information. ## mrc system extended with discourse relations in this paper, we present the novel discourse-aware attentive model designed to perform the mrc task. our approach is inspired by syntax-guided bert @xcite , while instead of encapsulating the syntactic dependencies among the words, we pre-process the discourse parse tree and observe the edus as long as the specific discourse relations connecting them. we introduce the architecture of the transformerbased encoder empowered with the discourse knowledge about the input text in section 3.1. as we aim to assess the influence of all three types of linguistic information, in section 3.2, we present the final mrc system designed as the ensemble of state-of-the-art syntactic, semantic, and the proposed discourse-aware attention components. ## experiments in this work, we rely on four qa datasets with long, complex, multi-hop questions to observe if/how syntactic, semantic, and, mainly, discourse-level features help to provide the correct answers. as the baseline, we use fine-tuned bert model. besides, we compare the performance of our system with the current state-of-the-art results published or obtained from the leaderboard for the corresponding dataset. ## conclusion in this paper, we analyzed various linguistically enriched deep neural models and assessed the influence of semantic, syntax, and discourse on their performance on mrc tasks. while, modern systems are usually linguistic-free or rely on some independent linguistic characteristic, such as syntax or semantic individually, we claim that their combination could provide even higher accuracy gain. we also introduce the approach to incorporate discourse structure into the transformer-based model, which has been proven to be necessary for answering complex multi-sentence questions. we have shown that the combination of three additional features encoded into a neural mrc is able to answer lengthy and complex questions better than the linguistic-free models, even the ones fine-tuned on the observed datasets. the introduced discourse-aware mrc model outperformed standalone syntax-guided @xcite and semantic-enhanced models @xcite ) for all the observed datasets. although our mrc system did not achieve state-ofthe-art results on some of the evaluation datasets (e.g., on the squad), it demonstrated the superiority of integrated syntax/semantic/discourse subsystems in multiple diverse qa domains with complex questions.