Incorporating Syntax and Frame Semantics in Neural Network for Machine Reading Comprehension (2020)

Abstract: Machine reading comprehension (MRC) is one of the most critical yet challenging tasks in natural language understanding(NLU), where both syntax and semantics information of text are essential components for text understanding. It is surprising that jointly considering syntax and semantics in neural networks was never formally reported in literature. This paper makes the first attempt by proposing a novel Syntax and Frame Semantics model for Machine Reading Comprehension (SS-MRC), which takes full advantage of syntax and frame semantics to get richer text representation. Our extensive experimental results demonstrate that SS-MRC performs better than ten state-of-the-art technologies on machine reading comprehension task.

## introduction machine reading comprehension (mrc) requires machines to read and understand a text passage, and answer relevant questions about it, where mrc systems must have the ability to infer the meanings of underlying natural language, where both syntax and semantics are critical for text understanding. traditional mrc methods are feature-based approaches, where they first manually generate syntactic or semantic features and subsequently apply a standard machine learning model to identify a best answer. for instance, @xcite use the abstract meaning representation (amr) formalism in a max-margin framework, which only focuses on semantic information. in addition, there are studies that take both syntax and semantics into account for mrc @xcite . however, these methods heavily rely on manually defined features and are difficult to generalize to other tasks. recently, as large qa datasets become available, neural-based methods have been proposed for mrc, where even without time consuming feature engineering, they can still achieve favorable results. especially significant progress has been achieved on mrc tasks by fine-tuning a pre-trained general purpose language model @xcite . despite the success of those neural models, a number of studies have found there is a huge gap between mrc deep learning models and human beings @xcite , as they might not really understand the natural language text @xcite . as such, some works employ semantic knowledge in neural network to facilitate sentence modelling @xcite @xcite . however, no work has been focused on integrating syntax and semantics into neural network for mrc. note that framenet @xcite , as a widely adopted knowledge base, provides rich schematic scenario representation that could be potentially leveraged to better understand sentences. in this paper, we proposed a syntax and frame semantics model for machine reading comprehension (ss-mrc), which fuses syntax and frame semantics into an end-to-end neural model for addressing mrc task. the key contributions of this work are summarized as follows: frame semantics choose to chocolate frosting make a chocolate cake with they . f1: choosing f2: manufacturing f3: food f4: food fe 23 : resource fe 22 : product fe 31 : food fe 11 : cognizer fe 12 : chosen fe 21 :manufacturer fe 41 : food nsubj xcomp mark obl compound case obj compound det syntax figure 1: an annotated sentence with syntax and frame semantics. 2. we propose a novel syntax and frame semantics for machine reading comprehension (ss-mrc) method, which takes full advantage of syntax and frame semantics for every token in the sentence/sequence to obtain richer and more comprehensive representation. 3. our extensive experimental results demonstrate our proposed ss-mrc method is significantly better than ten state-of-the-art methods across two benchmark datasets for mrc task. ## syntax and frame semantics labeling we employ stanford corenlp @xcite to analyze the syntactic structure of every sentence in given text passage. figure 1 shows dependency parse results (top part) of a example sentence. words on the arrows are dependency labels, e.g., the arrow from word choose to word they indicates that they is the subject of choose and the dependency label of they is nsubj -a nominal syntactic subject. in addition, we employ semafor @xcite to automatically process different sentences with multiple semantic annotations @xcite . in particular, a frame (f) is defined as a composition of lexical units (lus) and a set of frame elements (fes). given a sentence, if its certain word evokes a frame by matching a lu, then it is called target (t). figure 1 provides an example sentence (bottom part) with four t, namely choose, make, chocolate cake and chocolate frosting. each t has its evoked semantic frame right below it, i.e. f 1 , f 2 , f 3 , f 4 . for each frame, its corresponding fes are shown enclosed in the block. for example, t choose evokes the f 1 :choosing frame, and has two fes cognizer, chosen, fulfilled by they and to make a chocolate cake with chocolate frosting respectively. in addition, evoked f 2 :manufacturing frame has three fes, namely, manufacturer, product, resource, fulfilled by they, a chocolate cake and chocolate frosting, respectively. from this example, it is very clear that both syntax and frame semantics information are very useful for mrc task. ## syntax and frame semantics fusion module in this paper, we explore three different fusion methods to generate g sf by integrating x s and x f . note that our syntax and frame semantics fusion module is backbone-free, which indicates that we can use any existing neural models, i.e., lstm, gru and transformer. in this work, we use bi-lstm as our backbone model. ## conclusion we propose a novel syntax and frame semantic fusion method for mrc in a neural network, which, to our best knowledge, is the first attempt in this area. our extensive experimental results demonstrate it works better than ten state-of-the-art methods for the challenging machine reading comprehension task.