Improving Text Understanding via Deep Syntax-Semantics Communication (2020)

Abstract: Recent studies show that integrating syntactic tree models with sequential semantic models can bring improved task performance, while these methods mostly employ shallow integration of syntax and semantics. In this paper, we propose a deep neural communication model between syntax and semantics to improve the performance of text understanding. Local communication is performed between syntactic tree encoder and sequential semantic encoder for mutual learning of information exchange. Global communication can further ensure comprehensive information propagation. Results on multiple syntax-dependent tasks show that our model outperforms strong baselines by a large margin. In-depth analysis indicates that our method is highly effective in composing sentence semantics.

## introduction neural sequential models such as lstm @xcite , gru @xcite , transformer @xcite , elmo @xcite and bert @xcite , have been extensively applied for encoding the semantics of texts in natural language processing (nlp) @xcite @xcite . on the other hand, hierarchical tree models, such as treelstm @xcite and gcn @xcite , have been introduced to enrich sequence encoding with syntactic information, bringing further strengths in text modeling. such external syntactic structure knowledge provides enhanced features, which can facilitate a broad range of nlp tasks @xcite @xcite . recent studies show that integrating syntactic tree models with sequential semantic models can bring improved performance for syntax-dependent tasks @xcite , such as semantic role labeling (srl) @xcite and natural language inference (nli) @xcite @xcite , etc. intuitively, sequential semantic models and syntactic tree models play different roles in text modeling. sequential semantic models learn the representation via adjacency neighborhood, while syntactic tree models encode texts through structural connections. taking the two sentence pairs from the nli task in figure 1 as example, sentence a and b in example (a) share the same dependency structure but have irrelevant semantics, and tree models are more suitable and effective for capturing the semantic difference than sequential models in this case. in example (b), two sentences convey very similar semantics but have different syntactic structures. therefore, two types of models should interact closely in learning compositional representations for better understanding of the texts. however, existing efforts integrate tree and sequential models through a straightforward way such as representations concatenation @xcite or multi-task learning @xcite @xcite , limiting the performance of end tasks. we believe that a better integration can be achieved when adequate interactions between sequential semantic encoder and syntactic tree encoder can take place during learning, improving the performance of end tasks, and also alleviating the long-range dependency problem. to this end, we present a novel deep syntaxsemantics communication model, as shown in figure 2. in particular, sequential and dependencytree based submodels are used for encoding input sentence separately. local communication is performed between each submodel during learning for information exchange of consecutive words in a sentence. meanwhile, two submodels are considered as an entire unit, taking global propagation at sentence level over recurrent steps. in addition, we employ gate mechanism to control information flow of each node at each time step during global communication. experimental results on a wide range of syntaxdependent nlp tasks show that our model outperforms strong baselines by a large margin, offering an alternative for better integration of sequential and tree models. further analysis indicates that our method is highly effective in composing sentence semantics, verifying the importance of integrating syntax and semantics for text understanding. ## related work neural sequential models have been widely used for encoding texts in the nlp community, due to their effectiveness on capturing semantics. representative models such as lstm, gru, transformer, elmo and bert, have been employed for various nlp tasks, including language modeling @xcite , machine translation @xcite , question answering @xcite , etc. on the other hand, some efforts devote to develop hierarchical tree models such as treelstm and gcn, based on syntactic structures (e.g., dependency tree). such tree encoders equipped with external syntactic knowledge can bring further improvements for some nlp tasks, especially syntax-dependent ones @xcite @xcite , such as srl @xcite @xcite , nli @xcite and relation classification @xcite , etc. in recent years, exploring the correlation between syntax and semantics has become a hot research topic. previous work has shown a strong correlation between syntax and semantics, and proven that integrating syntactic tree models with sequential models could improve the performance of end tasks @xcite @xcite @xcite . for example, @xcite simultaneously conducted syntax parsing and semantic role labeling via multi-task training strategy. @xcite incorporated syntactic features into semantic parsing tasks by multi-task learning. @xcite concatenated the contextualized semantic representations with syntactic tree representations for improving the ability of word embeddings. more recently, @xcite added a multi-layer bilstm with shortcut connections to the pairwise word interaction model for capturing semantics and syntactic structure of sentences. however, these methods only use shallow integration of syntax and semantics, limiting the performance of end tasks. our model is inspired by @xcite , who introduce a novel method allowing the sufficient communication between different tree models for sentiment analysis. unlike their work, this paper is dedicated to realizing a deep communication between syntactic tree model and sequential semantic model for improving text understanding. the idea of sentence-level propagation in our work is partially related to @xcite , who propose a novel lstm architecture where a set of global states are used for sentence-level propagation along recurrent steps, rather than incremental reading of a sequence of words in vanilla sequential lstm. compared with their model, our model is more effective in composing semantic information of texts. ## model in this paper, we propose a deep neural communication model between syntax and semantics to improve the performance of text understanding. local communication is performed between two encoders for information exchange of each node, as illustrated in figure 3 . global communication is performed over the entire framework under recurrent steps for sufficient information propagation, as shown in figure 2 . ## conclusion we proposed a deep syntax-semantics communication model for improving text understanding. local communication was performed between syntactic tree encoder and sequential semantic encoder for mutual learning of information exchange. global communication was performed for ensuring information propagation throughout entire architecture over recurrent steps. results on multiple tasks showed that our model outperformed strong baselines. in-depth analysis further indicated that our method was highly effective on composing sentence semantics.