Sentence Matching with Syntax- and Semantics-Aware BERT (2020)

Abstract: Sentence matching aims to identify the special relationship between two sentences, and plays a key role in many natural language processing tasks. However, previous studies mainly focused on exploiting either syntactic or semantic information for sentence matching, and no studies consider integrating both of them. In this study, we propose integrating syntax and semantics into BERT with sentence matching. In particular, we use an implicit syntax and semantics integration method that is less sensitive to the output structure information. Thus the implicit integration can alleviate the error propagation problem. The experimental results show that our approach has achieved state-of-the-art or competitive performance on several sentence matching datasets, demonstrating the benefits of implicitly integrating syntactic and semantic features in sentence matching.

## introduction sentence matching aims to determine the specific relationship between two sentences. it plays a key role in many natural language processing tasks such as natural language inference @xcite , paraphrase identification @xcite , and answer selection @xcite . learning contextual representations is fundamental to many sentence matching tasks. earlier works explore contextual features of sentences individually in a neural network classifier to predict sentence relationship @xcite . however, they do not consider the more finegrained comparison information such as word-to-word correspondence between sentences. recent studies achieve better results by taking into account contextual features and fine-grained interaction between sentences at the same time @xcite @xcite . more recently, with the development of pre-training language models @xcite @xcite , substantial progress has been made in sentence matching tasks. for example, bert (bidirectional encoder representations from transformers) @xcite has proven to be a powerful representation of contexts for sentence relationship prediction, in that bert representations can implicitly capture a rich hierarchy of linguistic information within sentences @xcite . in addition to contextual representation, structural information such as syntactic structures and semantic roles has shown to be effective to enhance sentence matching tasks and other nlp systems @xcite ). on the one hand, with the maturity of dependency parsing and semantic role labeling technologies in recent years, it is relatively easier to obtain reliable structural information for sentence matching. on the other hand, integrating structural information with pre-training language models has been a promising way to enhance sentence matching performance. semantic role labeling (srl) is expressed as predicate-argument relationships to explore who did what to whom, when and why for representing the central meaning of the sentence, which is the same as judging their relationship by understanding two sentences. dependency parser can represent the dependency of words, which can compare from word level to reveal its syntactic structure and compare their difference. to enrich in semantic role labeling, they have different predicate-argument structures, which may lead to the wrong prediction. however, in dependency parsing, they have the same subject, predicate and object (i.e. person, pour, and tea), which makes the two sentences have the same backbone. thus, combining both syntax and semantics can help the model predict the correct result. the contextual sentence representation of bert, sembert @xcite has shown encouraging improvement, which first integrates the semantic labels into the bert and obtains state-of-the-art performance on sentence matching. moreover, sembert uses explicit method to integrate the semantic information, which highly depends on the accuracy of the semantic structure information. as the input of sentence matching, incorrect structure information may produce incorrect output, resulting in improper matching results. however, most previous studies on sentence matching have exploited syntactic information or semantic information alone, without considering both structural information. for example, as illustrated in figure 1 , we show an example of applying the sembert @xcite on the natural language inference task. the gold label of this example is entailment, but sembert predicts their relationship is contradiction. as the sembert only integrates semantic structure information that focuses on the different predicate-argument structures, the deeper meaning contained in each argument is not involved. we argue that syntactic and semantic structures are complementary cues for sentence matching, and can be combined with the bert representations to boost the performance of various sentence matching tasks. to this end, we propose to design a framework called syntax-and semantics-aware bert (ss-bert), which uses bert as the backbone and implicitly integrates syntactic and semantic information for sentence matching. in our model, the combination of syntax and semantics can improve the fault tolerance rate of the model to reduce the error propagation problem. moreover, our method further enhances its syntactic and semantic awareness based on the strong ability of bert to represent the context and enables ss-bert to learn deeper meaning representation. on the other hand, we implicitly integrate syntax and semantics at the same time. the advantage of implicit integration is to reduce the sensitivity of the model to the structural information of output, and thus to alleviate the error propagation problem. in summary, we mainly make the following three contributions in this work: (1) we propose a framework named ss-bert that integrates both syntactic and semantic information for the sentence matching. to the best of our knowledge, we are the first to integrate both syntax and semantics into bert for sentence matching; (2) we explore the differences between implicit and explicit integration. our experiments show that implicit integration achieves better results with bert. (3) we verify the performance of ss-bert on three sentence matching tasks, and the results show that ss-bert has achieved state-of-the-art or competitive performance on four datasets. ## conclusions in this paper, we propose syntax-and semantics-aware bert(ss-bert), which implicit integrates syntactic and semantic information for the first time. we have achieved state-of-the-art or competitive per-formance on four datasets. experiments show that our method of integrating syntax and semantics into bert is effective. on the other hand, results indicate that implicit integration has advantages over explicit integration. since ss-bert is an end-to-end framework, it is expected to be applied to other natural language tasks in the future.