Enhancing Generalization in Natural Language Inference by Syntax (2020)

Abstract: Pre-trained language models such as BERT have achieved the state-of-the-art performance on natural language inference (NLI). However, it has been shown that such models can be tricked by variations of surface patterns such as syntax. We investigate the use of dependency trees to enhance the generalization of BERT in the NLI task, leveraging on a graph convolutional network to represent a syntax-based matching graph with heterogeneous matching patterns. Experimental results show that, our syntax-based method largely enhance generalization of BERT on a test set where the sentence pair has high lexical overlap but diverse syntactic structures, and do not degrade performance on the standard test set. In other words, the proposed method makes BERT more robust on syntactic changes.

## introduction the task of natural language inference (nli) targets at determining whether one sentence entails another @xcite . recently, largescale pre-trained contextualized embeddings such as bert @xcite and xlnet @xcite have given the state-of-the-art accuracy for this task. it has been shown that pre-trained models help to better capture heuristic patterns in a set of training data and therefore enhance indomain performance @xcite . however, there are still limitations on the generalization of such models to examples under a different distribution. in particular, it has been shown that seemingly simple types of examples in a carefully designed evaluation set (i.e. hans) can lead to significant degeneration and large variability in performance @xcite . it has been shown that syntactic structures are useful for cross-domain generalization of nlp models @xcite . intuitively, a more robust nli model can be obtained by making use of structural information. we empirically investigate the effectiveness of syntactic features for enhancing the generalization of bert-based matching models. in particular, given a pair of sentences, the dependency syntax of each sentence is obtained using a neural parser @xcite . the parse trees are then extended using four types of edge patterns, including a soft co-attention matching pattern that links the sentence pair into an integrated graph. a graph convolutional network (gcn) @xcite is used to represent the whole matching graph structure. experiments show that the performance of the proposed model is much better than bert and other syntax-based baselines on the category in hans where the premise and non-entailment hypothesis have high lexical overlap but different syntactic structures, when both models are trained on mnli dataset. it proves that incorporating syntax by the proposed method enhances generalization of the bert model on syntactic changes ‡ . ## related work there has been much work based on deep neural networks for the nli task. one straight-forward solution is to independently encode the premise ‡ our code will be available at: https://github. com/heqi2015/ca_gcn premise hypothesis gold bert-cls bert+cagcn the student saw the managers. the managers saw the student. n e n the judge in front of the manager saw the doctors. the doctors saw the judge. ## method the overall architecture of the proposed method is shown in the top of figure 1 . at the bottom layer, contextualized representations of the two sentences are obtained by using bilstm, elmo or bert. the representation is then fed into gcn to initialize the representations in the first layer. ## conclusion we have investigated the effectiveness of introducing syntax into the nli task, by adopting gcn to enhance the text representation in existing models such as bert. results on hans show that our method can improve the generalization of bert, especially on examples where the sentence pair have high lexical overlap but different syntactic structures. it demonstrates that adding inductive biases such as dependency tree by gcn can make sentence encoding more robust.