Hy- NLI : a Hybrid system for Natural Language Inference (2020)

Abstract: Despite the advances in Natural Language Inference through the training of massive deep models, recent work has revealed the generalization difficulties of such models, which fail to perform on adversarial datasets with challenging linguistic phenomena. Such phenomena, however, can be handled well by symbolic systems. Thus, we propose Hy-NLI, a hybrid system that learns to identify an NLI pair as linguistically challenging or not. Based on that, it uses its symbolic or deep learning component, respectively, to make the final inference decision. We show how linguistically less complex cases are best solved by robust state-of-the-art models, like BERT and XLNet, while hard linguistic phenomena are best handled by our implemented symbolic engine. Our thorough evaluation shows that our hybrid system achieves state-of-the-art performance across mainstream and adversarial datasets and opens the way for further research into the hybrid direction.

## introduction natural language inference (nli), the task of determining whether a premise sentence entails, contradicts or is neutral with respect to a hypothesis sentence given a specific setting, has recently seen tremendous advances. the growing availability of increasingly large datasets has enabled the training of massive deep models, pushing the state-of-the-art (sota) to human performance @xcite . this performance, however, has triggered various questions: is the task now solved; is it possible to outperform humans; what do the models really understand, etc. much work has been undertaken, with researchers detecting bias or artifacts in the training sets @xcite and "breaking" the models with adversarial datasets that expose the generalization and compositionality difficulties of the models @xcite @xcite @xcite @xcite . this raises the question of how we can mitigate those shortcomings while taking advantage of the current advances in the field. in this paper we propose a hybrid nli system, which combines the strengths of a symbolic nli engine with a deep learning (dl) model. as argued elsewhere @xcite , we also believe in the division of semantic labour. distributional features are well suited for dealing with graded, fluid and robust conceptual aspects of the meaning of words, phrases, and sentences @xcite @xcite , but struggle with boolean and contextual phenomena such as negation, modals, quantifiers, implicatives. these are phenomena to which more symbolic approaches are well suited. thus, inference pairs of the former category that mostly rely on word/phrase similarity, and require taxonomic-and world-knowledge are considered easy, e.g., the dog is walking entails the animal is moving. in contrast, pairs of the latter category that require more complex inference around phenomena such as modals, negation, quantifiers, implicatives, conditionals, etc., are considered hard, e.g., the boy faked the illness contradicts the boy was sick (see appendix a for more such phenomena and relevant examples). to deal with such hard pairs, we build a symbolic nli engine, gkr4nli. we hybridize this engine by exploiting the power and robustness of sota language representation models, which achieve high performance in the mainstream, easy pairs. we then train a classifier, which determines whether the inference label of the symbolic or the dl component should be used, based on the nature of the pair, i.e., whether it involves challenging linguistic phenomena or not. our evaluation shows that this approach achieves sota results across mainstream and adversarial datasets. our contributions in the paper are three-fold: first, we implement a symbolic nli system, gkr4nli, and show its suitability for the task. second, we describe how our trained classifier learns to distinguish between the easy and hard cases. third, we show how such a hybrid setting reaches the current sota and propose the training of more hybrid models, able to assign each nli problem to the most suitable solver. ## related work in recent years, two broad methodologies have been used to tackle the problems posed by nli. on the one hand, there have been logic-based systems that involve linguistic methods such as syntactic and semantic parsing, and systems that employ a theorem prover or use natural logic and monotonicity principles @xcite @xcite . such systems have been largely evaluated on the semeval-2014 version of the sick dataset @xcite . on the other hand, nli has been tackled with machine-learning methods in two major directions. one strand of research has trained end-to-end deep models, which transform the premise and hypothesis sentences into n-dimensional vectors and learn to classify them into one of the inference relations. such methods range from attention architectures, e.g., @xcite , to approaches integrating linguistic features such as syntactic parses, e.g., @xcite , and external knowledge like wordnet, e.g., @xcite , to models trained on multiple tasks, e.g., @xcite , to name just a few. the other strand of work has focused on language representation models, building on the efforts of popular word @xcite , sentence @xcite @xcite and contextualized embeddings, such as elmo @xcite , open gpt @xcite , bert @xcite and xlnet @xcite . such representations are then efficient for supervised downstream tasks, like nli, by only fine-tuning them on the specific task. although such sota systems achieve high performance on massive multi-domain datasets like snli @xcite , mnli @xcite or scitail @xcite , it has repeatedly been shown that they lack the generalization and compositionality power needed to truly understand language. they are easily "broken" when presented with differently biased data @xcite or with challenging adversarial data, e.g., including semantic @xcite @xcite or structural phenomena @xcite @xcite . such phenomena may include comparatives, implicatives, conditionals, coordination, negation, modals (see appendix a for a fuller list of phenomena and relevant examples). for example, comparatives are hard because the models cannot yet efficiently capture the complex interaction between the order of the compared constituents and comparative notions like more or less. similarly, implicatives are challenging for models because they inherently possess an implication signature which is not only unique for each predicate and thus needs to be explicitly learnt for each of them, but also undergoes changes based on how the implicative is embedded and whether it is negated. pairs with conditionals, modals and coordination challenge the models due to the high lexical overlap between the sentences of the pair -which the models have learnt to interpret as entailment -and due to the inability of current models to exploit the notions of causality and modality. foot_0 most of this "breaking nli" work has proposed augmenting the training data with such complex phenomena and has shown that models can then successfully improve their performance. however, augmenting the training data with specific phenomena does not solve the bigger problem @xcite . due to the recursive nature of language, there are infinitely many ways of composing information into meaningful sentences. but each type of adversarial set can only improve performance for one specific linguistic phenomenon and already generating high-quality data for a single phenomenon is extremely costly. indeed, @xcite show that training on one type of adversaries does not improve performance for other types; in fact, it might even harm the overall performance due to over-fitting. @xcite also observe that augmenting the training data with only a subset of their adversarial set enables the model to learn this subset, but does not help it solve the withheld set. thus, we need to focus on the kind of learning process that will allow for the desired generalizations without having to include all possible constructions in the training data. one solution is to pursue a direction where such complex phenomena are treated separately by suitable symbolic components and other cases are treated by deep models. we propose training suitable hybrid models that can learn when to use which method. our work shows how such a hybrid approach can work in practice. recently, work in a similar direction was conducted by @xcite , who develop a monotonicity/natural logic system capable of producing entailing or contradictory sentences for a premise, in the search of the actual hypothesis. when combining their system with bert, they improve bert's performance on sick by 1%. however, this slight performance boost does not reveal the real value of such a hybrid system, compared to pure deep-learning models like bert, because it does not show its performance on adversarial sets. ## the hy-nli system the proposed system consists of a symbolic and a deep learning component, whose outputs are used for the training of the hybrid classifier. the overall architecture of hy-nli can be found in figure 1 . the explainability of our proposed system and an intuitive user interface are presented in @xcite . ## discussion overall, the reported results show the efficiency of the proposed approach. by utilizing general features about the semantic nature of the pair, hy-nli becomes competitive to the average ceiling performance across datasets. 12 the fact that the system does not reach the ceiling performance in two of the three datasets suggests that the training of the classifier did not include several of the feature combinations. sick, a training corpus of simple inferences, does not contain a large variety of feature combinations of complex phenomena. thus, the classifier could only learn a limited number of types of complex inferences, which is reflected in the lower performance for hans (which combines easy and hard cases, as discussed). on the other hand, the sampling process might also have disadvantaged the learning of easy patterns because the pairs on which only the dl component gave the correct label had to be severely downsampled to match the low number of pairs on which only gkr4nli gave the correct label (again, due to the nature of the sick corpus that mainly contains easy pairs). thus, this could have led to the performance difference for sick, where hy-nli's performance misses the best performance by 1.7%. in fact, an error analysis showed that the errors originate either from the hybrid classifier predicting the wrong component or the classifier predicting the correct component but the component itself predicting the wrong label. in the former case, for a pair like p: several people are in front of a building which is covered by colors. h: several people are in front of a colorful building, hy-nli predicted s and was thus mapped to the neutral label of gkr4nli; however, given the required robustness of colorful -covered by colors, the better label would have been dl which would have indeed mapped to entailment. in the latter case, there are pairs like p: the player is missing the basket and a crowd is in background. h: the player is dunking the basketball into the net and a crowd is in background, where hy-nli correctly predicts dl (no complex phenomenon involved, only lexical semantics), but the dl component incorrectly assigns the neutral label as it fails to recognize the contradiction. this shows that there is potential in improving all parts of the system: the hybrid classifier as such but also its distinct components. we can also observe that the average performance of hy-nli across datasets is higher only by a small margin than the average performance of gkr4nli. this is not so puzzling considering that hy-nli achieves higher performance than gkr4nli on sick but lower performance on hans, bringing the two average performances closer. hy-nli achieves its goal to combine the strengths of very different components and is able to perform across the board. by considering what types of inference each kind of component is best at, the system can successfully implement the idea of using the most suitable solver for a given problem. by exploiting features about the semantic nature of a pair, it can determine which pairs should be best handled by which component. the features that hy-nli exploits are generalizable semantic properties and not specific sentence combinations, so that the augmentation of the training data with further properties combinations should give a direct boost to the performance. in fact, given the small number of features used for training, this kind of classifier should be able to perfectly learn to classify the pairs, if presented with all combinations of the few used generic features. training and testing on the original sick (not the corrected version) does not change the overall picture: bert's accuracy remains the same for all three datasets and the accuracy of xlnet improves to 88% for sick but remains the same for hans and das. the accuracy of gkr4nli unsurprisingly drops to 76.3% for sick but remains stable for the hard sets. this means that the slight differences balance each other out and the overall performance of hy-nli remains the same. the reader might also wonder what happens with pairs where both dl robustness/world-knowledge and symbolic reasoning are required. currently, nli datasets contain pairs that are either linguistically complex or lexically/world-knowledge-wise complicated. thus, this otherwise very realistic situation does not occur and hy-nli does not need to deal with it. however, hy-nli has the potential of doing so by extending the initial term matching stage of gkr4nli: the matching process could be enhanced with embeddings, thus gaining on robustness and informativity. with such an improvement, dual cases, where symbolic reasoning and dl robustness are required, could still be best handled by gkr4nli. ## conclusion this paper presented a hybrid nli system that marries up the strengths of a symbolic and a dl component to close the gap between the performance of sota models on mainstream and adversarial datasets. the proposed system achieves sota results across the tested datasets, highlighting the advantages of the approach. thus, we propose concentrating on such hybrid settings, able to learn which technology is most suitable for the problem at hand. future work on hy-nli involves determining how the properties of each component are used and hence how to improve them. the modularity of the system allows each component to be improved independently by focusing on the problems each component is good at.