ERNIE - NLI : Analyzing the Impact of Domain-Specific External Knowledge on Enhanced Representations for NLI (2021)

Abstract: We examine the effect of domain-specific external knowledge variations on deep large scale language model performance. Recent work in enhancing BERT with external knowledge has been very popular, resulting in models such as ERNIE (Zhang et al., 2019a). Using the ERNIE architecture, we provide a detailed analysis on the types of knowledge that result in a performance increase on the Natural Language Inference (NLI) task, specifically on the Multi-Genre Natural Language Inference Corpus (MNLI). While ERNIE uses general TransE embeddings, we instead train domain-specific knowledge embeddings and insert this knowledge via an information fusion layer in the ERNIE architecture, allowing us to directly control and analyze knowledge input. Using several different knowledge training objectives, sources of knowledge, and knowledge ablations, we find a strong correlation between knowledge and classification labels within the same polarity, illustrating that knowledge polarity is an important feature in predicting entailment. We also perform classification change analysis across different knowledge variations to illustrate the importance of selecting appropriate knowledge input regarding content and polarity, and show representative examples of these changes.

## introduction recently, the selection and integration of external knowledge into large-scale language models has shown impressive improvements in several natural language understanding (nlu) tasks @xcite . understanding the relation between external knowledge and model performance is fundamental to understanding how best to select and integrate knowledge into nlu tasks. we focus specifically on natural language inference (nli), which requires understanding sentence semantics with respect to both the content and polarity. nli is motivated by recognizing textual entailment, or understanding whether a hypothesis entails, contradicts, or is neutral with respect to a premise. for example, given the premise: "some boys are playing soccer", the hypothesis "young men are playing a sport" is an entailment whereas the hypothesis "old men are playing a sport" is a contradiction. language modeling is a very common and important approach when considering the nli task. the nli state-of-the-art utilizes different language modeling techniques to learn the relations between the hypothesis and the premise. @xcite used dynamic self-attention (dsa) to learn sentence embeddings, @xcite proposed multi-task deep neural network (mt-dnn) for learning language representations in multiple nlu tasks, @xcite combined semantic role labeling and bert @xcite to explicitly absorb contextual semantics over a bert framework. however, these approaches limit the source of information available for representing both the premise and hypothesis. consider the following premise and hypothesis: people cut their expenses for the golden years. people decrease their expenses for retirement. it is challenging to know that "golden years" entails "retirement" if we rely only on the context within the two sentences. to illustrate how common this problem is, we conduct a manual analysis of bert classification errors on the nli task (specifically on the mnli corpus @xcite , more details in section 6), and find that at least 50% of misclassifications require external knowledge, specifically requiring domain-specific knowledge, world knowledge, jargon-based paraphrases, or commonsense knowledge to resolve the entailment. in the above example, a model that learns the relation between "golden years" and "retirement" from external knowledge can be used to enhance nli inference. on the basis of this idea, @xcite and @xcite used external knowledge from wordnet and transe @xcite and applied it to nli models. in their work, pre-trained representations of external knowledge from knowledge bases (e.g., transe) were directly applied; they did not tailor knowledge content or structure specifically to the nli task and did not improve nli performance @xcite . this finding motivates our investigation on how external knowledge can be efficiently used to improve nli models. the intention of our work is not to propose a new model that outperforms the state-of-the-art, but instead to focus on building a framework for investigating how different types and representations of external knowledge impact an nli model's decisions. consider our previous examples. we want to represent that the relation between "young men" and "boys" is positive for entailment, and that the relation between "old men" and "boys" is negative for entailment. similarly, we want to represent that the relation between "golden years" and "retirement" is positive for entailment. the interplay of external knowledge and entailment gives insight into the power of selecting relevant knowledge with respect to both content and polarity of the knowledge. here, content indicates the semantic meaning of external knowledge and polarity indicates whether the knowledge relation is positive or negative for entailment. the representation of external knowledge is required to be correct in both aspects for the nli task. the models learns (1) content via our knowledge extraction phase, by extracting concept edges from knowledge graphs, and (2) polarity via our knowledge training phase, by learning the polarity of the relationships between concepts. we define concepts as words or phrases througout this paper. in this work, we aim to show what type of external knowledge is useful for certain classes of nli. we examine how different types of knowledge impact neural language model decisions with respect to content and polarity. to this end, we propose ernie-nli, an nli model that integrates external knowledge to enhance and probe nli inference decisions. first, we adapt knowledge content in various sources to our setup: external knowledge relations are mapped to nli knowledge relations (section 4.2). in this step, we not only represent external knowledge from different sources in a unified way, but also convert external knowledge content to the nli task. second, the polarity is learned (section 4.3): nli knowl-edge embeddings are learned to predict whether they are positive or negative for entailment. in this step, we extend bert with a knowledge embedding layer and a classification layer. third, the content and polarity are applied to nli classification (section 4.4). all three phases listed above are depicted in fig. 1 . ernie-nli is developed on the basis of ernie @xcite , which did not improve performance on the nli task, although it was infused with transe embeddings. results show that our model ernie-nli enhanced with adapted knowledge achieves better performance than ernie for specific classes depending on knowledge input. we perform an in-depth analysis to examine how different types of knowledge impact nli model's decisions with respect to content and polarity. we conduct a series of experiments to investigate why and how the adapted knowledge enhances nli predictions. from the experiments, we find that: in summary, the proposed nli model enhanced with adapted external knowledge from various sources achieves better performance for respective classes, allows us to analyze the impact of knowledge type, and is robust when the knowledge at inference time has shifted. we examine this performance with detailed analysis throughout the paper. overall our contributions are as follows: ## nli corpus and external knowledge in this section, we introduce the particular nli corpus and external knowledge sources used throughout this work. ## methods we introduce our terminology in section 4.1. then, we introduce the three steps of ernie-nli: (1) knowledge extraction phase (content): extracting knowledge content from external knowledge sources (section 4.2), (2) knowledge training phase (polarity): learning knowledge embeddings adapted to the nli task (section 4.3), and (3) nli training phase: training our nli model with the integration of learned knowledge embeddings (section 4.4). the three phases are shown in fig. 1 . ## experiment setup as introduced in section 3, we examine various external knowledge sources. we describe the setups used in this work, all of which are combinations of these sources. the performance of each setup is reported in section 6. pc is the basic setup and includes paraphrase database (ppdb) and conceptnet. in this setup, we find that the number of positive nli knowledge relations is greater than the number of negative nli knowledge relations. thus, we design additional setups to balance the ratio of positive and negative relations. pc&bal balances the positive and negative nli knowledge relations to 50%-50% by downsampling positive relations. pcw adds negative nli knowledge relations from wordnet to pc. pcw&bal balances the positive and negative nli knowledge relations to 50%-50% on pcw by downsampling positive relations. ## examples in this section, we discuss the two examples depicted in ## conclusion we propose ernie-nli, an nli model that integrates external knowledge to enhance nli performance. our external knowledge representations are tailored to the nli task and trained to adapt to nli data requirements. we show that our model enhanced with external knowledge achieves better performance than the previous ernie model with non-adapted knowledge depending on the knowledge utilized. we examine these results with several analysis experiments to enable strong conclusions about the correlation between knowledge and nli classification. results also demonstrate that the model is able to handle unseen knowledge when the inference data shifts from training data.