Teach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference (2021)

Abstract: We present InferBert, a method to enhance transformer-based inference models with relevant relational knowledge. Our approach facilitates learning generic inference patterns requiring relational knowledge (e.g. inferences related to hypernymy) during training, while injecting on-demand the relevant relational facts (e.g. pangolin is an animal) at test time. We apply InferBERT to the NLI task over a diverse set of inference types (hypernymy, location, color, and country of origin), for which we collected challenge datasets. In this setting, InferBert succeeds to learn general inference patterns, from a relatively small number of training instances, while not hurting performance on the original NLI data and substantially outperforming prior knowledge enhancement models on the challenge data. It further applies its inferences successfully at test time to previously unobserved entities. InferBert is computationally more efficient than most prior methods, in terms of number of parameters, memory consumption and training time.

## introduction transformer-based pre-trained language models (lms), such as bert @xcite and gpt @xcite have recently achieved human-level performance on standard natural language inference (nli) benchmarks @xcite . however, the performance on this complex task is achieved in part thanks to large training sets that facilitate learning of dataset-specific biases and correlations, and thanks to the similar distributions between the training and test sets, that rewards such models @xcite . this contrasts with humans, who can learn a generalized solution from fewer examples (linzen, 2020). indeed, nli models often fail on examples involving various linguistic phenomena such as co-hyponymy @xcite and negation @xcite , which they are expected to acquire indirectly from the nli training set. prior work proposed to provide ("inoculate") nli models with a small number of phenomenonspecific training examples in order to teach the model to address them @xcite . however, @xcite showed that when the distributions of the training and test sets differ with respect to syntactic and lexical properties, the performance of such inoculated models drops, concluding that they do not learn a generalized notion of the phenomenon. in this paper we are motivated by the following question: how can we facilitate learning of generalized inference patterns, with respect to a given linguistic phenomenon, from a relatively small number of examples? ideally, we would like an nli model to learn inference patterns detached from their original context, and to be able to apply them in new contexts involving different concrete facts. for example, an nli model may learn that a word entails its hypernym in upward monotone sentences from training examples such as: alice ate a @xmath0. then, to be able to apply this rule to a test instance with the premise bob saw a pangolin and the hypothesis bob saw an animal, it needs to know that animal is a hypernym of pangolin. training a model on every possible hyponym-hypernym pair is incredibly inefficient and requires re-training a model whenever the vocabulary expands. instead, we propose to decouple the learning of generic inference patterns from that of the factual knowledge. to that end, we develop inferbert, a method to enhance language models with relational knowledge from a knowledge base (kb). in contrast to recent knowledge-enhancement approaches such as knowbert @xcite and ernie @xcite that incorporate into lms knowledge about individual entities (e.g. pangolin), we inform the lm of the relation between a pair of entities that are involved in an inference instance, e.g. hypernym(pangolin) = animal. this approach is agnostic to the identity of the specific entities, allowing models to learn inference patterns separately from the individual facts involved in particular instances. to evaluate the ability of nli models to learn inference patterns for specific linguistic phenomena, we follow the evaluation approach taken in previous work @xcite @xcite , which demonstrated the learning ability of models over a few chosen inference phenomena. we focus on 4 target semantic relations: hypernymy, location, country of origin, and color, for which we create challenge sets foot_0 (see our results confirm that inferbert manages to generalize inference patterns to new facts, substantially improving performance on the challenge sets upon the knowledge-enhanced baselines (up to +17.5 points in accuracy from the next best model), all while maintaining the performance on the original multinli test set @xcite . moreover, inferbert not only learns from a small number of training examples (which are insufficient for the baselines), it is also considerably more efficient than prior knowledge-enhanced lms in terms of training time, resources, and memory. in-ferbert doesn't require lm pre-training, which is a computationally expensive process, and doesn't embed entities, only a small number of relations, substantially reducing the number of parameters with respect to some of the prior work (e.g. only 23% of knowbert's parameters). finally, while inferbert is demonstrated on nli, it is a general method and may benefit additional tasks such as question answering and co-reference resolution which may rely on relational knowledge between words in given instances. ## data we focus on four types of semantic relations (section 3.1), each corresponding to a set of facts in the form of semantic relation triplets. an nli model may learn various inference patterns pertaining to the semantic relation type, such as "a word entails its hypernym in an upward monotone sentence". to evaluate the models' ability to learn and apply these rules, we create an nli challenge set for each semantic relation, that we derive from multinli (section 3.2). as usual, the goal is to determine the label of a premise-hypothesis pair (p, h) among entailment, neutral, and contradiction. for a given semantic relation, each instance in the corresponding challenge set requires applying an inference pattern associated with the semantic relation in order to determine the correct label (possibly along with other required inferences). ## inferbert we present inferbert, a bert-based nli model with a relational knowledge enhancement compo-5 all three annotations were given an equal weight. as we discussed in section 2.2, most prior work injects external knowledge into models through an entity's knowledge base embedding, which captures in a soft way its relationships with other kb entities. the limitation of such methods is the coupling of an inference pattern with the related factual knowledge. suppose that a model observed during training that "the boy ate an apple" entails "the boy ate a fruit". the test example with the premise "the woman has a dog" and the hypothesis "the woman has a pet" is represented differently from the training example due to the distance between the entities (e.g. apple and dog) in the kb. such a model is likely to fail on examples consisting of unseen entities. we propose to decouple learning the inference pattern from the facts by directly embedding the semantic relations between entities in the text. in the above example, inferbert can access the kb during both the training and inference phases, and add an indicator that @xmath1. after observing enough training examples with the hypernym indicator, the model can learn a general rule like "a word entails its hypernym in certain common context". during inference, the model can apply this rule to unseen entities in the kb. we first describe the knowbert model @xcite , section 4.1) which is the basis for in-ferbert. next, we describe how we replace know-bert's knowledge attention and recontextualization component (kar) by our simplified kar (s-kar, section 4.2). ## experiments we present the results of inferbert and the baselines on the various challenge sets (section 6.1). we also test the ability of models to learn relational knowledge about entities seen during training (section 6.2). finally, we analyze inferbert's efficiency in terms of memory and runtime compared to the baselines (section 6.3). ## conclusions we presented inferbert, a generic and efficient method to incorporate relational knowledge into transformer-based inference models. our approach targets specific inference phenomena that require external relational knowledge, allowing the model to learn generic inference patterns decoupled from the factual knowledge required for a particular instance, which is injected at inference time. our experiments show that inferbert successfully applies the learned patterns to unseen facts, where other knowledge enhancement models fail. unlike most prior work, inferbert does not require pre-training the lm on a kb, and consumes less memory. our work joins the effort of others to improve models by teaching them specific inference phenomena @xcite . a natural direction for future work would be to apply our methodology to a broader range of inference phenomena and adapt them for additional inference tasks.