How Fast can BERT Learn Simple Natural Language Inference? (2021)

Abstract: This paper empirically studies whether BERT can really learn to conduct natural language inference (NLI) without utilizing hidden dataset bias; and how efficiently it can learn if it could. This is done via creating a simple entailment judgment case which involves only binary predicates in plain English. The results show that the learning process of BERT is very slow. However, the efficiency of learning can be greatly improved (data reduction by a factor of 1,500) if task-related features are added. This suggests that domain knowledge greatly helps when conducting NLI with neural networks.

## introduction entailment judgment @xcite ) is a common test for natural language inference (nli) @xcite as it possesses the simplest form in related tasks such as question and answering @xcite . also, snli dataset @xcite is frequently adopted for nli evaluation because it is the first corpus to show the power of neural networks for the task that specifically targets nli. recently, bert @xcite , xlnet @xcite , and roberta @xcite have all shown excellent performances on snli, surpassing even human performance @xcite . however, @xcite and @xcite show that snli contains hidden bias. also, deep neural networks (dnns) have been shown to predominantly capture the statistical irregularities that go unnoticed by humans @xcite ; this surprisingly yields 69% accuracy on snli without being provided the associated premise (i.e., the required supporting evidence). furthermore, some studies @xcite @xcite have shown that bert mainly conducts nli with surface clues/patterns, but not those clues actually adopted by humans. to the best of our knowledge, none of previous studies carefully removes the dataset bias, and then investigates how efficiently that bert could learn nli if the surface clues are completely removed. in this paper, we empirically study whether bert is capable of learning nli without surface clues/bias appearing in the dataset; and if it is, whether it can learn nli efficiently. we carefully created absolutely unbiased datasets, in which the premises and hypotheses simply describe the relative position of two objects in plain english. that is, given a premise "john is on the left side of mary" (abbreviated as a predicate "left(john, mary)" from now on), the hypotheses predicates "left(john, mary)", "left(mary, john)" and "left(john, helen)" should be labeled as entailment, contradictory and neutral respectively. our experiment results show bert is very slow in learning this simple nli. we then further study if the learning efficiency can be improved with domain knowledge @xcite . inspired by @xcite , we think whether two entities are exactly the same is quite crucial in making the above nli. so, the task-related features, such as whether the first/second argument of the premise predicate exactly matches that of hypothesis, are fed into bert. the obtained results show that such task-related features are able to greatly benefit bert (reducing the data needed by a factor of 1,500), which is important as it is difficult to acquire enough data in many real-world applications. our main contributions are: (1) we are the first to quantitatively study how efficiently bert can learn to conduct nli without available surface clues/bias. (2) we design experiments to completely eliminate hidden bias while evaluating the inference capability of bert. (3) we show that adding task-related features greatly enhances the learning efficiency of bert. ## teaching bert binary predicates the sentence "john is on the left side of mary" describes a positional relation between two people. for conciseness, it will be denoted by a binary predicate "left(john,mary)" from now on, where "left" is the predicate name, "john" is the first argument and "mary" is the second argument. we seek to determine how much data are required to teach bert to truly understand this simple binary predicate (i.e., premise) and correctly judge that the hypothesis "left(mary,john)" is contradictory and that the hypothesis "left(john,helen)" is neutral. besides, we also seek to determine whether bert is also able to learn the antonymous predicate "right( ⋅,⋅ )", and judge that the hypothesis "right(mary,john)" is entailed by the above premise. ## incorporating human knowledge the previous section showed that many training examples are needed to teach bert for understanding two binary predicates "leftሺ⋅,⋅ሻ" and "rightሺ⋅,⋅ሻ" in a simple rte task. it thus naturally leads to a conjecture that training bert to understand more complicated predicates would very likely require infeasible amount of training data. one possible solution is to improve the learning curve of bert by feeding it useful/obvious features that have been identified in human/domain knowledge. ## conclusion this paper is the first quantitative study on whether bert could really learn to conduct nli without implicitly utilizing hidden dataset bias, and how quickly it does so if it could. we conduct experiments to evaluate the capability of bert on making inference without hidden bias, and show that bert learns nli inefficiently even for a simple case. we further add task-related features to greatly enhance bert's learning efficiency. as a result, it suggests that domain knowledge may be essential in conducting nli with neural networks (at least for bert).