Language Models for Lexical Inference in Context (2021)

Abstract: Lexical inference in context (LIiC) is the task of recognizing textual entailment between two very similar sentences, i.e., sentences that only differ in one expression. It can therefore be seen as a variant of the natural language inference task that is focused on lexical semantics. We formulate and evaluate the first approaches based on pretrained language models (LMs) for this task: (i) a few-shot NLI classifier, (ii) a relation induction approach based on handcrafted patterns expressing the semantics of lexical inference, and (iii) a variant of (ii) with patterns that were automatically extracted from a corpus. All our approaches outperform the previous state of the art, showing the potential of pretrained LMs for LIiC. In an extensive analysis, we investigate factors of success and failure of our three approaches.

## introduction lexical inference (li) denotes the task of deciding whether or not an entailment relation holds between two lexical items. it is therefore related to the detection of other lexical relations like hyponymy between nouns @xcite , e.g., dog ⇒ animal, or troponymy between verbs @xcite , e.g., to traipse ⇒ to walk. lexical inference in context (liic) adds the problem of disambiguating the pair of lexical items in a given context before reasoning about the inference question. this type of li is particularly interesting for entailments between verbs and verbal expressions because their meaning -and therefore their implications -can drastically change with different arguments. consider, e.g., run ⇒ lead in a person / company context ("bezos runs amazon") vs. run ⇒ execute in a computer / software context ("my mac runs macos"). liic is thus also closely related to the task of natural language inference (nli) -also called recognizing textual entailment @xcite -and can be seen as a focused variant of it. besides the important use case of evaluating nli systems, this kind of predicate entailment has also been shown useful for question answering @xcite , event coreference @xcite , and link prediction in knowledge graphs @xcite . despite its nli nature, previous systems for liic have primarily been models of lexical similarity @xcite or models based on verb argument inclusion @xcite . the reason is probably that supervised nli models need large amounts of training data, which is unavailable for liic, and that systems trained on available large-scale nli benchmarks (e.g., @xcite have been reported to insufficiently cover lexical phenomena @xcite . recently, transfer learning has become ubiquitous in nlp; transformer @xcite language models (lms) pretrained on large amounts of textual data @xcite form the basis of a lot of current stateof-the-art models. besides zero-and few-shot capabilities @xcite , pretrained lms have also been found to acquire factual and relational knowledge during pretraining @xcite . the entailment relation certainly stands out among previously explored semantic relations -such as the relation between a country and its capital -because it is very rarely stated explicitly and often involves reasoning about both the meaning of verbs and additional knowledge @xcite . it is unclear whether implicit clues during pretraining are enough to learn about liic and what the best way is to harness any such implicit knowledge. regarding these questions, we make the follow-ing contributions: (1) this work is the first to explore the use of pretrained lms for the liic task. ## related work lexical inference. there has been a lot of work on lexical inference for nouns, notably hypernymy detection, resulting in a variety of benchmarks @xcite and methods @xcite . although there has been work on predicate entailment before @xcite , @xcite were the first to create a general benchmark for evaluating entailment between verbs. in their evaluation, neither resource-based approaches @xcite nor vector space models @xcite achieved satisfying results. holt (2018) later published a re-annotated version, which was readily adopted by later work. @xcite put global constraints on top of directed local similarity scores @xcite @xcite based on distributional features of the predicates. @xcite replaced these scores by transition probabilities in a bipartite graph where edge weights are computed by a link prediction model. when @xcite created the sherliic benchmark, they also mainly focused on resource-and vector-based models for evaluation. their best model combines general-purpose word2vec representations @xcite with a vector representation of the arguments that co-occur with a predicate. all these works (i) base the probability of entailment validity on the similarity of the verbs and (ii) compute this similarity via (expected) co-occurrence of verbs and their arguments. our work differs in that our models solely reason about the sentence surface in an end-to-end nli task without access to previously observed argument pairs. this is possible because our models have learned about these surface forms during pretraining. patterns and entailment. pattern-based approaches have long been known for hypernymy detection @xcite . recent work combined them with vector space models @xcite @xcite . while there are effective patterns, such as x is a y , that are indicative for entailment between nouns, there is little work on comparable patterns for verbs. @xcite mine symmetric patterns for lexical similarity and achieve good results for verbs. entailment, however, is not symmetric. @xcite handcrafted 35 patterns to distinguish 6 semantic relations for pairs of distributionally similar verbs. some of their classes like strength (taint :: poison) or antonymy (ban :: allow) can be indicators of entailment and nonentailment but are, in general, much more narrowly defined than the patterns we use in our approach. another difference to our work is that verb pairs are scored based on co-occurrence counts on the web, while we employ an lm, which does not depend on a valid entailment pair actually appearing together in a document. patterns and language models. amrami and goldberg (2018) were the first to manipulate lm predictions with a simple pattern to enhance the quality of substitute words in a given context for word sense induction. @xcite found that large pretrained lms can be queried for factual knowledge, when presented with appropriate pattern-generated cloze-style sentences. this zeroshot factual knowledge has later been shown to be quite fragile @xcite . so we rather focus on approaches that fine-tune an lm on at least a few samples. @xcite train a binary classifier on top of a fine-tuned bert @xcite to predict the truth value of handwritten statements about objects and their properties. while their experiments investigate bert's physical common sense reasoning, we focus on the different phenomenon of entailment between two actions expressed by verbs in context. schick and schütze (2020) used handcrafted patterns and lms for few-shot text classification. based on manually defined label-token correspondences, the predicted classification label is determined by the token an lm estimates as most probable at a masked position in the cloze-style pattern. we differentiate entailment and non-entailment via compatibility scores for patterns and antipatterns and not via different predicted tokens. addressing relation induction, @xcite propose an automatic way of finding, given a relation, lm patterns that are likely to express it. they train a binary classifier per relation on the sentences generated by these patterns. while some of the relations they consider are related to verbal entailment (e.g., cook activity-goal eat), most of them concern common sense (e.g., library locationactivity reading) or encyclopedic knowledge (e.g., paris capital-of france). we adapt their method for the automatic retrieval of promising patterns for liic, but find that handcrafted patterns that capture the generality of the entailment relation still have an advantage over automatic patterns for liic. another important novelty we introduce is the use of antipatterns. while @xcite have to use negative samples for training their classifiers, they only consider patterns that exemplify the desired relation. in contrast, we also use antipatterns that exemplify what the entailment relation is not. we believe that antipatterns are particularly useful for entailment detection because they can help identify other kinds of semantic relations that often pose a challenge to vector space models @xcite . ## analysis ## conclusion we proposed and evaluated three approaches to the task of lexical inference in context (liic) based on pretrained language models (lms). in particular, we found that putting an inference candidate into a pattern-generated context mostly increases performance compared to a standard sequence classification approach. concrete performance, however, also depends on the particular dataset, used lm (variant), and pattern set. we introduced the concept of antipatterns, which express the negative class of a binary classification, and found that they often lead to performance gains for liic. we set a new state of the art for liic and conducted an extensive analysis of our approaches. notably, we found that automatically created patterns can perform nearly as well as handcrafted ones if we either use the right number n of patterns or manually identify the right subset of them. promising directions for future work are the investigation of alternative automatic pattern generation methods or a better modeling of the remaining challenges we described in our error analysis ( §6.3). batch size of 10 or 2 instances for roberta-base and -large, respectively. for autpat n approaches with @xmath0, we distribute the available patterns and antipatterns into chunks of size 5 for training to save memory. during evaluation, the predictions are based on all the patterns and antipatterns. we randomly sample 500 configurations for the remaining hyperparameters, i.e., initial learning rate lr, weight decay λ (l2 regularization), and the number of batches c the gradient is accumulated before each optimization step, which virtually increases the batch size by a factor of c. the hyperparameters are sampled from the following intervals: @xmath1, 5 • 10 -2 ], λ ∈ [10 -5 , 10 -1 ], @xmath2, 2, . . . , 10 }. lr and λ are sampled uniformly in log-space. for a fair comparison, we use the same 500 random configurations for all of our approaches. as usual for transformer models, we apply a learning rate schedule: lr decreases linearly such that it reaches 0 at the end of the last epoch. we do not employ warm-up. the best configurations can be seen in tables 8 and 10 for levy/holt's dataset and in tables 9 and 11 for sherliic.