Continuous Entailment Patterns for Lexical Inference in Context (2021)

Abstract: Combining a pretrained language model (PLM) with textual patterns has been shown to help in both zero- and few-shot settings. For zero-shot performance, it makes sense to design patterns that closely resemble the text seen during self-supervised pretraining because the model has never seen anything else. Supervised training allows for more flexibility. If we allow for tokens outside the PLM’s vocabulary, patterns can be adapted more flexibly to a PLM’s idiosyncrasies. Contrasting patterns where a “token” can be any continuous vector from those where a discrete choice between vocabulary elements has to be made, we call our method CONtinous pAtterNs (CONAN). We evaluate CONAN on two established benchmarks for lexical inference in context (LIiC) a.k.a. predicate entailment, a challenging natural language understanding task with relatively small training data. In a direct comparison with discrete patterns, CONAN consistently leads to improved performance, setting a new state of the art. Our experiments give valuable insights on the kind of pattern that enhances a PLM’s performance on LIiC and raise important questions regarding our understanding of PLMs using text patterns.

## introduction lexical inference in context (liic) -also called predicate entailment -is a variant of natural language inference (nli) or recognizing textual entailment @xcite with focus on the lexical semantics of verbs and verbal expressions @xcite . its goal is to detect entailment between two very similar sentences, i.e., sentences that share subject and object and only differ in the predicate, e.g., @xcite runs org(b) → person(a) leads org(b). nli models that were not specifically trained with lexical knowledge have been reported to struggle with this task @xcite , making liic an important evaluation criterion for general language understanding. other use cases for this kind of lexical entailment knowledge include question answering @xcite , event coreference @xcite , and link prediction in knowledge graphs @xcite . although liic is an inherently directional task, symmetric cosine similarity in a vector space, such as word2vec @xcite , has long been the state of the art for this task. only recently transfer learning with pretrained transformer @xcite language models @xcite , has led to large improvements for liic. @xcite combine natural language (nl) patterns with a pretrained language model (plm) and not only set a new state of the art but also beat baselines without access to such patterns. empirical findings suggest that a good pattern can be worth 100s of labeled training instances (le scao and rush, 2021), making pattern approaches interesting for low-resource tasks such as liic. but beyond the intuition that patterns serve as some sort of task instruction @xcite , little is known about the reasons for their success. recent findings that (i) plms can fail to follow even simple instructions @xcite , that (ii) plms can behave drastically different with paraphrases of the same pattern @xcite , and that (iii) performance increases if we train a second model to rewrite an input pattern with the goal of making it more comprehensible for a target plm @xcite , strongly suggest that patterns do not make sense to plms in the same way as they do to humans. our work sheds light on the interaction of patterns and plms and proposes a new method of improving pattern-based models fully automatically. on two popular liic benchmarks, our model (i) establishes a new state of the art without the need for handcrafting patterns or automatically identifying them in a corpus and (ii) does so more efficiently thanks to shorter patterns. our best model only uses 2 tokens per pattern. ## the conan model continuous patterns. liic is a binary classification task, i.e., given a premise @xmath0. . . p |p| and a hypothesis @xmath1. . . h |h| a model has to decide whether p entails h (@xmath2. a template-based approach to this task surrounds p and h with tokens t 1 t 2 . . . t m to bias the classifier for entailment detection, e.g., "p, which means that h".while in most approaches that leverage a plm m these tokens come from the plm's vocabulary, i.e., ∀i. t @xmath3, we propose a model based on continuous patterns (conan), i.e., surround the embedding representation of p and h with continuous vectors that may be close to but do not have to match the embedding of any vocabulary entry. for this, we first extend the plm's vocabulary by a finite set @xmath4, c 2 , . . . , c |c| of fresh tokens, i.e., σ = σ m ∪ c with c ∩ σ @xmath5. then, we distinguish two methods of surrounding p and h with these special tokens: α sets them both αround and between p and h (eq. ( 1 )) while β only sets them βetween the two input sentences (eq. ( 2 )). note that α divides its k tokens into three parts as equally as possible where any remaining tokens go between p and h if k is not a multiple of 3. in particular, this means that the same templates are produced by α and β for @xmath6. we chose this behavior as a generalization of the standard approach to fine-tuning a plm for sequence classification (such as nli) where there is only one special token and it separates the two input sequences. the template produced by α 1 and β 1 is very similar to this. a major difference is that the embeddings for c tokens are randomly initialized whereas the standard separator token has a pretrained embedding. pattern-based classifier. given γ ∈ { α, β }, we estimate the probability distribution p (ŷ | p, h) with a linear classifier on top of the pooled sequence representation produced by the plm m: inference with multiple patterns. previous work @xcite combined multiple patterns with the intuition that different nl patterns can capture different aspects of the task. this intuition makes also sense for conan. we conjecture that an efficient use of the model parameters requires different continuous patterns to learn different representations, which can detect different types of entailment. following the aforementioned work, we form our final score s by combining the probability estimates from different patterns γ by comparing the maximum probability for the two classes 0, 1 over all patterns: in conclusion, a conan model γ n k is characterized by three factors: (i) the type of pattern γ ∈ { α, β }, (ii) the number of patterns @xmath7, and (iii) the number of tokens @xmath8. training. while multiple patterns are combined for decision finding during inference, we treat all patterns separately during training -as did previous work @xcite . so, given a set of patterns γ, we minimize the negative loglikelihood of the training data t , i.e., in practice, we apply mini-batching to both t and γ and thus compute this loss only for a fraction of the available training data and patterns at a time. in this case, we normalize the loss by averaging over the training samples and patterns in the mini-batch. ## experiments we conduct experiments on two established liic benchmarks, sherliic (schmitt and schütze, 2019) and levy/holt @xcite , using the data splits as defined in (schmitt and schütze, 2021) for comparison. both benchmarks contain a majority of negative examples (sherliic: 67%, levy/holt: 81%), making the detection of the entailment (i.e., the minority) class a particular challenge. see following (schmitt and schütze, 2021), we use roberta as underlying plm and also use the same hyperparameters whenever possible for comparison. also following schmitt and schütze (2021), we instantiate the typed placeholders a, b in sherliic with freebase @xcite entities, making sure that a and b are not assigned the same entity. see appendix a for full training details. we evaluate model performance with two metrics: (i) the area under the precision-recall curve for precision @xmath9.5 (auc) as threshold-less metric using only the score s defined in the previous section and (ii) the f1 score of actual classification decisions after tuning a decision threshold ϑ on the respective dev portion of the data. our implementation is based on @xcite . ## results choosing n and k. the number n of patterns and the number k of c tokens per pattern are essential hyperparameters of a conan model. fig. 1 shows the impact on performance (measured in auc) on sherliic dev for different n-kcombinations. we observe that using too many, too long patterns harms performance w.r.t. the base case @xmath10. best results are obtained with either a small number of patterns or tokens or both. comparing the α and β settings, we notice that, rounded to one decimal, they produce identical results for both @xmath11, suggesting that the particular position of the c tokens does not matter much in these settings for sherliic. even with @xmath12, the two methods only begin to differ with @xmath13. evaluation on the levy/holt data (see fig. 2 in appendix b) shows more variation between α and β but, otherwise, confirms the trend that small n and k yield better performance. our results offer an explanation for the empirical finding in @xcite ) that patterns retrieved from a corpus lead to worse performance than handcrafted ones because the latter are generally shorter. conan models do not only yield better performance, they also provide an automatic way to test pattern properties, such as length, w.r.t. effect on performance for a given task. test performance. on both sherliic ( we find that with the very few train samples in sherliic the risk of overfitting to sherliic is indeed higher. when trained on levy/holt with around 4.4k train samples, however, conan clearly improves generalization to the sherliic domain. ## related work plms and text patterns. gpt-2 @xcite made the idea popular that a plm can perform tasks without access to any training data when prompted with the right nl task instructions. with @xcite adapted this idea to fewshot settings where the task prompt is extended by a few training samples. while this kind of few-shot adaptation with a frozen plm only works with very big models, schick and schütze (2021b) achieve similar performance with smaller models by finetuning the plm on the available training data and putting them into nl templates. recently, schmitt and schütze (2021) investigated the use of plms for liic. compared to a standard sequence classification fine-tuning approach, they were able to improve the plm roberta's performance by putting an entailment candidate into textual contexts that only make sense for either a valid or invalid example. patterns like "y because x." (valid) or "it does not mean that y just because x." (invalid) make intuitive sense to humans and outperform standard roberta on liic. a large problem with all these approaches, however, is to find well-functioning patterns, for which numerous solutions have been proposed @xcite @xcite @xcite . we argue that it is not optimal to constrain pattern search to the space of nl sequences if the primary goal is better task performance, and therefore abandon this constraint. plms and continuous patterns. li and liang (2021) and @xcite contemporaneously introduced the idea of mixing the input token embeddings of a plm with other continuous vectors that do not correspond to vocabulary elements. in the spirit of gpt-2 (see above), they keep the plm's parameters frozen and only fine-tune the embeddings of the "virtual tokens" to the target task. while this line of research offers certain appeals of its own, e.g., reusability of the frozen plm weights, this is not the focus of our work. in pursuit of the best possible performance, we instead compare the use of continuous vs. nl patterns in the process of fine-tuning all plm parameters and find that even carefully chosen nl patterns can be outperformed by our automatically learned ones. contemporaneously to our work, @xcite fine-tune entire plms with continuous patterns for superglue @xcite . besides reformulating the superglue tasks as cloze tasks, while we keep formalizing our task as classification, liu et al. (2021) also add more complexity by computing the continuous token representations with an lstm @xcite and adding certain "anchor tokens", such as a question mark, at manually chosen places. conan does not use any manual pattern design and embeds continuous tokens with a simple lookup table. another contemporaneous work by @xcite tests the influence of model size on the performance of a frozen plm with trained continuous prompts. their prompt ensembling is akin to our combining multiple patterns during inference (cf. §2). the key difference is that, instead of making predictions with different patterns and taking the majority vote, we rather compare the scores for different patterns to make our prediction. ## conclusion we presented conan, a method that improves finetuning performance of a plm with continuous patterns. conan does not depend on any manual pattern design and is efficient as the shortest possible patterns with good performance can be found automatically. it provides an automatic way of systematically testing structural properties of patterns, such as length, w.r.