SETI : Systematicity Evaluation of Textual Inference (2023)

Abstract: We propose SETI (Systematicity Evaluation of Textual Inference), a novel and comprehensive benchmark designed for evaluating pre-trained language models (PLMs) for their systematicity capabilities in the domain of textual inference. Specifically, SETI offers three different NLI tasks and corresponding datasets to evaluate various types of systematicity in reasoning processes. In order to solve these tasks, models are required to perform compositional inference based on known primitive constituents. We conduct experiments of SETI on six widely used PLMs. Results show that various PLMs are able to solve unseen compositional inferences when having encountered the knowledge of how to combine primitives, with good performance. However, they are considerably limited when this knowledge is unknown to the model (40-100 % points decrease). Furthermore, we find that PLMs are able to improve dramatically once exposed to crucial compositional knowledge in minimalistic shots. These findings position SETI as the first benchmark for measuring the future progress of PLMs in achieving systematicity generalization in the textual inference.

## introduction natural language inference (nli) determines whether a hypothesis follows from a premise @xcite @xcite and has been explored for decades. existing large pre-trained language models (plms) have shown remarkable performance on this task @xcite @xcite . to better assess the true capabilities of models to perform nli, various associated tasks and benchmarks have been proposed. these works concentrate on exploring how models make predictions, e.g. by establishing 'hard' nli datasets @xcite or asking models to 'explain' their predictions through highlighting @xcite , or by generating plausible explanations @xcite . but little is known about how well such models are able to address compositional generalization. compositional generalization focuses on how to combine primitive units to predict larger compounds @xcite . a key property underlying compositional generalization is systematicity @xcite , a hallmark of human cognition. systematicity concerns the ability of (re)combining known constituents and composing rules. for example, humans who understand 'red apple' and 'green train' are able to conceptualize 'red train' by recombining 'red' and 'train' into a new concept. similar effects of systematicity (generalization) can be studied in natural language understanding (nlu) @xcite . since plms have achieved results on par with human performance by fitting nli training data @xcite , we aim to evaluate to what extent these models can master different types of systematicity in textual inference. we propose a novel benchmark seti (systematicity evaluation of textural inference), which extensively explores systematicity in nli. seti contains three interrelated yet independent tasks covering various types of systematicity: 1) task1: @xmath0. 2) task2: @xmath1. 3) task3: primitives and @xmath2, while the other has only been encountered in compositions. seti can be used to explore systematicity in nli comprehensively since it considers all possibilities of how to construct a novel composition from known constituent types, derived from the 'permutation and combination' foot_0 theory acting between primitives and compositions. we introduce these tasks in detail in section §3. to make the instantiations of systematicity covered in seti easily accessible, we indicate three analogous visual tasks in to apply seti in practice, we define veridical inference @xcite and natural inference as primitives, and their combinations as compositions. for each systematicity task setting, we provide two instantiations: trivial and non-trivial, depending on the variety of instances presented to the model in training. while both settings fulfill the given task requirements, the non-trivial setting is more challenging because the compositional inference knowledge of how to combine constituents is not seen in training. we evaluate six well-known plms on all seti tasks. they show good performance in trivial settings, but inferior results in non-trivial settings, for all tasks. this indicates that models can generalize well to unseen compositions when constituents and compositional knowledge are known, while they are limited when they lack knowledge about how to compose constituents. hence, we further explore whether, and to what extent we can enhance the systematicity capabilities. our experiments indicate that all plms benefit greatly from being exposed to minimal doses of relevant compositional instances. our main contributions are as follows: plms. the results indicate that models generalize well to unseen compositions if they have previously acquired relevant compositional inference knowledge, but are limited when lacking such knowledge. ## related work textual inference natural language inference (nli) involves reasoning across a premise and hypothesis, determining the inferential relationship holding between them @xcite @xcite . as one of the major tasks for establishing natural language understanding (nlu), nli has been widely explored for decades. recently, large pre-trained language models @xcite @xcite exhibit remarkable performance on nli tasks, on par with humans. to better explore the true nli capabilities of models, various associated tasks and benchmarks have been proposed. some work has probed nli models by constructing hypothesis-only baselines @xcite , finding that models capture undesired biases. mccoy et al. (2019); zhou and bansal (2020); gubelmann et al. (2022) reveal that models rely on heuristics, e.g., lexical overlap, subsequence heuristics, etc. nie et al. (2020); chien and kalita (2020) evaluate models in adversarial settings and show robustness improvement by training on additional adversarial data. others focus on explainable nli, such as highlighting input words that are essential for the label @xcite , or generating plausible explanations @xcite . in this work, we focus on exploring the compositional generalization abilities of plms when performing textual inference. systematicity systematicity is a crucial property of compositionality, which was first introduced in cognitive science @xcite and recently formalized in @xcite . it is the ability to make use of known concepts to produce novel concept combinations that have not been encountered before. recently, systematicity has been widely explored in domains such as image caption generation @xcite , visual attribute recognition @xcite , question answering @xcite and semantic parsing @xcite @xcite . in this work, we focus on systematicity in the domain of textual inference. existing works that evaluate systematicity in textual inference only focus on one specific type. for example, @xcite evaluates systematicity by testing the transitivity of inference relations. others conduct experiments on novel compositions involving specific linguistic phenomena, such as systematicity of predicate replacements and embedding quantifiers @xcite , systematicity when combining lexical entailment and negation @xcite , and systematicity of quantifiers, negation and concerning the order between premises and hypotheses @xcite . compared to prior work, we propose a comprehensive systematicity evaluation benchmark seti, which: i) covers the full spectrum of systematicity; ii) evaluates various plms; and ii) showcases how plms can overcome limitations in systematicity. ## reasoning tasks for systematicity we now define primitive and compositional inferences and introduce three nli systematicity tasks. ## conclusion we propose the first comprehensive systematicity evaluation benchmark, seti, applied to natural language inference. experiments on six widely used plms show that they can distinguish novel compositions with known primitives and composing knowledge with high accuracy, but limited when lacking such knowledge. moreover, we show that models can quickly acquire missing inferential knowledge for systematicity by being presented with unique samples representing each missing case of inferential knowledge, in a few-shot setup. recently, @xcite dissect the notion of compositionality and define five theoretically grounded tests for generalization, in a taskagonistic manner. our work is limited to evaluating the systematicity of plms in textual inference. while the systematicity test is one of the most im-portant tests, the remaining ones (e.g., productivity and localism) are still worth to be explored in future works.