Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference (2022)

Abstract: The task of natural language inference (NLI), to decide if a hypothesis entails or contradicts a premise, received considerable attention in recent years. All competitive systems build on top of contextualized representations and make use of transformer architectures for learning an NLI model. When somebody is faced with a particular NLI task, they need to select the best model that is available. This is a time-consuming and resource-intense endeavour. To solve this practical problem, we propose a simple method for predicting the performance without actually fine-tuning the model. We do this by testing how well the pre-trained models perform on the aNLI task when just comparing sentence embeddings with cosine similarity to what kind of performance is achieved when training a classifier on top of these embeddings. We show that the accuracy of the cosine similarity approach correlates strongly with the accuracy of the classification approach with a Pearson correlation coefficient of 0.65. Since the similarity is orders of magnitude faster to compute on a given dataset (less than a minute vs. hours), our method can lead to significant time savings in the process of model selection.

## introduction abduction is a type of reasoning that infers an explanation for some observations @xcite . it is a particularly challenging type of inference; as opposed to deduction and induction, which derive the conclusion from only the information present in the observations, abduction requires making assumptions about an implicit context beyond just the given observations. abductive reasoning is therefore at the core of the way humans understand the world and how world knowledge is involved. abductive reasoning in the natural language domain has been introduced by @xcite who defined the abductive natural language inference (αnli) task. in it, we are given four sentences -two observations o 1 and o 2 and two hypotheses h 1 and h 2 , where we know that the sequence of events was o 1 → (h 1 |h 2 ) → o 2 . the task then is to decide which of the two hypotheses is the more plausible one. an example from @xcite is the following: o 1 : it was lunchtime and kat was hungry. o 2 : kat and her coworkers enjoyed a nice lunch outside. h 1 : kat went to get a salad. h 2 : kat decided to take a nap instead of eating. while it is not inconceivable that someone would decide to take a nap on their lunch break (h 2 ), given o 2 the first hypothesis becomes more likely. currently, transformer-based architectures @xcite are state of the art in a wide variety of natural language processing (nlp) tasks @xcite @xcite , including αnli. however, with an ever-changing landscape of transformer models and pre-training techniques (with over 10000 foot_0 different fine-tuned models available on the huggingface hub @xcite ), finding the best model for a given task has become a time-consuming process since, in order to compare multiple models, they each need to be separately fine-tuned on the task. this model selection process might lead to a prohibitive runtime, which has led to research on performance prediction, namely to predict the expected performance out of parameters of the model configuration, without actually training the model. this procedure has been evaluated for a set of nlp tasks, including span prediction @xcite and language modelling @xcite . however, we are not aware of any previous work that performed performance prediction for αnli. in this paper, we introduce a fast performance prediction method for the αnli task that allows a more guided way of choosing which models to fine-tune. we use various pre-trained transformer models to embed the observations and hypotheses with the approach introduced in @xcite , then compare which hypothesis is closer to the observations with cosine similarity. we find that the performance of the similaritybased approach is correlated to results obtained via fine-tuning. therefore, the similarity-based approach can serve as a performance prediction method. ## related work there are three research topics that need to be mentioned. approaches to abductive reasoning, pretrained language models, and performance prediction. in this section, we explore them in detail. abductive natural language inference. nli has been proposed as the task of recognizing textual entailment by @xcite and now constitutes a major challenge in nlp which has found application for other downstream tasks, including question answering or zero-shot classification @xcite . based on the initial goal of establishing inference relations between two short texts, a myriad of variants have been proposed @xcite @xcite . one such variant is abductive nli @xcite . transformer-based architectures dominate the αnli leaderboard, foot_1 including roberta-based models @xcite which explore how additional knowledge can improve performance on reasoning tasks, and @xcite who approach αnli as a ranking task. the task authors improved upon their result in @xcite by using a t5 model @xcite and experimenting with multi-task training and finetuning over multiple reasoning tasks. both the multi-task criteria and the text-to-text framework of t5 help the model generalize and understand the context better. the second-best model on the leaderboard is a deberta model @xcite . the model replaces the masked language modelling task with a replaced-token detection task. it also uses a disentangled attention mechanism to encode the position and content information. the current state of the art shows an accuracy of 91.18% using a new unified-modal pre-training method to leverage multi-modal data for singlemodal tasks @xcite . this result approaches the human baseline of 92.9%. pre-trained language models. the αnli task requires the model to successfully "understand" the context of both the observations and use that understanding to identify the more likely hypothesis entailing it. most semantic representations in practical applications rely on distributional semantics. such methods include the word-level embedding methods word2vec @xcite and glove @xcite and language model-based word representation like elmo @xcite , ulmfit @xcite , and gpt @xcite . the current state of the art are pre-trained transformer architectures @xcite like bert @xcite , which use a masked language modelling and next sentence prediction objective. this not only helps the model understand the context within a sentence but also inbetween consecutive sentences. there is however a trade-off in terms of the time it takes to train transformer models. for example, a from-scratch training of bert takes 6.4 days on an 8 gpu nvidia v100 server foot_2 . @xcite recommend finetuning the language model between 2-4 epochs for a given task. however, in practice, multiple trials are required to find the optimal hyperparameters. these long training times and multiple fine-tuning runs make model selection a time-intensive process @xcite . performance prediction. the task of performance prediction is to estimate the performance of a specific system without explicitly running it. it helps in setting hyperparameters, finding feature sets, or identifying candidate language models for a downstream nlp task. chen (2009) develop, for instance, a generalized method for predicting the performance of exponential language models. they analyze the backoff features in an exponential n-gram model. @xcite leverage meta-learning to identify candidate model performance on the task of span identification. they train a linear regressor as a meta-model to predict span id performance based on model features and task metrics for an unseen task. @xcite propose performance prediction methods particularly suited for fine-grained evaluation metrics. they also develop methods for estimating the reliability of these performance predictions. contrary to the previously mentioned papers, @xcite build regression models to predict the performance across a variety of nlp tasks, however, they do not consider nli as one of them. in contrast to our work, all these previous methods build on top of the idea to train a surrogate model for performance prediction and depend on the information about past runs of these models. our approach focuses solely on the embeddings provided by the language model and leverages those as a predictor of performance. this particular setup is also motivated by the αnli task itself, in which a sentence needs to be chosen for a given set of other sentences. ## methods our paper investigates how well a fine-tuned transformer model's performance on the αnli task @xcite is approximated by the cosine similarity of embeddings of the input sentences which we obtain from the pre-trained models before fine-tuning them. intuitively, if a model embeds the correct hypothesis close to the observations in some latent space (not necessarily a semantic similarity space), then a classification model built on top of that latent space should have an easier time discerning which is the correct hypothesis, because apparently that latent space captured some features that were salient for the αnli task. ## experiments we compare the similarity-prediction-based αnli approach and the classification-based αnli approach to evaluate if the first can act as an approximation for the performance expected by the second. we use the pre-trained transformer models which are available on the huggingface @xcite hub. the full list of models we use is listed in ## conclusions & future work in this paper, we have shown that similarity measures based on the distributional semantic representation in pre-trained transformer models serve as an effective proxy for fine-tuned transformer-based classification in αnli. since fine-tuning a transformer model takes notably more time than performing similarity comparisons, our approach supports efficient model selection procedures. future work should investigate the suitability of similaritybased performance prediction for other similar tasks, like next sentence prediction, question answering, summarization.