Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Features (2023)

Abstract: Detecting out-of-distribution (OOD) inputs is crucial for the safe deployment of natural language processing (NLP) models. Though existing methods, especially those based on the statistics in the feature space of fine-tuned pre-trained language models (PLMs), are claimed to be effective, their effectiveness on different types of distribution shifts remains underexplored. In this work, we take the first step to comprehensively evaluate the mainstream textual OOD detection methods for detecting semantic and non-semantic shifts. We find that: (1) no existing method behaves well in both settings; (2) fine-tuning PLMs on in-distribution data benefits detecting semantic shifts but severely deteriorates detecting non-semantic shifts, which can be attributed to the distortion of task-agnostic features. To alleviate the issue, we present a simple yet effective general OOD score named GNOME that integrates the confidence scores derived from the task-agnostic and task-specific representations. Experiments show that GNOME works well in both semantic and non-semantic shift scenarios, and further brings significant improvement on two cross-task benchmarks where both kinds of shifts simultaneously take place. Our code is available at https://github.com/lancopku/GNOME .

## introduction the pre-training and fine-tuning paradigm based on transformers @xcite has achieved tremendous success in various natural language understanding (nlu) tasks @xcite @xcite . however, fine-tuned pre-trained language models (plms) notoriously suffer from over-confident predictions on out-ofdistribution (ood) inputs @xcite . as this issue threats the reliability of nlp models deployed in the open world, textual ood detection has attracted great attention recently @xcite @xcite , 2022, etc.), which aims to enable the model to abstain from making unreasonable predictions on ood data and resort to human intervention. nonetheless, almost all of the current approaches are assessed under certain assumptions about the type of ood texts. one line of works creates in-distribution (id) and ood pairs from arbitrary datasets for different tasks @xcite , while another line assumes that ood data belong to classes in the id task but unseen during training, e.g., in intent recognition @xcite . @xcite reveal the inconsistency among the evaluation protocols and category the distribution shifts to non-semantic shifts (nss) and semantic shifts (ss), but a thorough comparison of existing methods in different settings is missing as later works either focus on either detecting nss @xcite or ss @xcite . in this work, we systematically evaluate the mainstream textual ood detection methods on a comprehensive suite of benchmarks covering both nss and ss scenarios. as shown in figure 1 , no single method wins across the board. notably, the detec-tors based on the pre-trained features, e.g., the mahalanobis distance detector md pre @xcite , excel at detecting non-semantic shifts but fail in detecting semantic shifts. in contrast, when the plm is fine-tuned on annotated id data, the detectors based on fine-tuned features, e.g., md ft @xcite , perform well in the ss scenario but disastrously fail in the nss setting. these observations uncover an intriguing trade-off: fine-tuning contributes to the detection of semantic shifts but impairs the detection of non-semantic shifts. this trade-off raises two critical research questions: rq1: why does fine-tuning undermine the detection of non-semantic shifts? it is relatively easy to attribute the positive effect of fine-tuning in the ss setting to the learned class-discriminative features @xcite , but it remains unknown why fine-tuning plays a negative role in the nss setting. we empirically find that the adverse effect comes from the fact that fine-tuning gradually destructs the pre-trained task-agnostic knowledge about general linguistic properties, which are useful cues for the detection of non-semantic shifts. rq2: how to develop a general textual ood detection method? since the type of distribution shifts is unknown in practice, our findings suggest that a practical method able to detect different kinds of ood texts is yet to be developed. to this end, we aggregate the distance scores estimated in the feature space of both pre-trained and fine-tuned models to derive a general textual ood measurement score (gnome) capable of detecting both nss and ss. on the suite of benchmarks covering both nss and ss settings, gnome (the green star in figure 1 ) surpasses the previous sota by 8.13 far95 points on average; on two cross-task benchmarks where both kinds of shifts happen simultaneously, gnome reduces the average far95 by 4.88 points. note that gnome is not meant to be a sota method in all settings but rather a simple, principled way to get reasonable detection performance under various kinds of distribution shifts-we hope our analysis inspires better approaches for general textual ood detection. ## related work ood detection aims to detect abnormalities coming from a different distribution from the training data so that the model can refuse to make predictions on them @xcite . since it is essential for the security of machine learning models deployed in the open-world environment, ood detection has gained great attention, first in computer vision (cv). we categorize the mainstream ood detection methods into three groups by way to derive confidence scores: (1) confidence-based methods using the output probabilities of classifiers trained on in-distribution (id) data @xcite @xcite ; (2) density-based methods using density scores derived from generation models @xcite @xcite ; (3) distance-based methods using the distance statistics in the feature space of neural networks @xcite @xcite . following the progress in cv, textual ood detection based on plms has also attracted increasing attention. @xcite show that the maximum softmax probability (msp) score @xcite is a strong baseline for plms, followed by a group of works on confidence-based textual ood detection @xcite @xcite . as for the density-based branch, @xcite and @xcite apply the idea to textual ood detection by leveraging language models such as lstm @xcite and gpt-2 @xcite . regarding the distance-based methods, @xcite revisit the mahalanobis distance-based detector @xcite for textual ood detection based on fine-tuned plms and achieve performance gains over confidence-based methods, which is then further improved by introducing contrastive regularization @xcite , utilizing nearestneighbor distance @xcite , and leveraging intermediate features @xcite . nonetheless, the nlp community lacks uniform evaluation criteria for ood detection. generally, id/ood pairs for evaluation are constructed in three ways: (1) the non-semantic shift (nss) setting (a.k.a., the background shift setting) @xcite @xcite , where id and ood data consist of the same semantic classes but differ in background information, 1 e.g., tweets as id and wikipedia comments as ood in toxicity detection; (2) the semantic shift (ss) setting @xcite , where ood data are composed of unseen classes belonging to the id task, e.g., new classes in intent classification; (3) the cross-task setting @xcite , where the id and ood data are from datasets for different tasks and both semantic and non-semantic shifts happen, e.g., sentiment analysis data as id and news classification data as ood. @xcite first notice the inconsistency and compare confidencebased and density-based methods in nss and ss settings, but they neglect the crucial branch of distance-based methods and the cross-task setting. in this work, we fill in this gap by presenting a comprehensive evaluation and developing a general textual ood score motivated by our observations. ## observations and explanations in this section, we first give preliminaries in § 3.1. then we introduce our benchmark for evaluating textual ood detection ( § 3.2) and the evaluated methods ( § 3.3). finally, we present the evaluation results ( § 3.4) and our interpretation of the observed trade-off between nss and ss scenarios ( § 3.5). ## gnome for textual ood detection in view of the observed trade-off, we are motivated to combine the strengths of task-agnostic and task-specific representations to obtain a confidence score capable of modeling both non-semantic shifts and semantic shifts. a straightforward way is to take the mean of md pre and md ft scores. 3 however, given that the norm of features can fluctuate and thus the distance scores are not comparable across different spaces, simple averaging may cause the integrated score to be skewed towards the side with the larger norm. to alleviate the issue, we normalize md pre and md ft before aggregation: ## conclusion aware of the lack of a fair and comprehensive evaluation of current textual ood detection methods, we take the first step to systematically assess them under different distribution shifts. interestingly, we find that no single method works well in both the non-semantic shift setting and the semantic shift setting, and there exists a trade-off: fine-tuning pre-trained language models on in-distribution data benefits detecting semantic shifts but undermines detecting non-semantic shifts. after presenting empirical explanations for the trade-off from the perspective of feature distortion, we are then motivated to fully utilize both the pre-trained and fine-tuned features to obtain an efficient measurement score gnome for better detecting diverse distribution shifts. extensive experimental results demonstrate the efficacy and generality of gnome. overall, gnome is a first step in leveraging the intuition from our observations and analysis, and we hope that this work sheds light on the behavior of pre-trained language models upon detecting different kinds of distribution shifts and inspires new methods for general textual ood detection.