Types of Out-of-Distribution Texts and How to Detect Them (2021)

Abstract: Despite agreement on the importance of detecting out-of-distribution (OOD) examples, there is little consensus on the formal definition of the distribution shifts of OOD examples and how to best detect them. We categorize these examples as exhibiting a background shift or semantic shift, and find that the two major approaches to OOD detection, calibration and density estimation (language modeling for text), have distinct behavior on these types of OOD data. Across 14 pairs of in-distribution and OOD English natural language understanding datasets, we find that density estimation methods consistently beat calibration methods in background shift settings and perform worse in semantic shift settings. In addition, we find that both methods generally fail to detect examples from challenge data, indicating that these examples constitute a different type of OOD data. Overall, while the categorization we apply explains many of the differences between the two methods, our results call for a more explicit definition of OOD to create better benchmarks and build detectors that can target the type of OOD data expected at test time.

## introduction current nlp models work well when the training and test distributions are the same (e.g. from the same benchmark dataset). however, it is common to encounter out-of-distribution (ood) examples that diverge from the training data once the model is deployed to real settings. when training and test distributions differ, current models tend to produce unreliable or even catastrophic predictions that hurt user trust @xcite . therefore, it is important to identify ood inputs so that we can modify models' inference-time behavior by abstaining, asking for human feedback, or gathering additional information @xcite . current work in nlp either focuses on specific tasks like intent classification in taskoriented dialogue @xcite , or arbitrary in-distribution (id) and ood dataset pairs @xcite given by the density estimation method (left) and the calibration method (right). the calibration method fails to detect ood examples due to background shift. @xcite @xcite , e.g. taking a sentiment classification dataset as id and a natural language inference dataset as ood. however, getting inputs intended for a different task is rare in realistic settings as users typically know the intended task. in practice, an example is considered ood due to various reasons, e.g. being rare @xcite , out-ofdomain @xcite , or adversarial @xcite . this broad range of distribution shifts makes it unreasonable to expect a detection algorithm to work well for arbitrary ood examples without assumptions on the test distribution @xcite . in this paper, we categorize ood examples by common types of distribution shifts in nlp problems inspired by @xcite and @xcite . specifically, we assume an input (e.g. a movie review) can be represented as background features (e.g. genre) that are invariant across different labels, and semantic features (e.g. sentiment words) that are discriminative for the prediction task. correspondingly, at test time we consider two types of ood examples characterized by a major shift in the distribution of background and semantic features, respectively. while the two types of shifts often happen simultaneously, we note that there are realistic settings where distribution shift is dominated by one or the other. for example, background shift dominates when the domain or the style of the text changes @xcite , e.g. from news to tweets, and semantic shift dominates when unseen classes occur at test time, as in open-set classification @xcite . 1 we use this categorization to evaluate two major approaches to ood detection, namely calibration methods that use the model's prediction confidence @xcite and density estimation methods that fit a distribution of the training inputs @xcite @xcite . we show that the two approaches make implicit assumptions on the type of distribution shift, and result in behavioral differences under each type of shift. by studying id/ood pairs constructed from both simulations and real datasets, we find that the density estimation method better accounts for shifts in background features, consistently outperforming the calibration method on background shift pairs. we further see the opposite in semantic shift pairs, with the calibration method consistently yielding higher performance. in addition, we analyze the detection performance on challenge datasets @xcite through the lens of background/semantic shift. we find that these challenge datasets provide interesting failure cases for both methods. calibration methods completely fail when the model is over-confident due to spurious semantic features. while density estimation methods are slightly more robust, language models are easily fooled by repetitions that significantly increase the probability of a piece of text. together, our findings suggest that better definitions of ood and corresponding evaluation datasets are required for both model development and fair comparison of ood detection methods. 1 we exclude task shift where the ood examples are from a different task, e.g. textual entailment inputs for a text classification model, because it is less likely to happen in realistic settings where users are often aware of the intended use of the model. ## ood detection methods to classify an input @xmath0, we produce a score s(x) and classify it as ood if s(x) < γ, where γ is a pre-defined threshold. most methods differ by how they define s(x). below we describe two types of methods commonly used for ood detection. calibration methods. these methods use the model's prediction confidence as the score. a well-calibrated model's confidence score reflects the likelihood of the predicted label being correct. since the performance on ood data is usually lower than on id data, lower confidence suggests that the input is more likely to be ood. the simplest method to obtain the confidence score is to directly use the conditional probability produced by a probabilistic classifier p model , referred to as maximum softmax probability (msp; @xcite . formally, while there exist more sophisticated methods that take additional calibration steps @xcite , msp proves to be a strong baseline, especially when p model is fine-tuned from pretrained transformers @xcite . density estimation methods. these methods use the likelihood of the input given by a density estimator as the score. for text or sequence data, a language model p lm is typically used to estimate p(x) @xcite . to avoid bias due to the length of the sequence (see analysis in appendix a), we use the token perplexity (ppl) as the score. formally, given a sequence @xmath1, . . . , x t ), while there are many works on density estimation methods using flow-based models in computer vision (e.g. @xcite , there is limited work experimenting with density estimation methods for ood detection on text @xcite . implicit assumptions on ood. one key question in ood detection is how the distribution shifts at test time, i.e. what characterizes the difference between id and ood examples. without access to ood data during training, the knowledge must be incorporated into the detector through some inductive bias. calibration methods rely on p(y | x) estimated by a classifier, thus they are more influenced by the semantic features which are correlated with the label. we can see this formally by in contrast, density estimation methods are sensitive to all components of the input, including both background and semantic features, even in situations where distribution shifts are predominately driven by one particular type. in the following sections, we examine how these implicit assumptions impact performance on different id/ood pairs. ## simulation of distribution shifts as an illustrative example, we construct a toy ood detection problem using a binary classification setting similar to the one depicted in figure 1 . this allows us to remove estimation errors and study optimal calibration and density estimation detectors under controlled semantic and background shifts. ## experiments and analysis we perform head-to-head comparisons of calibration and density estimation methods on 14 id/ood pairs categorized as either background shift or semantic shift, as well as 8 pairs from challenge datasets. ## conclusion despite the extensive literature on outlier and ood detection, previous work in nlp tends to lack consensus on a rigorous definition of ood examples, instead relying on arbitrary dataset pairs from different tasks. in our work, we approach this problem in natural text and simulated data by categorizing ood examples as either background or semantic shifts and study the performance of two common ood detection methods-calibration and density estimation. for both types of data, we find that density estimation methods outperform calibration methods under background shifts while the opposite is true under semantic shifts. however, we find several failure cases from challenge examples that target model shortcomings. as explained in section 2, we assume that φ s and φ b map x to two disjoint sets of components for simplicity. this assumption helps us simplify the framework and compare the two types of detection methods in relation to the two types of shifts. while this simplified framework explains much of the differences between the two methods, failure cases from challenge examples highlight the room for better frameworks and a more explicit definition of ood to progress the development of ood detection methods. such a definition can inform the creation of benchmarks on ood detection that reflect realistic distribution shifts. defining (or at least explicitly stating) the types of ood examples that predictors are designed to target can also guide future modeling decisions between using calibration and density estimation methods, and help improve detection. some promising directions include test-time fine-tuning @xcite and data augmentation @xcite , which can be guided towards a specific type of distribution shift for improved detection performance against it. finally, the methods we studied work well for one type of shift, which motivates the use of hybrid models @xcite ) that use both calibration and density estimation when both types of shift occur at the same time.