D2U: Distance-to-Uniform Learning for Out-of-Scope Detection
ehyilmaz,ctoraman}@aselsan.com.tr
Abstract
Supervised training with cross-entropy loss implicitly forces models to produce probability distributions that follow a discrete delta distribution.
Model predictions in test time are expected to be similar to delta distributions if the classiﬁer determines the class of an input correctly.
However, the shape of the predicted probability distribution can become similar to the uniform distribution when the model cannot infer properly.
We exploit this observation for detecting out-of-scope (OOS) utterances in conversational systems.
Speciﬁcally, we propose a zero-shot post-processing step, called Distance-to-Uniform (D2U), exploiting not only the classiﬁcation conﬁdence score, but the shape of the entire output distribution.
We later combine it with a learning procedure that uses D2U for loss calculation in the supervised setup.
We conduct experiments using six publicly available datasets.
Experimental results show that the performance of OOS detection is improved with our post-processing when there is no OOS training data, as well as with D2U learning procedure when OOS training data is available.
Introduction
Automated conversational systems have recently received attention from the research community (Dopierre et al., 2021;
Mehri et al., 2020; Qin et al., 2021).
In applications such as voice assistants, Spoken Language Understanding (Young et al., 2013) aims to extract meaning from the user inputs, called utterances , in order to process and execute desired functionalities.
The task of Intent Detection, or
Intent Classiﬁcation, aims to classify user utterance into a set of system-identiﬁable intents.
However, supervised training of such systems can only cover a restricted set of classes, i e. in-scope (INS) classes.
To enhance user experience, the task of
Out-of-Scope (OOS) detection (Lin and Xu, 2019a;
Xu et al., 2021; Zhan et al., 2021;
Shen et al., 2021)
R a t e B o o k P
l
a y M u s i
c G e t W e a t
h e r A d d T o P l a y li s t
B o o k R e s t a u r a n t
S e a r c h C r e a t
i v e W
o r k
S e a r c h S c
r e e n i n g E v e n t 0.
P r o b a b il t y S
c
o r e s
INS Utterance
R a t e B o o k P
l
a y M u s i
c G e t W e a t
h e r A d d T o P l a y li s t
B o o k R e s t a u r a n t
S e a r c h C r e a t
i v e W
o r k
S e a r c h S c
r e e n i n g E v e n t 0.
OOS Utterance distinguishes INS utterances from those that do not belong to the scope of the classiﬁer with dedicated model architectures and loss functions.
Existing methods in OOS detection utilize the classiﬁer conﬁdence score for a given utterance with thresholding to classify highly conﬁdent predictions as INS and lower conﬁdence predictions as OOS.
However, softmax classiﬁers suffer from overconﬁdent predictions for OOS data (Hendrycks and Gimpel, 2017), which makes it difﬁcult to accurately determine a threshold value.
Conﬁdence loss (Lee et al., 2018) mitigates this by calculating the KL Divergence between model prediction and the uniform distribution to decrease the conﬁdence for OOS input.
We adapt a similar idea to the zero-shot setup with a novel post-processing step and exploit it jointly in the supervised setup with a learning procedure.
The joint application of supervised D2U learning and D2U post-processing forms a novel OOS detection pipeline.
Figure 1 illustrates output probability distributions of a classiﬁer for predicting the intent class for an INS and OOS utterance.
The classiﬁer, trained only on INS utterances, is confused when OOS utterance is given.
The model assigns closer probabilities for different classes since there is no correct class for this OOS utterance, hence the resulting distribution gets closer to a uniform distribution
[Figure]
r a
[Figure]
c
[Figure]
than a discrete delta distribution.
Based on this observation, we propose to measure the dissimilarity from or distance to the uniform distribution (D2U).
Statistical distance calculations between the prediction and uniform distribution enable the decision boundary to be more accurate.
Figure 2 illustrates possible beneﬁts of using distance to the uniform distribution with crossentropy.
The subplot at the left shows the distribution of the number of utterances according to their
Maximum Likelihood Estimate (MLE) score.
The subplot at the right shows the same distribution according to cross-entropy score between prediction probability and uniform distribution.
A decision boundary or threshold can be easily determined using D2U’s cross-entropy as a post-processing step without any OOS training data.
When OOS training data is available (Larson et al., 2019), D2U can be used as a loss function to minimize the distance between OOS predictions and the uniform distribution.
Such a loss function forces OOS predictions to be less conﬁdent, and beneﬁt D2U
post-processing further.
To test our hypothesis that D2U is a useful method for OOS detection, we answer following research questions:
• RQ1: Does the application of D2U as a postprocessing step on INS classiﬁer predictions increase OOS detection performance when there is no OOS training data?
• RQ2: Does incorporating D2U into the training procedure as a particular loss function boost performance when OOS training data is available?
• RQ3: Is the performance of OOS detection signiﬁcantly improved by D2U over existing stateof-the-art methods?
Related Work
We divide OOS detection studies into three categories: (i) Conﬁdence-based, (ii) representationbased, and (iii) distance-based methods.
2. Conﬁdence-based OOS detection
Threshold-based Methods Thresholding is a common approach in OOS detection (Larson et al., 2019;
Feng et al., 2020; Zhang et al., 2020), which reﬂects the intuition that a classiﬁer output is more conﬁdent for a sample that follows its training distribution.
The overconﬁdence problem of softmax classiﬁers (Hendrycks and Gimpel, 2017), although less apparent in Transformer-based (Vaswani et al., 2017) models (Hendrycks et al., 2020), hinders threshold-based OOS detection performance.
Post-processing Methods
The overconﬁdence problem of softmax classiﬁers is tackled by postprocessing predictions.
ODIN (Liang et al., 2018) and SofterMax (Lin and Xu, 2019b) apply temperature scaling for enlarging the conﬁdence gap between INS and OOS instances, since INS logits are ideally further away on the positive axis of the softmax input.
Gangal et al.
(2020) utilize likelihood ratios with generative classiﬁers to distinguish
OOS predictions.
Our method, D2U, employs a conﬁdence-based post-processing method.
2. Representation-based OOS detection
Dedicated model architectures or loss functions help represent utterances in a high-dimensional space suitable for OOS detection.
Large Margin
Cosine Loss (LMCL) ensures that INS intents are tightly clustered (Zeng et al., 2021a), so that OOS utterances are exposed for outlier detection algorithms, such as Local Outlier Factor (Lin and Xu, 2019a).
Intent class embeddings (Cavalin et al., 2020) model OOS detection as a reverse dictionary task by mapping intent classes and utterances to the same space.
Yilmaz and Toraman (2020) propose a feature representation mechanism that uses KL
Divergence to capture the changes in model predictions during sequential processing of utterances.
In order to mitigate data scarcity, Marek et al.
(2021) propose a method to generate OOS data with Generative Adversarial Networks.
GANs are also utilized to generate high-dimensional representations that are hard to distinguish from that of real utterances, providing adversarial signals to the
INS classiﬁer which increases the robustness of the model (Zeng et al., 2021b;
Liang et al., 2021).
2. Distance-based OOS detection
Distances and divergences are useful tools in OOS detection, since they provide a measure of dissimilarity that can distinguish INS and OOS samples.
Xu et al.
(2020) utilize Euclidean and Mahalanobis distances with generative classiﬁers to identify outliers with Gaussian Discriminative Analysis.
Mahalanobis distance calculated using representations from the intermediate layers of BERT (Devlin et al., 2019) increases OOS detection performance (Shen et al., 2021).
Lee et al.
(2018) introduce the conﬁdence loss in Computer Vision for GANs that calculates KL Divergence between the training predictions for OOS samples and uniform distribution.
The idea of measuring the distance between prediction distribution and uniform distribution is utilized in different learning architectures (Lee et al., 2018; Gangal et al., 2020), but not extensively studied for OOS intent detection.
Besides, we explore various distance metrics in zero-shot OOS detection and different distance-to-uniform training procedures in supervised setup.
Distance-to-Uniform OOS Detection 3. D2U post-processing for zero-shot setup
Supervised classiﬁers
trained
on INS data model the ground truth labels with a discrete delta function that corresponds to the label, given as follows.
[Equation]
For the data instance i , c i is the ground-truth label indicating the correct class.
The cross-entropy loss between softmax model output and discrete delta function is given as follows.
[Equation]
Here, ˆ P ( u i ) is the output probability distribution of the model for utterance u i in a batch of N utterances, and c i is the correct class label for the given utterance.
This criterion implicitly forces the model to generate conﬁdent predictions for a given data point with maximal conﬁdence score assigned to the ground-truth class label, and low prediction scores for the other classes.
When an
OOS utterance is given to an intent classiﬁer that is trained using only INS data, the classiﬁer gets confused, i.e., the output probability distribution is more dissimilar to a delta distribution than what an INS utterance would result in.
In other words, output distributions of OOS samples get closer to the uniform distribution than that of INS samples, an observation that we exploit for OOS detection.
The conventional methods for OOS detection make use of a pre-determined threshold value on the Maximum Likelihood Estimate (MLE) score assigned to the predicted label, given as follows.
[Equation]
Here, θ is a pre-deﬁned threshold value between 0 and 1, and max ( ˆ P ( u i )) is the MLE score, which considers only the conﬁdence and ignores the shape of the distribution.
We exploit the information conveyed by the shape of the entire prediction distribution by ﬁrst calculating a distance between the output distribution ˆ P and the uniform distribution
U before applying the threshold, given as follows.
[Equation]
The distance determined by the dst ( . ) function between ˆ P ( u i ) and U can be calculated with various distance metrics.
We experiment with geometric distance calculations, such as Euclidean distance and Cosine distance; as well as statistical distance calculations, such as Jensen-Shannon distance and symmetrized Kullback-Leibler divergence.
The distance value calculated by the dst ( . ) function can be intuitively interpreted as the level of conﬁdence of the model.
When the distance value is low, the model is less conﬁdent and more confused, since the output distribution assigns closer scores for each class.
This is an architecture-agnostic zero-shot postprocessing step which can be generalized to any classiﬁcation model trained with cross-entropy loss with no need for OOS training data.
OOS detection in test time is achieved by a function of the prediction distribution given by D2U.
3. Distance metrics for post-processing
We examine a number of geometric and statistical distance measures listed as follows.
• Bray Curtis Distance (BC): For two probability distributions, u and v , the Bray Curtis distance is given as (cid:80) i | u
i − v i | / (cid:80) i | u i + v i | .
INS Utterances
OOS Utterances
BERT
P(x) x P(x) x
Prediction
Ground-Truth
P(x) x P(x) x
Prediction
Ground-Truth
D2U
Cross Entropy
S o ft m a x
• Canberra Distance (Cbr): Canberra distance between u and v is (cid:80) i
( | u i − v i | / ( u i + v i )) .
• Cosine Distance
Cos): Derived from the Cosine similarity, the Cosine distance is formulated as 1 − ( u · v/
||
u
|| 2 ||
v || 2 )
where || .
|| 2 is the L 2 norm.
• Euclidean Distance
Euc): The Euclidean distance between u and v is given as || u − v || 2 .
• Hellinger Distance (Helng): The Hellinger distance between u and v is ||√ u − √ v
|| 2 / √ 2 .
• Cross-Entropy (CE): Cross-Entropy is a measure of dissimilarity between distributions u and v given as − (cid:80) i u i log v i .
• Symmetrized KL Divergence (KL): The symmetrized Kullback-Leibler divergence is given as [ KL
[Equation]
[Equation]
2 where KL ( u, v ) =
[Equation]
i u i log ( u i /v i ) .
• Jenshen Shannon Distance (JS): JS distance between u and v is KL ( u, m ) / 2 + KL ( v, m ) /where
m is the mean of two distributions.
3. D2U training for supervised setup
When OOS training data is available, we modify the ﬁne-tuning procedure as in Figure 3, to increase the similarity between OOS prediction and uniform distribution.
We use pretrained BERT (Devlin et al., 2019) as the classiﬁer network.
The loss function for INS utterances, L ins , is still cross-entropy between true label and prediction, given as follows.
[Equation]
For OOS utterances, the loss L oos , is calculated against the uniform distribution, given as follows.
[Equation]
[Table]
The total loss is the weighted average over a batch of utterances containing N ins number of INS utterances and N oos number of OOS utterances, given as follows.
[Equation]
As the dst ( . ) function in Equation 6, we experiment with differentiable functions; such as crossentropy, KL divergence, and Sinkhorn distance (Cuturi, 2013), named as D2U-CE, D2U-KL, and D2US, respectively.
These functions treat the model output and ground truth as probability distributions and provide a differentiable measure.
We do not modify the loss calculation for INS utterances so as not to affect the INS classiﬁcation performance.
Note that this architecture does not model the
OOS intent as a separate class.
Therefore, postprocessing is applied as described in Section 3.
in test time.
Since the loss function incorporates
D2U into training, the performance gain by the post-processing is expected to increase.
Experiments 4. Datasets
We use six publicly available intent classiﬁcation datasets, some of which include labeled OOS data.
We give the main statistics of the datasets in Table 1.
CLINC (Larson et al., 2019) is a dataset with 150 INS intent classes targeting various domains with curated OOS data.
We use the OOS split of CLINC to augment other existing intent detection datasets that do not include labeled OOS data; which are ACID (Acharya and Fung, 2020),
Banking (Casanueva et al., 2020), HWU64 (Liu et al., 2019), and SNIPS (Coucke et al., 2018).
We observe that HWU64 has many short and noisy utterances, we therefore remove any utterances with length less than or equal to three words.
TOP (Gupta et al., 2018) is an intent detection dataset that generalizes conventional intent labeling with semantic parsing.
The intent labels follow a hierarchical structure with potentially many labels for an utterance.
However, we take only root intent class label into account to be consistent with other datasets.
The utterances with the intent labels "UNSUPPORTED" and "UNSUPPORTED_NAVIGATION" are treated as OOS.
The variety of the number of classes, average length (number of words), and vocabulary size provide a wide spectrum for understanding different OOS detection scenarios.
For instance, TOP dataset can be considered a low resource setup since the number of OOS utterances is signiﬁcantly lower than INS utterances.
4. Evaluation metrics
To assess the performance of OOS detection, we report the scores of Receiver Operating Curve
Area Under Curve (ROC AUC), False Positive
Rate at 90% OOS True Positive Rate (FPR90), and False Negative Rate at 90% OOS True Negative Rate (FNR90) using sklearn (Pedregosa et al., 2011).
These metrics are independent of the threshold value used for decision boundary, providing a means of fair comparison.
We also report weighted
OOS Recall and weighted OOS F1 based on the threshold value that maximizes the Youden’s J statistic (Youden, 1950) on a validation set.
Compared to Precision, Recall is arguably a more critical performance metric for OOS detection; since Recall considers Type II error, meaning that OOS utterances are mislabeled as INS.
In this case, the voice assistant would execute a task that the user does not intent to do.
We argue that ROC is a more generic measure that considers the performances of varying thresholds, than Recall and
F1 considering only a ﬁxed threshold.
4. Baseline approaches
In the experiments, BERT (Devlin et al., 2019) with softmax layer is used as the classiﬁer network.
For RQ1, the baseline zero-shot post-processing approaches are listed below.
• MLE (Hendrycks and Gimpel, 2017; Hendrycks et al., 2020):
The conﬁdence score of a classiﬁer trained only on INS data is used for thresholding.
• Softmax temperature scaling (Temp) (Liang et al., 2018; Lin and Xu, 2019b):
As a modiﬁcation to the MLE setup, the softmax input is applied a temperature value of 10 3 .
• Standard deviation (Stdev) :
We use the stdev of the distribution before thresholding since OOS predictions would have lower standard deviation.
[Figure]
with
splits
• Entropy (Ent) (Shen et al., 2021):
The entropy of the prediction distribution, H ( ˆ P ( u i )) , is calculated before applying the threshold, as follows.
[Equation]
For RQ2, we use D2U zero-shot cross-entropy post-processing (D2U-zero) as the baseline method, since we examine any improvement in supervised setup over zero-shot.
For RQ3, we compare supervised D2U with
the following
baselines.
• Large Margin Cosine Loss (LMCL) (Zeng et al., 2021b):
Cosine distance among INS class centroids is increased up to a margin.
We set the margin as 0.35, and scaling factor as 30.
• Domain Regularization Module (DRM)
(Shen et al., 2021):
DRM introduces domain logits for regularization during INS training.
We slightly modify the design and apply sigmoid to domain logits before dividing the classiﬁcation logits for training stability.
• BERT-Binary (Binary) (Devlin et al., 2019):
The "bert-base-uncased" model ﬁne-tuned as a binary classiﬁer for OOS detection.
• Entropy Regularization (Reg.)
(Zheng et al
., 2020): Entropy of OOS predictions are maximized while minimizing INS training loss.
4. Experimental design
To avoid potential annotator-dependent effects as noted by Larson et al.
and comply with the original splits, we modify 10-fold leave-one-out cross-validation as illustrated in Figure 4.
The validation splits are used to ﬁnd conﬁdence threshold values for Recall and F1 calculations.
We validate statistically signiﬁcant differences in the average performances of 10-folds with the two-tailed paired t-test at a 95% interval with Bonferroni correction.
[Table]
Note that the test splits do not overlap in order to satisfy the independence criterion of t-test.
The experiments are designed with respect to our research questions (RQ 1-3).
First, we ﬁnetune a BERT classiﬁer (Devlin et al., 2019) using huggingface implementation (Wolf et al., 2019) for
INS intent detection with cross-entropy loss, and apply different D2U post-processing methods for
RQ1.
Then, we ﬁx the post-processing method, and examine the effect of supervised D2U training for RQ2.
Lastly, we compare D2U with state-ofthe art baselines for RQ3 to assess the performance gain of our method.
4. Experimental results
RQ1: D2U in zero-shot setup.
In Table 2, we report ROC AUC, FPR90, and FNR90 scores for different post-processing methods applied to a BERTbased INS classiﬁer with no OOS training data.
Our proposed method, D2U-zero, statistically signiﬁcantly outperforms all baselines in all datasets with respect to ROC AUC score.
Using crossentropy for D2U-zero has better performance in majority of cases, compared to other distance metrics.
The reason for its success might be that crossentropy is the loss function used in the training procedure of the model.
In terms of FPR90 and
FNR90, D2U-zero does not always outperform all baselines.
Though, the cases when baselines outperform are not statistically signiﬁcant.
This shows that the baseline methods can optimize FPR90 and
FNR90 individually but cannot outperform D2U in terms of ROC which considers Type I and Type II statistically signiﬁcant
[Table]
errors simultaneously.
Entropy (Shen et al., 2021) is a strong baseline that performs better than other baselines with respect to all performance metrics.
RQ2: D2U in supervised setup.
Next, we report the effect of D2U training on OOS detection in Table 3.
Since our concern here is to ob-
[Table]
serve any improvement over zero-shot setup, we ﬁx post-processing method as cross-entropy for all methods due to its performance in the previous experiment.
The results show that using D2U as a loss function statistically signiﬁcantly improves the performance of D2U-zero in almost all cases.
KL divergence loss (D2U-KL) and Cross-Entropy loss (D2U-CE) are effective D2U methods in all datasets, except that Sinkhorn distance (D2U-S) is effective in CLINC dataset.
The choice of loss function is a hyperparameter that can be tuned according to speciﬁc use cases and datasets.
RQ3: D2U versus state-of-the-art.
The performances of state-of-the-art baseline OOS detection models, regardless of zero-shot or supervised, and D2U methods are compared in Table 4, with extensive results reported in the Appendix.
MLE, softmax temperature (Temp.), Entropy, LMCL, and
DRM are zero-shot OOS detection setups, whereas entropy regularization (Reg.) and BERT-Binary (Binary) are supervised setups.
D2U statistically signiﬁcantly outperforms most baselines, although
Binary is a strong baseline method that outperforms
D2U in ACID and Banking datasets and challenges it in HWU64 and TOP, which is not statistically signiﬁcant.
The reason for this might be the prevalent domain difference between INS and OOS utterances in ACID, Banking and TOP datasets; which belong to the insurance, banking, and navigation applications, respectively.
It causes a trivial detection for the BERT-based binary classiﬁer.
HWU contains generic utterances like queries and questions which may coincide with the OOS split and disturb the training process of D2U.
The combination of D2U training and D2U post-processing demonstrates its advantage in CLINC where INS and OOS utterances span a wide spectrum.
Discussion 5. Domain analysis
To validate our hypothesis that domain-speciﬁc datasets provide an advantage to the Binary method, we apply UMAP (Becht et al., 2019) dimension reduction on the CLS embeddings of Binary and
D2U CE models and plot them in Figure 5.
It is apparent that the OOS utterances are separated from
INS utterances when there is a clear domain difference as in ACID and Banking.
However, when this separation becomes fuzzy, Binary fails to properly distinguish INS and OOS utterances as in CLINC.
There is also an overlapping set of INS and OOS utterances in SNIPS for Binary.
In D2U-CE plots, the clusters of INS intents are easily identiﬁable since the model is trained for intent detection, however, OOS utterances do not
[Figure]
R T
B i n a r y
[Figure]
form a separate cluster.
We see that the training procedure does not necessarily enforce a clustering on the OOS embeddings since the model is trained to output a uniform distribution for OOS inputs.
D2U achieves competitive performance even though there are overlapping embeddings of INS and OOS utterances, highlighting the importance of D2U post-processing in the supervised setup.
5. Qualitative analysis
We provide a qualitative analysis on the effect of D2U training.
We illustrate the model output distributions for INS utterance " get me to ritzville by 4 via the freeway.
" belonging to the "GET_DIRECTIONS" intent, and the OOS utterance " how many skating rinks are available in the south paciﬁc tomorrow at 10 " taken from the TOP dataset in Figure 6.
We observe that the OOS utterance results in an overconﬁdent prediction in the
BERT MLE model whereas the prediction distribution of D2U-CE is similar to uniform distribution.
5. INS performance
In Table 5, we analyze if OOS detection models deteriorate INS performance.
MLE baseline does not modify the training procedure.
The results show that INS classiﬁcation performance is not dramatically deteriorated by the supervised models including D2U in SNIPS and TOP, whereas it is even improved in the remaining datasets.
Although
D2U’s INS performance is similar to other supervised models, D2U has better OOS performance than others, as observed in Table 4.
The reason for the increase in INS detection performance could be the regularization signal provided by the OOS loss as observed by Shen et al.
Note that this effect becomes more apparent when domain difference is prevalent (in ACID and Banking).
We do not include Binary, which has no capability of INS classiﬁcation.
Binary has a challenging
OOS performance in Table 4, but D2U has advan0.
P r o b a b ili t y S c
o r e s
BERT MLE 0.
D2U-CE
Uniform INS
OOS
[Table]
tage of showing state-of-the-art performances for both INS and OOS detection.
5. Limitations
We acknowledge some limitations to our study.
Except for CLINC and TOP, the datasets we use are augmented with the OOS data from CLINC.
However, we argue that the majority of the data remains
OOS for other datasets since it is sampled from
Wikipedia (Larson et al., 2019).
Moreover, D2U has effective performance on CLINC and TOP datasets which are designed with OOS utterances.
We leave the selection of the distance metric in post-processing and supervised learning as a hyperparameter of D2U.
In the results, this might provide an advantage to D2U in comparisons since we do not apply hyperparameter tuning for baselines.
However, we use default or suggested parameter settings for baselines.
We adopt transparency in reporting the results that are also detailed in Appendix.
Zero-shot D2U
post-processing emphasizes the distinction between INS and OOS utterances when conﬁdence score becomes misleading.
However,
D2U struggles in the ultimate case where an OOS utterance is mapped to an INS class with ~100% conﬁdence (see Figure 6 BERT MLE).
Nonetheless, D2U suffers from such overconﬁdent predictions less than existing methods (see Table 2).
5. Ethical considerations
We list a number of ethical concerns related to environmental impact, explainability, and transparency in this section.
We employ BERT ﬁne-tuning with small modiﬁcations, therefore the environmental impact can be considered small.
Our work focuses on well-known OOS detection task with established use cases, therefore there would be no risk for unintended use.
We use publicly available datasets with licences suitable for academic research.
To assure explainability and transparency, we report the length of utterances and domains of datasets in Section 4.1.
We report the statistics of datasets and ﬁguratively report the split strategy used in the experiments in Section 4.1.
There are two setups in our study.
The zero-shot setting does not include any training.
In the supervised setup, the complexity of our method is quite similar to regular ﬁne-tuning procedure of BERT.
The thresholding hyperparameter is decided by maximizing the Youden’s J statistic as explained in Section 4.2.
In Section 4.3, we also report the hyperparameters of the baseline methods.
We employ a modiﬁed 10-fold cross-validation strategy as explained in
Section 4.4 and apply t-test with Bonferroni correction to all experimental results.
Conclusion
We propose an OOS detection pipeline with a distance calculation between classiﬁer prediction and uniform distribution, called D2U.
In the zero-shot setup, D2U serves as an architecture-agnostic postprocessing step to emphasize the distinction between INS and OOS.
In the supervised setup, we bring closer OOS predictions to uniform distribution with a modiﬁed loss function.
Experimental results, supported by statistical tests, show that
D2U outperforms existing baselines in zero-shot, and has challenging performance in the supervised setup.
We plan to extend our study to different architectures and deep learning tasks in the future.
A Appendix
We report Receiver Operating Curve Area Under
Curve, False Positive Rate at 90% OOS True Positive Rate, False Negative Rate at 90% OOS True
Negative Rate, weighted OOS Recall, and weighted
OOS F1 scores in Tables 6, 7, 8, 9, 10 respectively.
Different training procedures, baseline and proposed, are reported in rows and different postprocessing methods, baseline and proposed, are reported in columns.
Baseline training methods are BERT-based inscope classiﬁer (MLE) (Larson et al., 2019;
Devlin et al., 2019), Large Margin Cosine Loss (LMCL) (Zeng et al., 2021a), Domain Regularization Module (DRM) (Shen et al., 2021), entropy regularization (Reg.)
(Zheng et al., 2020), and BERTbinary classiﬁer (Binary) (Devlin et al., 2019).
Postprocessing methods are not applicable for Binary training since it models OOS detection as a binary classiﬁcation problem.
Baseline post-processing methods are Maximum Likelihood Estimate (MLE) (Gangal et al., 2020; Zhang et al., 2020), softmax temperature (Temp)
(Liang et al., 2018;
Lin and
Xu, 2019b), standard deviation (Stdev), and entropy (Ent) (Shen et al., 2021).
[Table]
[Table] [Table]
[Table] [Table]