Beyond Mahalanobis-Based Scores for Textual OOD Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep learning methods have boosted the adoption of NLP systems in real-lifeapplications.
However, they turn out to be vulnerable to distribution shifts overtime which may cause severe dysfunctions in production systems, urging practitioners to develop tools to detect out-of-distribution (OOD) samples through thelens of the neural network.
In this paper, we introduce TRUSTED , a new OODdetector for classifiers based on Transformer architectures that meets operationalrequirements: it is unsupervised and fast to compute.
The efficiency of TRUSTEDrelies on the fruitful idea that all hidden layers carry relevant information to detect
OOD examples.
Based on this, for a given input, TRUSTED consists in (i) aggregating this information and (ii) computing a similarity score by exploiting thetraining distribution, leveraging the powerful concept of data depth .
Our extensivenumerical experiments involve 51k model configurations, including various checkpoints, seeds, and datasets, and demonstrate that TRUSTED achieves state-of-the-artperformances.
In particular, it improves previous AUROC over 3 points.
Introduction
The number of AI systems put into production has steadily increased over the last few years.
Thisis because advanced techniques of Machine Learning (ML) and Deep Learning (DL) have broughtsignificant improvements over previous state-of-the-art (SOTA) methods in many areas such asfinance [16, 43], transportation [45] and medicine [14, 62].
However, the increasing use of black-boxmodels has raised concerns about their societal impact: privacy [22, 50], security [6, 3], safety[29, 37], fairness [ and explainability [
49] which became areas of active research in the
ML community.
This paper is about a critical safety issue, namely Out-Of-Distribution (OOD) detection [11] whichrefers to a change of distribution of incoming data that may cause failures of in-production AIsystems.
When data are tabular and of small dimension, simple statistical methods can be efficient,and one may, for instance, monitor the mean and variance of each marginal over time.
However,these traditional methods do not work anymore when data are high-dimensional and/or unstructured.
Thus, the need to design new techniques that incorporate incoming data and the neural networksthemselves.
Distinguishing OOD examples from in-distribution (ID) examples is challenging for modern deepneural architectures.
DL models transform incoming data into latent representations from whichreliable information extraction is cumbersome.
Because of that, designing new tools of investigationfor large pretrained models, judiciously named foundation models by [9], is an essential line ofresearch for the years to come.
In computer vision, methods are more mature thanks to the availabilityof appropriate deformation techniques that allow for sensitivity analysis.
In contrast, the nature of tokens in Natural Language Processing (NLP) makes it more difficult to develop such suitablemethods.
In this paper, we focus on classifiers for textual data and on the ubiquitous BERT [23], DistilBERT[68], and RoBERTa [54] architectures.
Existing methods can be grouped according to their positioningwith respect to the network.
Some works exploit only the incoming data and compare them within-distribution examples through likelihood ratios [30, 17].
Another line of research consists inincorporating robust constraints during training, with [35] or without [85, 46] access to some available
OOD examples.
Another line of research focuses on post-processing methods that can be used onany pretrained models.
In our view, these are the most promising tools because typical users rely ontransformers without retraining.
Within post-processing methods, one can distinguish softmax-basedtools that compute a confidence score based on the predicted probabilities and threshold to decidewhether a sample is OOD or not.
Notice that this does not require direct access to in-distribution data.
The seminal work is due to Hendrycks [34] who uses the maximum soft-probability, and has beenpushed further in [48, 39].
In [53], authors suggest looking one step deeper into the network, namely,to compute a confidence score based on the projections of the pre-softmax layer.
More recently,[60] achieved SOTA results on transformer-based encoder by computing the Mahalanobis distance[57, 21] between a test sample and the in-distribution law [44], estimated through accessible trainingdata points.
Nevertheless, the distance-based score is computed on the last-layer embedding only, suggesting thatgoing deeper inside the network might improve OOD detection power.
Moreover, the computation of
Mahalanobis-based scores requires inverting the covariance matrix of the training data, which canbe prohibitive in high dimensions.
It is worth noticing that the Mahalanobis-based scores can beseen as a data depth [80, 86] through a simple re-scaling [52], that is a statistical function measuringthe centrality of an observation with respect to a probability distribution.
Although data depths arequite natural in the context of OOD detection, they remain overlooked by the ML community.
In thepresent work, we rely on the recently introduced Integrated Rank-Weighted depth [61, 77] in order toremedy the drawbacks of the Mahalanobis-based scores for OOD detection.
1.1 Our contribution
We first leverage the observation introduced by previous work that all hidden layers of a neuralnetwork carry useful information to perform textual OOD detection.
For a given input x , our methodconsists in computing its average latent representation x and then its OOD score through the depthscore of x with respect to the averaged in-distribution law (see Fig.
1 for an illustration).
Notice thatthe ability to compute averaged latent representations crucially relies on the structure of transformerslayers that share the same dimension.
The depth function we are using is based on the computationof the projected ranks of the test inputs using randomly sampled directions.
From a theoreticalviewpoint, this novel method requires fewer assumptions on the data structure than the Mahalanobisscore.
We conduct extensive numerical experiments on three transformers architectures and eight datasetsand benchmark our method with previous approaches.
To ensure reliable results, we introduce a newframework for evaluating OOD detection that considers hyperparameters that were unreported before.
It consists of computing performances for various choices of checkpoints and seeds, which allows usto report a variance term that makes some previous methods fall within the same performance range.
Our conclusions are drawn by considering over 51k configurations , and show that our new detectorbased on data depth improves SOTA methods by 3 AUROC points while having less variance.
Thisresult supports the intuition that OOD detection is a matter of looking at the information availableacross the entire network .
Our contribution can be summarized as follows:
We introduce a novel OOD detection method for textual data.
Our detector TRUSTED 1 relieson the full information contained in pretrained transformers and leverages the concept of data depth:a given input is detected as being in-distribution or OOD sample based on its depth score with respectto the training distribution.2.
We conduct extensive numerical experiments and prove that our method improves over SOTAmethods.
Our evaluation framework is more reliable than previous studies
as it includes the variance
We release open-source code and data to ease future research , ensure reproducibility andreduce computation overhead.
Problem Formulation
Training distribution and classifier.
Let us denote by X the textual input space.
Consider themulticlass classification setup with target space Y = { 1 , . .
, C } of size C ≥ 2 .
We assume thedataset under consideration is made of N ≥ 1 i.i.d.
samples ( x 1 , y 1 ) , . .
, ( x N , y N ) with probabilitylaw denoted by p XY and defined on X × Y .
Accordingly, we will denote by p X and p Y the marginallaws of p XY .
Finally, we denote by f N : X → Y the classifier that has been trained using ( x i , y i ) .
Open world setting.
In real-life scenarios, the trained model f N is deployed into production andwill certainly be faced with input data whose law is not p XY .
To each test point ( x , y ) , we associatea variable z ∈ { 0 , 1 } such that z =
if ( x , y ) stems from p XY and z = 1 otherwise.
It is worthemphasizing that in our setting f N has never been faced with OOD examples before deployment.
This is usually referred to as the open-world setting.
From a probabilistic viewpoint, the test setdistribution of the input data p test X is a mixture of in-distribution and OOD samples:
[Equation]
where α ∈ (0 , 1) .
In this work, we will not make any further assumptions on the proportion α of
OOD samples and on the OOD pdf p X | Z
( x | z = 1) , making the problem more difficult but at thesame time more well suited for practical use.
Indeed, for textual data, it does not appear to be realisticto model how a corpus can evolve.
OOD detection.
The objective of OOD detection is to construct a similarity function s : X → R +that accounts for the similarity of any element in X with respect to the training in-distribution.
For agiven test input x , we then classify x as in-distribution or OOD according to the magnitude of s ( x ) .
Therefore, one fixes a threshold γ and classifies IN ( i.e. ˆ z = 0 ) if s ( x ) > γ or OOD ( i.e. ˆ z = 1 ) ifs ( x ) ≤ γ .
Formally, denoting g ( · , γ ) the decision function, we take:
[Equation]
Performance evaluation.
The OOD problem is a (unbalanced) classification problem, and classically,two quantities allow to measure the performance of a method.
The false alarm rate is the proportionof samples that are classified as OOD while they are IN.
For a given threshold γ , it is theoreticallygiven by Pr (cid:0) s ( X ) ≤ γ | Z = 0 (cid:1) .
The true detection rate is the proportion of samples that arepredicted OOD while being OOD.
For a given threshold γ , it is theoretically given by Pr (cid:0)
s ( X ) ≤γ | Z = 1 (cid:1) .
There exist several ways to measure the effectiveness of an OOD method.
We will focus on fourmetrics.
The first two are specifically designed to assess the quality of the similarity function s .
Area Under the Receiver Operating Characteristic curve ( AUROC ) [10].
It is the area under the
ROC curve γ (cid:55)→ (Pr (cid:0) s ( X ) > γ | Z = 0 (cid:1) , Pr (cid:0) s ( X ) ≤ γ | Z = 1 (cid:1) ) , which plots the true detectionrates against the false alarm rates.
The AUROC corresponds to the probability that an in-distributionexample X in has higher score than an OOD sample X out : AUROC =
Pr( s ( X in ) > s ( X out )) , as canbe checked from elementary computations.
Area Under the Precision-Recall curve ( AUPR-IN / AUPR-OUT ) [20] .
It is the area under the precisionrecall curve γ (cid:55)→ (Pr (cid:0) Z = 1 | s ( X ) ≤ γ (cid:1) , Pr (cid:0) s ( X ) ≤ γ | Z = 1 (cid:1) ) which plots the recall (truedetection rate) against the precision (actual proportion of OOD amongst the predicted OOD).
The
AUPR is more relevant to unbalanced situations.
The third metric we will use is more operational as it computes the performance at a specific thresholdγ corresponding to a security requirement.
False Positive Rate at 95% True Positive Rate ( FPR ) .
In practice, one wishes to achieve reasonablelevel of OOD detection.
For a desired detection rate r , this incites to fix a threshold γ r such that thecorresponding TPR equals r .
At this threshold, one then computes:
[Equation]
In our work, we set r = 0 .
95 in (2).
Error of the best classifier ( Err (%)) .
This refers to the lowest classification error obtained bychoosing the best threshold.
TRUSTED : Textual OOD-Detection using Integrated Rank-Weighted Depth
In this work, we focus on OOD detection when using a contextual encoder ( e.g. , BERT ).
We denoteby { ϕ 1 , . . .
, ϕ L } the L functions corresponding to the layers of the encoder: for every 1 ≤ l ≤ L anda given textual input x ,
ϕ l ( x ) ∈ R d is the embedding of x in the l -th layer, where d is the dimensionof the corresponding embedding space.
Notice that all layers share the same dimension d .3.
TRUSTED in a nutshell.
Our OOD detection method is composed of three steps.
For a given input x with predicted label ˆ y :1.
We first aggregate the latent representations of x via an aggregation function F : (cid:0) R d (cid:1) L → R d .
We choose to take the mean and compute
[Equation]
We will further elaborate on this choice of aggregation function in Sec.
We compute a similarity score D ( F PM ( x ) , F PM ( S train n, ˆ y )) between F PM ( x ) and the distribution ofthe mean-aggregation of the training distribution samples with same predicted target as x ( i.e. ˆ y ) thatwe denote by F PM ( S train n, ˆ y ) .
Formally, if x 1 , . . .
, x n are the training data, this distribution is givenby (1 /n ˆ y ) (cid:80)
i :ˆ y i
=ˆ y δ F PM ( x i ) , with n ˆ y =
|{ i : ˆ y i =
ˆ y }| and δ x is the Dirac measure in x .
We take assimilarity score D a depth function, namely the integrated rank-weighted depth, that we introduce in
Sec. 3.3.3.
The last step consists in thresholding the previous similarity score D ( F PM ( x ) , F PM ( S train n, ˆ y )) :under a given threshold γ , we classify x as an OOD example.
3.2 Layer aggregation choice
Most recent work in textual OOD detection with a pretrained transformer solely relies on the lastlayer of the encoder [83, 60].
Although detectors using information available in multiple layershave been proposed previously, mostly for image data, they rely on post-score aggregation heuristicsthat are either supervised [44, 31] (and thus require having access to OOD samples) or heavily usearbitrary heuristics [70].
TRUSTED differentiates from previous OOD detection methods as it relieson a pre-score aggregation function.
Most popular layer aggregation techniques for Transformer based architecture involve either Power
Means [33, 67] or Wasserstein barycenters [18].
Motivated by both simplicity and computationalefficiency, we discard the Wasserstein barycenters and decide to work with Power Mean (case p = 1 ).
3.3 OOD Score Computation via Integrated Rank-Weighted Depth
Since its introduction by John Tukey in 1975 [80] to extend the notion of median to the multivariatesetting, the concept of statistical depth has become increasingly popular in multivariate data analysis.
Multivariate data depths are nonparametric statistics that measure the centrality of any element of 168 R d , where d ≥ 2 , w.r.t.
a probability distribution (respectively a random variable) defined on anysubset of R d .
Let X be a random variable.
We denote by P X the law of X .
Formally, a data depth isdefined as follows:
[Equation]
The higher D ( x , P X ) , the deeper x is in P X .
Data depth finds many applications in statistics and MLranging from anomaly detection [72,
Numerous definitions have been proposed, such as, among others, the halfspace depth [80],the simplicial depth [51], the projection depth [52] or the zonoid depth [42], see [75, Ch. 2] for anexcellent account of data depth.
The halfspace depth is the most popular depth function probablydue to its attractive theoretical properties [25, 86].
However, it is defined as the solution of anoptimization problem (over the unit hypersphere) of a non-differentiable quantity and is therefore noteasy to compute in practice [66, 27].
Furthermore, it has been show in [59] that the approximationof the halfspace depth suffers from the curse of dimensionality involving statistical rates of order
O( (log( n ) /n )
1 / ( d − 1) )
(see Equation (12) in [59]) where n is the sample size.
Recently, the Integrated
Rank-Weighted (IRW) depth has been introduced in [61], replacing the infimum with an expectation(see also [15, 77]) in order to remedy this drawback.
In contrast to the halfspace depth, it has beenshow in [77] that the approximation of the IRW depth doesn’t suffer from the curse of dimensionality(see Corollary B.3 in [77]).
The IRW depth of x ∈ R d w.r.t.
to a probability distribution P X on R d isgiven by:
[Equation]
where F u ( t ) =
Pr( ⟨ u , X ⟩ ≤ t ) and S d − 1 is the unit hypersphere.
In practice, the expectation can beapproximated by means of Monte-Carlo.
Given a sample S n = { x 1 , . . .
, x n } , the approximation ofthe IRW depth is defined as:
[Equation]
where u k ∈ S
d − 1 and n proj is the number of direction sampled on the sphere.
The approximationversion of the IRW depth can be computed in O ( n proj nd ) and is then linear in all of its parameters.
In addition, the IRW depth has many appealing properties such as invariance to scale/translationtransformations or robustness [61, 15].
Furthermore, it has been successfully applied to anomalydetection [77] making it a natural choice for OOD detection.
Connection to Mahalanobis-based score.
Interestingly enough, the Mahalanobis distance [57]can be seen as a data depth via an appropriate rescaling as suggested in [52].
It measures thedistance between an element in R d and a probability distribution having finite expectation andinvertible covariance matrix differing from the Euclidean perspective by taking account of correlations.
Precisely, the Mahalanobis depth function D M ( x , P X ) is defined as: D M ( x , p X )
= 199 (cid:0)
+ ( x − E
[ X ]) ⊤ Σ − 1 ( x − E
[ X ]) (cid:1) − 1 , where Σ − 1 is the precision matrix of the r.v. X .
Eventhough interesting results relying on this notion have been highlighted in [60] for OOD detection, weexperimentally observe better results with the IRW depth.
Additionally, the Mahalanobis distancerequires the first two moments to be finite and to compute Σ − 1 in high dimension, which can beill-conditioned in low data regimes.
Last, inverting Σ requires O ( d 3 ) operations or storing C matrix,which can become a burden when the number of classes grows [7].
Application to TRUSTED .
The second step of TRUSTED uses an OOD score on the aggregated features.
Thanks to its appealing properties, we choose to rely on the
Integrated Rank-Weighted Depth D
IRWthat measures the “similarity” between a test sample x and a training dataset S train n .
One independent
D IRW is computed per class on the final aggregated layer.
The decision is taken by taking the Dscore of the predicted class.
Formally the final score is taken as:
[Equation]
where we recall that F PM ( S train n, ˆ y ) is the distribution of the mean-aggregation of the training distribution samples with same predicted target as x ( i.e. ˆ y ).
Experimental Settings
In this section, we first discuss the limitation of the previous works on textual OOD detection, thenpresent the chosen benchmark, the pretrained encoders, and baseline methods.
4.1 Previous works and their limitations
Previous works in OOD detection [48, 69, 39, 53, 60] mostly rely on a single model to determinewhich methods are the best.
This undermines the soundness of the conclusions that may only hold for x
[Figure]
x
[Figure]
(cid:80)
[Figure] [Figure]
[Figure]
the particular instance of the chosen model ( e.g. for a specific checkpoint trained with a specific seed).
To the best of our knowledge, no work studies the impact of the several sources of randomness thatare involved, such as checkpoint and seed selections [71]).
Nevertheless, these hyperparameters doimpact the OOD detectors’ performances.
This is illustrated in Fig.
9 of the supplementary material,which gathers several Mahalanobis scores for various checkpoints of the same model.
In the light of Fig.
9, we choose to study both the impact of the checkpoint and the seed choice inour experiment.
Specifically, for each model, we consider 5 different checkpoints.
We save andprobe models after 1k, 3k, 5k, 10k, 15k, and 20k finetuning steps.
We additionally reproduce thisexperiment for 3 different seeds.
As the reuse of checkpoints reduces the cost of research and allowsfor easy head-to-head comparison, our library also contains the probed models to draw general robustconclusions about the performance of the considered class of models [19, 26, 84].
4.2 Dataset selection
Dataset selection is instrumental for OOD detection evaluation as it is unreasonable to expect adetection method to achieve good results on any type of OOD data [1].
Since there is a lack ofconsensus on which benchmark to use for OOD detection in NLP, we choose to rely on the benchmarkintroduced by [85] which is an extension of the one proposed by [36].
Benchmark description.
The considered benchmark is composed of three different types of indistribution datasets (referred to as IN-DS ) which are used to train the classifiers: sentiment analysis( i.e. , SST2 [73] and IMDB [56]), topic classification ( i.e. , 20Newsgroup [40]) and question answering( i.e. , TREC-10 [47]).
For splitting we use either the standard split or the one provided by [85].
Forthe OOD datasets (referred to as OUT-DS ), we first consider the aforementioned datasets ( i.e. , any pairof datasets can be considered as OOD).
Then, we also rely on four other datasets: a concatenationof premises and respective hypotheses from two NLI datasets ( i.e. , RTE [12, 38] and MNLI [82]),
Multi30K [28] and the source of the English-German WMT16 [8].
We gather in Tab.
5 the statisticsof the various data-sets and refer the reader to reference [85] for further details.
4.3 Baseline methods and pretrained models
Baseline methods.
We consider the three following baselines 2 :
Maximum Soft-max Probability (MSP).
This method has been proposed by [34].
Given an input x ,it relies on the final score s MSP defined by s MSP ( x ) = 1 − max y ∈Y p Y | X
( y | x ) , where p Y | X ( ·| x ))is the soft-probability predicted by the classifier after x has been observed.
Energy-based score (E) [53] is defined as the score s E ( x ) =
T × log (cid:104) (cid:80)
[Equation]
,where g y ( x ) represents the logit corresponding to the class label y .
Mahalanobis ( D M ) .
Following [60, 46, 85], the last layer of the encoder is considered leadingto the score: s M ( x ) =
− D M ( F PM ( x ) , F PM ( S train n, ˆ y )) where ˆ y represents the label predicted by theclassifier based on the observation of x .
Aggregation procedures.
Both D M and D IRW rely on feature representations of the data which areextracted from the neural networks.
Our goal is to demonstrate that our aggregation procedure F PMdefined in Eq.
3 is a relevant choice to be plugged in Eq. 5.
To do so, we also perform experiments onother natural aggregation strategies we introduce in the following.
Logits layer selection .
We use the raw non-normalized predictions of the classifier.
In this caseF
Logits ≡ F ( ϕ 1 ( x ) , . .
ϕ L ( x )) =
ϕ L +1 ( x ) .2.
Last layer selection .
Following previous work in textual OOD detection [83], we also consider thelast layer of the network.
Formally F L ≡ F ( ϕ 1 ( x ) , . .
ϕ L ( x )) =
ϕ L ( x ) .3.
Layer concatenation .
We follow the BERT pooler and explore the concatenation of all layers.
Formally, F cat ≡ F ( ϕ 1 ( x ) , . . .
, ϕ L ( x )) =
[ ϕ 1 ( x ) , · · · , ϕ L ( x )] represents the concatenated vector.
The main limitation of layer concatenation is that the dimension of the considered features linearlyincreases with the number of layers which can be problematic for very deep networks [81].
Pretrained encoders.
To provide an exhaustive comparison, we choose to work with different types of pretrained encoders.
We test the various methods on DISTILBERT ( DIS. )
[68], BERT [24] and ROBERTA ( ROB. )
We trained all models with a dropout rate [74] of 0 . 2 , a batch size of 32 , we use ADAMW [41].
Additionally, the weight decay is set to 0 .
01 , the warmup ratio is set to 0 .and
the learning rate to 10 − 5 .
[Table]
Static Experimental Results
In this section
[Table]
we demonstrate the effectiveness of our proposed detector using various pretrainedmodels.
Due to space limitations, additional tables are reported in the Supplementary Material.
5.1 Methods comparison
[Table]
In Tab.
[Table]
we report the aggregated score obtained by each method combined with a differentaggregation function.
We observe that TRUSTED obtains the best overall scores followed by D IRWusing F cat .
Similarly to previous works [60]
[Table]
we notice in general that score leveraging information available from the training set ( i.e. , D M and D IRW ) achieve stronger results than those relying onoutput of softmax scores solely ( i.e. , E and MSP).
Interestingly, we observe that D M achieves the best results when relying on the last layer solely ( i.e. ,using F L +1 ).
Considering additional layers through concatenation or mean hurts the performances of
D M .
This is not the case when relying on D IRW .
Indeed layer aggregation improves the performanceof the detector demonstrating the relevance of using D IRW over D M as an OOD score.
Relyingon Mahalanobis as OOD score suppose that the representation follows a multivariate Gaussiandistribution which might be too strong assumption in the case of layer aggregation.
On the contrary
D IRW do not rely on any distributional assumption.5.
On the pretrained encoder choice
[Table]
When training and deploying a classifier
[Table]
a key question is choosing a pretrained encoder.
It can bebeneficial in critical applications to trade off the main task accuracy to ensure better OOD detection.
In Tab.
[Table]
we report the individual performance of OOD methods on three types of classifiers.
Although TRUSTED achieves state-of-the-art on all configurations, it is worth noticing a difference inperformance concerning the type of pretrained model.
It is also important to remark that for ROB both
MSP and E achieve on-par performances with D M while not requiring any extra training information.
Overall, for a given method ( i.e. D M or D IRW ), the ranking of detectors performances according tothe type of feature extractor remain still.
This validates the use of the mean-aggregation procedure of
TRUSTED .
Overall, based on the difference of OOD detection performance of Tab. 3, we recommendto use TRUSTED on BERT or DIS.
if accurate OOD detection is required.
5.3 Impact of the training dataset
OOD detection performance depends on the nature of what is considered in-distribution (the trainingdistribution in our case).
Thus, it is interesting to study the performance per-
IN-DS as reported in
Tab. 4.
Even though TRUSTED achieves strong results in terms of AUROC , we observe a high FPR on
SST2.
From Fig.
4, we observe that IMDB and SST2 are harder to detect, especially for D M .
Finally,since high AUROC does not necessarily imply a low FPR , it is crucial to take both into account whendesigning an OOD detector.
[Figure] [Figure]
[Table]
Dynamic Experimental Results
[Figure]
Most OOD detection methods are tested on specific checkpoints, where the selection criterion is oftenunclear.
The consequences of this selection on OOD detection are rarely studied.
This section aimsto respond to this by measuring the OOD detection performance of methods on various checkpointfinetuning of the pretrained encoder.
We will use 5 different checkpoints taken after 1k, 3k, 5k,10k, 15k, and 20k iterations.
Training curves of the models are given in Fig.
Notice that after 3kiterations models have converged and no over-fitting is observed even after 20k iterations ( i.e., we donot observe an increase in validation loss).
[Figure]
Overall analysis.
We report the results of the dynamic analysis on the various pretrained modelsin Fig.
For all the methods and models (except D M on ROB ), we observe that training longerthe classifier hurts detection.
Interestingly, this drop in performance has a higher impact on FPRcompared to AUROC .
Thus, it is better to use an early stopping criterion to ensure proper OODdetection performance.
In addition, we observed that TRUSTED (corresponding to D IRW ) achievesbetter detection results and that D M outperforms TRUSTED for checkpoints larger than 10k on ROB .
[Figure]
Analysis Per In-Dataset.
Fig.
6 reports the results of the dynamical analysis per IN-DS .
We observe that D IRW is consistently better than D M with sensible improvement on IMDB and SST2, which are the hardest benchmarks.
We observe a similar trend to the previous experiment: training longer the classifiers hurt their OOD detection performances .
Similar observations hold for FPR ,
AUPR-IN
[Figure]
and AUPR-OUT that are postponed to the
Supplementary Material (see Fig. 10)
[Figure]
Conclusions and Future Directions
In this work, we introduced TRUSTED a novel OOD detector that relies on information availablein all the hidden layers of a network.
TRUSTED leverages a novel similarity score built on top ofthe Integrate Rank-Weighted depth.
We conduct extensive numerical experiments proving that itconsistently outperforms previous approaches, including detection based on the Mahalanobis distance.
Our comprehensive evaluation framework demonstrates that, in general, OOD performances varydepending on several hyperparameters of the models, the datasets, and the detector’s feature extractionstep.
Thus, we would like to promote the use of such exhaustive evaluation frameworks for futuresearch to assess AI systems’ safety tools properly.
Another interesting question is the detectioninference-time / accuracy trade-off, which is instrumental for the practitioner.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We have backed all of our claims in the abstract and
introduction throughout the work.
See the summary of our contributions at Section 1.1.
(b) Did you describe the limitations of your work?
[Yes] Please see Section CITE.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our
works aim at developing safer AI systems, so we believe our does not have any negative
societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include
extensive instructions to reproduce the main experimental results in both the supple-
mentary material and main paper.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We provide an extensive analysis on the hyperparamters for
training the models and reproducing the results in the supplementary material.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We report error bars in all of our main results (please see
Sections 5 and 6).
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We provide a rough estimate of
the amount of compute used in the supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all the
datasets we use in the experimental setup of this work in the main paper.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We open-source our code in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
Appendices
A Experimental Details
A.1 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Benchmark Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Training Parameters . .
A.4 Training Curves . .
B Additional Static Experimental Results
B.1 Analysis Per In-Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Trade-off between metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Analysis Per Out-Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Analysis using [2] Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
Additional Dynamical Experimental Results
[-Results 
-]
C.
On the importance of dynamical probing . . . . . . . . . . . . . . . . . . . . . . .
C.2 Analysis Per IN-Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Impact of the pretrained encoder . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.4 All Combinations
24D Computation time comparison between IRW and Mahalanobis depths
A Experimental Details
In this section we gather additional experimental details.
For completeness we provide the usedalgorithms for both TRUSTED and D IRW (see Sec.
A.1), we also gather additional benchmarks details(see
Sec.
A.2), hyperparameter used during training (see Sec.
A.3) as well as the training curves (see
Sec.
A.4).
A.1 Algorithms
In this part, we present algorithms to compute D IRW (see Algorithm 1) and TRUSTED (see Algorithm2). Algorithm Initialization: test sample x , n proj X =
x
, x n
proj vectors Output [Table]
Initialization: x , n proj , S n , ˆ y .
1: Compute F PM ( x ) 2: for y = 1 , . . .
C do
Compute F PM ( S train n,y ) 3: end for 4: Compute D IRW ( F PM ( x ) , F PM ( S train n, ˆ y )) using Algorithm 1 with F PM ( x ) , n proj and
F PM ( S train n, ˆ y )
Output : s TRUSTED ( x ) =
D IRW ( F PM ( x ) , F PM ( S train n, ˆ y ))
A.2 Benchmark Details
We report in Tab.
5 statistics related to the datasets of our benchmark.
The only difference betweenour work and the one from [85] is that we considered the pair IMDB and SST2 a valid OOD pair of
IN-DS/OOD-DS as this can be seen as a background shift (see [63]).
Remark
We initially started to work with the appealing benchmark introduced by [46] whichintroduces an alternative standard that addresses the limitation of the benchmark proposed by [83](e.g.
there is no category overlap between training and OOD test examples in the non-semantic shiftdataset).
Additionally, [46] put effort into designing a dataset that can categorize shifts as belongingto semantic or background shift [2, 63].
However, we failed to reproduce the baseline results from[46] even after contacting the authors.
A.3 Training Parameters
In this section, the detail the main hyper-parameters that were used for finetuning the pretrainedencoders.
It is worth noting that we use the same set of hyperparameters for all the different encoders.
The dropout rate [74] is set to 0 .
2 , we train with a batch size of 32 , we use ADAMW [41, 55, 4].
Additionally, we set the weight decay to 0 .
01 , the warmup ratio to 0 .
06 , and the learning rate to 10 − 5 .
All the models were trained during 20k iterations with different seeds.
A.4 Training Curves
In order to understand the change of performance while finetuning the model, it is crucial to understandwhen and if the different models have converged.
Thus we report in Fig.
7 dev losses and dev andtest accuracy.
From Fig.
7, we can observe that a pretrained encoder finetuned on 20ng takes moretime to converge compared to the same pretrained encoder finetuned on either SST2, IMDB, ortrec.
Additionally, after 2k updates, both dev and test accuracy are stable on SST2, IMDB, andtrec, while for 20ng, it requires 6k steps.
Last, it is worth noting (see Tab. 1) that ROB.
achieves thebest accuracy overall.
BERT is the second best and achieves stronger test accuracy than DIS.
on thedifferent datasets.
[Figure]
D e v A cc u r a c y
[Figure]
D e v A cc u r a c y
[Figure]
D e v A cc u r a c y
[Figure]
T e s t
A cc u r a c y 20ngtrecimdbsst (i) ROB.
Test Accuracy.
B Additional Static Experimental Results
In this section, we report additional static experiment results.
Formally, we aim to gain understanding:
• on the role of the IN-DS (see Sec.
B.1)
and on the impact of the choice of the pretrained
encoder.
• on the different trade-off that exists between the different metrics (see Sec.
B.2)
• on the impact of performance of the detection methods depending on the OOD distribution
(see Sec. B.3).
B.1 Analysis Per In-Dataset
We present in Tab.
the average performance per IN-DS .
On 3 out of 4 datasets TRUSTED achievesthe best results and outperforms other methods.
Interestingly, the table shows the key importance ofthe training corpus.
As an example, we observe that detection methods applied to classifiers trainedon sst2 are less efficient (by several AUROC points) compared to other IN-DS .
A finer analysis of this phenomenon can be conducted using Tab. 7.
From Tab. 7, we observe thatthis phenomenon is consistent accross all the pretrained classifiers ( i.e., BERT , ROB. and DIS. ).
Weobserve that different TRUSTED does not work uniformly better on all pretrained models.
For example,the best results on 20ng are obtained for BERT while on sst2 it is obtained for DIS. .
Takeaways: Both IN-DS and pretrained model choices are essential to ensure good detectionperformance.
[Table]
B.2 Trade-off between metrics
We report Fig.
8 the different trade-off that exist between the considered metrics.
Interestingly, ahigh AUROC does not imply a low FPR .
Similar conclusions can be drawn regarding AUPR-IN and
AUPR-OUT .
Takeaways.
Fig.
8 illustrates that all metrics matters and should be considered when comparingdetection methods.
[Figure]
B.3 Analysis Per Out-Dataset
We report on Tab. 8 the detailled results of the detection methods for different OUT-DS .
It is worthnoting that TRUSTED achieves best results on 22 configurations.
Takeaways.
Tab. 8 illustrates the impact of the type of OOD sample on the detection performances.
B.4 Analysis using [2] Framework
Following [2] framework we evaluate our detector TRUSTED .
The results reported in Tab.
10 support the superiority of our TRUSTED detector, even in the case of
LM. [Table] C Additional Dynamical Experimental Results
In this section, we gather additional dynamical analysis.
We further illustrate the importance ofdynamical probing in Sec.
C.1.
We then conduct a dynamical study of the impact of the OOD dataset [Table] in Sec.
C.2.
In Sec.
C.3, we study the role of the pretrained encoder and in Sec.
C.4.
Last, we gatherper encoder, per IN-DS and per OUT-DS analysis.
C.1 On the importance of dynamical probing
In Fig.
9, we report histograms for TRUSTED and Mahanalobis distance.
Interestingly, the shape ofhistograms is changing across checkpoints demonstrating the need for dynamical probing.
C.2 Analysis Per IN-Dataset
In Fig.
10, we conduct a dynamical analysis per IN-DS .
We observe a variation of performance ( e.g.,up to 10 AUROC points for sst2 for D M ) while probing accross time.
Takeaways: Although our method achieves strong results a key dimension when deploying OODdetection methods is to carefully select the checkpoints when learning the classifier. [Table] ID OOD
[Table]
C.3 Impact of the pretrained encoder
In Fig.
[Table] we report the results of the dynamical analysis and study the influence of the encoderchoice on the detection performance.
Takeaways Although, TRUSTED achieves strong results on a large number of configurations.
Weobserve different behaviours while considering different success criterion or different pretrainedmodels.
For examples, on BERT and DIS. , TRUSTED is uniformly better across checkpoints whenconsidering Fig.
For ROB. , TRUSTED is not better on all metrics on last checkpoint ( e.g., 20k).
C.4 All Combinations
For completeness of the paper, we report all the results of the dynamical analysis for all consideredcombinations in this section (see Fig.
12 Fig.
13 Fig.
14 Fig. 15).
We believe this will allow thecurious reader to gain more intuition and draw nuanced conclusions from our experiments.
D Computation time comparison between IRW and Mahalanobis depths
In this part, we investigate the computation time of the IRW and the Mahalanobis depths.
Precisely,we want to compare the time of computing these data depths for an element x ∈ R d w.r.t.
a dataset.
To that end, this experiment is conducted as follows: we sample several datasets (varying dimensionin { 800 , 1000 , 1200 , 1500 , 2000 , 2500 , 5000 } and sample size { 100 , 1000 , 2000 , 3000 , 4000 , 5000 } )from centered Gaussian distributions N (0 , Σ) where Σ follows a Wishart distribution.
Thus, wecompute the depth of 0 w.r.t.
each dataset.
This procedure is repeated 10 times.
We report theirmean computation time as well as 10-90% quantiles in Figure 16 highlighting the computationaladvantages of using the IRW depth over the Mahalanobis distance.
The dimension is fixed to 5000 onthe left graphic while the sample size is fixed to 100 on the right graphic.
For all our experiments, then p roj has been set to 1000.
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]