Beyond Mahalanobis-Based Scores for Textual OOD Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep learning methods have boosted the adoption of NLP systems in real-lifeapplications.
However, they turn out to be vulnerable to distribution shifts over timewhich may cause severe dysfunctions in production systems, urging practitionersto develop tools to detect out-of-distribution (OOD) samples through the lens ofthe neural network.
In this paper, we introduce TRUSTED , a new OOD detector for Transformer architectures that meets operational requirements: it is unsupervisedand fast to compute.
The efficiency of TRUSTED relies on the fruitful idea thatall hidden layers carry relevant information to detect OOD examples.
Based onthis, for a given input, TRUSTED consists in (i) aggregating this information and(ii) computing a similarity score by exploiting the training distribution, leveragingthe powerful concept of data depth .
Our extensive numerical experiments involve51k model configurations, including various checkpoints, seeds, and datasets, anddemonstrate that TRUSTED achieves state-of-the-art performances.
In particular, itimproves previous AUROC over 3 points.
Introduction
The number of AI systems put into production has steadily increased over the last few years.
Thisis because advanced techniques of Machine Learning (ML) and Deep Learning (DL) have broughtsignificant improvements over previous state-of-the-art (SOTA) methods in many areas such asfinance [16, 43], transportation [45] and medicine [14, 61].
However, the increasing use of black-boxmodels has raised concerns about their societal impact: privacy [22, 50], security [6, 3], safety[29, 37], fairness [5, 58] and explainability [13, 49] which became areas of active research in the
ML community.
This paper is about a critical safety issue, namely Out-Of-Distribution (OOD) detection [11] whichrefers to a change of distribution of incoming data that may cause failures of in-production AIsystems.
When data are tabular and of small dimension, simple statistical methods can be efficient,and one may, for instance, monitor the mean and variance of each marginal over time.
However,these traditional methods do not work anymore when data are high-dimensional and/or unstructured.
Thus, the need to design new techniques that incorporate incoming data and the neural networksthemselves.
Distinguishing OOD examples from in-distribution (ID) examples is challenging for modern deepneural architectures.
DL models transform incoming data into latent representations from whichreliable information extraction is cumbersome.
Because of that, designing new tools of investigationfor large pre-trained models, judiciously named foundation models by [9], is an essential line ofresearch for the years to come.
In computer vision, methods are more mature thanks to the availabilityof appropriate deformation techniques that allow for sensitivity analysis.
In contrast, the nature of tokens in Natural Language Processing (NLP) makes it more difficult to develop such suitablemethods.
In this paper, we focus on textual data and on the ubiquitous BERT [23], DistilBERT [67], and RoBERTa [54] architectures.
Existing methods can be grouped according to their positioning withrespect to the network.
Some works exploit only the incoming data and compare them with indistribution examples through likelihood ratios [30, 17].
Another line of research consists in incorporating robust constraints during training, with [35] or without [82, 46] access to some available
OOD examples.
Another line of research focuses on post-processing methods that can be used onany pre-trained models.
In our view, these are the most promising tools because typical users rely ontransformers without retraining.
Within post-processing methods, one can distinguish softmax-basedtools that compute a confidence score based on the predicted probabilities and threshold to decidewhether a sample is OOD or not.
Notice that this does not require direct access to in-distribution data.
The seminal work is due to Hendrycks [34] who uses the maximum soft-probability, and has beenpushed further in [48, 39].
In [53], authors suggest looking one step deeper into the network, namely,to compute a confidence score based on the projections of the pre-softmax layer.
More recently,[59] achieved SOTA results on transformer-based encoder by computing the Mahalanobis distance[57, 21] between a test sample and the in-distribution law [44], estimated through accessible trainingdata points.
Nevertheless, the distance-based score is computed on the last-layer embedding only, suggesting thatgoing deeper inside the network might improve OOD detection power.
Moreover, the computation of
Mahalanobis-based scores requires inverting the covariance matrix of the training data, which canbe prohibitive in high dimensions.
It is worth noticing that the Mahalanobis-based scores can beseen as a data depth [77, 83] through a simple re-scaling [52], that is a statistical function measuringthe centrality of an observation with respect to a probability distribution.
Although data depths arequite natural in the context of OOD detection, they remain overlooked by the ML community.
In thepresent work, we rely on the recently introduced Integrated Rank-Weighted depth [60, 76] in order toremedy the drawbacks of the Mahalanobis-based scores for OOD detection.
1.1 Our contribution
We first leverage the observation introduced by previous work that all hidden layers of a neuralnetwork carry useful information to perform textual OOD detection.
For a given input x , our methodconsists in computing its average latent representation x and then its OOD score through the depthscore of x with respect to the averaged in-distribution law (see Fig.
1 for an illustration).
Notice thatthe ability to compute averaged latent representations crucially relies on the structure of transformerslayers that share the same dimension.
The depth function we are using is based on the computationof the projected ranks of the test inputs using randomly sampled directions.
From a theoreticalviewpoint, this novel method requires fewer assumptions on the data structure than the Mahalanobisscore.
We conduct extensive numerical experiments on three transformers architectures and eight datasetsand benchmark our method with previous approaches.
To ensure reliable results, we introduce a newframework for evaluating OOD detection that considers hyperparameters that were unreported before.
It consists of computing performances for various choices of checkpoints and seeds, which allows usto report a variance term that makes some previous methods fall within the same performance range.
Our conclusions are drawn by considering over 51k configurations , and show that our new detectorbased on data depth improves SOTA methods by 3 AUROC points while having less variance.
Thisresult supports the intuition that OOD detection is a matter of looking at the information availableacross the entire network .
Our contribution can be summarized as follows:
We introduce a novel OOD detection method for textual data.
Our detector TRUSTED 1 relieson the full information contained in pre-trained transformers and leverages the concept of data depth:a given input is detected as being in-distribution or OOD sample based on its depth score with respectto the training distribution.2.
We conduct extensive numerical experiments and prove that our method improves over SOTAmethods.
Our evaluation framework is more reliable than previous
studies as it includes the variance with respect to seeds and checkpoints
We release open-source code and data to ease future research , ensure reproducibility andreduce computation overhead.
Problem Formulation
Training distribution and classifier.
Let us denote by X the textual input space.
Consider themulticlass classification setup with target space Y = { 1 , . .
, C } of size C ≥ 2 .
We assume thedataset under consideration is made of N ≥ 1 i.i.d.
samples ( x 1 , y 1 ) , . .
, ( x N , y N ) with probabilitylaw denoted by p XY and defined on X × Y .
Accordingly, we will denote by p X and p Y the marginallaws of p XY .
Finally, we denote by f N : X → Y the classifier that has been trained using ( x i , y i ) .
Open world setting.
In real-life scenarios, the trained model f N is deployed into production andwill certainly be faced with input data whose law is not p XY .
To each test point ( x , y ) , we associatea variable z ∈ { 0 , 1 } such that z =
if ( x , y ) stems from p XY and z = 1 otherwise.
It is worthemphasizing that in our setting f N has never been faced with OOD examples before deployment.
This is usually referred to as the open-world setting.
From a probabilistic viewpoint, the test setdistribution of the input data p test X is a mixture of in-distribution and OOD samples:
[Equation]
where α ∈ (0 , 1) .
In this work, we will not make any further assumptions on the proportion α of
OOD samples and on the OOD pdf p X | Z
( x | z = 1) , making the problem more difficult but at thesame time more well suited for practical use.
Indeed, for textual data, it does not appear to be realisticto model how a corpus can evolve.
OOD detection.
The objective of OOD detection is to construct a similarity function s : X → R +that accounts for the similarity of any element in X with respect to the training in-distribution.
For agiven test input x , we then classify x as in-distribution or OOD according to the magnitude of s ( x ) .
Therefore, one fixes a threshold δ and classifies IN ( i.e. ˆ z = 0 ) if s ( x ) > δ or OOD ( i.e. ˆ z = 1 ) ifs ( x ) ≤ δ .
Formally, denoting g ( · , δ ) the decision function, we take:
[Equation]
Performance evaluation.
The OOD problem is a (unbalanced) classification problem, and classically,two quantities allow to measure the performance of a method.
The false alarm rate is the proportionof samples that are classified as OOD while they are IN.
For a given threshold δ , it is theoretically givenby Pr (cid:0) s ( X ) ≤ δ | Z = 0 (cid:1) .
The true detection rate is the proportion of samples that are predicted OOD while being OOD.
For a given threshold δ , it is theoretically given by Pr (cid:0)
s ( X ) ≤ δ | Z = 1 (cid:1) .
There exist several ways to measure the effectiveness of an OOD method.
We will focus on fourmetrics.
The first two are specifically designed to assess the quality of the similarity function s .
Area Under the Receiver Operating Characteristic curve ( AUROC ) [10].
It is the area under the
ROC curve δ (cid:55)→ (Pr (cid:0) s ( X ) > δ | Z = 0 (cid:1) , Pr (cid:0) s ( X ) ≤ δ | Z = 1 (cid:1) ) , which plots the true detectionrates against the false alarm rates.
The AUROC corresponds to the probability that an in-distributionexample X in has higher score than an OOD sample X out : AUROC =
Pr( s ( X in ) > s ( X out )) , as canbe checked from elementary computations.
Area Under the Precision-Recall curve ( AUPR-IN / AUPR-OUT ) [20] .
It is the area under the precisionrecall curve δ (cid:55)→ (Pr (cid:0) Z = 1 | s ( X ) ≤ δ (cid:1) , Pr (cid:0) s ( X ) ≤ δ | Z = 1 (cid:1) ) which plots the recall (truedetection rate) against the precision (actual proportion of OOD amongst the predicted OOD).
The
AUPR is more relevant to unbalanced situations.
The third metric we will use is more operational as it computes the performance at a specific thresholdδ corresponding to a security requirement.
False Positive Rate at 95% True Positive Rate ( FPR ) .
In practice, one wishes to achieve reasonablelevel of OOD detection.
For a desired detection rate r , this incites to fix a threshold δ r such that thecorresponding TPR equals r .
At this threshold, one then computes:
[Equation]
In our work, we set r = 0 .
95 in (2).
Error of the best classifier ( Err (%)) .
This refers to the lowest classification error obtained bychoosing the best threshold.
TRUSTED : Textual OOD-Detection using Integrated Rank-Weighted Depth
In this work, we focus on OOD detection when using a contextual encoder ( e.g. , BERT ).
We denoteby { ϕ 1 , . . .
, ϕ L } the L functions corresponding to the layers of the encoder: for every 1 ≤ l ≤ L anda given textual input x ,
ϕ l ( x ) ∈ R d is the embedding of x in the l -th layer, where d is the dimensionof the corresponding embedding space.
Notice that all layers share the same dimension d .3.
TRUSTED in a nutshell.
Our OOD detection method is composed of three steps.
For a given input x with predicted label ˆ y :1.
We first aggregate the latent representations of x via an aggregation function F : (cid:0) R d (cid:1) L → R d .
We choose to take the mean and compute
[Equation]
We will further elaborate on this choice of aggregation function in Sec.
We compute a similarity score D ( F PM ( x ) , F PM ( S train n, ˆ y )) between F PM ( x ) and the distributionof the mean-aggregation of the training distribution samples with same predicted target as x ( i.e.ˆ y ) that we denote by F PM ( S train n, ˆ y ) .
Formally, if x 1 , . . .
, x n are the training data, this distribution isgiven by (1 /n ˆ y ) (cid:80) i :ˆ y i
= ˆ y δ F PM ( x i ) , with n ˆ y =
|{ i : ˆ y i =
ˆ y }| .
We take as similarity score D a depthfunction, namely the integrated rank-weighted depth, that we introduce in Sec. 3.3.3.
The last step consists in thresholding the previous similarity score D ( F PM ( x ) , F PM ( S train n, ˆ y )) :under a given threshold δ , we classify x as an OOD example.
3.2 Layer aggregation choice
Most recent work in textual OOD detection with a pre-trained transformer solely relies on the lastlayer of the encoder [80, 59].
Although detectors using information available in multiple layershave been proposed previously, mostly for image data, they rely on post-score aggregation heuristicsthat are either supervised [44, 31] (and thus require having access to OOD samples) or heavily usearbitrary heuristics [69].
TRUSTED differentiates from previous OOD detection methods as it relieson a pre-score aggregation function.
Most popular layer aggregation techniques for Transformer based architecture involve either Power
Means [33, 66] or Wasserstein barycenters [18].
Motivated by both simplicity and computationalefficiency, we discard the Wasserstein barycenters and decide to work with Power Mean (case p = 1 ).
3.3 OOD Score Computation via Integrated Rank-Weighted Depth
Since its introduction by John Tukey in 1975 [77] to extend the notion of median to the multivariatesetting, the concept of statistical depth has become increasingly popular in multivariate data analysis.
Multivariate data depths are nonparametric statistics that measure the centrality of any element of 166 R d , where d ≥ 2 , w.r.t.
a probability distribution (respectively a random variable) defined on anysubset of R d .
Let X be a random variable.
We denote by P X the law of X .
Formally, a data depth isdefined as follows:
[Equation]
The higher D ( x , P X ) , the deeper x is in P X .
Data depth finds many applications in statistics and MLranging from anomaly detection [71, 64, 75] to regression [63, 32].
Numerous definitions have beenproposed, such as, among others, the halfspace depth [77], the simplicial depth [51], the projectiondepth [52] or the zonoid depth [42], see [74, Ch. 2] for an excellent account of data depth.
Thehalfspace depth is the most popular depth function probably due to its attractive theoretical properties [Figure]
[Figure] However, it is defined as the solution of an optimization problem (over the unit hypersphere)of a non-differentiable quantity and is therefore not easy to compute in practice [65, 27].
Recently,the Integrated Rank-Weighted (IRW) depth has been introduced in [60], replacing the infimum withan expectation (see also [15, 76]) in order to remedy this drawback.
More precisely, the IRW depthof x ∈ R d w.r.t.
to a probability distribution P X on R d is given by:
[Equation]
where F u ( t ) =
Pr( ⟨ u , X ⟩ ≤ t ) and S d − 1 is the unit hypersphere.
In practice, the expectation can beapproximated by means of Monte-Carlo.
Given a sample S n = { x 1 , . . .
, x n } , the approximation ofthe IRW depth is defined as:(cid:101) D [Equation] proj
[Equation]
proj [Equation] min [Equation]
n [Equation] where u k ∈ S d − 1 .
The approximation version of the IRW depth can be computed in O ( n proj nd ) andis then linear in all of its parameters.
In addition, the IRW depth has many appealing properties suchas invariance to scale/translation transformations or robustness [60, 15].
Furthermore, it has beensuccessfully applied to anomaly detection [76] making it a natural choice for OOD detection.
Connection to Mahalanobis-based score.
Interestingly enough, the Mahalanobis distance [57]can be seen as a data depth via an appropriate rescaling as suggested in [52].
It measures thedistance between an element in R d and a probability distribution having finite expectation andinvertible covariance matrix differing from the Euclidean perspective by taking account of correlations.
Precisely, the Mahalanobis depth function D M ( x , P X ) is defined as: D M ( x , p X )
= 191 (cid:0)
+ ( x − E
[ X ]) ⊤ Σ − 1 ( x − E
[ X ]) (cid:1) − 1 , where Σ − 1 is the precision matrix of the r.v. X .
Eventhough interesting results relying on this notion have been highlighted in [59] for OOD detection, weexperimentally observe better results with the IRW depth.
Additionally, the Mahalanobis distancerequires the first two moments to be finite and to compute Σ − 1 in high dimension, which can beill-conditioned in low data regimes.
Last, inverting Σ requires O ( d 3 ) operations or storing C matrix,which can become a burden when the number of classes grows [7].
Application to TRUSTED .
The second step of TRUSTED uses an OOD score on the aggregated features.
Thanks to its appealing properties, we choose to rely on the
Integrated Rank-Weighted Depth D
IRWthat measures the “similarity” between a test sample x and a training dataset S train n .
One independent
D IRW is computed per class on the final aggregated layer.
The decision is taken by taking the Dscore of the predicted class.
Formally the final score is taken as:
[Equation]
[Equation] where we recall that F PM ( S train n, ˆ y ) is the distribution of the mean-aggregation of the training distribution samples with same predicted target as x ( i.e. ˆ y ).
Experimental Settings
In this section, we first discuss the limitation of the previous works on textual OOD detection, thenpresent the chosen benchmark, the pre-trained encoders, and baseline methods.
4.1 Previous works and their limitations
Previous works in OOD detection [48, 68, 39, 53, 59] mostly rely on a single model to determinewhich methods are the best.
This undermines the soundness of the conclusions that may only hold forthe particular instance of the chosen model ( e.g. for a specific checkpoint trained with a specific seed).
To the best of our knowledge, no work studies the impact of the several sources of randomness thatare involved, such as checkpoint and seed selections [70]).
Nevertheless, these hyperparameters doimpact the OOD detectors’ performances.
This is illustrated in Fig.
9 of the supplementary material,which gathers several Mahalanobis scores for various checkpoints of the same model.
In the light of Fig.
9, we choose to study both the impact of the checkpoint and the seed choice inour experiment.
Specifically, for each model, we consider 5 different checkpoints.
We save andprobe models after 1k, 3k, 5k, 10k, 15k, and 20k finetuning steps.
We additionally reproduce thisexperiment for 3 different seeds.
As the reuse of checkpoints reduces the cost of research and allowsfor easy head-to-head comparison, our library also contains the probed models to draw general robustconclusions about the performance of the considered class of models [19, 26, 81].
4.2 Dataset selection
Dataset selection is instrumental for OOD detection evaluation as it is unreasonable to expect adetection method to achieve good results on any type of OOD data [1].
Since there is a lack ofconsensus on which benchmark to use for OOD detection in NLP, we choose to rely on the benchmarkintroduced by [82] which is an extension of the one proposed by [36].
Benchmark description.
The considered benchmark is composed of three different types of indistribution datasets (referred to as IN-DS ) which are used to train the classifiers: sentiment analysis( i.e. , SST2 [72] and IMDB [56]), topic classification ( i.e. , 20Newsgroup [40]) and question answering( i.e. , TREC-10 [47]).
For splitting we use either the standard split or the one provided by [82].
Forthe OOD datasets (referred to as OUT-DS ), we first consider the aforementioned datasets ( i.e. , any pairof datasets can be considered as OOD).
Then, we also rely on four other datasets: a concatenationof premises and respective hypotheses from two NLI datasets ( i.e. , RTE [12, 38] and MNLI [79]),
Multi30K [28] and the source of the English-German WMT16 [8].
We gather in Tab.
5 the statisticsof the various data-sets and refer the reader to reference [82] for further details.
4.3 Baseline methods and pre-trained models
Baseline methods.
We consider the three following baselines 2 :
Maximum Soft-max Probability (MSP).
This method has been proposed by [34].
Given an input x ,it relies on the final score s MSP defined by s MSP ( x ) = 1 − max y ∈Y p Y | X
( y | x ) , where p Y | X ( ·| x ))is the soft-probability predicted by the classifier after x has been observed.2.
Energy-based score (E) [53] is defined as the score s E ( x ) =
T × log (cid:104)(cid:80)
y ∈Y exp (cid:16) g y ( x ) T (cid:17)(cid:105) ,where g y ( x ) represents the logit corresponding to the class label y .3.
Mahalanobis ( D M ) .
Following [59, 46, 82], the last layer of the encoder is considered leadingto the score: s M ( x ) =
− D M ( F PM ( x ) , F PM ( S train n, ˆ y )) where ˆ y represents the label predicted by theclassifier based on the observation of x .[-
-]
Aggregation procedures.
Both D M and D IRW rely on feature representations of the data which areextracted from the neural networks.
For feature extraction we test four different strategies:
Logits layer selection .
We use the raw non-normalized predictions of the classifier.
In this case
F Logits ≡ F ( ϕ 1 ( x ) , . . .
, ϕ L ( x )) =
ϕ L +1 ( x ) .2.
Last layer selection .
Following previous work in textual OOD detection [80], we also consider thelast layer of the network.
Formally F L ≡ F ( ϕ 1 ( x ) , . .
ϕ L ( x )) =
ϕ L ( x ) .3.
Layer concatenation .
We follow the BERT pooler and explore the concatenation of all layers.
Formally, F cat ≡ F ( ϕ 1 ( x ) , . .
ϕ L ( x )) =
[ ϕ 1 ( x ) , · · · , ϕ L ( x )] represents the concatenated vector.
The main limitation of layer concatenation is that the dimension of the considered features linearlyincreases with the number of layers which can be problematic for very deep networks [78].
Pretrained encoders.
To provide an exhaustive comparison, we choose to work with different types of pre-trained encoders.
We test the various methods on DISTILBERT ( DIS. )
[67], BERT [24] and ROBERTA ( ROB. )
We trained all models with a dropout rate [73] of 0 . 2 , a batch size of 32 , we use ADAMW [41].
Additionally, the weight decay is set to 0 .
01 , the warmup ratio is set to 0 .and
the learning rate to 10 − 5 .
[Table]
Static Experimental Results
In this section, we demonstrate the effectiveness of our proposed detector using various pre-trainedmodels.
Due to space limitations, additional tables are reported in the Supplementary Material.
5.1 Methods comparison
[Table]
In Tab. 2, we report the aggregated score obtained by each method combined with a differentaggregation function.
We observe that TRUSTED obtains the best overall scores followed by D IRWusing F cat .
Similarly to previous works [59], we notice in general that score leveraging informationavailable from the training set ( i.e. , D M and D IRW ) achieve stronger results than those relying onoutput of softmax scores solely ( i.e. , E and MSP).
Interestingly, we observe a large variation of performance of both D M and D IRW , e.g. , relying onthe logits representations instead of using the last layer ( i.e. , using F L +1 instead of F L ) hurts theperformances of the methods.
We observe a significant decrease in performance for D M whenaveraging all the layers (case of F PM ), suggesting that the multivariate Gaussian assumption onwhich Mahalanobis approaches rely [-on any distributional assumption.5.
-]no longer hold for the resulting distribution.
5.2 On the pre-trained encoder choice
When training and deploying a classifier, a key question is choosing a pre-trained encoder.
It can bebeneficial in critical applications to trade off the main task accuracy to ensure better OOD detection.
In Tab. 3, we report the individual performance of OOD methods on three types of classifiers.
[Table]
Although TRUSTED achieves state-of-the-art on all configurations [Table] it is worth noticing a differencein performance concerning the type of pre-trained model.
It is also important to remark that for
ROB both MSP and E achieve on-par performances with D M while not requiring any extra traininginformation.
Overall, for a given method ( i [Table] e. D M or D IRW ), the ranking of detectors performancesaccording to the type of feature extractor remain still.
This validates the use of the mean-aggregationprocedure of TRUSTED .
Overall, based on the difference of OOD detection performance of Tab. 3, werecommend to use TRUSTED on BERT or DIS.
if accurate OOD detection is required.
5.3 Impact of the training dataset
OOD detection performance depends on the nature of what is considered in-distribution (the trainingdistribution in our case).
Thus, it is interesting to study the performance per-
IN-DS as reported in
Tab. 4.
Even though TRUSTED achieves strong results in terms of AUROC , we observe a high FPR on
SST2.
From Fig.
[Figure] we observe that IMDB and SST2 are harder to detect, especially for D M .
Finally,since high AUROC does not necessarily imply a low FPR [Figure] it is crucial to take both into account whendesigning an OOD detector. [Figure]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
Dynamic Experimental Results
Most OOD detection methods are tested on specific checkpoints, where the selection criterion is oftenunclear.
The consequences of this selection on OOD detection are rarely studied.
This section aimsto respond to this by measuring the OOD detection performance of methods on various checkpointfinetuning of the pre-trained encoder.
We will use 5 different checkpoints taken after 1k, 3k, 5k,10k, 15k, and 20k iterations.
Training curves of the models are given in Fig.
Notice that after 3kiterations models have converged and no over-fitting is observed even after 20k iterations ( i.e., we donot observe an increase in validation loss).
Overall analysis.
We report the results of the dynamic analysis on the various pre-trained modelsin Fig.
For all the methods and models (except D M on ROB ), we observe that training longerthe classifier hurts detection.
Interestingly, this drop in performance has a higher impact on FPRcompared to AUROC .
Thus, it is better to use an early stopping criterion to ensure proper OODdetection performance.
In addition, we observed that TRUSTED (corresponding to D IRW ) achievesbetter detection results and that D M outperforms TRUSTED for checkpoints larger than 10k on ROB .
[Figure]
Analysis Per In-Dataset.
Fig.
6 reports the results of the dynamical analysis per IN-DS .
We observe that D IRW is consistently better than D M with sensible improvement on IMDB and SST2, which are the hardest benchmarks.
We observe a similar trend to the previous experiment: training longer the classifiers hurt their OOD detection performances .
Similar observations hold for FPR ,
AUPR-IN
[Figure]
and AUPR-OUT that are postponed to the
Supplementary Material (see Fig. 10)
[Figure]
Conclusions and Future Directions
In this work, we introduced TRUSTED a novel OOD detector that relies on information availablein all the hidden layers of a network.
TRUSTED leverages a novel similarity score built on top ofthe Integrate Rank-Weighted depth.
We conduct extensive numerical experiments proving that itconsistently outperforms previous approaches, including detection based on the Mahalanobis distance.
Our comprehensive evaluation framework demonstrates that, in general, OOD performances varydepending on several hyperparameters of the models, the datasets, and the detector’s feature extractionstep.
Thus, we would like to promote the use of such exhaustive evaluation frameworks for futuresearch to assess AI systems’ safety tools properly.
Another interesting question is the detectioninference-time / accuracy trade-off, which is instrumental for the practitioner.
[-Results 
-]
[-C.
-]
[-Remark 
-]