Original: I NTERPRETABLE O UT - OF -D ISTRIBUTION D ETECTION USING P ATTERN D ETECTION
Revised: I NTERPRETABLE O UT - OF -D ISTRIBUTION D ETECTION USING P ATTERN I DENTIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Out-of-distribution (OoD) detection for data-based programs is a goal of paramount importance.
Revised: Out-of-distribution (OoD) detection for data-based programs is a goal of paramount importance.

Original: Common approaches in the literature tend to train binary classiﬁers requiring inside-of-distribution (IoD) and OoD validation samples, and/or implement conﬁdence metrics that are often abstract and therefore difﬁcult to interpret.
Revised: Common approaches in the literature tend to train binary classiﬁers requiring inside-of-distribution (in-distribution, or IoD) and OoD validation samples, and/or implement conﬁdence metrics that are often abstract and therefore difﬁcult to interpret.

Original: In this work, we propose to use the PARTICUL pattern detection algorithm in order to build more interpretable and robust OoD detectors for visual classiﬁers.
Revised: In this work, we propose to use the PARTICUL pattern identiﬁcation algorithm in order to build more interpretable and robust OoD detectors for visual classiﬁers.

Original: Crucially, this approach does not require to retrain the classiﬁer and is tuned directly to the IoD dataset, making it applicable to domains where OoD does not have a clear deﬁnition.
Revised: Crucially, this approach does not require to retrain the classiﬁer and is tuned directly to the IoD dataset, making it applicable to domains where OoD does not have a clear deﬁnition.

Original: Moreover, pattern detection allows us to provide images from the IoD dataset as reference points to better explain our conﬁdence scores.
Revised: Moreover, pattern identiﬁcation allows us to provide images from the IoD dataset as reference points to better explain our conﬁdence scores.

Original: We illustrate the generalization abilities of our approach through an extensive benchmark across four datasets and two deﬁnitions of OoD. Our experiments show that the robustness of all metrics under test does not solely depend on the nature of the IoD dataset or the OoD deﬁnition, but also on the architecture of the classiﬁer, which stresses the need for thorough experimentations for future work on OoD detection
Revised: We illustrate the generalization abilities of our approach through an extensive benchmark across four datasets and two deﬁnitions of OoD. Our experiments show that the robustness of all metrics under test does not solely depend on the nature of the IoD dataset or the OoD deﬁnition, but also on the architecture of the classiﬁer, which stresses the need for thorough experimentations for future work on OoD detection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A fundamental aspect of software safety is arguably the modelling of its expected operational domain through a formal or semi-formal speciﬁcation, giving clear boundaries on when it is sensible to deploy the program, and when it is not.
Revised: A fundamental aspect of software safety is arguably the modelling of its expected operational domain through a formal or semi-formal speciﬁcation, giving clear boundaries on when it is sensible to deploy the program, and when it is not.

Original: It is however difﬁcult to deﬁne such boundaries for machine learning programs, especially for visual classiﬁers based on artiﬁcial neural networks (ANN), which are the subject of this paper.
Revised: It is however difﬁcult to deﬁne such boundaries for machine learning programs, especially for visual classiﬁers based on artiﬁcial neural networks (ANN), which are the subject of this paper.

Original: Indeed, such programs process high-dimensional data (images, videos) and are the result of a complex optimization procedure, but they do not embed clear failure modes that could get trigged in the case of an unknown distribution, with potentially dire consequences in critical applications.
Revised: Indeed, such programs process high-dimensional data (images, videos) and are the result of a complex optimization procedure, but they do not embed clear failure modes that could get trigged in the case of an unknown distribution, with potentially dire consequences in critical applications.

Original: Although it is difﬁcult to characterize the operational distribution, one could still measure its dissimilarity with other distributions.
Revised: Although it is difﬁcult to characterize an operational distribution, one could still measure its dissimilarity with other distributions.

Original: In this context, Out-of-Distribution (OoD) detection - which aims to detect whether an input of an ANN is Inside-of-distribution (IoD) or outside of it - serves several purposes.
Revised: In this context, Out-of-Distribution (OoD) detection - which aims to detect whether an input of an ANN is Inside-of-distribution (IoD) or outside of it - serves several purposes.

Original: It helps characterize the extent to which the ANN can operate outside a bounded dataset (which is important due to the incompleteness of the training set w.r.t.
Revised: It helps characterize the extent to which the ANN can operate outside a bounded dataset (which is important due to the incompleteness of the training set w.r.t.

Original: the operational domain).
Revised: the operational domain).

Original: It also constitutes a surrogate measure of the generalization abilities of the ANN.
Revised: It also constitutes a surrogate measure of the generalization abilities of the ANN.

Original: Finally, OoD detection can help assess when an input is too far away from the operational domain, which prevents misuses of the program and increases its safety.
Revised: Finally, OoD detection can help assess when an input is too far away from the operational domain, which prevents misuses of the program and increases its safety.

Original:  C ONTRIBUTION AND RELATED WORK
Revised:  C ONTRIBUTION AND RELATED WORK

Original: Out-of-distribution detection The maximum class probability (MCP) obtained after softmax normalization of the classiﬁer logits already constitutes a good baseline for OoD detection Hendrycks & Gimpel (2017).
Revised: Out-of-distribution detection The maximum class probability (MCP) obtained after softmax normalization of the classiﬁer logits already constitutes a good baseline for OoD detection Hendrycks & Gimpel (2017).

Original: However, neural networks tend to be overconﬁdent on their predictions Szegedy et al.
Revised: However, neural networks tend to be overconﬁdent on their predictions Szegedy et al.

Original: Lee et al.
Revised: Lee et al.

Original: (2018a);
Revised: (2018a);

Original: Hein et al.
Revised: Hein et al.

Original: (2019), even when they are wrong, which may result in false claims of conﬁdence.
Revised: (2019), even when they are wrong, which may result in false claims of conﬁdence.

Original: Hence, the development of enhancements like temperature scaling Liang et al.
Revised: Hence, the development of enhancements like temperature scaling Liang et al.

Original: (2017), ensemble learning Nguyen et al.
Revised: (2017), ensemble learning Nguyen et al.

Original: (2020) or True-Class Probability learning Corbiere et al.
Revised: (2020) or True-Class Probability learning Corbiere et al.

Original: (2021) (assuming such information is known).
Revised: (2021) (assuming such information is known).

Original: As an alternative, other types of conﬁdence measures have also been developed.
Revised: As an alternative, other types of conﬁdence measures have also been developed.

Original: Approaches such as the
Revised: Approaches such as the

Original: Fractional Neuron Region Distance Hond et al.
Revised: Fractional Neuron Region Distance Hond et al.

Original: (2021) (FNRD) or the Attribution-Based Conﬁdence Jha et al.
Revised: (2021) (FNRD) or the Attribution-Based Conﬁdence Jha et al.

Original: (2019) (ABC) require only access to a trained network and, for FNRD, to the training set of the classiﬁer.
Revised: (2019) (ABC) require only access to a trained network and, for FNRD, to the training set of the classiﬁer.

Original: FNRD computes the range of activations for each neuron on the training set, then provides a score describing how many neurons are  outside their boundaries for a given input, while ABC equates the conﬁdence of the network to its local stability by sampling the neighbourhood of a given input and measuring stability through attribution methods.
Revised: FNRD computes the range of activations for each neuron on the training set, then provides a score describing how many neurons are activated outside their boundaries for a given input, while ABC equates the conﬁdence of the network to its local stability by sampling the neighbourhood of a given input and measuring stability through attribution methods.

Original: Although this last method relies on fewer prerequisites, it is computationally expensive ( e.g., gradient computation) and may not be suited to runtime constraints.
Revised: Although this last method relies on fewer prerequisites, it is computationally expensive ( e.g., gradient computation) and may not be suited to runtime constraints.

Original: Other approaches also attempt to capture the ”normal” behaviour of the classiﬁer: Huang et al.
Revised: Other approaches also attempt to capture the ”normal” behaviour of the classiﬁer: Huang et al.

Original: (2021) states that the latent representations of OoD samples through a CNN classiﬁer are clustered around a point called the feature-space singularity (FSS) which serves as a reference point for OoD detection.
Revised: (2021) states that the latent representations of OoD samples through a CNN classiﬁer are clustered around a point called the feature-space singularity (FSS) which serves as a reference point for OoD detection.

Original: Similarly, Lee et al.
Revised: Similarly, Lee et al.

Original: (2018b) proposes a conﬁdence score based on the Mahalanobis distance between a new sample and class conditional Gaussian distributions inferred from the training set.
Revised: (2018b) proposes a conﬁdence score based on the Mahalanobis distance between a new sample and class conditional Gaussian distributions inferred from the training set.

Original: However, both approaches operate upon multiple layers of the network and require the use of a set of OoD samples for calibrating the relative importance of each layer in the ﬁnal conﬁdence score.
Revised: However, both approaches operate upon multiple layers of the network and require the use of a set of OoD samples for calibrating the relative importance of each layer in the ﬁnal conﬁdence score.

Original: As we will see later in this work, this reliance on additional samples can lead to very unstable results depending on their representativeness of OoD inputs.
Revised: As we will see later in this work, this reliance on additional samples can lead to very unstable results depending on their representativeness of OoD inputs.

Original: A ﬁnal line of research proposes to integrate the learning of a conﬁdence measure directly during the training of the classiﬁer itself (Lee et al.
Revised: A ﬁnal line of research proposes to integrate the learning of a conﬁdence measure directly during the training of the classiﬁer itself (Lee et al.

Original: (2018a);
Revised: (2018a);

Original: Hein et al.
Revised: Hein et al.

Original: Similarly, the whole ﬁeld of Bayesian DeepLearning aims to capture uncertainty by design in the model, using for instance dropout as an approximation for Bayesian Inference Gal & Ghahramani (2016).
Revised: Similarly, the whole ﬁeld of Bayesian DeepLearning aims to capture uncertainty by design in the model, using for instance dropout as an approximation for Bayesian Inference Gal & Ghahramani (2016).

Original: While efﬁcient, these approaches requireto retrain the classiﬁer, which can prove costly or impractical in an industrial context.
Revised: While efﬁcient, these approaches require the classiﬁer to be retrained, which can prove costly or impractical in an industrial context.

Original: Importantly, all these methods were evaluated on different datasets and deﬁnitions of OoD ( e.g., different datasets, distribution shifts), which only gives a partial picture on their robustness Tajwar et al.
Revised: Importantly, all these methods were evaluated on different datasets and deﬁnitions of OoD ( e.g., different datasets, distribution shifts), which only gives a partial picture on their robustness Tajwar et al.

Original: Pattern detection A lot of work has been focused on part detection, in supervised (using annotations Farhadi et al.
Revised: Part detection Many object recognition methods have focused on part detection, in supervised (using annotations Farhadi et al.

Original: ; Han et al.
Revised: ; Han et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: , weakly-supervised (using class labels Li et al.
Revised: , weakly-supervised (using class labels Li et al.

Original:  Peng et al.
Revised: ; Peng et al.

Original: or unsupervised settings (Zheng et al.
Revised: or unsupervised settings (Zheng et al.

Original: ; Han et al.
Revised: ; Han et al.

Original: (2022); Xu-Darme et al.
Revised: Xu-Darme et al.

Original: (2022)), primarily with the goal of improving accuracy on hard classiﬁcation tasks such as ﬁne-grained recognition Welinder et al.
Revised: (2022)), primarily with the goal of improving accuracy on hard classiﬁcation tasks such as ﬁne-grained recognition Welinder et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: To our knowledge, the PARTICUL algorithm described in Xu-Darme et al.
Revised: To our knowledge, the PARTICUL algorithm described in Xu-Darme et al.

Original: (2022) is the only method including a conﬁdence measure associated with the detected parts (in the paper, it is used to infer the visibility of a given part).
Revised: (2022) is the only method including a conﬁdence measure associated with the detected parts (in the paper, it is used to infer the visibility of a given part).

Original: PARTICUL aims at mining recurring patterns in the latent representation of a set of images processed through a CNN, in an unsupervised manner.
Revised: PARTICUL aims at mining recurring patterns in the latent representation of a set of images processed through a CNN, in an unsupervised manner.

Original: It is however restricted to ﬁne-grained recognition datasets where all images belong to the same macro-category.
Revised: It is however restricted to ﬁne-grained recognition datasets where all images belong to the same macro-category.

Original: Our contributions In this paper, we aim to investigate the following research tracks: 1) How well can OoD detection methods generalize across multiple deﬁnitions of OoD?
Revised: Our contributions In this paper, we aim to investigate the following research tracks: 1) How well can OoD detection methods generalize across multiple deﬁnitions of OoD?

Original: 2) How to enhance the overall performance (precision, recall, genericity) of OoD detection?
Revised: 2) How to enhance the overall performance (precision, recall, genericity) of OoD detection?

Original: How to improve the interpretability of conﬁdence measures in the context of visual classiﬁers?
Revised: How to improve the interpretability of conﬁdence measures in the context of visual classiﬁers?

Original: As an answer to those questions, we make the following contributions:
Revised: As an answer to those questions, we make the following contributions:

Original: We present a novel OoD measure relying on part detection learning based on work (Xu-Darme et al.
Revised: We present a novel OoD measure relying on part detection learning based on work (Xu-Darme et al.

Original: in explainable AI (xAI) that can use any deﬁnition of in-distribution dataset, does not depend on a deﬁnition of OoD, can provide visual justiﬁcations for conﬁdence scores, is modular for the problem at hand and requires no retraining of the original classiﬁer, and, in fact, needs to access only one layer of the model which allows it to be used in an almost black-box setting; 2.
Revised: in explainable AI (xAI) that can use any deﬁnition of in-distribution dataset, does not depend on a deﬁnition of OoD, can provide visual justiﬁcations for conﬁdence scores, is modular for the problem at hand and requires no retraining of the original classiﬁer, and, in fact, needs to access only one layer of the model which allows it to be used in an almost black-box setting; 2.

Original: We demonstrate the ability of our approach to consistently capture various notions of OoD over an extensive benchmark comprising four datasets, two architectures and several evaluation metrics.
Revised: We demonstrate the ability of our approach to consistently capture various notions of OoD over an extensive benchmark comprising four datasets, two architectures and several evaluation metrics.

Original: This paper is organized as follows:
Revised: This paper is organized as follows:

Original: Sec. 3 formulates the problem and our OoD deﬁnition; Sec. 4 introduces our modiﬁcations to the PARTICUL algorithm with the goal of providing a more interpretable and robust conﬁdence measure; Sec. 5 describes our experimental setup and results.
Revised: Sec. 3 formulates the problem and the two OoD modalities we will explore; Sec. 4 introduces our modiﬁcations to the PARTICUL algorithm with the goal of providing a more interpretable and robust conﬁdence measure; Sec. 5 describes our experimental setup and results.

Original: Finally, Sec. 6 presents our conclusions and future work.
Revised: Finally, Sec. 6 presents our conclusions and future work.

Original:  P ROBLEM FORMULATION
Revised:  P ROBLEM FORMULATION

Original: Let I s be the space of all RGB images of size s .
Revised: Let I s be the space of all RGB images of size s .

Revised: Let D iod ⊆

Original:  Let M be a deep convolutional neural network (CNN) classiﬁer trained on dataset D iod⊆ I s , approximating the distribution D iod of images belonging to N categories.
Revised: I s be a dataset approximating the distribution D iod of images belonging to N categories. Let M be a deep convolutional neural network (CNN) classiﬁer trained on a subset of D iod , X train .

Original: For any image x ∈ I s , M outputs a vector of logits M ( x ) ∈ R N , where the index of the highest value corresponds to the most probable category (or class ) of x - relatively to all other categories.
Revised: For any image x ∈ I s , M outputs a vector of logits M ( x ) ∈ R N , where the index of the highest value corresponds to the most probable category (or class ) of x - relatively to all other categories.

Original: Since any image is assigned a category regardless of its similarity to the training distribution, M ( x ) is often normalized using a softmax function in order to indicate a level of conﬁdence in the prediction.
Revised: Since any image is assigned a category regardless of its similarity to the training distribution, M ( x ) is often normalized using a softmax function in order to indicate a level of conﬁdence in the prediction.

Original: More generally, a conﬁdence measure C M : I s → [0 , 1] is a function assigning a score to each image x ∈
Revised: More generally, a conﬁdence measure C M : I s → [0 , 1] is a function assigning a score to each image x ∈

Original: I s processed through M .
Revised: I s processed through M .

Original: We denote C M ( D iod ) the distribution of values computed over D iod using the conﬁdence measure C M .
Revised: We denote C M ( D iod ) the distribution of values computed over D iod using the conﬁdence measure C M .

Original: As indicated above, C M ( D iod ) is approximated by C M ( D iod ) 
Revised: As indicated above, C M ( D iod ) is approximated by C M ( D iod ) ≈

Original: ≈ C M ( D iod ) =
Revised: C M ( D iod ) =

Original: (cid:8) C M ( x ) (cid:12) (cid:12) x
Revised: (cid:8) C M ( x ) (cid:12)(cid:12) x

Original: ∈ D iod (cid:9) .
Revised: ∈ D iod (cid:9) .

Original: Ideally, images belonging to the training distribution D iod should have a higher conﬁdence score than images outside D iod .
Revised: Ideally, images belonging to the training distribution D iod should have a higher conﬁdence score than images outside D iod .

Revised: We propose to take an empirical approach, which will only attempt to provide evidence that a conﬁdence measure tends to return a greater score for images in

Revised: D iod than images outside D iod .

Original: However, it is not possible to obtain a dataset representative of I s \ D iod , i.e., encompassing all possible OoD inputs.
Revised: However, it is not possible to obtain a dataset representative of I s \ D iod , i.e., encompassing all possible OoD inputs.

Original: Therefore, in this work we use two weaker modalities of OoD detection that serve as a benchmark for  various proposals of conﬁdence measures.
Revised: Therefore, in this work we use two weaker modalities of OoD detection that serve as benchmarks for assessing the various proposed conﬁdence measures.

Original: Cross-dataset detection In this experiment, we evaluate the separability of the distributions C M
Revised: Cross-dataset detection In this experiment, we evaluate the separability of the distributions C M

Original: ( D iod ) and C M
Revised: ( D iod ) and C M

Original: ( D ood ) , where D ood is a dataset different  from D iod .
Revised: ( D ood ) , where D ood is a dataset different (not drawn from the same distribution) from D iod .

Original: We use three complementary metrics that do not require to determine an acceptance threshold between IoD and OoD inputs, and that are suitable to compare differentproposals of conﬁdence measures: the Area Under the ROC curve (AUROC ↑ ); the Area Under the Precision-Recall curve (AUPR ↑ ); the False Positive Rate when the true positive rate is 80% (FPR80 ↓ ).
Revised: We use three complementary metrics that do not require determination of an acceptance threshold between IoD and OoD inputs, and that are suitable for comparing the different conﬁdence measures proposed: the Area Under the ROC curve (AUROC); the Area Under the Precision-Recall curve (AUPR) and the False Positive Rate when the true positive rate is 80% (FPR80).

Original: Sensitivity to dataset perturbation For this second experiment, we build the OoD dataset D ood by applying a perturbation to all images of D iod .
Revised: Sensitivity to dataset perturbation For this second experiment, we generate an OoD dataset D ood by applying a perturbation to all images of D iod .

Original: More precisely, a perturbation P λ is a function applying a transformation on an image x ∈
Revised: More precisely, a perturbation P λ is a function which applies a transformation to an image x ∈

Original: I s with a variable intensity
Revised: I s with a variable magnitude λ

Original: λ ( e.g., for a rotation, λ might correspond to the angle of the rotation).
Revised: ( e.g., for a rotation, λ might correspond to the angle of the rotation).

Revised: Note that there exist no general value for the magnitude of the perturbation that would result in D ood to be OoD in all cases.

Revised: In this paper, maximum perturbation amplitude is chosen when the resulting image is qualitatively considered considerably different than the base image.

Original: For a perturbation P λ applied over D iod , we deﬁne the average value of conﬁdence measure C M as
Revised: For a perturbation P λ applied over D iod , we deﬁne the average value of conﬁdence measure C M as

Original: [Equation]
Revised: [Equation]

Original: which is extended to a set of perturbation intensities Λ
Revised: which is extended to a set of perturbation magnitudes Λ

Original: = ( λ 0 , . . .
Revised: = ( λ 0 , . . .

Original: , λ n ) as Γ( C M , D iod , P, Λ) =
Revised: , λ n ) as Γ( C M , D iod , P, Λ) =

Original: (cid:0) Γ( C M , D iod , P, λ 0 ) , . .
Revised: (cid:0) Γ( C M , D iod , P, λ 0 ) , . .

Original: , Γ( C M , D iod , P, λ n ) (cid:1) .
Revised: , Γ( C M , D iod , P, λ n ) (cid:1) .

Original: Although it wouldbe again  possible to measure the separability of IoD and OoD conﬁdence distributions, perturbations of small intensities would result in almost identical distributions.
Revised: Although it would again be possible to measure the separability of IoD and OoD conﬁdence distributions, perturbations of small intensities would result in almost identical distributions.

Original: Instead, we evaluate the correlation between the intensity of the perturbation and the average conﬁdence value of the perturbed dataset by computing the Spearman Rank correlation coefﬁcient r s between Λ and Γ( C M , D iod , P, Λ) (cid:1) .
Revised: Instead, we evaluate the correlation between the intensity of the perturbation and the average conﬁdence value of the perturbed dataset by computing the Spearman Rank correlation coefﬁcient r s between Λ and Γ( C M , D iod , P, Λ) (cid:1) .

Original: Indeed, r s (cid:0) Λ , Γ( C M , D iod , P, Λ) (cid:1) = [Equation] (resp.
Revised: Indeed, r s (cid:0) Λ , Γ( C M , D iod , P, Λ) (cid:1) = 1 (resp.

Original: − 1 ) indicates that the average conﬁdence measure increases (resp.
Revised: − 1 ) indicates that the average conﬁdence measure increases (resp.

Original: decreases) monotonously with the value of λ .
Revised: decreases) monotonically with the value of λ , which indicates that the measure is at least correlated with an increase in the perturbation.

Original:  P ROPOSED MODEL
Revised:  P ROPOSED MODEL

Original: In this section, we present two proposals for building a conﬁdence measure based on the PARTICUL algorithm,  described in Xu-Darme et al.
Revised: In this section, we present two proposals for building a conﬁdence measure based on the PARTICUL algorithm, as described in Xu-Darme et al.

Original: (2022), which aims at mining recurring patterns in the latent representation of a set of images processed through a CNN.
Revised: PARTICUL is intented to mine recurring patterns in the latent representation of a set of images processed through a CNN.

Original: In the context of this paper,we learn patterns  from the last convolutional layer of  classiﬁer M over the training set D iod , in a plug-in fashion that does not requireto retrain the classiﬁer.
Revised: In the context of this paper, patterns are learnt from the last convolutional layer of a classiﬁer M over the training set D iod , in a plug-in fashion that does not require the classiﬁer to be retrained.

Original: Let F be the restriction of classiﬁer M up to its last convolutional layer, i.e., M = L ◦ F , where L corresponds to the last pooling layer followed by one or several fully connected layers.
Revised: Let F be the restriction of classiﬁer M up to its last convolutional layer, i.e., M = L ◦ F , where L corresponds to the last pooling layer followed by one or several fully connected layers.

Original: ∀ x ∈ I s , F ( x ) ∈
Revised: ∀ x ∈ I s , F ( x ) ∈

Original: R H
Revised: R H

Original: × W × D is a convolutional map of D -dimensional vectors.
Revised: × W × D is a convolutional map of D -dimensional vectors.

Original: Let X train ⊆

Original: D iod be the subset of the

Original: IoD dataset D iod used during the training of the classiﬁer.

Original: Vanilla PARTICUL
Revised: Vanilla PARTICUL

Original: The purpose of the PARTICUL algorithm is to learn p distinct 1 × 1 × D convolutional kernels K =
Revised: The purpose of the PARTICUL algorithm is to learn p distinct 1 × 1 × D convolutional kernels K =

Original: [ k (1) , . .
Revised: [ k (1) , . .

Original: k ( p ) ] , such that ∀ x ∈ X train , each kernel k ( i ) strongly correlates with exactly one vector F [ h,w ] ( x ) in F ( x ) ( Locality constraint).
Revised: k ( p ) ] (or detectors ), such that ∀ x ∈ X train , each kernel k ( i ) strongly correlates with exactly one vector F [ h,w ] ( x ) in F ( x ) ( Locality constraint).

Original: After training, PARTICUL uses the function H ( i ) ( x ) = max h,w (cid:0) F [ h,w ] ( x ) ∗ k ( i ) (cid:1) returning the maximum correlation score between kernel k ( i ) and the convolutional map F ( x ) .
Revised: After the training of said kernels, PARTICUL uses the function H ( i ) ( x ) = max h,w (cid:0) F [ h,w ] ( x ) ∗ k ( i ) (cid:1) returning the maximum correlation score between kernel k ( i ) and the convolutional map F ( x ) .

Original: In Xu-Darme et al.
Revised: In Xu-Darme et al.

Original: (2022), the distribution H ( i ) ( D iod ) is modeled as a random variable following a normal distribution N ( µ ( i ) , σ ( i ) ) and estimated  over X train .
Revised: (2022), the distribution H ( i ) ( D iod ) is modeled as a random variable following a normal distribution N ( µ ( i ) , σ ( i ) ) and estimated (or calibrated ) over X train .

Original: Then, the corresponding conﬁdence measure is computed using the cumulative distribution function (CDF) associated with N ( µ ( i ) , σ ( i ) ) .
Revised: Then, the corresponding conﬁdence measure is computed using the cumulative distribution function (CDF) associated with N ( µ ( i ) , σ ( i ) ) .

Original: However, such CDF does not have a closed-form solution, which can slow down operations on tensors.
Revised: However, such CDF does not have a closed-form solution, which can slow down operations on tensors.

Original: Therefore, in this work, we choose to model H ( i ) ( D iod ) as a logistic distribution L ( µ ( i ) , σ ( i ) ) , where µ ( i ) and σ ( i ) are again estimated over X train , and deﬁne the conﬁdence associated with detector i for image x as:
Revised: Therefore, in this work, we choose to model H ( i ) ( D iod ) as a logistic distribution L ( µ ( i ) , σ ( i ) ) , where µ ( i ) and σ ( i ) are again estimated over X train , and deﬁne the conﬁdence associated with detector i for image x as:

Original: [Equation]
Revised: [Equation]

Original: Finally, we average the conﬁdence measures of all detectors and deﬁne, ∀ 
Revised: Finally, we average the conﬁdence measures of all detectors and deﬁne, ∀ x ∈ I s : C vPM [Equation] =

Original: [Equation]
Revised: [Equation]

Original:  PARTICUL is well-suited for datasets with low intra-variance - i.e., when images in X train are very similarwhich is the case for ﬁne-grained recognition datasets (Welinder et al.
Revised: C [Equation] ( x ) . PARTICUL is well-suited for datasets with low intra-set variance - i.e., when images in X train are very similar - which is the case for ﬁne-grained recognition datasets (Welinder et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: However, for more heterogeneous datasets (Krizhevsky (2009); Li et al.
Revised: However, for more heterogeneous datasets (Krizhevsky (2009); Li et al.

Original: , it becomes more difﬁcult to ﬁnd recurring patterns that are present across the entire training set.
Revised: , it becomes more difﬁcult to ﬁnd recurring patterns that are present across the entire training set.

Original: Moreover, in order to learn more consistent patterns, it might be interesting to learn detectors that are speciﬁc to each of the N classes of D iod .
Revised: Moreover, in order to learn more consistent patterns, it might be beneﬁcial to learn detectors that are speciﬁc to each of the N classes of D iod .

Original: Class-based PARTICUL
Revised: Class-based PARTICUL

Original: Here, we introduce a variant of PARTICUL that uses the labels of the training set X train in order to learn p detectors per class .
Revised: Here, we introduce a variant of PARTICUL that uses the labels of the training set X train in order to learn p detectors per class .

Original: During the training and calibration phases, for each pair of images and  label ( x, c ) ∈ X train  , only the detectors of class c are modiﬁed.
Revised: During the training and calibration phases, for each pair of images and class label ( x, c ) ∈ X train × N , only the detectors of class c are modiﬁed.

Original: More precisely, let K c =
Revised: More precisely, let K c =

Original: [ k (1) c , . . .
Revised: [ k (1) c , . . .

Original: k ( p ) c ] be the set of kernel detectors for class c .
Revised: k ( p ) c ] be the set of kernel detectors for class c .

Original: As in Xu-Darme et al.
Revised: As in Xu-Darme et al.

Original: , K c is learned by minimizing the weighted combination of a locality function L l and an unicity function L u ,
Revised: , K c is learned by minimizing the weighted combination of a locality function L l and an unicity function L u ,

Original: L ( K c ) =
Revised: L ( K c ) =

Original: L l ( K c ) + λ L u ( K c ) , restricted to X train,c
Revised: L l ( K c ) + λ L u ( K c ) , restricted to X train,c

Original: = (cid:8) x | ( x, c ) ∈
Revised: = (cid:8) x | ( x, c ) ∈

Original: X train (cid:9) .
Revised: X train (cid:9) .

Original: Importantly, since patterns may be similar across different classes ( e.g., the wheels on a car or a bus), we do not treat images from other classes as negative samples when learning our class-based detectors.
Revised: Importantly, since patterns may be similar across different classes ( e.g., the wheels on a car or a bus), we do not treat images from other classes as negative samples when learning our class-based detectors.

Original: Therefore, although this approach may allowlearning more speciﬁc detectors  than the vanilla -PARTICUL approach, it effectively restricts the number of training samples per detector to the images of a single class, which is potentially detrimental to the  training process.
Revised: Therefore, although this approach may allow more speciﬁc detectors to be learnt than the vanilla -PARTICUL approach, it effectively restricts the number of training samples per detector to the images of a single class, which is potentially detrimental to the PARTICUL training process.

Original: The calibration of the detectors for class c also uses a logistic distribution model, which is estimated over training samples from class c only.
Revised: The calibration of the detectors for class c also uses a logistic distribution model, which is estimated over training samples from class c only.

Original: Then, as shown in Fig. 1, for any image x ∈
Revised: Then, as shown in Fig. 1, for any image x ∈

Original: I s , and with no access to the ground truth label of x , the conﬁdence measure C cPM ( x ) is obtained by averaging the conﬁdence scores from the detectors associated with the predicted class z = argmax (cid:0) M ( x ) (cid:1) of image x , i.e., 
Revised: I s , and with no access to the ground truth label of x , the conﬁdence measure C cPM ( x ) is obtained by averaging the conﬁdence scores from the detectors associated with the predicted class z = argmax (cid:0) M ( x ) (cid:1) of image x , i.e., C cPM ( x ) = 1 p p (cid:80)

Original: [Equation]
Revised: [Equation]

Original: As a consequence, for an image x ∈ D iod , the quality of our class-based conﬁdence measure is also dependent on the ability of the classiﬁer to assign the correct class to x , i.e., on the accuracy of M .
Revised: As a consequence, for an image x ∈ D iod , the quality of our class-based conﬁdence measure is also dependent on the ability of the classiﬁer to assign the correct class to x , i.e., on the accuracy of M .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present the experimental results obtained on our PARTICUL-based conﬁdence measure, in the context of cross-dataset OoD detection and perturbation OoD detection.
Revised: In this section, we present the experimental results obtained for our PARTICUL-based conﬁdence measure, in the context of cross-dataset OoD detection and perturbation OoD detection.

Original: Setup We performed our evaluation using a ResNet50 He et al.
Revised: Setup We performed our evaluation using a ResNet50 He et al.

Original: (2016) classiﬁer trained on various datasets, with different image sizes: the Caltech-101 Li et al.
Revised: (2016) classiﬁer trained on various datasets, with different image sizes: the Caltech-101 Li et al.

Original: (2022) dataset contains 9,144 images - from 101 different categories - that are resized to 224 × 224 during training (acc. 86.8%); the CUB-200 Welinder et al.
Revised: (2022) dataset contains 9,144 images - from 101 different categories - that are resized to 224 × 224 during training (test accuracy 86.8%); the CUB-200 Welinder et al.

Original: (2010) dataset contains 11,788 images - from 200 different bird species - that are resized to 448 × 448 during training (acc. 83.6%); the Stanford Cars Yang et al.
Revised: (2010) dataset contains 11,788 images - from 200 different bird species - that are resized to 448 × 448 during training (test accuracy 83.6%); the Stanford Cars Yang et al.

Original: (2015) dataset contains 16,185 images - from 196 different car models - that are resized to 448 × 448 (acc. 90.5%).
Revised: (2015) dataset contains 16,185 images - from 196 different car models - that are resized to 448 × 448 (test accuracy 90.5%).

Original: Both CUB-200 and Stanford Cars are ﬁne-grained classiﬁcation datasets and contain very homogeneous data from a single macro-category, while Caltechcontains more heterogeneous data from diverse categories.
Revised: Both CUB-200 and Stanford Cars are ﬁne-grained classiﬁcation datasets and contain very homogeneous data from a single macro-category, while Caltech-101 contains more heterogeneous data from diverse categories.

Original: Each dataset is split between a training set X train (used for  calibration) and a test set X test (used for evaluation).
Revised: Each dataset is split between a training set X train (also used for PARTICUL calibration) and a test set X test (used for evaluation).

Original:  Using the trained classiﬁers for feature extraction, we implemented our PARTICUL detectors using the two approaches described above: for vanilla PARTICUL, we trained 4 or 6 global detectors, ignoring image labels during training; for class-based PARTICUL, we trained 4 or 6 detectors per class , using the image labels of the training set.
Revised: We follow the dataset providers’ proposed splits. Using the trained classiﬁers for feature extraction, we implemented our PARTICUL detectors using the two approaches described above: for vanilla PARTICUL, we trainedor 6 global detectors, ignoring image labels; for class-based PARTICUL, we trained 4 or 6 detectors per class , using the image labels of the training set.

Original: As a comparison, we also implemented a conﬁdence measure C fnrdM ( x ) = 1 − fNRD ( x ) based on the fractional Neuron Region Distance Hond et al. (2021) (fNRD), as well as the FSSD measure C fssdM presented in Huang et al.
Revised: As a comparison, we also implemented a conﬁdence measure C fnrdM ( x ) = 1 − fNRD ( x ) based on the fractional Neuron Region Distance Hond et al. (2021) (fNRD), as well as the Feature Space Singularity Distance (FSSD) measure C fssd M presented in Huang et al.

Original: It is important to indicate that the FSSD is weakly supervised as it uses a validation OoD dataset D ood,val in order to learn a linear regressor with both positive and negative samples.
Revised: It is important to indicate that the FSSD is weakly supervised as it uses a validation OoD dataset D ood,val in order to learn a linear regressor with both positive and negative samples.

Original: In Huang et al.
Revised: In Huang et al.

Original: (2021), the authors choose D ood,val ∼ D ood , effectively training a binary classiﬁer between D iod and D ood .
Revised: (2021), the authors choose D ood,val ∼ D ood , effectively training a binary classiﬁer between D iod and D ood .

Original: For a fair comparison between methods, we perform several experiments using different validation OoD datasets to perform FSSD calibration.
Revised: For a fair comparison between methods, we perform several experiments using different validation OoD datasets to perform FSSD calibration.

Original: Finally, since the output of the classiﬁer can also constitute a valid conﬁdence measure, we also compare our results with the maximum class probability [Equation]
Revised: Finally, since the output of the classiﬁer can also constitute a valid conﬁdence measure, we also compare our results with the maximum class probability C MCPM ( x ) = max (cid:0) softmax (cid:0) M ( x ) (cid:1)(cid:1) .

Original: Cross- dataset out-of distribution In this section, we evaluate the ability of various conﬁdence measures to distinguish the dataset D iod- used when training the classiﬁer - from another dataset D ood (cid:54) =
Revised: Cross-dataset out-of-distribution In this section, we evaluate the ability of various conﬁdence measures to distinguish the dataset D iod from another dataset D ood (cid:54) =

Original: D iod .
Revised: D iod .

Original: As shown in Table 1 and illustrated in Fig.
Revised: As shown in Table 1 and illustrated in Fig.

Original: 2, the ability to discriminate D iod from D ood widely differs, depending on the choice of measure and the  datasets themselves.
Revised: 2, the ability to discriminate D iod from D ood widely differs, depending on the choice of measure and the [Table] datasets themselves.

Original: In particular, when used as reference distributions,  homogeneous datasets such as CUB200 and Stanford Cars are in general easier to discriminate than  more heterogenous datasets, across all measures except C fnrdM .
Revised: In particular, when used as reference distributions, the homogeneous datasets CUB and Stanford Cars are in general easier to discriminate than the more heterogenous dataset Caltech-101, across all measures except C fnrdM .

Original: This is consistent with the intuition that Caltech101 distribution covers a broader area of the latent space and is therefore harder to circumscribe.
Revised: This is consistent with the intuition that Caltech101 distribution covers a broader area of the latent space and is therefore harder to circumscribe.

Original: As[Table] expected, when using D ood,val ∼ D ood , C fssdM becomes a binary classiﬁer with fully-supervised training and displays great discriminative properties.
Revised: As expected, when using D ood,val ∼ D ood , C fssdM becomes a binary classiﬁer with fully-supervised training and displays a high-level of discrimination.

Original: However, when using D ood,val (cid:54)∼ D ood , the ability of this conﬁdence measure to generalize to unseen datasets can be questioned.
Revised: However, when using D ood,val (cid:54)∼ D ood , the ability of this conﬁdence measure to generalize to unseen datasets is more questionable.

Original: Indeed, for Caltech101 v. CUB and Caltech101 v. Stanford Cars, we obtain results with a high variance depending on the choice of validation dataset.
Revised: Indeed, for Caltech101 v. CUB200 and Caltech101 v. Stanford Cars, we obtain results which vary markedly depending on the choice of validation dataset.

Original: Surprisingly, the conﬁdence measure C MCPM shows good discriminative properties for most pairs ( D iod , D ood ) .
Revised: Surprisingly, the conﬁdence measure C MCPM shows good discriminative properties for most pairs ( D iod , D ood ) .

Original: Without contradicting the conclusions of Hein et al.
Revised: Without contradicting the conclusions of Hein et al.

Original: (2019), this indicates that - outside an adversarial setting - using the normalized output logits of a classiﬁer is often enough to distinguish OoD samples.
Revised: (2019), this indicates that - outside an adversarial setting - using the normalized output logits of a classiﬁer is often enough to distinguish OoD samples.

Original: On the contrary, the measure C fnrdM is a poor discriminator of OoD datasets.
Revised: On the contrary, the measure C fnrdM is a poor discriminator of OoD datasets.

Original: Results of our PARTICUL-based conﬁdence measure are contrasted and also depend on the choice of architecture ( vanilla or class-based ) w.r.t. to the IoD dataset.
Revised: Results of our PARTICUL-based conﬁdence measure are contrasted and also depend on the choice of architecture ( vanilla or class-based ) with respect to the IoD dataset.

Original: For CUB200 and Stanford Cars, not only does a class-based PARTICUL architecture constitute a good OoD detector (with AUROC scores greater than 90% in 67% of the cases), but vanilla PARTICUL fully exploits the homogeneity of the IoD dataset to train robust detectors, with AUROC scores greater than 90% in all cases, using fewer detectors compared to the class-based approach.
Revised: For CUB200 and Stanford Cars as reference sets, not only does a class-based PARTICUL architecture constitute a good OoD detector (with AUROC scores greater than 90% in 67% of the cases), but vanilla PARTICUL fully exploits the homogeneity of the IoD dataset to train robust detectors, with AUROC scores greater than 90% in all cases, using fewer detectors compared to the class-based approach.

Original: For Caltech101, class-based PARTICUL provides similar (CUB200) or better (Stanford Cars) results than vanilla PARTICUL, but at the cost of a greater number of detectors, and with an AUROC score lower than C MCPM .
Revised: For Caltech101 as the reference set, class-based PARTICUL provides similar (CUB200) or better (Stanford Cars) results than vanilla PARTICUL, but at the cost of a greater number of detectors, and with an AUROC score lower than C MCPM .

Original: This is probably due to the fact that Caltech101 already contains images of birds and cars with similar resolution, making the distinction between distributions harder.
Revised: This is probably due to the fact that Caltech101 already contains images of birds and cars with similar size, making the distinction between distributions harder.

Original: This ﬁrst experiment conﬁrms that our PARTICUL-based conﬁdence measure is, in general, suited to perform
Revised: This ﬁrst experiment conﬁrms that our PARTICUL-based conﬁdence measure is, in general, suited to perform

Original: OoD detection in a cross-dataset setting.
Revised: OoD detection in a cross-dataset setting.

Original: For homogeneous training datasets, vanilla -PARTICUL surpasses the other two unsupervised measures and provides results on par with a weakly supervised approach (FSSD).
Revised: For homogeneous training datasets, vanilla -PARTICUL surpasses the other two unsupervised measures and provides results on par with a weakly supervised approach (FSSD).

Original: For heterogeneous training datasets, our class-based PARTICUL approach improves upon vanilla -PARTICUL by learning more speciﬁc detectors.
Revised: For heterogeneous training datasets, our class-based PARTICUL approach improves upon vanilla -PARTICUL by learning more speciﬁc detectors.

Original: Finally, it is interesting to point out that our results are stable, with a standard deviation of the AUROC/AUPR/FPR80 scores of less than 10% in the vast majority of our experiments, and that the number of detectors (4 or 6) does not have a signiﬁcant impact on the results.
Revised: Finally, it is interesting to point out that our results are stable, with a standard deviation of the AUROC/AUPR/FPR80 scores of less than 10% in the vast majority of our experiments, and that the number of detectors (4 or 6) does not have a signiﬁcant impact on the results.

Original: [Table]
Revised: [Table]

Original: Sensitivity to dataset perturbation In this experiment, we study the behaviour of conﬁdence measures using  various perturbations described in Table 2.
Revised: Sensitivity to dataset perturbation In this experiment, we study the behaviour of conﬁdence measures using the various perturbations described in Table 2.

Original: Due to its instability during the previous experiment and its weakly supervised setting, C fssdM is not considered here.
Revised: Due to its instability during the previous experiment and its weakly supervised setting, C fssdM is not considered here.

Original: As shown in Table [Table] the three conﬁdence measures under test exhibit very different behaviors depending on the nature of the perturbation, but also sometimes depending on the target dataset.
Revised: As shown in Table 3, the three conﬁdence measures under test exhibit very different behaviors depending on the nature of the perturbation, but also sometimes depending on the reference dataset.

Original: Indeed, although C MCPM seems sensitive to blur and brightness perturbations, it is not correlated with gaussian noise, opening the door to adversarial attacks (Szegedy et al.
Revised: Indeed, although C MCPM seems sensitive to blur and brightness perturbations, it is not correlated with gaussian noise, opening the door, if it were employed, to adversarial attacks (Szegedy et al.

Original: Hein et al.
Revised: Hein et al.

Original: Conversely, while C fnrd M is strongly sensitive to noise, any perturbation reducing the amplitude of neuron activation values (blur, brightness) has the opposite effect of increasing this conﬁdence measure.
Revised: Conversely, while C fnrdM is strongly sensitive to noise, any perturbation reducing the amplitude of neuron activation values (blur, brightness) has the opposite effect of increasing this conﬁdence measure.

Original: In general, both C vPM and C cPM correlate with each type perturbation.
Revised: In general, both C vPM and C cPM correlate with each type perturbation.

Original: However, we notice that results on CUB200 (and Stanford Cars, in a lesser extent) for blur and brightness suffer the same issue as C fnrdM (inverse  correlation) when using our baseline classiﬁer (ResNet50).
Revised: However, we notice that results on CUB200 (and Stanford Cars, in a lesser extent) for blur and brightness suffer the same issue as C fnrdM (inverse correlation compared to the correct correlation) when using our baseline classiﬁer (ResNet50).

Original: Finally, rotations seem harder to detect, with softmax and PARTICUL-based measures achieving similar results, while C fnrdM is only weakly correlated to the rotation angle. 
Revised: Finally, rotations seem harder to detect, with softmax and PARTICUL-based measures achieving similar results, while C fnrdM is only weakly correlated to the rotation angle. Interestingly,

Original: Interestingly, when using a VGG19BN (Simonyan & Zisserman (2015)) classiﬁer trained on CUB200, we obtain better results for C MCPM (noise sensitivity) and our PARTICUL-based approach (blur, brightness), while conﬁrming the drawbacks of C fnrdM (blur, brightness).
Revised: [Table] when using a VGG19BN (Simonyan & Zisserman (2015)) classiﬁer trained on CUB200, we obtain better results for C MCPM (noise sensitivity) and our PARTICUL-based approach (blur, brightness), while conﬁrming the drawbacks of C fnrdM (blur, brightness).

Original: This indicates that not only does the robustness of a given measure depends on the type of perturbation and on the IoD dataset, but it  seems toalso rely on the architecture of the classiﬁer.[Table]
Revised: This indicates that not only does the robustness of a given measure depends on the type of perturbation and on the IoD dataset, but it also seems to rely on the architecture of the classiﬁer.

Original: A step towards more explainable conﬁdence measure One of the key advantages of the PARTICULbased conﬁdence measure resides in the possibility of visualizing the detected patterns during the inference of the classiﬁer in order to trace back the contribution of each detector.
Revised: A step towards more explainable conﬁdence measure One of the key advantages of the PARTICULbased conﬁdence measure resides in the possibility for a human to visualize the detected patterns during the inference of the classiﬁer in order to trace back the contribution of each detector.

Original: As in Xu-Darme et al.
Revised: As in Xu-Darme et al.

Original: (2022), we use the Smoothgrads (Smilkov et al.
Revised: (2022), we use the Smoothgrads (Smilkov et al.

Original: algorithm to highlight the most salient area of the image for each detector.
Revised: algorithm to highlight the most salient area of the image for each detector.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 3a, images from D iod can be used to extract a visual reference of the pattern highlighted by each detector.
Revised: 3a, images from D iod can be used to extract a visual reference of the pattern highlighted by each detector.

Original: When processing a new image through the classiﬁer M , patterns can be compared to these visual references to better understand the conﬁdence score.
Revised: When processing a new image through the classiﬁer M , patterns can be compared to these visual references to better understand the conﬁdence score.

Original: For example, some detectors trained on the StanfordCars dataset seem to be able to recognize similar patterns on an image of a motorcycle (row 3), which is to be expected from a visual point of view.
Revised: For example, some detectors trained on the StanfordCars dataset seem to be able to recognize similar patterns on an image of a motorcycle (row 3), which is to be expected from a visual point of view.

Original: Secondly, visualization can also help illustrate the behaviour of each detector w.r.t.
Revised: Secondly, visualization can also help illustrate the behaviour of each detector w.r.t.

Original: a perturbation of the image, as illustrated in Fig.
Revised: a perturbation of the image, as illustrated in Fig.

Original: 3c and Fig.
Revised: 3b and Fig.

Original: 3b.
Revised: 3c.

Original: In particular, it is interesting to notice that contrary to our initial intuition, our PARTICUL-based conﬁdence measure does not steadily decrease for angles ranging from 0° to 180°, but rather displays some invariance to horizontal symmetry (Fig.3d).
Revised: In particular, it is interesting to notice that contrary to our initial intuition, our PARTICUL-based conﬁdence measure does not steadily decrease for angles ranging from 0° to 180°, but rather displays some invariance to horizontal symmetry (Fig.3d).

Original: Finally, as shown in Fig.
Revised: Finally, as shown in Fig.

Original: 3b, although each of our detectors might not be strongly sensitive to a given perturbation, averaging scores across all detectors partially mitigates the variations of individual detectors and creates an ensemble effect that improves the overall stability of the metric, without the cost of training several classiﬁers (Nguyen et al. 
Revised: 3b, although each of our detectors might not be strongly sensitive to a given perturbation, averaging scores across all detectors partially mitigates the variations of individual detectors and creates an ensemble effect that improves the overall stability of the metric, without the cost of training several classiﬁers (Nguyen et al. [Figure]

Original:  C ONCLUSION & FUTURE WORKS
Revised:  C ONCLUSION & FUTURE WORKS

Original: In this paper, we  demonstrated how the detection of recurring patterns, using the PARTICUL algorithm, can help to build conﬁdence measures with the ability to discriminate between IoD and OoD inputs, while allowing a form of visualisation of the detected patterns.
Revised: In this paper, we have demonstrated how the detection of recurring patterns, using the PARTICUL algorithm, can be exploited to develop conﬁdence measures with the ability to discriminate between IoD and OoD inputs, while enabling a form of visualisation of the detected patterns.

Original: We tested our approach in two modalities (crossdataset OoD and perturbation OoD) and showed that it offers mostly consistent results across all experiments.
Revised: We tested our approach in two modalities (cross-dataset OoD and perturbation OoD) and showed that it offers mostly consistent results across all experiments.

Original: However, we also showed that the robustness of all tested metrics can depend on the architecture of the underlying classiﬁer, and we believe that experiments with multiple architectures should be systematically  in future proposals for 
Revised: However, we also showed that the robustness of all tested metrics can depend on the architecture of the underlying classiﬁer and the processed dataset. We believe that experiments with multiple architectures should be systematically conducted in future proposals for measuring Out-of-Distribution.

Original: Out-of- the datasets we used were not strictly exclusive ( images  both birds and cars) a proper  study ( classiﬁer without these classes) should be performed to verify whether the  actually improves
Revised: Moreover, since the datasets we used were not strictly exclusive (Caltech101 contains images of both birds and cars), a proper ablation study (retraining a classiﬁer without these classes) should be performed to verify whether the separability of distributions actually improves.

Original: Reproducibility All the code necessary to reproduce our experimental results is available at the following link: https://anonymous.4open.science/r/iclr23 submission 596/
Revised: Reproducibility All the code necessary to reproduce our experimental results is available at the following link: https://anonymous.4open.science/r/iclr23 submission 596/

Original: [Table]
Revised: [Table]

Original: Table 4 summarizes the parameters used when training our baseline classiﬁers.
Revised: Table 4 summarizes the parameters used when training our baseline classiﬁers.

Original: Note that for CUB200 and
Revised: Note that for CUB200 and

Original: Stanford Cars, we perform data augmentation through a random cropping of size 448 × 448 on the training images (for CIFAR100 and Caltech101, no data augmentation is used) and reduce the learning rate by a factor 10 every 30 epochs.
Revised: Stanford Cars, we perform data augmentation through a random cropping of size 448 × 448 on the training images (for CIFAR100 and Caltech101, no data augmentation is used) and reduce the learning rate by a factor 10 every 30 epochs.

Original: For reproducibility purposes, we performed 3 random initializations of the  detectors for both cross-dataset and dataset perturbation OoD settings.
Revised: For reproducibility purposes, we performed 3 random initializations of the PARTICUL detectors for both crossdataset and dataset perturbation OoD settings.

Original: We are providing averaged scores along with corresponding unbiased standard deviation.
Revised: We are providing averaged scores along with corresponding unbiased standard deviation.

Original: In our tables, average score is on the left of the ± symbol, standard deviation is on the right.
Revised: In our tables, average score is on the left of the ± symbol, standard deviation is on the right.

Original: When training our PARTICUL detectors (either in the vanilla or class-based versions), we use  RMSprop optimizer, with  learning rate  10 − 4 and  decay  10 − 5 , for 30 epochs.
Revised: When training our PARTICUL detectors (either in the vanilla or class-based versions), we use the RMSprop optimizer, with a learning rate of 10 − 4 and a decay of 10 − 5 , for 30 epochs.

Original: We do not use any data augmentation except resizing each image to the appropriate size.
Revised: We do not use any data augmentation except resizing each image to the appropriate size.

Original: In all experiments, we try to maximize the diversity of our detectors by setting the ratio λ of the Unicity constraint to 1 (see Xu-Darme et al.
Revised: In all experiments, we try to maximize the diversity of our detectors by setting the ratio λ of the Unicity constraint to 1 (see Xu-Darme et al.

Original: For the implementation of FSSD, we used the code available on GitHub (https://github.com/megviiresearch/FSSD OoD Detection), with the following modiﬁcations:
Revised: For the implementation of FSSD, we used the code available on GitHub (https://github.com/megviiresearch/FSSD OoD Detection), with the following modiﬁcations:

Original: • Allow the choice of a validation set D ood,val (cid:54)∼ D ood for a fairer comparison betwen methods • Use the entire test sets during the evaluation rather than subsampling D ood to match the size of D iod (would improve AUPR score in some cases).
Revised: • Allow the choice of a validation set D ood,val (cid:54)∼ D ood for a fairer comparison betwen methods • Use the entire test sets during the evaluation rather than subsampling D ood to match the size of D iod (would improve AUPR score in some cases).

Original: Note that during training, we did not perform hyperparameter tuning due to its computational cost.
Revised: Note that during training, we did not perform hyperparameter tuning due to its computational cost.