Original: A F UNCTIONAL P ERSPECTIVE ON M ULTI -L AYER O UT - OF -D ISTRIBUTION D ETECTION
Revised: A F UNCTIONAL P ERSPECTIVE ON M ULTI -L AYER O UT - OF -D ISTRIBUTION D ETECTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A crucial component for implementing reliable classifiers is detecting examples far from the reference (training) distribution, referred to as out-of-distribution (OOD) samples.
Revised: A crucial component for implementing reliable classiﬁers is detecting examples far from the reference (training) distribution, referred to as out-of-distribution (OOD) samples.

Original: A key feature of OOD detection is to exploit the network by extracting statistical patterns and relationships through the pre-trained multi-layer classifier.
Revised: A key feature of OOD detection is to exploit the network by extracting statistical patterns and relationships through the pre-trained multi-layer classiﬁer.

Original: Despite achieving solid results, state-of-the-art methods require either additional OOD examples, expensive computation of gradients, or are tightened to a particular architecture, limiting their applications.
Revised: Despite achieving solid results, state-of-the-art methods require either additional OOD examples, expensive computation of gradients, or are tightened to a particular architecture, limiting their applications.

Original: This work adopts an original approach based on a functional view of the network that exploits the sample’s trajectories through the various layers and their statistical dependencies.
Revised: This work adopts an original approach based on a functional view of the network that exploits the sample’s trajectories through the various layers and their statistical dependencies.

Original: In this new framework, OOD detection translates into detecting samples whose trajectories differ from the typical behavior characterized by the training set.
Revised: In this new framework, OOD detection translates into detecting samples whose trajectories differ from the typical behavior characterized by the training set.

Original: Our method significantly decreases the OOD detection error of classifiers trained on ImageNet and outperforms the state-of-the-art methods on average AUROC and TNR at 95%
Revised: Our method signiﬁcantly decreases the OOD detection error of classiﬁers trained on ImageNet and outperforms the state-of-the-art methods on average AUROC and TNR at 95%

Original: TPR.
Revised: TPR.

Original: We demonstrate that the functional signature left by a sample in a network carries relevant information for OOD detection.
Revised: We demonstrate that the functional signature left by a sample in a network carries relevant information for OOD detection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The current ability of a Deep Neural Network (DNN) to generalize to new data is mostly restricted to priorly known concepts in the training dataset.
Revised: The current ability of a Deep Neural Network (DNN) to generalize to new data is mostly restricted to priorly known concepts in the training dataset.

Original: In real-world scenarios, Machine Learning (ML) models may encounter Out-Of-Distribution (OOD) samples, such as data belonging to novel concepts (classes) (Pimentel et al., 2014), abnormal samples (Tishby & Zaslavsky, 2015), or even carefully crafted attacks designed to exploit the model (Szegedy et al., 2013).
Revised: In real-world scenarios, Machine Learning (ML) models may encounter Out-Of-Distribution (OOD) samples, such as data belonging to novel concepts (classes) (Pimentel et al., 2014), abnormal samples (Tishby & Zaslavsky, 2015), or even carefully crafted attacks designed to exploit the model (Szegedy et al., 2013).

Original: The behavior of ML systems on unseen data is of great concern for safety-critical applications (Amodei et al., 2016b;a), such as medical diagnosis in healthcare (Subbaswamy & Saria, 2020), autonomous vehicle control in transportation (Bojarski et al., 2016) among others.
Revised: The behavior of ML systems on unseen data is of great concern for safety-critical applications (Amodei et al., 2016b;a), such as medical diagnosis in healthcare (Subbaswamy & Saria, 2020), autonomous vehicle control in transportation (Bojarski et al., 2016) among others.

Original: To address safety issues arising from the presence of OOD samples, a successful line of work aims at augmenting ML models with an
Revised: To address safety issues arising from the presence of OOD samples, a successful line of work aims at augmenting ML models with an

Original: OOD binary detector to distinguish between abnormal and in-distribution examples (Hendrycks &
Revised: OOD binary detector to distinguish between abnormal and in-distribution examples (Hendrycks &

Original: Gimpel, 2017).
Revised: Gimpel, 2017).

Original: An analogy to the detector is the human body’s immune system, with the task of differentiating between antigens and the body itself.
Revised: An analogy to the detector is the human body’s immune system, with the task of differentiating between antigens and the body itself.

Original: Distinguishing OOD samples is challenging, and previous works developed detectors by combining scores at the various layers of the multi-layer pre-trained classifier (Sastry & Oore, 2020; Lee et al., 2018;
Revised: Distinguishing OOD samples is challenging, and previous works developed detectors by combining scores at the various layers of the multi-layer pre-trained classiﬁer (Sastry & Oore, 2020; Lee et al., 2018;

Original: Gomes et al., 2022; Huang et al., 2021).
Revised: Gomes et al., 2022; Huang et al., 2021).

Original: These detectors require either a held-out OOD dataset (e.g., adversarially generated or OOD data) or ad-hoc methods to linearly combine OOD scores computed on each layer embedding tightened to a particular architecture.
Revised: These detectors require either a held-out OOD dataset (e.g., adversarially generated or OOD data) or ad-hoc methods to linearly combine OOD scores computed on each layer embedding tightened to a particular architecture.

Original: A key observation is that existing aggregation techniques overlook the sequential nature of the underlying problem and, thus, limit the discriminative power of those methods.
Revised: A key observation is that existing aggregation techniques overlook the sequential nature of the underlying problem and, thus, limit the discriminative power of those methods.

Original: Indeed, an input sample passes consecutively through each layer and generates a highly correlated signature that can be statistically characterized.
Revised: Indeed, an input sample passes consecutively through each layer and generates a highly correlated signature that can be statistically characterized.

Original: Our observations in this work motivate the fact that:
Revised: Our observations in this work motivate the fact that:

Original: The input’s trajectory through a network is key for discriminating typical from atypical samples.
Revised: The input’s trajectory through a network is key for discriminating typical from atypical samples

Original: In this paper, we introduce a significant change of perspective.
Revised: In this paper, we introduce a signiﬁcant change of perspective.

Original: Instead of looking at each layer score independently, we cast them into a sequential representation that captures the statistical trajectory of an input sample through the various layers of a multi-layer neural network.
Revised: Instead of looking at each layer score independently, we cast them into a sequential representation that captures the statistical trajectory of an input sample through the various layers of a multi-layer neural network.

Original: To this end, we adopt a functional point of view by considering the sequential representation as curves parametrized by l
Revised: To this end, we adopt a functional point of view by considering the sequential representation as curves parametrized by l

Original: OOD
Revised: OOD

Original: I ND
Revised: I ND

Original: Trajectory [Figure]
Revised: Trajectory Extraction

Original: OOD Score Computation
Revised: OOD Score Computation

Original: L a y e r-
Revised: L a y e r-

Original: w i  S c
Revised: w i s e S c

Original: o r e s
Revised: o r e s

Original: Projection each layer.
Revised: Projection each layer.

Original: Consequently, we redefine the problem of OOD detection as that of detecting samples whose trajectories are abnormal (or atypical) compared to reference trajectories characterized by the training set.
Revised: Consequently, we redeﬁne the problem of OOD detection as that of detecting samples whose trajectories are abnormal (or atypical) compared to reference trajectories characterized by the training set.

Original: Our method, which requires little parameter tuning and, perhaps more importantly, no additional OOD data, can identify OOD samples from their trajectories.
Revised: Our method, which requires little parameter tuning and, perhaps more importantly, no additional OOD data, can identify OOD samples from their trajectories.

Original: Furthermore, we show that typical multivariate detection methods fail to detect OOD patterns, which may manifest in an isolated fashion by shifting in magnitude or overall shape.
Revised: Furthermore, we show that typical multivariate detection methods fail to detect OOD patterns, which may manifest in an isolated fashion by shifting in magnitude or overall shape.

Original: Figure 1 summarizes our method.
Revised: Figure 1 summarizes our method.

Original: Contributions.
Revised: Contributions.

Original: This work presents a new principle and unsupervised method for detecting OOD samples that do not require OOD (or extra) data and brings novel insights into the problem of OOD detection.
Revised: This work presents a new principle and unsupervised method for detecting OOD samples that do not require OOD (or extra) data and brings novel insights into the problem of OOD detection.

Original: Our main contributions can be summarized as follows.
Revised: Our main contributions can be summarized as follows.

Original: A novel problem formulation .
Revised: A novel problem formulation .

Original: We reformulate the problem of OOD detection through a functional perspective that effectively captures the statistical dependencies of an input sample’s path across a multi-layer neural classifier.
Revised: We reformulate the problem of OOD detection through a functional perspective that effectively captures the statistical dependencies of an input sample’s path across a multi-layer neural classiﬁer.

Original: Moreover, we propose a map from the multivariate feature space (at each layer) to a functional space that relies on the Cosine similarity between the test sample and a training reference at the layer and the sample’s ℓnorm.
Revised: Moreover, we propose a map from the multivariate feature space (at each layer) to a functional space that relies on the Cosine similarity between the test sample and a training reference at the layer and the sample’s `norm.

Original: It is computationally efficient and straightforward.
Revised: It is computationally efﬁcient and straightforward.

Original: Computing OOD scores from trajectories .
Revised: Computing OOD scores from trajectories .

Original: We extract the training trajectories to have an average reference trajectory on which we project test trajectories to measure their similarity w.r.t the training centroid.
Revised: We extract the training trajectories to have an average reference trajectory on which we project test trajectories to measure their similarity w.r.t the training centroid.

Original: Low similarity indicates that the test sample is likely sampled from OOD.
Revised: Low similarity indicates that the test sample is likely sampled from OOD.

Original: Empirical evaluation .
Revised: Empirical evaluation .

Original: We validate the value of the proposed method using a mid-size OOD detection benchmark on ImageNet-1k.
Revised: We validate the value of the proposed method using a mid-size OOD detection benchmark on ImageNet-1k.

Original: We obtain competitive results, demonstrating an average ROC gain of up to 3% across three architectures and four OOD datasets.
Revised: We obtain competitive results, demonstrating an average ROC gain of up to 3% across three architectures and four OOD datasets.

Original: We release our code anonymized online.
Revised: We release our code anonymized online.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: This section briefly discusses prior work in OOD detection, highlighting confidence-based and feature-based methods without special training as they resonate the most with our work.
Revised: This section brieﬂy discusses prior work in OOD detection, highlighting conﬁdence-based and feature-based methods without special training as they resonate the most with our work.

Original: Another thread of research relies on re-training for learning more appropriate representations adapted to OOD detection (Mohseni et al., 2020; Bitterwolf et al.
Revised: Another thread of research relies on re-training for learning more appropriate representations adapted to OOD detection (Mohseni et al., 2020; Bitterwolf et al.

Original: ; Mahmood et al., 2021), either through contrastive  (Hendrycks et al., 2019; Winkens et al., 2020), regularization (Lee et al., 2021; Nandy et al., 2021; Hein et al., 2019), generative (Schlegl et al., 2017; ood, 2019; Xiao et al.;
Revised: ; Mahmood et al., 2021), either through contrastive training (Hendrycks et al., 2019; Winkens et al., 2020; Sehwag et al., 2021), regularization (Lee et al., 2021; Nandy et al., 2021; Hein et al., 2019; Du et al., 2022), generative (Schlegl et al., 2017; ood, 2019; Xiao et al.;

Original: Ren et al.; Zhang et al., 2021) or ensemble (Vyas et al., 2018; Choi & Jang,2018)
Revised: Ren et al.; Zhang et al., 2021), or ensemble (Vyas et al., 2018; Choi & Jang,

Original:  based approaches.
Revised: 2018) based approaches.

Original: Related subfields are open set recognition (Geng et al., 2021), novelty detection (Pimentel et al., 2014), anomaly detection (Chandola et al., 2009; Chalapathy & Chawla, 2019), outlier detection (Hodge & Austin, 2004), and adversarial attacks detection (Akhtar & Mian, 2018).
Revised: Related subﬁelds are open set recognition (Geng et al., 2021), novelty detection (Pimentel et al., 2014), anomaly detection (Chandola et al., 2009; Chalapathy & Chawla, 2019), outlier detection (Hodge & Austin, 2004), and adversarial attacks detection (Akhtar & Mian, 2018).

Original: Confidence-based OOD detection.
Revised: Conﬁdence-based OOD detection.

Original: A natural measure of uncertainty of a sample’s label is the classification model’s softmax output.
Revised: A natural measure of uncertainty of a sample’s label is the classiﬁcation model’s softmax output.

Original: Hendrycks & Gimpel (2017) observed that the maximum of the softmax output could be used as a discriminative score between in-distribution and OOD samples.
Revised: Hendrycks & Gimpel (2017) observed that the maximum of the softmax output could be used as a discriminative score between in-distribution and OOD samples.

Original: Even though this metric is effective in some cases, Hein et al.
Revised: Hein et al.

Original: (2019) observed that it may still assign overconfident values to OOD examples.
Revised: (2019) observed that it may still assign overconﬁdent values to OOD examples.

Original: Liang et al.
Revised: Liang et al.

Original: (2018) and Hsu et al.
Revised: (2018) and Hsu et al.

Original: (2020)are incremental steps towards ameliorating this method. They propose re-scaling the softmax response with a temperature value and a pre-processing technique that further separates in- from out-of-distribution examples.
Revised: (2020) propose re-scaling the softmax response with a temperature value and a pre-processing technique that further separates in- from out-of-distribution examples.

Original: Authors of Liu et al.
Revised: Authors of Liu et al.

Original: (2020) propose an energy-based OOD detection score by substituting the softmax confidence score with the free energy function.
Revised: (2020) propose an energy-based OOD detection score by replacing the softmax conﬁdence score with the free energy function.

Revised: Sun & Li (2022) proposes sparsiﬁcation of the classiﬁcation layer weights to improve OOD detection by regularizing predictions.

Original: Feature-based OOD detection.
Revised: Feature-based OOD detection.

Original: This line of research focuses on exploring latent representations for OOD detection.
Revised: This line of research focuses on exploring latent representations for OOD detection.

Original: For instance, Haroush et al.
Revised: For instance, Haroush et al.

Original: (2021) considers using statistical tests, Sastry &
Revised: (2021) considers using statistical tests, Sastry &

Original: Oore (2020) rely on higher-order Grams matrices, Quintanilha et al.
Revised: Oore (2020) rely on higher-order Grams matrices, Quintanilha et al.

Original: (2019) uses mean and standard deviation within feature maps, Sun et al.
Revised: (2019) uses mean and standard deviation within feature maps, Sun et al.

Original: (2021) proposes clipping the activations to boost OOD detection performance, while recent work Huang et al.
Revised: (2021) proposes clipping the activations to boost OOD detection performance, while recent work Huang et al.

Original: (2021) also explores the gradient space.
Revised: (2021) also explores the gradient space.

Original: Normalizing and residual flows to estimate the probability distribution of the feature space were proposed in Kirichenko et al.
Revised: Normalizing and residual ﬂows to estimate the probability distribution of the feature space were proposed in Kirichenko et al.

Original: and Zisselman & Tamar (2020).
Revised: and Zisselman & Tamar (2020).

Original: Perhaps one of the most widely used techniques relies on the Gaussian mixture assumption for the hidden representations and the
Revised: Perhaps one of the most widely used techniques relies on the Gaussian mixture assumption for the hidden representations and the

Original: Mahalanobis distance (Lee et al., 2018;
Revised: Mahalanobis distance (Lee et al., 2018;

Original: Ren et al., 2021) or further information geometry tools (Gomes et al., 2022).
Revised: Ren et al., 2021) or further information geometry tools (Gomes et al., 2022).

Original: The reasons are its simplicity, performance, and geometrical explanation.
Revised: The reasons are its simplicity, performance, and geometrical explanation.

Original: Efforts toward combining multiple features to improve performance were previously explored (Lee et al., 2018; Sastry & Oore, 2020; Gomes et al., 2022).
Revised: Efforts toward combining multiple features to improve performance were previously explored (Lee et al., 2018; Sastry & Oore, 2020; Gomes et al., 2022).

Original: The strategy relies heavily upon having additional data for tuning the detector or focusing on specific model architectures, which are limiting factors in real-world applications.
Revised: The strategy relies heavily upon having additional data for tuning the detector or focusing on speciﬁc model architectures, which are limiting factors in real-world applications.

Revised: Lin et al.

Revised: (2021) rely on networks with multiple classiﬁers and introduces a dynamic OOD detection method which selects the index of the layer in test time to compute the score from.

Revised: Sun et al.

Revised: (2022) proposes non-parametric nearest-neighbor based on the

Revised: Euclidean distance for OOD detection.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We start by recalling the general setting of the OOD detection problem from a mathematical point of view.
Revised: We start by recalling the general setting of the OOD detection problem from a mathematical point of view.

Original: Then, we motivate our method through a simple illustrative example.
Revised: Then, we motivate our method through a simple illustrative example.

Original: 3. B ACKGROUND
Revised: 3. B ACKGROUND

Original: Let ( X, Y ) be a random variable valued in a space X ×Y with unknown probability density function (pdf) p XY and probability distribution P XY .
Revised: Let ( X, Y ) be a random variable valued in a space X ⇥ Y with unknown probability density function (pdf) p XY and probability distribution P XY .

Original: Here, X ⊆
Revised: Here, X ✓

Original: R d represents the covariate space and
Revised: R d represents the covariate space and

Original: Y = { 1 , . . .
Revised: Y = { 1 , . . .

Original: , C } corresponds to the labels attached to elements from X .
Revised: , C } corresponds to the labels attached to elements from X .

Original: The training dataset S N = { ( x i , y i ) }
Revised: The training dataset S N = { ( x i , y i ) }

Original: Ni =1 is defined as independent and identically distributed (i.i.d) realizations of P XY .
Revised: Ni =1 is deﬁned as independent and identically distributed (i.i.d) realizations of P XY .

Original: From this formulation, detecting OOD samples boils down to building a binary rule g : X → { 0 , 1 } through a soft scoring function s : X → R and a threshold γ ∈ R .
Revised: From this formulation, detecting OOD samples boils down to building a binary rule g : X ! { 0 , 1 } through a soft scoring function s : X ! R and a threshold  2 R .

Original: Namely, a new observation x ∈ X is then considered as in-distribution , i.e., generated by P XY , when g ( x ) = [-
-] and as OOD when g ( x ) = 1 .
Revised: Namely, a new observation x 2 X is then considered as in-distribution , i.e., generated by P XY , when g ( x ) = [-
-]0 and as OOD when g ( x ) = 1 .

Original: Finding this rule g from X can become intractable when the dimension d is large.
Revised: Finding this rule g from X can become intractable when the dimension d is large.

Original: Thus, previous work rely on a multi-layer pre-trained classifier f θ : X → Y defined as:
Revised: Thus, previous work rely on a multi-layer pre-trained classiﬁer f ✓ : X ! Y deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: with L ≥ 1 layers, where f ℓ : R d ℓ − 1 → R d ℓ is the ℓ -th layer of the multi-layer neural classifier, d ℓ denotes the dimension of the latent space induced by the ℓ -th layer ( d 0 = d ), and h indicates the classifier that outputs the logits.
Revised: with L  1 layers, where f ` : R d `  1 ! R d ` is the ` -th layer of the multi-layer neural classiﬁer, d ` denotes the dimension of the latent space induced by the ` -th layer ( d 0 = d ), and h indicates the classiﬁer that outputs the logits.

Original: We also define z ℓ = ( f ℓ ◦ · · · ◦ f 1 )( x ) as the latent vectorial representation at the ℓ − th layer for an input sample x .
Revised: We also deﬁne z ` = ( f `  · · ·  f 1 )( x ) as the latent vectorial representation at the `  th layer for an input sample x .

Original: We will refer to the logits as z L +1 and h as f L +1 to homogenize notation.
Revised: We will refer to the logits as z L +1 and h as f L +1 to homogenize notation.

Original: It is worth emphasizing that the trajectory of ( z 1 , z 2 , . . .
Revised: It is worth emphasizing that the trajectory of ( z 1 , z 2 , . . .

Original: , z L +1 )
Revised: , z L +1 )

Original: corresponding to a test input x 0 are dependent random variables whose joint distribution strongly depends on the underlying distribution of the input.
Revised: corresponding to a test input x 0 are dependent random variables whose joint distribution strongly depends on the underlying distribution of the input.

Original: Therefore, the constructed function g ( · ) is typically based on the three key steps: 
Revised: Therefore, the design of function g ( · ) is typically based on the three key steps: [Figure] [Figure] [Figure]

Original: (i)
Revised: [Figure] well OOD distribution OOD have disparate trajectories through a network (Fig. 2b) a clear separation the features.

Revised: Also, these features are correlated in a sequential fashion, as observed in Fig.

Revised: 2c

Original:  A similarity measure d( · ; · ) (e.g., Cosine similarity, Mahalanobis distance, etc.) between a sample and a population is applied at each layer to measure the similarity (or dissimilarity) of a test input x 0 at the ℓ -th layer z ℓ, 0 =( f ℓ ◦ · · · ◦ f 1 )( x 0 ) w.r
Revised: (i) A similarity measure d( · ; · ) (e g., Cosine similarity, Mahalanobis distance, etc.) between a sample and a population is applied at each layer to measure the similarity (or dissimilarity) of a test input x 0 at the ` -th layer z ` , 0 =

Original:  t. the population of the training examples observed at the same layer (cid:8) z ℓ = ( f ℓ ◦ · · · ◦ f 1 )( x ) : x ∈ S N (cid:9) .
Revised: ( f `  · · ·  f 1 )( x 0 ) w r t. the population of the training examples observed at the same layer [Equation]

Original: (ii)
Revised: (ii)

Original: The layer-wise score obtained is then mapped to the real line collecting the OOD scores.Feature Score F e a t u r e S c o r e
Revised: The layer-wise score obtained is then mapped to the real line collecting the OOD scores.

Original: Unreliable

Original: Held-out IND Held-out OOD data Linear D.B.

Original: 1 2 3 4 5 6 7 8 9 1011121314 Features 0. S c o r e Train set In test set iNaturalist
Revised: (iii) Lastly, a threshold is chosen to build the ﬁnal decision function.

Original: SUN Places

Original: Textures A fundamental ingredient remains in step (ii):
Revised: A fundamental ingredient remains in step (ii):

Original: How to consistently leverage the information collected from multiple layers  in an unsupervised way, i.e., without resorting to OOD or pseudo-OOD examples?
Revised: How to consistently leverage the information collected from multiple layers outputs in an unsupervised way, i.e., without resorting to OOD or pseudo-OOD examples

Original:  3. F ROM I NDEPENDENT M ULTI -L AYER S CORES TO A S EQUENTIAL P ERSPECTIVE OF OOD D ETECTION
Revised: ? 3. F ROM I NDEPENDENT M ULTI -L AYER S CORES TO A S EQUENTIAL P ERSPECTIVE OF OOD D ETECTION

Original: State-of-the-art methods treat step (ii) as a supervised learning problem  for which the solution is a linear binary classifier.
Revised: Previous multi-feature OOD detection works treat step (ii) as a supervised learning problem (Lee et al., 2018; Gomes et al., 2022) for which the solution is a linear binary classiﬁer.

Original: The objective is to find a linear combination of the scores obtained at each layer that will sufficiently separate in-distribution from OOD samples.
Revised: The objective is to ﬁnd a linear combination of the scores obtained at each layer that will sufﬁciently separate in-distribution from OOD samples.

Original: A held-out OOD dataset is collected from true (or pseudo-generated)
Revised: A held-out OOD dataset is collected from true (or pseudogenerated)

Original: OOD samples.
Revised: OOD samples.

Original: The linear soft novelty score function s α writes:
Revised: The linear soft novelty score function s ↵ writes:

Original: [Equation]
Revised: [Equation]

Original: The shortcomings of this method are the need for extra data or ad-hoc parameters, which results in decision boundaries that underfit the problem and fail to capture certain types of OOD samples.
Revised: The shortcomings of this method are the need for extra data or ad-hoc parameters, which results in decision boundaries that underﬁt the problem and fail to capture certain types of OOD samples.

Original: To illustrate this phenomenon, we designed a toy example (see Figure 2a) where scores are extracted from two features fitting a linear discriminator on held-out in-distribution (IND) and OOD samples.
Revised: To illustrate this phenomenon, we designed a toy example (see Figure 2a) where scores are extracted from two features ﬁtting a linear discriminator on held-out in-distribution (IND) and OOD samples.

Original: We show an area of unreliable predictions where OOD samples cannot be detected due to the misspecification of the linear model.
Revised: We show an area of unreliable predictions where OOD samples cannot be detected due to the misspeciﬁcation of the linear model.

Original: Of course, one could introduce a non-linear discriminator that better captures the geometry of the data for this 2D toy example.
Revised: Of course, one could introduce a non-linear discriminator that better captures the geometry of the data for this 2D toy example.

Original: However, it becomes challenging as we move to higher dimensions with limited data.
Revised: However, it becomes challenging as we move to higher dimensions with limited data.

Original: The sequential nature of feature extraction matters.
Revised: The sequential nature of feature extraction matters.

Original: By reformulating the problem from a functional point of view, we can identify trends and typical trajectories presented in the data.
Revised: By reformulating the problem from a functional point of view, we can identify trends and typical trajectories presented in the data.

Original: Figureshows the dispersion of trajectories at step (ii) coming from the in-distribution and OOD samples.
Revised: Figureshows the dispersion of trajectories at step (ii) coming from the in-distribution and OOD samples.

Original: These patterns are extracted from multiple latent representations and aligned on a time-series-like object, clearly distinguishing training samples.
Revised: These patterns are extracted from multiple latent representations and aligned on a time-series-like object, clearly distinguishing training samples.

Original: We observed that trajectories coming from OOD samples exhibit a different shape when compared to typical trajectories from training data .
Revised: We observed that trajectories coming from OOD samples exhibit a different shape when compared to typical trajectories from training data .

Original: Thus, to determine if an instance belongs to a reference distribution, we can test if the observed path is close to the functional data extracted from the training set.
Revised: Thus, to determine if an instance belongs to a reference distribution, we can test if the observed path is close to the functional data extracted from the training set.

Original: We will do so class-wise as we plan to retain the most structural information about the downstream task while showcasing the relevant shape differences.
Revised: We will do so class-wise as we plan to retain the most structural information about the downstream task while showcasing the relevant shape differences.

Original:  F UNCTIONAL O UT -O F -D ISTRIBUTION D ETECTION
Revised:  F UNCTIONAL O UT -O F -D ISTRIBUTION D ETECTION

Original: This section presents our OOD detection framework, which applies to any pre-trained multi-layer neural network with no requirements for OOD samples.
Revised: This section presents our OOD detection framework, which applies to any pre-trained multi-layer neural network with no requirements for OOD samples.

Original: We describe our method through two key steps: functional representation of the input sample (see Section 4.1) and test time OOD score computation (see Section 4.2).
Revised: We describe our method through two key steps: functional representation of the input sample (see Section 4.1) and test time OOD score computation (see Section 4.2).

Original: 4. T OWARDS A F UNCTIONAL R EPRESENTATION
Revised: 4. T OWARDS A F UNCTIONAL R EPRESENTATION

Original: The first step to obtaining a functional representation of the data from the multivariate hidden representations is to reduce each one to a scalar or vector.
Revised: The ﬁrst step to obtaining a functional representation of the data from the multivariate hidden representations is to reduce each one to a scalar or vector.

Original: To do so in a meaningful way for the task in hand, we first compute the training population references defined as the layer-wise class-conditional average vectors:
Revised: To do so, we ﬁrst compute the training population references deﬁned as the layer-wise class-conditional average vectors:

Original: [Equation]
Revised: [Equation]

Original: where N y = 
Revised: where N y =

Original: (cid:12)(cid:12) { z ℓ,i | y i = y, ∀ i ∈ { 1 ..N }} (cid:12)(cid:12) , 1 ≤ ℓ ≤ L + 1 and z ℓ,i =
Revised: [Equation]

Original: ( f ℓ ◦ · · · ◦ f 1 )( x i ) . Given an arbitrary input example, we compute its features  and their similarity w.r.t.
Revised: Given an arbitrary input example, we compute its features (including the logits) and their similarity w.r.t.

Original: therespective training average for the predicted classˆ y (including the logits), leading to L + 1 scores.
Revised: the training average for each class y weighted averaged by the prediction probability vector f ✓ ( · ) , leading to L + 1 scores.

Original: We rely on the  scalar projection 1  the sample features’ z ℓ onto the direction of the average training feature for the predicted class as similarity score:
Revised: We rely on the probability weighted scalar projection 1 of the sample features’ z ` onto the direction of the average training features as similarity score:

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ is the ℓ 2 -norm and ∠ ( · , · ) is the angle between two vectors.
Revised: where k · k is the ` 2 -norm, \ ( · , · ) is the angle between two vectors, and  y ( x ; f ✓ ) is the softmax function on the logits f ✓ ( x ) .

Original: Hence, our layer-wise scores rely on the notions of vector length and angle between vectors, which can be generalized to any n -dimensional inner product space without imposing any geometrical constraints.
Revised: Hence, our layer-wise scores rely on the notions of vector length and angle between vectors, which can be generalized to any n -dimensional inner product space without imposing any geometrical constraints.

Original: It is worth emphasizing that our layer score has some advantages compared to the class conditional
Revised: It is worth emphasizing that our layer score has some advantages compared to the class conditional

Original: Gaussian model first introduced in Lee et al.
Revised: Gaussian model ﬁrst introduced in Lee et al.

Original: and the Gram matrix based-method introduced in Sastry & Oore (2020).
Revised: and the Gram matrix based-method introduced in Sastry & Oore (2020).

Original: Our layer score encompasses a broader class of distributions as we do not suppose an specific underlying probability distribution.
Revised: Our layer score encompasses a broader class of distributions as we do not suppose an speciﬁc underlying probability distribution.

Original: We avoid computing covariance matrices, which are often ill-conditioned for latent representations of DNNs.
Revised: We avoid computing covariance matrices, which are often ill-conditioned for latent representations of DNNs.

Original: Since we do not store covariance matrices, our functional approach has a negligible overhead regarding memory requirements.
Revised: Since we do not store covariance matrices, our functional approach has a negligible overhead regarding memory requirements.

Original: Also, our method can be applied to any vector-based hidden representation, not being restricted to matrixbased representations as in Sastry & Oore (2020).
Revised: Also, our method can be applied to any vector-based hidden representation, not being restricted to matrixbased representations as in Sastry & Oore (2020).

Original: Thus, our approach applies to a broader range of models, including transformers.
Revised: Thus, our approach applies to a broader range of models, including transformers.

Original: By computing the scalar projection at each layer, we obtain the following functional neuralrepresentation extraction function:
Revised: By computing the scalar projection at each layer, we obtain the following functional neuralrepresentation extraction function:

Original: [Equation]
Revised: [Equation]

Original: We apply ϕ to the training input x i to obtain the representation of the training sample across the network u i =ϕ
Revised: We apply  to the training input x i to obtain the representation of the training sample across the network u i =

Original:  ( x i ) .
Revised:  ( x i ) .

Original: We consider the related vectors u i , ∀ i ∈ [1 : N ] as curves parameterized by the layers of the network.
Revised: We consider the related vectors u i , 8 i 2 [1 : N ] as curves parameterized by the layers of the network.

Original: Thus, we can map samples from the representation to a functional space while retaining information on the typicality of the sample w.r.t the training dataset.
Revised: Thus, we can map samples from the representation to a functional space while retaining information on the typicality of the sample w.r.t the training dataset.

Original: We build a training dataset  from these functional representations that will be useful for detecting OOD samples during test time: [Equation]
Revised: We build a training dataset U = { u i } Ni =1 from these functional representations that will be useful for detecting OOD samples during test time.

Original: We then rescale the training set trajectories w.r.t the maximum value found at each coordinate to obtain layer-wise scoresmore or less on the same scaling for each coordinate.
Revised: We then rescale the training set trajectories w.r.t the maximum value found at each coordinate to obtain layer-wise scores on the same scaling for each coordinate.

Original: Hence, for j ∈ { 1 , . . .
Revised: Hence, for j 2 { 1 , . . .

Original: , L + 1 } , let max ( U ) :=
Revised: , L +1 } , let max ( U ) : =

Original: [max i u
Revised: [max i u

Original: i, 1 , . .
Revised: i, 1 , . .

Original: , max i u
Revised: , max i u

Original: i,L +1 ] ⊤ , weapply the following re-scaling on the functional training set: [Equation] [Equation] From the re-scaled set, we can compute a reference trajectory ¯ u for the entire training dataset defined in equation 6 that will serve as a global typical reference to compare test trajectories with.
Revised: i,L +1 ] > , we can compute a reference trajectory ¯ u for the entire training dataset deﬁned in equation 4 that will serve as a global typical reference to compare test trajectories with.

Original: [Equation]
Revised: [Equation]

Original: 4. C OMPUTING THE OOD S CORE AT T EST T IME
Revised: 4. C OMPUTING THE OOD S CORE AT T EST T IME

Original: At inference time, we first re-scale the test sample’s trajectory as we did with the training reference:
Revised: At inference time, we ﬁrst re-scale the test sample’s trajectory as we did with the training reference ˜

Original: [Equation]
Revised:  ( x ) =

Revised:  ( x ) / max ( U ) .

Original: Then, we compute a similarity score w.r.t this typical reference that will be our OOD score.
Revised: Then, we compute a similarity score w.r.t this typical reference that will be our OOD score.

Original: We choose as metric also the scalar projection of the test vector to the training reference.
Revised: We choose as metric also the scalar projection of the test vector to the training reference.

Original: In practical terms, it boils down to the inner product between the test sample’s trajectory and the training set’s typical reference trajectory since the norm of the average trajectory is constant for all test samples.
Revised: In practical terms, it boils down to the inner product between the test sample’s trajectory and the training set’s typical reference trajectory since the norm of the average trajectory is constant for all test samples.

Original: Mathematically, our scoring function s : X (cid:55)→ R writes:
Revised: Mathematically, our scoring function s : X 7! R writes:

Original: [Equation]
Revised: [Equation]

Original: which is bounded by Cauchy-Schwartz’s inequality.
Revised: which is bounded by Cauchy-Schwartz’s inequality.

Original: From this OOD score, we can derive a binary classifier g by fixing a threshold γ ∈ R :
Revised: From this OOD score, we can derive a binary classiﬁer g by ﬁxing a threshold  2 R :

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) = 1 means that the input sample x is classified as being out-of-distribution.
Revised: where g ( x ) = 1 means that the input sample x is classiﬁed as being out-of-distribution.

Original: Please refer to Appendix (see Section A.1) for further details on the algorithm.
Revised: Please refer to Appendix (see Section A.1) for further details on the algorithm.

Original:  E XPERIMENTAL S ETTING
Revised:  E XPERIMENTAL S ETTING

Original: This section describes the experimental setting, including the datasets used, the pre-trained DNN architectures, the evaluation metrics, and other essential information.
Revised: This section describes the experimental setting, including the datasets used, the pre-trained DNN architectures, the evaluation metrics, and other essential information.

Original: We open-source our code anonymized in https://github.com/ood-trajectory/ood-trajectory .
Revised: We open-source our code anonymized in https://github.com/ood-trajectory/ood-trajectory .

Original: 5. D ATASETS
Revised: 5. D ATASETS

Original: We set as in-distribution dataset
Revised: We set as in-distribution dataset

Original: ImageNet-1k (=
Revised: ImageNet-1k (=

Original: ILSVRC2012;
Revised: ILSVRC2012;

Original: Deng et al., 2009) for all of our experiments, which is a challenging mid-size and realistic dataset that have been incorporated recently to OOD detection (Huang & Li, 2021).
Revised: Deng et al., 2009) for all of our experiments, which is a challenging mid-size and realistic dataset that have been incorporated recently to OOD detection (Huang & Li, 2021).

Original: It contains around 1.28M training samples and 50,000 test samples belonging to 1000 different classes.
Revised: It contains around 1.28M training samples and 50,000 test samples belonging to 1000 different classes.

Original: For the out-of-distribution datasets, we take the same dataset splits introduced by Huang & Li (2021).
Revised: For the out-of-distribution datasets, we take the same dataset splits introduced by Huang & Li (2021).

Original: The iNaturalist (Horn et al., 2017) dataset contains over 5,000 species of plant and animals.
Revised: The iNaturalist (Horn et al., 2017) dataset contains over 5,000 species of plant and animals.

Original: We consider a split with 10,000 test samples with concepts from 110 classes different from the in-distribution ones.
Revised: We consider a split with 10,000 test samples with concepts from 110 classes different from the in-distribution ones.

Original: The Sun (Xiao et al., 2010) dataset is a scene dataset of 397 categories.
Revised: The Sun (Xiao et al., 2010) dataset is a scene dataset of 397 categories.

Original: We considered a split with 10,000 randomly sampled test examples belonging to 50 categories.
Revised: We considered a split with 10,000 randomly sampled test examples belonging to 50 categories.

Original: The Places365 (Zhou et al., 2017) is also a scenes dataset withdifferent concepts.
Revised: The Places365 (Zhou et al., 2017) is also a scenes dataset withdifferent concepts.

Original: We also considered a random split with 10,000 samples from 50 disjoint categories.
Revised: We also considered a random split with 10,000 samples from 50 disjoint categories.

Original: For the DTD or Textures (Cimpoi et al., 2014) dataset is composed of textural patterns.
Revised: For the DTD or Textures (Cimpoi et al., 2014) dataset is composed of textural patterns.

Original: We considered all of the 5,640 available test samples.
Revised: We considered all of the 5,640 available test samples.

Original: Note that there is a couple of overlaps between the semantics of classes of this dataset and ImageNet.
Revised: Note that there is a couple of overlaps between the semantics of classes of this dataset and ImageNet.

Original: We decided to keep the entire dataset in order to be comparable with Huang et al.
Revised: We decided to keep the entire dataset in order to be comparable with Huang et al.

Original: We provide a study case on this in Section 6.
Revised: We provide a study case on this in Section 6.

Original: 5. M ODELS
Revised: M ODELS

Original: We ran experiments with three models of different architectures.
Revised: We ran experiments with three models of different architectures.

Original: A DenseNet-121 (Huang et al., 2017) pre-trained on ILSVRC-2012 with 8M parameters and test set top-1 accuracy of 74.43%.We 6 Features 0. S c
Revised: A DenseNet-121 (Huang et al., 2017) pre-trained on ILSVRC-2012 with 8M parameters and test set top-1 accuracy of 74.43%.

Original: o r e Train set In test set Textures (a) Functional trajectories. 2.5 Score 0.
Revised: We reduced the intermediate representations from the transition blocks with an adaptive max pooling

Original: D e n s i t y

Original: In Score Textures (b) OOD score histogram.

Original: 1.0 False Positive Rate 0.

Original: T r u e P o s i t i v e R a t e MSP, auc=0.79 Penult.
Revised: [Figure]

Original: Proj., auc=0. Ours, auc=0. (c) ROC curve. reduced the intermediate representations from the transition blocks with an adaptive max pooling operation obtaining a final vector with a dimension equal to the number of channels of each output.
Revised: operation obtaining a ﬁnal vector with a dimension equal to the number of channels of each output.

Original: The dimensions are 128, 256, 512, and 1024, respectively.
Revised: The dimensions are 128, 256, 512, and 1024, respectively.

Original: We resize input images to 224 ×pixels.
Revised: We resize input images to 224 ⇥pixels.

Original: A BiT-S-101 (Kolesnikov et al., 2020) model based on a ResNet-101 (He et al., 2016) architecture with top-1 test set accuracy of 77.41% and 44.5M parameters.
Revised: A BiT-S-101 (Kolesnikov et al., 2020) model based on a ResNetv2-101 architecture with top-1 test set accuracy of 77.41% and 44.5M parameters.

Original: We extract features from the outputs of layers 1 to 4 and the penultimate layer, obtaining representations with sizes 256, 512, 1024, and 2048, respectively after max pooling.
Revised: We extract features from the outputs of layers 1 to 4 and the penultimate layer, obtaining representations with sizes 256, 512, 1024, and 2048, respectively after max pooling.

Original: We resize input images to 480 × 480.
Revised: We resize input images to 480 ⇥ 480.

Original: We also ran experiments with a Vision Transformer ( ViT-B-16 ; Dosovitskiy et al., 2021), which is trained on the ILSVRC2012 dataset with 82.64% top-1 test accuracy and 70M parameters.
Revised: We also ran experiments with a Vision Transformer ( ViT-B-16 ; Dosovitskiy et al., 2021), which is trained on the ILSVRC dataset with 82.64% top-1 test accuracy and 70M parameters.

Original: We take the output’s class tokens for layers 1 to 13 and the encoder’s output as latent representations, totalingfeatures of dimension 768 each.
Revised: We take the output’s class tokens for layers 1 to 13 and the encoder’s output as latent representations, totaling 14 features of dimension 768 each.

Original: We resize images to 224 × 224.
Revised: We resize images to 224 ⇥ 224.

Original: We download the DenseNet and Vision Transformer weights from PyTorch (Paszke et al., 2019) hub and the weights of Big Transfer(ResNet-101) from Kolesnikov et al.
Revised: We download the DenseNet and Vision Transformer weights from PyTorch (Paszke et al., 2019) hub and the weights of Big Transfer from Kolesnikov et al.

Original: They areall trained from scratch on ImageNet-1k.
Revised: All models are trained from scratch on ImageNet-1k.

Original: 5. E VALUATION M ETRICS
Revised: 5. E VALUATION M ETRICS

Original: We apply the established OOD evaluation metrics introduced in previous works (Hendrycks & Gimpel, 2017) for the primary benchmark.

Original: We evaluate the methods in terms of ROC and TNR .The
Revised: We evaluate the methods in terms of ROC and TNR .

Original:  Area Under The Receiving Operation Curve ( ROC ) is the Area Under the curve representing the true negative rate against the false positive rate when considering a varying threshold.
Revised: The Area Under The Receiving Operation Curve ( ROC ) is the Area Under the curve representing the true negative rate against the false positive rate when considering a varying threshold.

Original: It measures how well can the OOD score distinguish between out- and in-distribution data in a threshold-independent manner.
Revised: It measures how well can the OOD score distinguish between out- and in-distribution data in a threshold-independent manner.

Original: The True Negative Rate at 95% True Positive Rate (TNR at 95% TPR or TNR for short) is a threshold-dependent metric that provides the detector’s performance in a reasonable choice of novelty threshold.
Revised: The True Negative Rate at 95% True Positive Rate (TNR at 95% TPR or TNR for short) is a threshold-dependent metric that provides the detector’s performance in a reasonable choice of novelty threshold.

Original: It measures the accuracy in detecting OOD samples when the accuracy of detecting in-distribution samples is fixed at 95%.
Revised: It measures the accuracy in detecting OOD samples when the accuracy of detecting in-distribution samples is ﬁxed at 95%.

Original: For both measures, higher is better.
Revised: For both measures, higher is better.

Original: 5. B ASELINE H YPERPARAMETERS
Revised: 5. B ASELINE H YPERPARAMETERS

Original: For ODIN (Liang et al., 2018), we set the temperature to 1000 and the noise magnitude to zero.
Revised: For ODIN (Liang et al., 2018), we set the temperature to 1000 and the noise magnitude to zero.

Original: We take a temperature equal to one for Energy (Liu et al., 2020).
Revised: We take a temperature equal to one for Energy (Liu et al., 2020).

Original: We set the temperature to one for
Revised: We set the temperature to one for

Original: GradNorm (GradN. for short; Huang et al., 2021).
Revised: GradNorm (GradN. for short; Huang et al., 2021).

Original: For Mahalanobis (Maha.
Revised: For Mahalanobis (Maha.

Original: for short; Lee et al., 2018), we take only the scores of the outputs of the penultimate layer of the network.
Revised: for short; Lee et al., 2018), we take only the scores of the outputs of the penultimate layer of the network.

Original: The MSP (Hendrycks & Gimpel, 2017) does not have any hyperparameters.
Revised: The MSP (Hendrycks & Gimpel, 2017) does not have any hyperparameters.

Revised: For ReAct (Sun et al., 2021), we compute the activation clipping threshold with a percentile equals to 90.

Original: These hyperparameters choices are identical to the ones presented in Huang et al.
Revised: For KNN (Sun et al., 2022) we set as hyperparameters ↵ = 1% and k = 10 .

Original:  R ESULTS AND D ISCUSSION
Revised:  R ESULTS AND D ISCUSSION

Original: We report our main results in Table 1, which includes the performance for the three pre-trained architectures, four OOD datasets, and six detection methods.
Revised: We report our main results in Table 1, which includes the performance for the three pre-trained architectures, four OOD datasets, and seven detection methods.

Original: Our results are across the board consistent, obtaining an AUROC of 90% on average (see Table 1).
Revised: Our results are across the board consistent, obtaining an AUROC of 90% on average (see Table 1).

Original: It improves existing methods by a margin on all datasets and models.
Revised: It improves existing methods by a margin on all datasets and models.

Original: Multivariate OOD Scores is Not Enough.
Revised: Multivariate OOD Scores is Not Enough.

Original: Even though well-known multivariate novelty (or anomaly) detection techniques, such as One-class SVM (Cortes & Vapnik, 1995), Isolation Forest 
Revised: Even though well-known multivariate novelty (or anomaly) detection techniques, such as One-class SVM (Cortes & Vapnik, 1995), Isolation Forest (Liu et al., 2008), Extended Isolation Forest (Hariri et al., 2021), Local Outlier Factor (LOF;

Original: [Table]
Revised: [Table]

Original: (Liu et al., 2008), Extended Isolation Forest (Hariri et al., 2021), Local Outlier Factor (LOF; Breunig et al., 2000), k-NN approaches (Fix & Hodges, 1989) and distance-based approaches (Mahalanobis, 1936) are adapted to various scenarios, they showed to be inefficient for integrating layers’ information.
Revised: Breunig et al., 2000), k-NN approaches (Fix & Hodges, 1989) and distance-based approaches (Mahalanobis, 1936) are adapted to various scenarios, they showed to be inefﬁcient for integrating layers’ information.

Original: Unfortunately, classic multivariate methods cannot capture this interlayer dependence effectively.

Original: A hypothesis that explains this failure is the important sequential dependence pattern we noticed in the in-distribution layer-wise scores.
Revised: A hypothesis that explains this failure is the important sequential dependence pattern we noticed in the in-distribution layer-wise scores.

Original: Table 2 shows the performance of a few unsupervised aggregation methods based on a multivariate OOD detection paradigm.
Revised: Table 2 shows the performance of a few unsupervised aggregation methods based on a multivariate OOD detection paradigm.

Original: We tried typical methods: evaluating the Euclidean distance w.r.t the training set Gaussian representation, fitting an Isolation Forest, and a One-class SVM.
Revised: We tried typical methods: evaluating the Euclidean distance w.r.t the training set Gaussian representation, ﬁtting an Isolation Forest, and a One-class SVM.

Original: We compare the results with the performance of taking only the penultimate layer scores, and we observe that the standard multivariate aggregation fails to improve the scores for DenseNet-121. 
Revised: We compare the results with the performance of taking only the penultimate layer scores, and we observe that the standard multivariate aggregation fails to improve the scores for DenseNet-121. iNaturalist

Original: [Table]
Revised: [Table]

Original: Qualitative Evaluation of the Functional Dataset.
Revised: Qualitative Evaluation of the Functional Dataset.

Original: The test in-distribution trajectories follow a well-defined trend very similar to the training distribution (see Figure 3a).
Revised: The test in-distribution trajectories follow a well-deﬁned trend very similar to the training distribution (see Figure 3a).

Original: While the OOD trajectories manifest mainly as shape and magnitude anomalies (w.r.t.
Revised: While the OOD trajectories manifest mainly as shape and magnitude anomalies (w.r.t.

Original: the taxonomy of Hubert et al., 
Revised: the taxonomy of Hubert et al., 2015).

Revised: These characteristics reﬂects on the histogram of our detection score (see Figure 3b).

Revised: The

Original: [Figure]
Revised: [Figure]

Original: Study Case.
Revised: Study Case.

Original: There are a few overlaps in terms of the semantics of class names in the Textures and
Revised: There are a few overlaps in terms of the semantics of class names in the Textures and

Original: ImageNet datasets.
Revised: ImageNet datasets.

Original: In particular, “honeycombed” in Textures versus “honeycomb” in ImageNet, “stripes” vs. “zebra”, ”tiger”, and ”tiger cat”, and “cobwebbed” vs. “spider web”.
Revised: In particular, “honeycombed” in Textures versus “honeycomb” in ImageNet, “stripes” vs. “zebra”, ”tiger”, and ”tiger cat”, and “cobwebbed” vs. “spider web”.

Original: We showed that our method decreases the false negatives in this OOD benchmark.
Revised: We showed that our method decreases the false negatives in this OOD benchmark.

Original: In order to better understand how our method can discriminate where baselines often fail, we designed a simple study case.
Revised: In order to better understand how our method can discriminate where baselines often fail, we designed a simple study case.

Original: Take the
Revised: Take the

Original: Honeycombed vs. Honeycomb, for instance (the first row of Fig.
Revised: Honeycombed vs. Honeycomb, for instance (the ﬁrst row of Fig.

Original: 4a shows four examples of this partition and a couple of examples from ImageNet).
Revised: 4a shows four examples of this partition and a couple of examples from ImageNet).

Original: The honeycomb from ImageNet references natural honeycombs, usually found in nature, while honeycombed have a wider definition attached to artificial patterns.
Revised: The honeycomb from ImageNet references natural honeycombs, usually found in nature, while honeycombed have a wider deﬁnition attached to artiﬁcial patterns.

Original: In this class, the Energy baseline makes 108 false negative mistakes, while we only make 20 mistakes.
Revised: In this class, the Energy baseline makes 108 false negative mistakes, while we only make 20 mistakes.

Original: We noticed that some of our mistakes are aligned with real examples of honeycombs (e.g., the second example from the first row), whilst we confidently classify other patterns correctly as OOD.
Revised: We noticed that some of our mistakes are aligned with real examples of honeycombs (e.g., the second example from the ﬁrst row), whilst we conﬁdently classify other patterns correctly as OOD.

Original: For the Spider webs class, most examples from Textures are visually closer to ImageNet.
Revised: For the Spider webs class, most examples from Textures are visually closer to ImageNet.

Original: For the striped case (middle row), our method flags only 16 examples as being in-distribution, but we noticed an average higher score for the trajectories.
Revised: For the striped case (middle row), our method ﬂags only 16 examples as being in-distribution, but we noticed an average higher score for the trajectories.

Original: Overall, the study shows that our scores are aligned with the semantic proximity between samples and the training set.
Revised: Overall, the study shows that our scores are aligned with the semantic proximity between samples and the training set.

Original: Note that, for the animal classes, the context and head are essential features for classifying them.
Revised: Note that, for the animal classes, the context and head are essential features for classifying them.

Original: Potential Shortcomings of our Method.
Revised: Potential Shortcomings of our Method.

Original: We believe this work is only the first step in paving the way for efficient post-score aggregation as we have tackled an open and challenging problem of combining multi-layer information.
Revised: We believe this work is only the ﬁrst step in paving the way for efﬁcient post-score aggregation as we have tackled an open and challenging problem of combining multi-layer information.

Original: However, we believe there is room for improvement since our metric lives in the inner product space, which is a case for more general structures found in Hilbert spaces that might contain more adequate metrics.
Revised: However, we believe there is room for improvement since our metric lives in the inner product space, which is a case for more general structures found in Hilbert spaces that might contain more adequate metrics.

Original: Another concern that may arise is that, from a practical point of view, current third-party ML services often restrict the practitioner from accessing the intermediate outputs of the model in production.
Revised: Another concern that may arise is that, from a practical point of view, current third-party ML services often restrict the practitioner from accessing the intermediate outputs of the model in production.

Original: Nonetheless, the service provider has access to this information and could leverage it to deliver OOD detection as a service, for instance.
Revised: Nonetheless, the service provider has access to this information and could leverage it to deliver OOD detection as a service, for instance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduced an original approach to OOD detection based on a functional view of the pre-trained multi-layer neural network.
Revised: In this work, we introduced an original approach to OOD detection based on a functional view of the pre-trained multi-layer neural network.

Original: Our method detects samples whose trajectories differ from the typical behavior characterized by the training set.
Revised: Our method detects samples whose trajectories differ from the typical behavior characterized by the training set.

Original: The key ingredient relies on the statistical dependencies of the scores extracted at each layer, using a purely self-supervised algorithm.
Revised: The key ingredient relies on the statistical dependencies of the scores extracted at each layer, using a purely self-supervised algorithm.

Original: Beyond the novelty and practical advantages of the algorithm, our results establish the value of using a functional approach as an unsupervised technique to combine multiple scores, which offers an exciting alternative to usual single-layer approaches.
Revised: Beyond the novelty and practical advantages of the algorithm, our results establish the value of using a functional approach as an unsupervised technique to combine multiple scores, which offers an exciting alternative to usual single-layer approaches.

Original: We hope this work will be a first step to pave a new way for future research to enhance the safety of AI systems.R A A PPENDIX A. A LGORITHM AND C OMPUTATIONAL D ETAILS
Revised: We hope this work will be a ﬁrst step to pave a new way for future research to enhance the safety of AI systems.

Original: This section introduces further details on the computation algorithm and resources.

Original: Algorithmsand 2 describe how to extract the neural functional representations from the samples and compute the OOD score from test samples, respectively.

Original: Algorithm

Original: Input : dataset S N i )

Original: DNN f f L +1 ◦ · · · ◦ f 1 .

Original: Output: Reference trajectory ¯ u , scaling vector max ( U ) , and neural functional trajectory extraction function ϕ ( · ; { µ ℓ,y : y ∈ Y and ℓ ∈ { 1 , . . .

Original: , L + 1 }} ) .

Original: Training dataset

Original: feature

Original: extraction for ℓ ∈ { 1 , . . .

Original: , L + 1 } do z ℓ,i ← ( f ℓ ◦ · · · ◦ f 1 )( x i ) for y ∈ Y do µ ℓ,y ← 1 N y (cid:80) N y

Original: i

Original: =1 z ℓ,i // Class

Original: conditional features references

Original: end endfor i ∈ { 1 , . . .

Original: , N } do //

Original: Functional trajectory extractor ϕ ( · ) u i ← [d( z 1 ,i ; µ 1 ,y i ) , . . .

Original: , d( z L +1 ,i ; µ L +1 ,y i )] end

Original: U ← { u } Ni = max ( U ) ← (cid:101)

Original: [-U ← { u i max( U ) } Ni = ¯ u = 1 N (cid:80) Ni =
-]
Revised: [-U ← { u i max( U ) } Ni = ¯ u = 1 N (cid:80) Ni =
-]

Original: ˜ u i return ¯ u , max ( U ) , and ϕ ( · ; { µ ℓ,y : y ∈ Y and ℓ ∈ { 1 , . . .

Original: , L + 1 }} ) sample’s

Original: OOD score s ( x 0 ) .

Original: for ℓ ∈ { 1 , . . .

Original: , L + 1 } do z ℓ, 0 ← ( f ℓ ◦ · · · ◦ f 1 )( x 0 ) end ˆ y ← arg max f ( x 0 ) u 0 ← ϕ

Original: ( z 0 ; { µ ℓ, ˆ y : ℓ ∈ { 1 , . . .

Original: , L + 1 }} )

Original: ˜ u 0 ← u 0 max( U ) s ( x 0 )

Original: ← 1 ∥

Original: ¯ u ∥ 2 (cid:80) L +1 i =1 ˜ u 0 ,i ¯

Original: u i return s ( x 0 )

Original: A.1. C OMPUTING R ESOURCES

Original: We run our experiments on an internal cluster.

Original: Since we use pre-trained popular models, it was not necessary to retrain the deep models.

Original: Thus, our results should be reproducible with a single GPU.

Original: Since we are dealing with ImageNet datasets (approximately 150GB), a large storage capacity is expected.

Original: We save the features in memory to speed up downstream analysis of the algorithm, which may occupy around 200GB of storage.

Original: [Figure]

Original: [Figure]

Original: . L ATENT F EATURES A.2. B I T-S-

Original: We used the outputs of the following features to calculate our scores for the BiT-S-101 model: [body.block1, body.block2, body.block3, body.block4, head.flatten]

Original: A.2.

Original: D ENSE N ETWe used the outputs of the following features to calculate our scores for the DenseNet-121 model: [features.transition1.pool, features.transition2.pool, features.transition3.pool, features.norm5, flatten, classifier]

Original: A

Original: .2. V I T-B-

Original: We used the outputs of the following features to calculate our scores for the ViT-B-16 model: [encoder.layers.encoder layer 0, encoder.layers.encoder layer 1, encoder.layers.encoder layer 2, encoder.layers.encoder layer 3, encoder.layers.encoder layer 4, encoder.layers.encoder layer 5, encoder.layers.encoder layer 6, encoder.layers.encoder layer 7, encoder.layers.encoder layer 8, encoder.layers.encoder layer 9, encoder.layers.encoder layer 10, encoder.layers.encoder layer 11, encoder.ln, getitem 5, heads.head]

Original: A

Original: D ATASETS

Original: Figure 5 shows a few examples from the in-distribution and out-of-distributions datasets.

Original: We also list below the subsets of classes used for the OOD datasets following Huang & Li (2021):

Original: A.

Original: I N ATURALIST

Original: The classes considered for the iNaturalist dataset were: [Coprosma lucida, Cucurbita foetidissima, Mitella diphylla, Selaginella bigelovii, Toxicodendron vernix, Rumex obtusifolius, Ceratophyllum demersum, Streptopus amplexifolius, Portulaca oleracea, Cynodon dactylon,

Original: Agave lechuguilla, Pennantia corymbosa, Sapindus saponaria, Prunus serotina, Chondracanthus exasperatus, Sambucus racemosa, Polypodium vulgare, Rhus integrifolia, Woodwardia areolata, Epifagus virginiana,

Original: Rubus idaeus, Croton setiger, Mammillaria dioica, Opuntia littoralis,

Original: Cercis canadensis, Psidium guajava, Asclepias exaltata, Linaria purpurea,

Original: Ferocactus wislizeni, Briza minor, Arbutus menziesii, Corylus americana,

Original: Pleopeltis polypodioides, Myoporum laetum, Persea americana, Avena fatua, Blechnum discolor, Physocarpus capitatus, Ungnadia speciosa,

Original: Cercocarpus betuloides, Arisaema dracontium, Juniperus californica,

Original: Euphorbia prostrata, Leptopteris hymenophylloides, Arum italicum,

Original: Raphanus sativus, Myrsine australis, Lupinus stiversii, Pinus echinata,

Original: Geum macrophyllum, Ripogonum scandens, Echinocereus triglochidiatus,

Original: Cupressus macrocarpa, Ulmus crassifolia, Phormium tenax, Aptenia cordifolia, Osmunda claytoniana, Datura wrightii, Solanum rostratum,

Original: Viola adunca, Toxicodendron diversilobum, Viola sororia, Uropappus lindleyi, Veronica chamaedrys, Adenocaulon bicolor, Clintonia uniflora,

Original: Cirsium scariosum, Arum maculatum, Taraxacum officinale officinale,

Original: Orthilia secunda, Eryngium yuccifolium, Diodia virginiana, Cuscuta gronovii, Sisyrinchium montanum, Lotus corniculatus, Lamium purpureum,

Original: Ranunculus repens, Hirschfeldia incana, Phlox divaricata laphamii, Lilium martagon, Clarkia purpurea, Hibiscus moscheutos, Polanisia dodecandra,

Original: Fallugia paradoxa, Oenothera rosea, Proboscidea louisianica, Packera glabella, Impatiens parviflora, Glaucium flavum, Cirsium andersonii,

Original: Heliopsis helianthoides, Hesperis matronalis, Callirhoe pedata, Crocosmia crocosmiiflora, Calochortus albus, Nuttallanthus canadensis, Argemone albiflora, Eriogonum fasciculatum, Pyrrhopappus pauciflorus, Zantedeschia aethiopica, Melilotus officinalis, Peritoma arborea, Sisyrinchium bellum,

Original: Lobelia siphilitica, Sorghastrum nutans, Typha domingensis, Rubus laciniatus, Dichelostemma congestum, Chimaphila maculata, Echinocactus

Original: texensis

Original: A.3. SUN

Original: The classes considered for the SUN dataset were: [badlands, bamboo forest, bayou, botanical garden, canal (natural), canal (urban), catacomb, cavern (indoor), corn field, creek, crevasse, desert (sand), desert (vegetation), field (cultivated), field (wild), fishpond, forest (broadleaf), forest (needleleaf), forest path, forest road, hayfield, ice floe, ice shelf, iceberg, islet, marsh, ocean, orchard, pond, rainforest, rice paddy, river, rock arch, sky, snowfield, swamp, tree farm, trench, vineyard, waterfall (block), waterfall (fan), waterfall (plunge), wave, wheat field, herb garden, putting green, ski slope, topiary garden, vegetable garden, formal garden]

Original: A.3. P LACES

Original: The classes considered for the Places365 dataset were: [badlands, bamboo forest, canal (natural), canal (urban), corn field, creek, crevasse, desert (sand), desert (vegetation), desert road, field (cultivated), field (wild), field road, forest (broadleaf), forest path, forest road, formal garden, glacier, grotto, hayfield, ice floe, ice shelf, iceberg, igloo, islet, japanese garden, lagoon, lawn, marsh, ocean, orchard, pond, rainforest, rice paddy, river, rock arch, ski slope, sky, snowfield, swamp, swimming hole, topiary garden, tree farm, trench, tundra, underwater (ocean deep), vegetable garden waterfall wave field

Original: A.3. T EXTURES

Original: The classes considered for the Textures dataset were: [banded, blotchy, braided, bubbly, bumpy, chequered, cobwebbed, cracked, crosshatched, crystalline, dotted, dibrous, flecked, freckled, frilly, gauzy, grid, grooved, honeycombed, interlaced, knitted, lacelike, lined, marbled, matted, meshed, paisley, perforated, pitted, pleated, polka-dotted, porous, potholed, scaly, smeared, spiralled, sprinkled, stained, stratified, striped, studded, veined, waffled, woven, wrinkled, zigzagged]

Original: A. A DDITIONAL P LOTS , F UNCTIONAL D ATASET , H ISTOGRAMS AND ROC C URVES

Original: We display additional plots in Figures 7 and 6 for the observed functional data, the histogram of our scores showing separation between in-distribution and OOD data and the ROC curves for all of our experiments.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]