Original: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection
Revised: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The task of out-of-distribution (OOD) detection is crucial for deploying machinelearning models in real-world settings.
Revised: The task of out-of-distribution (OOD) detection is crucial for deploying machine learning models in real-world settings.

Original: In this paper, we observe that the singularvalue distributions of the in-distribution (ID) and OOD features are quite different:the OOD feature matrix tends to have a larger dominant singular value than the
Revised: In this paper, we observe that the singular value distributions of the in-distribution (ID) and OOD features are quite different: the OOD feature matrix tends to have a larger dominant singular value than the

Original: ID feature, and the class predictions of OOD samples are largely determined byit.
Revised: ID feature, and the class predictions of OOD samples are largely determined by it.

Original: Based on this observation, we assume that the first singular value of the OODfeature is larger than that of ID feature. This  motivates us to propose RankFeat ,a simple yet effective post hoc approach for OOD detection by removing therank-1 matrix composed of the largest singular value and the associated singularvectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).
Revised: This observation motivates us to propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).

Original: RankFeat achieves thestate-of-the-art performance and reduces the average false positive rate (FPR95) by17.90% compared with the previous best method.
Revised: RankFeat achieves the state-of-the-art performance and reduces the average false positive rate (FPR95) by 17.90% compared with the previous best method.

Original: Extensive ablation studies andcomprehensive theoretical analyses are presented to support the empirical results.
Revised: Extensive ablation studies and comprehensive theoretical analyses are presented to support the empirical results.

Original:  Introduction
Revised:  Introduction

Original: In the real-world applications of deep learning, understanding whether a test sample belongs to thesame distribution of training data is critical to the safe deployment of machine learning models.
Revised: In the real-world applications of deep learning, understanding whether a test sample belongs to the same distribution of training data is critical to the safe deployment of machine learning models.

Original: Themain challenge stems from the fact that current deep learning models can easily give over-confidentpredictions for out-of-distribution (OOD) data [42].
Revised: The main challenge stems from the fact that current deep learning models can easily give over-conﬁdent predictions for out-of-distribution (OOD) data [43].

Original: Recently a rich line of literature has emerged toaddress the challenge of OOD detection [59, 26, 3, 7, 51, 14, 11, 53, 65, 36, 13, 67, 10, 15, 18].
Revised: Recently a rich line of literature has emerged to address the challenge of OOD detection [60, 27, 3, 7, 52, 14, 11, 54, 66, 37, 13, 68, 10, 15, 18].

Original: Previous OOD detection approaches either rely on the feature distance [37], activation abnormality [51] or gradient norm [26].
Revised: Previous OOD detection approaches either rely on the feature distance [38], activation abnormality [52] or gradient norm [27].

Original: In this paper, we tackle the problem of OOD detection from anotherperspective: by analyzing the spectrum of the high-level feature matrices ( e.g., the output of Block 3or Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite differentsingular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)). The OOD featuretends to have a much larger dominant singular value than the ID feature, whereas the magnitudes ofthe rest singular values are very similar.This implies that the OOD feature allocates more energyalong with the dominant singular vector direction.
Revised: In this paper, we tackle the problem of OOD detection from another perspective: by analyzing the spectrum of the high-level feature matrices ( e g., the output of Blockor Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite different singular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)): the OOD feature tends to have a much larger dominant singular value than the ID feature, whereas the magnitudes of the rest singular values are very similar.

Original: The peculiar behavior motivates us to removethe rank-1 matrix composed of the dominant singular value and singular vectors from the feature.
Revised: This peculiar behavior motivates us to remove the rankmatrix composed of the dominant singular value and singular vectors from the feature.

Original: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of OOD samples; a majority of predictions have been changed.
Revised: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of OOD samples; a majority of predictions have been changed.

Original: On the contrary, most ID samples haveconsistent classification results before and after removing the subspace.
Revised: On the contrary, most ID samples have consistent classiﬁcation results before and after removing the subspace.

Original: This phenomenon indicatesthat the over-confident prediction of OOD samples might be largely determined by the dominantsingular value and the corresponding singular vectors.
Revised: This phenomenon indicates that the over-conﬁdent prediction of OOD samples might be largely determined by the dominant singular value and the corresponding singular vectors.

Original: Based on this observation, we assume that the first singular value of OOD feature tends to be muchlarger than that of ID feature.
Revised: Based on this observation, we assume that the ﬁrst singular value of OOD feature tends to be much larger than that of ID feature.

Original: The intuition behind is that the OOD feature corresponds to a larger
Revised: The intuition behind is that the OOD feature corresponds to a larger

Original: PCA explained variance ratio (less informative), and the well-trained network weights might cause 
Revised: PCA explained variance ratio (being less informative), and the well-trained network weights might cause and amplify the difference (see Sec.

Revised: E of the supplementary for the detailed illustration).

Revised: Hence, we conjecture that leveraging this gap might help to better distinguish ID and OOD samples.

Revised: To this

Original: After removing the rank-1 matrix composed by the dominant singular value and singular vectors the class predictions of OOD data are severely perturbed, while those of ID data are moderately influenced.
Revised: After removing the rank-1 matrix composed by the dominant singular value and singular vectors, the class predictions of OOD data are severely perturbed, while those of ID data are moderately inﬂuenced.

Original: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.
Revised: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.

Original: In light of this finding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In light of this ﬁnding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Both observations also hold for other OOD datasets and amplify the difference (see Sec. E of the supplementary for the detailed illustration). Hence, weconjecture that leveraging this gap might help to better distinguish ID and OOD samples. To thisend, we propose RankFeat , a simple but effective post hoc approach for OOD detection.
Revised: Both observations also hold for other OOD datasets end, we propose RankFeat , a simple but effective post hoc approach for OOD detection.

Original: RankFeatperturbs the high-level feature by removing its rank-1 matrix composed of the dominant singularvalue and vectors.
Revised: RankFeat perturbs the high-level feature by removing its rank-1 matrix composed of the dominant singular value and vectors.

Original: Then the logits derived from the perturbed features are used to compute the OODscore function.
Revised: Then the logits derived from the perturbed features are used to compute the OOD score function.

Original: By removing the rank-1 feature, the over-confidence of OOD samples is mitigated,and consequently the ID and OOD data can be better distinguished (see Fig. 2).
Revised: By removing the rank-1 feature, the over-conﬁdence of OOD samples is mitigated, and consequently the ID and OOD data can be better distinguished (see Fig. 2).

Original: Our RankFeatestablishes the state-of-the-art performance on the large-scale ImageNet benchmark and a suite ofwidely used OOD datasets across different network depths and architectures.
Revised: Our RankFeat establishes the state-of-the-art performance on the large-scale ImageNet benchmark and a suite of widely used OOD datasets across different network depths and architectures.

Original: In particular, RankFeatoutperforms the previous best method by 17.90% in the average false positive rate (FPR95) andby 5.44% in the area under curve (AUROC).
Revised: In particular, RankFeat outperforms the previous best method by 17.90% in the average false positive rate (FPR95) and by 5.44% in the area under curve (AUROC).

Original: Extensive ablation studies are performed to revealimportant insights of RankFeat , and comprehensive theoretical analyses are conducted to explainthe working mechanism.
Revised: Extensive ablation studies are performed to reveal important insights of RankFeat , and comprehensive theoretical analyses are conducted to explain the working mechanism.

Original: Code will be publicly available.
Revised: Code is publicly available via https://github.com/KingJamesSong/RankFeat .

Original: The key results and main contributions are threefold:
Revised: The key results and main contributions are threefold:

Original: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection byremoving the rank-1 matrix from the high-level feature.
Revised: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90%and improving the average AUROC by 5.44% compared to the previous best method.• We perform extensive ablation studies to illustrate the impact of (1) removing or keepingthe rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat atvarious network depths, (4) the number of iterations to iteratively derive the approximaterank-1 matrix for acceleration but without performance degradation, and (5) different fusionstrategies to combine multi-scale features for further performance improvements.
Revised: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90% and improving the average AUROC by 5.44% compared to the previous best method. • We perform extensive ablation studies to illustrate the impact of (1) removing or keeping the rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat at various network depths, (4) the number of iterations to iteratively derive the approximate rank-1 matrix for acceleration but without performance degradation, and (5) different fusion strategies to combine multi-scale features for further performance improvements.

Original: • Comprehensive theoretical analyses are conducted to explain the working mechanism andto underpin the superior empirical results.
Revised: • Comprehensive theoretical analyses are conducted to explain the working mechanism and to underpin the superior empirical results.

Original: We show that (1) removing the rank-1 matrixreduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [51]work by optimizing the upper bound containing the largest singular value.
Revised: We show that (1) removing the rank-1 matrix reduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [52] work by optimizing the upper bound containing the largest singular value.

Original: ReAct [51]indirectly and manually clips the underlying term, while RankFeat directly subtracts it.
Revised: ReAct [52] indirectly and manually clips the underlying term, while RankFeat directly subtracts it.

Original:  RankFeat: Rank-1 Feature Removal for OOD Detection
Revised:  RankFeat: Rank-1 Feature Removal for OOD Detection

Original: In this section, we introduce the background of OOD detection task and our proposed RankFeat thatperforms the OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In this section, we introduce the background of OOD detection task and our proposed RankFeat that performs the OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Preliminary: OOD detection.
Revised: Preliminary: OOD detection.

Original: The OOD detection is often formulated as a binary classificationproblem with the goal to distinguish between ID and OOD data.
Revised: The OOD detection is often formulated as a binary classiﬁcation problem with the goal to distinguish between ID and OOD data.

Original: Let f denote a model trained onsamples from the ID data D in .
Revised: Let f denote a model trained on samples from the ID data D in .

Original: For the unseen OOD data D out at test time, OOD detection aims todefine a decision function G ( · ) :
Revised: For the unseen OOD data D out at test time, OOD detection aims to deﬁne a decision function G ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function,and γ is a chosen threshold to make a large portion of ID data correctly classified ( e.g., 95% ).
Revised: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function, and γ is a chosen threshold to make a large portion of ID data correctly classiﬁed ( e.g., 95% ).

Original: Thedifficulty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the scoredistributions of ID and OOD data overlap as little as possible.
Revised: The difﬁculty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the score distributions of ID and OOD data overlap as little as possible.

Original: RankFeat: rank-1 feature removal.
Revised: RankFeat: rank-1 feature removal.

Original: Consider the reshaped high-level feature map X ∈ R C × HWof a deep network (the batch size is omitted for simplicity).
Revised: Consider the reshaped high-level feature map X ∈ R C × HW of a deep network (the batch size is omitted for simplicity).

Original: Here ’high-level feature’ denotes thefeature map that carries rich semantics in the deeper layers of a deep model ( e.g., the output of Block3 or Block 4 of a typical deep model like ResNet).
Revised: Here ’high-level feature’ denotes the feature map that carries rich semantics in the later layers of a network ( e.g., the output of Blockor Block 4 of a typical deep model like ResNet).

Original: Our RankFeat first performs the Singular Value
Revised: Our RankFeat ﬁrst performs the Singular Value

Original: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:
Revised: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and
Revised: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and

Original: V ∈ R
Revised: V ∈ R

Original: HW × HW are left and right orthogonal singular vector matrices, respectively.
Revised: HW × HW are left and right orthogonal singular vector matrices, respectively.

Original: Then RankFeatremoves the rank-1 matrix from the feature as:
Revised: Then RankFeat removes the rank-1 matrix from the feature as:

Original: [Equation]
Revised: [Equation]

Original: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singularvectors, respectively.
Revised: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singular vectors, respectively.

Original: The perturbed feature is fed into the rest of the network to generate the logitpredictions y ′ .
Revised: The perturbed feature is fed into the rest of the network to generate the logit predictions y (cid:48) .

Original: Finally, RankFeat computes the energy score of the logits for the input x as:
Revised: Finally, RankFeat computes the energy score of the logits for the input x as:

Original: [Equation]
Revised: [Equation]

Original: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-confidentpredictions of OOD data are largely perturbed.
Revised: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-conﬁdent predictions of OOD data are largely perturbed.

Original: In contrast, the decisions of ID data are mildlyinfluenced.
Revised: In contrast, the decisions of ID data are mildly inﬂuenced.

Original: This could help to separate the ID and OOD data better in the logit space.
Revised: This could help to separate the ID and OOD data better in the logit space.

Original: Acceleration by Power Iteration.
Revised: Acceleration by Power Iteration.

Original: Since RankFeat only involves the dominant singular value andvectors, there is no need to compute the full SVD of the feature matrix.
Revised: Since RankFeat only involves the dominant singular value and vectors, there is no need to compute the full SVD of the feature matrix.

Original: Hence our method can bepotentially accelerated by Power Iteration (PI).
Revised: Hence our method can be potentially accelerated by Power Iteration (PI).

Original: The PI algorithm is originally used to approximatethe dominant eigenvector of a Hermitian matrix.
Revised: The PI algorithm is originally used to approximate the dominant eigenvector of a Hermitian matrix.

Original: With a slight modification, it can also be applied togeneral rectangular matrices.
Revised: With a slight modiﬁcation, it can also be applied to general rectangular matrices.

Original: Given the feature X , the modified PI takes the coupled iterative update:
Revised: Given the feature X , the modiﬁed PI takes the coupled iterative update:

Original: [Equation]
Revised: [Equation]

Original: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and rightsingular vectors, respectively.
Revised: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and right singular vectors, respectively.

Original: After certain iterations, the dominant singular value is computed ass 1 = v Tk Xu k .
Revised: After certain iterations, the dominant singular value is computed as s 1 = v Tk Xu k .

Original: As will be illustrated in Sec.
Revised: As will be illustrated in Sec.

Original: 4.3, the approximate solution yielded by PI achieves verycompetitive performance against the SVD but with  less time overhead.
Revised: 4.3, the approximate solution yielded by PI achieves very competitive performance against the SVD but with much less time overhead.

Original: Combination of multi-scale features.
Revised: Combination of multi-scale features.

Original: Our RankFeat works at various later depths of a model, i.e.,
Revised: Our RankFeat works at various later depths of a model, i.e.,

Original: Block 3 and Block 4.
Revised: Block 3 and Block 4.

Original: Since intermediate features might focus on different semantic information, thedecision cue is very likely to be different.
Revised: Since intermediate features might focus on different semantic information, their decision cues are very likely to be different.

Original: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for better performance.
Revised: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for further performance improvements.

Original: Let y ′ and y ′′ denote thelogit predictions of Block 3 and Block 4 features, respectively.
Revised: Let y (cid:48) and y (cid:48)(cid:48) denote the logit predictions of Block 3 and Block 4 features, respectively.

Original: RankFeat performs the fusion at thelogit space and computes the score function as log (cid:80) exp( ( y ′ + y ′′ ) / 2 ) .
Revised: RankFeat performs the fusion at the logit space and computes the score function as log (cid:80) exp( ( y (cid:48) + y (cid:48)(cid:48) ) / 2 ) .

Original: Different fusion strategies areexplored and discussed in Sec. 4.3.
Revised: Different fusion strategies are explored and discussed in Sec. 4.3.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.
Revised: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.

Original: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper boundof RankFeat score more.
Revised: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Then based on Random Matrix Theory (RMT), we show that removingthe rank-1 matrix makes the statistics of OOD features closer to random matrices.
Revised: Then based on Random Matrix Theory (RMT), we show that removing the rank-1 matrix makes the statistics of OOD features closer to random matrices.

Original: Finally, thetheoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches workby optimizing the score upper bound determined by s 1 .
Revised: Finally, the theoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches work by optimizing the score upper bound determined by s 1 .

Original: ReAct manually uses a pre-defined thresholdto clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.
Revised: ReAct manually uses a pre-deﬁned threshold to clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.

Original: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.
Revised: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.

Original: For our RankFeat score function, we can express its upper bound in an analytical form.
Revised: For our RankFeat score function, we can express its upper bound in an analytical form.

Original: Moreover,the upper bound analysis implies that removing the rank-1 matrix with a larger first singular valuewould reduce the upper bound more.
Revised: Moreover, the upper bound analysis explicitly indicates that removing the rank-1 matrix with a larger ﬁrst singular value would reduce the upper bound more.

Original: Specifically, we have the following proposition.
Revised: Speciﬁcally, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The upper bound of RankFeat score is defined as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni
Revised: The upper bound of RankFeat score is deﬁned as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni

Original: =

Revised: =

Original: s i −s 1 (cid:17) || W || ∞ + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are theweight and bias of the last layer, respectively.
Revised: s i − s 1 (cid:17) || W || ∞ + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are the weight and bias of the last layer, respectively.

Original: A larger s i would reduce the upper bound more.
Revised: A larger s 1 would reduce the upper bound more.

Original: Proof.
Revised: Proof.

Original: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation ofrank-1 matrices X = (cid:80) s i 
Revised: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation of rank-1 matrices X = (cid:80) s i u

Original: u i v Ti .
Revised: i v Ti .

Original: The feature perturbed by RankFeat can be computed as:
Revised: The feature perturbed by RankFeat can be computed as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where N denotes the shorter side of the matrix (usually N = HW ).
Revised: where N denotes the shorter side of the matrix (usually N = HW ).

Original: In most deep models [19, 20],usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse thewidth and height dimensions.
Revised: In most deep models [19, 20], usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse the width and height dimensions.

Original: The GAP layer can be represented by a vector
Revised: The GAP layer can be represented by a vector

Original: [Equation]
Revised: [Equation]

Original: The pooled feature map is calculated as X ′ m .
Revised: The pooled feature map is calculated as X (cid:48) m .

Original: Then the output logits are computed by the matrixvector product with the classification head as:
Revised: Then the output logits are computed by the matrixvector product with the classiﬁcation head as:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and
Revised: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and

Original: y ′ ∈ R Q × 1 isthe output logits that correspond to the perturbed feature X ′ .
Revised: y (cid:48) ∈ R Q × 1 is the output logits that correspond to the perturbed feature X (cid:48) .

Original: Our RankFeat score is computed as:
Revised: Our RankFeat score is computed as:

Original: [Equation]
Revised: [Equation]

Original: where x is the input image, and Q denotes the number of categories.
Revised: where x is the input image, and Q denotes the number of categories.

Original: Here we choose Energy [39] asthe base function due to its theoretical alignment with the input probability density and its strongempirical performance.
Revised: Here we choose Energy [40] as the base function due to its theoretical alignment with the input probability density and its strong empirical performance.

Original: Eq.
Revised: Eq.

Original: (9) can be re-formulated by the Log-Sum-Exp trick
Revised: (9) can be re-formulated by the Log-Sum-Exp trick

Original: [Equation]
Revised: [Equation]

Original: The above equation directly yields the tight bound as:
Revised: The above equation directly yields the tight bound as:

Original: [Equation]
Revised: [Equation]

Original: Since max( y ′ ) ≤ max( | y ′ | )
Revised: Since max( y (cid:48) ) ≤ max( | y (cid:48) | )

Original: =
Revised: =

Original: ||
Revised: ||

Original: y ′
Revised: y (cid:48)

Original: || ∞ , we have
Revised: || ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||
Revised: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||

Original: + || c
Revised: + || c

Original: || holds for anyvectors a and c .
Revised: || holds for any vectors a and c .

Original: Moreover, since both u and v are orthogonal vectors, we have the relation ||
Revised: Moreover, since both u and v are orthogonal vectors, we have the relation ||

Original: u
Revised: u

Original: i || ∞ ≤ 1and || v i || ∞ ≤ 1 .
Revised: i || ∞ ≤and || v i || ∞ ≤ 1 .

Original: Relying on these two properties, injecting eq.
Revised: Relying on these two properties, injecting eq.

Original: (8) into eq.
Revised: (8) into eq.

Original: (12) leads to
Revised: (12) leads to

Original: RankFeat ( x ) <
Revised: RankFeat ( x ) <

Original: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Since m is a scaled all-ones vector, we have ||
Revised: Since m is a scaled all-ones vector, we have ||

Original: Wm || ∞
Revised: Wm || ∞

Original: =
Revised: =

Original: || W || ∞ / HW .
Revised: || W || ∞ / HW .

Original: The bound is simplified as:
Revised: The bound is simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.
Revised: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Our bound explicitly indicates the impact of s 1 on the score.
Revised: Remark:

Original: Considering that OOD feature usuallyhas a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.
Revised: Considering that OOD feature usually has a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.

Original: Notice that our bound analysis strives to improve the understanding of OOD methods from newperspectives instead of giving a strict guarantee of the score.
Revised: Notice that our bound analysis strives to improve the understanding of OOD methods from new perspectives instead of giving a strict guarantee of the score.

Original: For example, the upper bound can beused to explain the shrinkage and skew of score distributions in Fig.
Revised: For example, the upper bound can be used to explain the shrinkage and skew of score distributions in Fig.

Original: Subtracting s 1 would largelyreduce the numerical range of both ID and OOD scores, which could squeeze score distributions.
Revised: Subtracting s 1 would largely reduce the numerical range of both ID and OOD scores, which could squeeze score distributions.

Original: Since the dominant singular value s 1 contributesthe most to the score, removing s 1 is likely to makemany samples have similar scores.
Revised: Since the dominant singular value s 1 contributes most to the score, removing s 1 is likely to make many samples have similar scores.

Original: This would concentrate samples in a smaller region and furtherskew the distribution.
Revised: This would concentrate samples in a smaller region and further skew the distribution.

Original: Consider that the OOD feature tends to have a much larger s 1 . This wouldhave a greater impact on OOD data and skew the OOD score distribution more.
Revised: Given that the OOD feature tends to have a much larger s 1 , this would have a greater impact on OOD data and skew the OOD score distribution more.

Original: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.
Revised: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.

Original: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to randommatrices.
Revised: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to random matrices.

Original: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.
Revised: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.

Original: Fora random matrix of a given shape, the density of its eigenvalue asymptotically converges to the
Revised: For a random matrix of a given shape, the density of its eigenvalue asymptotically converges to the

Original: Manchenko-Pastur (MP) distribution [40, 49].
Revised: Manchenko-Pastur (MP) distribution [41, 50].

Original: Formally, we have:
Revised: Formally, we have:

Original: Theorem 1 (Manchenko-Pastur Law [40, 49]) .
Revised: Theorem 1 (Manchenko-Pastur Law [41, 50]) .

Original: Let X be a random matrix of shape t × n whose entriesare random variables with E ( X ij = 0) and E ( X 2 ij = 1) .
Revised: Let X be a random matrix of shape t × n whose entries are random variables with E ( X ij = 0) and E ( X 2 ij = 1) .

Original: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ ) = tn
Revised: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ [Equation] = [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Original: for λ ∈[ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .
Revised: for [Equation] [ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .

Original: This theorem indicates the possibility to measure the statistical distance between ID/OOD matrixand random matrices.
Revised: This theorem implies the possibility to measure the statistical distance between ID/OOD features and random matrices.

Original: To this end, we randomly sample 1 , 000 ID and OOD feature matrices andcompute the KL divergence between the actual eigenvalue distribution and the fitted MP distribution.
Revised: To this end, we randomly sample 1 , 000 ID and OOD feature matrices and compute the KL divergence between the actual eigenvalue distribution and the ﬁtted MP distribution.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of
Revised: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of

Original: Block 4 and Block 3 features, respectively.
Revised: Block 4 and Block 3 features, respectively.

Original: For the original feature, the OOD and ID feature matricesexhibit similar behaviors: the distances to the fitted MP distribution are roughly the same ( diff.
Revised: For the original feature, the OOD and ID feature matrices exhibit similar behaviors: the distances to the ﬁtted MP distribution are roughly the same ( diff.

Original: ≈ 0 . 1 ).
Revised: ≈ 0 . 1 ).

Original: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the
Revised: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the

Original: KL divergence.
Revised: KL divergence.

Original: This indicates that removing the rank-1 matrix makes the statistics of OOD featurecloser to random matrices, i.e., the OOD feature is very likely to become less informative than the IDfeature.
Revised: This indicates that removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, i.e., the OOD feature is very likely to become less informative than the ID feature.

Original: The result partly explains the working mechanism of RankFeat : by removing the featurematrix where OOD data might convey more information than ID data, the two types of distributionshave a larger discrepancy and therefore can be better separated.
Revised: The result partly explains the working mechanism of RankFeat : by removing the feature matrix where OOD data might convey more information than ID data, the two types of distributions have a larger discrepancy and therefore can be better separated.

Original: Connection with ReAct [51].
Revised: Connection with ReAct [52].

Original: ReAct clips the activations at the penultimate layer of a model todistinguish ID and OOD samples.
Revised: ReAct clips the activations at the penultimate layer of a model to distinguish ID and OOD samples.

Original: Given the feature X and the pooling layer m , the perturbation canbe defined as:
Revised: Given the feature X and the pooling layer m , the perturbation can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 15) where τ is a pre-defined threshold.
Revised: (15) where τ is a pre-deﬁned threshold.

Original: Their method shares some similarity with RankFeat formulation
Revised: Their method shares some similarity with RankFeat formulation

Original: Xm − s 1 u 1 v T 1 m .
Revised: Xm − s 1 u 1 v T 1 m .

Original: Both approaches subtract from the feature a portion of information that is mostlikely to cause the over-confidence of OOD prediction.
Revised: Both approaches subtract from the feature a portion of information that is most likely to cause the over-conﬁdence of OOD prediction.

Original: ReAct selects the manually-defined thresholdτ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrixfrom the feature itself.
Revised: ReAct selects the manually-deﬁned threshold τ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrix from the feature itself.

Original: Taking a step further, ReAct has the score inequality following eq.
Revised: Taking a step further, ReAct has the score inequality following eq

Original: [Equation]
Revised: [Equation]

Original: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .
Revised: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .

Original: Exploiting the vectornorm inequality ||
Revised: Exploiting the vector norm inequality ||

Original: X
Revised: X

Original: || F ≥|| X
Revised: || F ≥|| X

Original: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .
Revised: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .

Original: Relying on this property,the above inequality can be re-formulated as:
Revised: Relying on this property, the above inequality can be re-formulated as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .
Revised: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .

Original: Incontrast, the upper bound of our RankFeat can be expressed as:
Revised: In contrast, the upper bound of our RankFeat can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The upper bounds of both methods resemble each other with the only different term boxed.
Revised: The upper bounds of both methods resemble each other with the only different term boxed.

Original: From thispoint of view, both methods distinguish the ID and OOD data by eliminating the impact of the termcontaining s 1 in the upper bound.
Revised: From this point of view, both methods distinguish the ID and OOD data by eliminating the impact of the term containing s 1 in the upper bound.

Original: ReAct optimizes it by clipping the term with a manually-definedthreshold, which is indirect and might be sub-optimal.
Revised: ReAct optimizes it by clipping the term with a manually-deﬁned threshold, which is indirect and might be sub-optimal.

Original: Moreover, the threshold selection requiresstatistics of the whole ID set.
Revised: Moreover, the threshold selection requires statistics of the whole ID set.

Original: In contrast, our RankFeat does not require any extra data and directlysubtracts this underlying term which is likely to cause the over-confidence of OOD samples.
Revised: In contrast, our RankFeat does not require any extra data and directly subtracts this underlying term which is likely to cause the over-conﬁdence of OOD samples.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we first discuss the setup in Sec. 4.1, and then present the main experimental resultson ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.
Revised: In this section, we ﬁrst discuss the setup in Sec. 4.1, and then present the main experimental results on ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Datasets.
Revised: Datasets.

Original: In line with [25, 51, 26], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].
Revised: In line with [26, 52, 27], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].

Original: The large-scale dataset is more challenging than the traditional CIFAR benchmark [35] because the images are more realistic and diverse ( i.e., 1 .
Revised: The large-scale dataset is more challenging than the traditional CIFAR benchmark [36] because the images are more realistic and diverse ( i.e., 1 .

Original: 28 M images of 1 , 000 classes).
Revised: 28 M images of 1 , 000 classes).

Original: Forthe OOD datasets, we select four testsets from subsets of iNaturalist [57], SUN [62], Places [69],and Textures [5].
Revised: For the OOD datasets, we select four testsets from subsets of iNaturalist [58], SUN [63], Places [70], and Textures [5].

Original: These datasets are crafted by [25] with non-overlapping categories from ImageNet1k.
Revised: These datasets are crafted by [26] with non-overlapping categories from ImageNet1k.

Original: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of ourapproach with the CIFAR dataset as the ID dataset (see Supplementary Material).
Revised: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of our approach on Species [24] and CIFAR [36] benchmark. (see Supplementary Material).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with 6 recent post hoc OOD detection methods, namely 
Revised: We compare our method with 6 recent post hoc OOD detection methods, namely MSP [22],

Original: MSP [22], ODIN[38], Energy [39],  Mahalanobis [37], GradNorm [26], and ReAct [51].
Revised: ODIN [39], Energy [40], Mahalanobis [38], GradNorm [27], and ReAct [52].

Original: Thedetailed illustration and settings of these methods are kindly referred to Supplementary Material.
Revised: The detailed illustration and settings of these methods are kindly referred to Supplementary Material.

Original: Architectures.
Revised: Architectures.

Original: In line with [26], the main evaluation is done using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 [20].
Revised: In line with [27], the main evaluation is done using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 [20].

Original: We also evaluate the performance on
Revised: We also evaluate the performance on

Original: SqueezeNet [28], an alternative tiny architecture suitable for mobile devices and on [-T2T-ViT-
 [66],a-] tokens-to-tokens vision transformer that has impressive performance when trained from scratch.
Revised: SqueezeNet [29], an alternative tiny architecture suitable for mobile devices and on [-T2T-ViT-
 [66],a-]T2T-ViT-24 [67], a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.

Original: For the implementation details and evaluation metrics, please refer to Supplementary Material .
Revised: For the implementation details and evaluation metrics, please refer to Supplementary Material .

Original: [Table]
Revised: [Table]

Original: 4.2 Results
Revised: 4.2 Results

Original: Main results.
Revised: Main results.

Original: Following [26], the main evaluation is conducted using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 architecture [20].
Revised: Following [27], the main evaluation is conducted using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 architecture [20].

Original: Table 2 compares the performanceof all the post hoc methods.
Revised: Table 2 compares the performance of all the post hoc methods.

Original: For both Block 3 and Block 4 features, our RankFeat achieves thebest evaluation results across datasets and metrics.
Revised: For both Block 3 and Block 4 features, our RankFeat achieves the best evaluation results across datasets and metrics.

Original: More specifically, RankFeat based on the Block4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.
Revised: More speciﬁcally, RankFeat based on the Block 4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block 3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.

Original: Theircombination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in theaverage AUROC.
Revised: Their combination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in the average AUROC.

Original: The superior performances at various depths demonstrate the effectiveness andgeneral applicability of RankFeat .
Revised: The superior performances at various depths demonstrate the effectiveness and general applicability of RankFeat .

Original: The Block 3 feature has a higher AUROC but slightly falls behindthe 4 feature in the FPR95, which can be considered a compromise between the two metrics.[Table]
Revised: The Block 3 feature has a higher AUROC but slightly falls behind the Block 4 feature in the FPR95, which can be considered a compromise between the two metrics.

Original: Our RankFeat is also effective on alternative CNN architectures.
Revised: Our RankFeat is also effective on alternative CNN architectures.

Original: Besides the experiment on
Revised: Besides the experiment on

Original: ResNetv2 [20], we also evaluate our method on SqueezeNet [28], an alternative tiny network suitablefor mobile devices and on-chip applications.
Revised: ResNetv2 [20], we also evaluate our method on SqueezeNet [29], an alternative tiny network suitable for mobile devices and on-chip applications.

Original: This network is more challenging because the tinynetwork size makes the model prone to overfit the training data, which could increase the difficultyto distinguish between ID and OOD samples.
Revised: This network is more challenging because the tiny network size makes the model prone to overﬁt the training data, which could increase the difﬁculty to distinguish between ID and OOD samples.

Original: Table 3  presents the performance of all the methods.
Revised: Table 3 top presents the performance of all the methods.

Original: Collectively, the performances of RankFeat are very competitive at both depths, as well as the scorefusion.
Revised: Collectively, the performances of RankFeat are very competitive at both depths, as well as the score fusion.

Original: Our RankFeat achieves the state-of-the-art performances, outperforming the second-bestbaseline by 14.22% in FPR95 and by 7.48% in AUROC.
Revised: Our RankFeat achieves the state-of-the-art performances, outperforming the second-best baseline by 14.22% in FPR95 and by 7.48% in AUROC.

Original: Our RankFeat also suits transformer-based architectures.
Revised: Our RankFeat also suits transformer-based architectures.

Original: To further demonstrate the applicabilityof our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)
Revised: To further demonstrate the applicability of our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)

Original: [66],a popular transformer architecture that can achieve competitive performance against CNNs whentrained from scratch.
Revised: [67], a popular transformer architecture that can achieve competitive performance against CNNs when trained from scratch.

Original: Similar to the CNN, RankFeat removes the rank-1 matrix from the finaltoken of T2T-ViT before the last normalization layer and the classification head.
Revised: Similar to the CNN, RankFeat removes the rank-1 matrix from the ﬁnal token of T2T-ViT before the last normalization layer and the classiﬁcation head.

Original: Table 3 comparesthe performance on T2T-ViT-24.
Revised: Table 3 bottom compares the performance on T2T-ViT-24.

Original: Our RankFeat outperforms the second-best method by 9.49%in FPR95 and by 3.68% in AUROC.
Revised: Our RankFeat outperforms the second-best method by 6.11% in FPR95 and by 2.05% in AUROC.

Original: Since the transformer models [9, 66] do not have increasingreceptive fields like CNNs, we do not evaluate the performance at alternative network depths. 
Revised: Since the transformer models [9, 67] do not have increasing receptive ﬁelds like CNNs, we do not evaluate the performance at alternative network depths. [Table]

Revised: Comparison against training-needed approaches.

Revised: Since our method is post hoc , we only compare it with other post hoc baselines.

Original: 4.3 Ablation Studies
Revised: MOS [26] and KL Matching [24] are not taken into account because MOS needs extra training processes and KL Matching requires the labeled validation set to compute distributions for each class. Nonetheless, we note that our method can still hold an advantage against those approaches. Table 4 presents the average FPR95 and AUROC on the ImageNet-1k benchmark.

Revised: Our RankFeat achieves the best performance without any extra training or validation set.

Revised: [Table]

Revised: 4.3 Ablation Studies

Original: In this subsection, we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.
Revised: In this subsection [Table] we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.

Original: Unless explicitly specified, we apply RankFeat on the Block 4 feature by default.
Revised: Unless explicitly speciﬁed, we apply RankFeat on the Block 4 feature by default.

Original: [Table]
Revised: [Table]

Revised: Removing Rank46.

Original: Removing the rank-1 matrix outperforms keeping only it.
Revised: Removing the rank-1 matrix outperforms keeping only it.

Original: Instead of removing the rank-1 matrix,another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest ofthe matrix.
Revised: Instead of removing the rank-1 matrix, another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest of the matrix.

Original: Table 4 presents the evaluation results of keeping only the rank-1 matrix.
Revised: Table 5 presents the evaluation results of keeping only the rank-1 matrix.

Original: The performancefalls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keepingonly the rank-1 feature is inferior to removing it in distinguishing the two distributions.
Revised: The performance falls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keeping only the rank-1 feature is inferior to removing it in distinguishing the two distributions.

Original: Nonetheless,it is worth noting that even keeping only the rank-1 matrix achieves very competitive performanceagainst previous best methods, such as GradNorm [26] and ReAct [51].
Revised: Nonetheless, it is worth noting that even keeping only the rank-1 matrix achieves very competitive performance against previous best methods, such as GradNorm [27] and ReAct [52].

Original: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).
Revised: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).

Original: We evaluate theimpact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni
Revised: We evaluate the impact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni

Original: =

Revised: =

Original: s i
Revised: s i

Original: u
Revised: u

Original: i v Ti where n> 1 forthe high-level feature X .
Revised: i v Ti where n> 1 for the high-level feature X .

Original: Table 4 compares the performance of removing the rank-2 matrix andrank-3 matrix.
Revised: Table 5 compares the performance of removing the rank-2 matrix and rank-3 matrix.

Original: When the rank of the removed matrix is increased, the average performance degradesaccordingly.
Revised: When the rank of the removed matrix is increased, the average performance degrades accordingly.

Original: This demonstrates that removing the rank-1 matrix is the most effective approach toseparate ID and OOD data.
Revised: This demonstrates that removing the rank-1 matrix is the most effective approach to separate ID and OOD data.

Original: This result is coherent with the finding in Fig.
Revised: This result is coherent with the ﬁnding in Fig.

Original: 1(a): only the largestsingular value of OOD data is significantly different from that of ID data.
Revised: 1(a): only the largest singular value of OOD data is signiﬁcantly different from that of ID data.

Original: Therefore, removing therank-1 matrix achieves the best performance.
Revised: Therefore, removing the rank-1 matrix achieves the best performance.

Original: [Table]
Revised: [Table]

Original: Block 3 and Block 4 features are the most informative.
Revised: Block 3 and Block 4 features are the most informative.

Original: In addition to exploring the high-levelfeatures at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to featuresat shallow network layers.
Revised: In addition to exploring the high-level features at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to features at shallow network layers.

Original: As shown in Table 5, the performances of RankFeat at the Block 1 and
Revised: As shown in Table 6, the performances of RankFeat at the Block 1 and

Original: Block 2 features are not comparable to those at deeper layers.
Revised: Block 2 features are not comparable to those at deeper layers.

Original: This is mainly because the shallowlow-level features do not embed as rich semantic information as the deep features.
Revised: This is mainly because the shallow low-level features do not embed as rich semantic information as the deep features.

Original: Consequently,removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.
Revised: Consequently, removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.

Original: [Table]
Revised: [Table]

Original: The approximate solution by PI yields competitive performances.
Revised: The approximate solution by PI yields competitive performances.

Original: Table 6 compares the timeconsumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods
Revised: Table 7 compares the time consumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods

Original: ReAct and GradNorm .
Revised: ReAct and GradNorm .

Original: The performance of PI starts to become competitive against that of SVD ( < 0 .
Revised: The performance of PI starts to become competitive against that of SVD ( < 0 .

Original: 1% ) from 20 iterations on with 48.41% time reduction.
Revised: 1% ) from 20 iterations on with 48.41% time reduction.

Original: Compared with ReAct , the PI-based
Revised: Compared with ReAct , the PI-based

Original: RankFeat only requires marginally 4 .
Revised: RankFeat only requires marginally 4 .

Original: 89% more time consumption.
Revised: 89% more time consumption.

Original: GradNorm is not comparableagainst other baselines in terms of time cost because it does not support the batch mode.
Revised: GradNorm is not comparable against other baselines in terms of time cost because it does not support the batch mode.

Original: Fusion at the logit space achieves the best performance.
Revised: Fusion at the logit space achieves the best performance.

Original: Fig.
Revised: Fig.

Original: 4 displays the performance ofdifferent fusion strategies in combining RankFeat at the Block 3 and Block 4 features.
Revised: 4 displays the performance of different fusion strategies in combining RankFeat at the Block 3 and Block 4 features.

Original: As can beobserved, averaging logits outperforms other fusion strategies in most datasets and metrics.
Revised: As can be observed, averaging logits outperforms other fusion strategies in most datasets and metrics.

Original: Thisindicates that the fusing the logits can best coordinate the benefit of both features.
Revised: This indicates that the fusing the logits can best coordinate the beneﬁt of both features.

Original:  Related Work
Revised:  Related Work

Original: Distribution shifts.
Revised: Distribution shifts.

Original: Distribution shifts have been a long-standing problem in the machine learningresearch community [17, 44, 33, 61].
Revised: Distribution shifts have been a long-standing problem in the machine learning research community [17, 45, 34, 62].

Original: The problem of distributions shifts can be generally categorizedas shifts in the input space and shifts in the label space.
Revised: The problem of distributions shifts can be generally categorized as shifts in the input space and shifts in the label space.

Original: Shifts only in the input space are oftendeemed as covariate shifts [21, 43].
Revised: Shifts only in the input space are often deemed as covariate shifts [21, 44].

Original: In this setting, the inputs are corrupted by perturbations or shiftedby domains, but the label space stays the same [24, 52].
Revised: In this setting, the inputs are corrupted by perturbations or shifted by domains, but the label space stays the same [25, 53].

Original: The aim is mainly to improve the robustnessand generalization of a model [23].
Revised: The aim is mainly to improve the robustness and generalization of a model [23].

Original: For OOD detection, the labels are disjoint and the main concernis to determine whether a test sample should be predicted by the pre-trained model [38, 24].
Revised: For OOD detection, the labels are disjoint and the main concern is to determine whether a test sample should be predicted by the pre-trained model [39, 25].

Original: Some related sub-fields also tackle the problem of distribution shifts in the label space, such as novelclass discovery [16, 68], open-set recognition [47, 58], and novelty detection [1, 55].
Revised: Some related sub-ﬁelds also tackle the problem of distribution shifts in the label space, such as novel class discovery [16, 69], open-set recognition [48, 59], and novelty detection [1, 56].

Original: These sub-fieldstarget specific distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.
Revised: These sub-ﬁelds target speciﬁc distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.

Original: OOD detection with discriminative models.
Revised: OOD detection with discriminative models.

Original: The early work on discriminative OOD detection datesback to the classification model with rejection option [4, 12].
Revised: The early work on discriminative OOD detection dates back to the classiﬁcation model with rejection option [4, 12].

Revised: The OOD detection methods can be generally divided into training-need methods and post hoc approaches.

Revised: Compared with trainingneeded approaches, post hoc methods do not require any extra training processes and could be directly applied to any pre-trained models.

Original: For the wide body of research on OODdetection, please refer to [64] for the comprehensive survey.
Revised: For the wide body of research on OOD detection, please refer to [65] for the comprehensive survey.

Original: Here we  highlight the representative posthoc methods that do not require extra training processes and can be directly applied to any pre-trainedmodels.
Revised: Here we only highlight the representative post hoc methods.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: [42] first observed the phenomenon that neural networks easily give overconfident predictions for OOD samples.
Revised: [43] ﬁrst observed the phenomenon that neural networks easily give over-conﬁdent predictions for OOD samples.

Original: The following researches attempted to improve the OODuncertainty estimation by proposing ODIN score [38], OpenMax score [2], Mahalanobis distance [37],and Energy score [39].
Revised: The following researches attempted to improve the OOD uncertainty estimation by proposing ODIN score [39], OpenMax score [2], Mahalanobis distance [38], and Energy score [40].

Original: Huang et al.
Revised: Huang et al.

Original: [25] pointed out that the traditional CIFAR benchmark does notextrapolate to real-world settings and proposed a large-scale ImageNet benchmark.
Revised: [26] pointed out that the traditional CIFAR benchmark does not extrapolate to real-world settings and proposed a large-scale ImageNet benchmark.

Original: More recently,
Revised: More recently,

Original: Sun et al.
Revised: Sun et al.

Original: [51] and Huang et al.
Revised: [52] and Huang et al.

Original: [26] proposed to tackle the challenge of OOD detection from thelens of activation abnormality and gradient norm, respectively.
Revised: [27] proposed to tackle the challenge of OOD detection from the lens of activation abnormality and gradient norm, respectively.

Original: In contrast, based on the empiricalobservation of singular value distributions, we propose a simple yet effective post hoc solution byremoving the rank-1 subspace from the high-level features.
Revised: In contrast, based on the empirical observation of singular value distributions, we propose a simple yet effective post hoc solution by removing the rank-1 subspace from the high-level features.

Original: OOD detection with generative models.
Revised: OOD detection with generative models.

Original: Different from discriminative models, generative modelsdetect the OOD samples by estimating the probability density function [31, 54, 46, 56, 8, 27, 3, 29].
Revised: Different from discriminative models, generative models detect the OOD samples by estimating the probability density function [32, 55, 47, 57, 8, 28, 3, 30].

Original: A sample with a low likelihood is deemed as OOD data.
Revised: A sample with a low likelihood is deemed as OOD data.

Original: Recently, a multitude of methods haveutilized generative models for OOD detection [45, 50, 60, 63, 32, 48, 30].
Revised: Recently, a multitude of methods have utilized generative models for OOD detection [46, 51, 61, 64, 33, 49, 31].

Original: However, as pointed outin [41], generative models could assign a high likelihood to OOD data.
Revised: However, as pointed out in [42], generative models could assign a high likelihood to OOD data.

Original: Furthermore, generativemodels can be prohibitively harder to train and optimize than their discriminative counterparts, andthe performance is often inferior.
Revised: Furthermore, generative models can be prohibitively harder to train and optimize than their discriminative counterparts, and the performance is often inferior.

Original: This might limit their practical usage.
Revised: This might limit their practical usage.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present RankFeat , a simple yet effective approach for  OOD detection by removingthe rank-1 matrix composed by the largest singular value from the high-level feature.
Revised: In this paper, we present RankFeat , a simple yet effective approach for post hoc OOD detection by removing the rank-1 matrix composed by the largest singular value from the high-level feature.

Original: We demonstratethe superior empirical results and the general applicabilityof RankFeat across architectures, networkdepths, and benchmarks.
Revised: We demonstrate its superior empirical results and the general applicability across architectures, network depths, and benchmarks.

Original: Extensive ablation studies and comprehensive theoretical analyses areconducted to reveal the important insights and to explain the working mechanism of our method.Checklist
Revised: Extensive ablation studies and comprehensive theoretical analyses are conducted to reveal the important insights and to explain the working mechanism of our method.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]
Revised: Acknowledgments and Disclosure of Funding

Original: (b) Did you describe the limitations of your work?

Original: [Yes] We briefly discuss the slightly

Original: higher time consumption of RankFeat in Sec. 4.3.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do notanticipate any negative societal impacts of our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [N/A] Since our method is post hoc , it will not be influenced

Original: by random seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: This research was supported by the EU H2020 projects AI4Media (No. 951911) and SPRING (No.

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: The used data do not contain this kind of

Original: information.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]
Revised: We thank our colleague Zhun Zhong for the fruitful discussion and valuable suggestions.