Original: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection
Revised: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

Original: Abstract
Revised: Abstract

Original: The task of out-of-distribution (OOD) detection is crucial for deploying machine learning models in real-world settings.
Revised: The task of out-of-distribution (OOD) detection is crucial for deploying machine learning models in real-world settings.

Original: In this paper, we observe that the singular value distributions of the in-distribution (ID) and OOD features are quite different: the OOD feature matrix tends to have a larger dominant singular value than the
Revised: In this paper, we observe that the singular value distributions of the in-distribution (ID) and OOD features are quite different: the OOD feature matrix tends to have a larger dominant singular value than the

Original: ID feature, and the class predictions of OOD samples are largely determined by it.
Revised: ID feature, and the class predictions of OOD samples are largely determined by it.

Original: This observation motivates us to propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).
Revised: This observation motivates us to propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature.

Original: RankFeat achievesthe state-of-the-art performance and reduces the average false positive rate (FPR95) by 17.90% compared with the previous best method.
Revised: RankFeat achieves state-of-the-art performance and reduces the average false positive rate (FPR95) by 17.90% compared with the previous best method.

Original: Extensive ablation studies and comprehensive theoretical analyses are presented to support the empirical results.
Revised: Extensive ablation studies and comprehensive theoretical analyses are presented to support the empirical results.

Original:  Introduction
Revised:  Introduction

Original: In the real-world applications of deep learning, understanding whether a test sample belongs to the same distribution of training data is critical to the safe deployment of machine learning models.
Revised: In the real-world applications of deep learning, understanding whether a test sample belongs to the same distribution of training data is critical to the safe deployment of machine learning models.

Original: The main challenge stems from the fact that current deep learning models can easily give over-conﬁdent predictions for out-of-distribution (OOD) data [43].
Revised: The main challenge stems from the fact that current deep learning models can easily give over-conﬁdent predictions for out-of-distribution (OOD) data [43].

Original: Recently a rich line of literature has emerged to address the challenge of OOD detection [60, 27, 3, 7, 52, 14, 11, 54, 66, 37, 13, 68, 10, 15, 18].
Revised: Recently a rich line of literature has emerged to address the challenge of OOD detection [60, 27, 3, 7, 52, 14, 11, 54, 66, 37, 13, 68, 10, 15, 18].

Original: Previous OOD detection approaches either rely on the feature distance [38], activation abnormality [52] or gradient norm [27].
Revised: Previous OOD detection approaches either rely on the feature distance [38], activation abnormality [52] or gradient norm [27].

Original: In this paper, we tackle the problem of OOD detection from another perspective: by analyzing the spectrum of the high-level feature matrices ( e g., the output of Blockor Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite different singular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)): the OOD feature tends to have a much larger dominant singular value than the ID feature, whereas the magnitudes of the rest singular values are very similar.
Revised: In this paper, we tackle the problem of OOD detection from another perspective: by analyzing the spectrum of the high-level feature matrices ( e g., the output of Blockor Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite different singular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)): the OOD feature tends to have a much larger dominant singular value than the ID feature, whereas the magnitudes of the rest singular values are very similar.

Original: This peculiar behavior motivates us to remove the rankmatrix composed of the dominant singular value and singular vectors from the feature.
Revised: This peculiar behavior motivates us to remove the rankmatrix composed of the dominant singular value and singular vectors from the feature.

Original: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of OOD samples; a majority of predictions have been changed.
Revised: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of OOD samples; a majority of predictions have been changed.

Original: On the contrary, most ID samples have consistent classiﬁcation results before and after removing the subspace.
Revised: On the contrary, most ID samples have consistent classiﬁcation results before and after removing the subspace.

Original: This phenomenon indicates that the over-conﬁdent prediction of OOD samples might be largely determined by the dominant singular value and the corresponding singular vectors.
Revised: This phenomenon indicates that the over-conﬁdent prediction of OOD samples might be largely determined by the dominant singular value and the corresponding singular vectors.

Original: Based on this observation, we assume that the ﬁrst singular value of OOD feature tends to be much larger than that of ID feature.
Revised: Based on this observation, we assume that the ﬁrst singular value of OOD feature tends to be much larger than that of ID feature.

Original: The intuition behind is that the OOD feature corresponds to a larger
Revised: The intuition behind is that the OOD feature corresponds to a larger

Original: PCA explained variance ratio (being less informative), and the well-trained network weights might cause and amplify the difference (see Sec.
Revised: PCA explained variance ratio (being less informative), and the well-trained network weights might cause and amplify the difference (see Sec.

Original: E of the supplementary for the detailed illustration).
Revised: E of the supplementary for the detailed illustration).

Original: Hence, we conjecture that leveraging this gap might help to better distinguish ID and OOD samples.
Revised: Hence, we conjecture that leveraging this gap might help to better distinguish ID and OOD samples.

Original: To this
Revised: To this

Original: After removing the rank-1 matrix composed by the dominant singular value and singular vectors, the class predictions of OOD data are severely perturbed, while those of ID data are moderately inﬂuenced.
Revised: After removing the rank-1 matrix composed by the dominant singular value and singular vectors, the class predictions of OOD data are severely perturbed, while those of ID data are moderately inﬂuenced.

Original: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.
Revised: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.

Original: In light of this ﬁnding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In light of this ﬁnding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Both observations also hold for other OOD datasets end, we propose RankFeat , a simple but effective post hoc approach for OOD detection.
Revised: Both observations also hold for other OOD datasets end, we propose RankFeat , a simple but effective post hoc approach for OOD detection.

Original: RankFeat perturbs the high-level feature by removing its rank-1 matrix composed of the dominant singular value and vectors.
Revised: RankFeat perturbs the high-level feature by removing its rank-1 matrix composed of the dominant singular value and vectors.

Original: Then the logits derived from the perturbed features are used to compute the OOD score function.
Revised: Then the logits derived from the perturbed features are used to compute the OOD score function.

Original: By removing the rank-1 feature, the over-conﬁdence of OOD samples is mitigated, and consequently the ID and OOD data can be better distinguished (see Fig. 2).
Revised: By removing the rank-1 feature, the over-conﬁdence of OOD samples is mitigated, and consequently the ID and OOD data can be better distinguished (see Fig. 2).

Original: Our RankFeat establishes the state-of-the-art performance on the large-scale ImageNet benchmark and a suite of widely used
Revised: Our RankFeat establishes the state-of-the-art performance on the large-scale ImageNet benchmark and a suite of widely used

Original: OOD datasets across different network depths and architectures.
Revised: OOD datasets across different network depths and architectures.

Original: In particular, RankFeat outperforms the previous best method by 17.90% in the average false positive rate (FPR95) and by 5.44% in the area under curve (AUROC).
Revised: In particular, RankFeat outperforms the previous best method by 17.90% in the average false positive rate (FPR95) and by 5.44% in the area under curve (AUROC).

Original: Extensive ablation studies are performed to reveal important insights of
Revised: Extensive ablation studies are performed to reveal important insights of

Original: RankFeat , and comprehensive theoretical analyses are conducted to explain the working mechanism.
Revised: RankFeat , and comprehensive theoretical analyses are conducted to explain the working mechanism.

Original: Code is publicly available via https://github.com/KingJamesSong/RankFeat .
Revised: Code is publicly available via https://github.com/KingJamesSong/RankFeat .

Original: The key results and main contributions are threefold:
Revised: The key results and main contributions are threefold:

Original: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90% and improving the average AUROC by 5.44% compared to the previous best method.
Revised: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90% and improving the average AUROC by 5.44% compared to the previous best method.

Original: • We perform extensive ablation studies to illustrate the impact of (1) removing or keeping the rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat at various network depths, (4) the number of iterations to iteratively derive the approximate
Revised: • We perform extensive ablation studies to illustrate the impact of (1) removing or keeping the rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat at various network depths, (4) the number of iterations to iteratively derive the approximate

Original: rank-1 matrix for acceleration but without performance degradation, and (5) different fusion strategies to combine multi-scale features for further performance improvements.
Revised: rank-1 matrix for acceleration but without performance degradation, and (5) different fusion strategies to combine multi-scale features for further performance improvements.

Original: • Comprehensive theoretical analyses are conducted to explain the working mechanism and to underpin the superior empirical results.
Revised: • Comprehensive theoretical analyses are conducted to explain the working mechanism and to underpin the superior empirical results.

Original: We show that (1) removing the rank-1 matrix reduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [52] work by optimizing the upper bound containing the largest singular value.
Revised: We show that (1) removing the rank-1 matrix reduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [52] work by optimizing the upper bound containing the largest singular value.

Original: ReAct [52] indirectly and manually clips the underlying term, while RankFeat directly subtracts it.
Revised: ReAct [52] indirectly and manually clips the underlying term, while RankFeat directly subtracts it.

Original:  RankFeat: Rank-1 Feature Removal for OOD Detection
Revised:  RankFeat: Rank-1 Feature Removal for OOD Detection

Original: In this section, we introduce the background of OOD detection task and our proposed RankFeat that performs the OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In this section, we introduce the background of OOD detection task and our proposed RankFeat that performs the OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Preliminary: OOD detection.
Revised: Preliminary: OOD detection.

Original: The OOD detection is often formulated as a binary classiﬁcation problem with the goal to distinguish between ID and OOD data.
Revised: The OOD detection is often formulated as a binary classiﬁcation problem with the goal to distinguish between ID and OOD data.

Original: Let f denote a model trained on samples from the ID data D in .
Revised: Let f denote a model trained on samples from the ID data D in .

Original: For the unseen OOD data D out at test time, OOD detection aims to deﬁne a decision function G ( · ) :
Revised: For the unseen OOD data D out at test time, OOD detection aims to deﬁne a decision function G ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function, and γ is a chosen threshold to make a large portion of ID data correctly classiﬁed
Revised: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function, and γ is a chosen threshold to make a large portion of ID data correctly classiﬁed

Original: ( e.g., 95% ).
Revised: ( e.g., 95% ).

Original: The difﬁculty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the score distributions of ID and OOD data overlap as little as possible.
Revised: The difﬁculty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the score distributions of ID and OOD data overlap as little as possible.

Original: RankFeat: rank-1 feature removal.
Revised: RankFeat: rank-1 feature removal.

Original: Consider the reshaped high-level feature map X ∈ R C × HW of a deep network (the batch size is omitted for simplicity).
Revised: Consider the reshaped high-level feature map X ∈ R C × HW of a deep network (the batch size is omitted for simplicity).

Original: Here ’high-level feature’ denotes the feature map that carries rich semantics in the later layers of a network ( e.g., the output of Blockor Block 4 of a typical deep model like ResNet).
Revised: Here ’high-level feature’ denotes the feature map that carries rich semantics in the later layers of a network ( e.g., the output of Blockor Block 4 of a typical deep model like ResNet).

Original: Our RankFeat ﬁrst performs the Singular Value
Revised: Our RankFeat ﬁrst performs the Singular Value

Original: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:
Revised: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and
Revised: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and

Original: V ∈ R
Revised: V ∈ R

Original: HW × HW are left and right orthogonal singular vector matrices, respectively.
Revised: HW × HW are left and right orthogonal singular vector matrices, respectively.

Original: Then RankFeat removes the rank-1 matrix from the feature as:
Revised: Then RankFeat removes the rank-1 matrix from the feature as:

Original: [Equation]
Revised: [Equation]

Original: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singular vectors, respectively.
Revised: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singular vectors, respectively.

Original: The perturbed feature is fed into the rest of the network to generate the logit predictions y (cid:48) .
Revised: The perturbed feature is fed into the rest of the network to generate the logit predictions y (cid:48) .

Original: Finally, RankFeat computes the energy score of the logits for the input x as:
Revised: Finally, RankFeat computes the energy score of the logits for the input x as:

Original: [Equation]
Revised: [Equation]

Original: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-conﬁdent predictions of OOD data are largely perturbed.
Revised: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-conﬁdent predictions of OOD data are largely perturbed.

Original: In contrast, the decisions of ID data are mildly inﬂuenced.
Revised: In contrast, the decisions of ID data are mildly inﬂuenced.

Original: This could help to separate the ID and OOD data better in the logit space.
Revised: This could help to separate the ID and OOD data better in the logit space.

Original: Acceleration by Power Iteration.
Revised: Acceleration by Power Iteration.

Original: Since RankFeat only involves the dominant singular value and vectors, there is no need to compute the full SVD of the feature matrix.
Revised: Since RankFeat only involves the dominant singular value and vectors, there is no need to compute the full SVD of the feature matrix.

Original: Hence our method can be potentially accelerated by Power Iteration (PI).
Revised: Hence our method can be potentially accelerated by Power Iteration (PI).

Original: The PI algorithm is originally used to approximate the dominant eigenvector of a Hermitian matrix.
Revised: The PI algorithm is originally used to approximate the dominant eigenvector of a Hermitian matrix.

Original: With a slight modiﬁcation, it can also be applied to general rectangular matrices.
Revised: With a slight modiﬁcation, it can also be applied to general rectangular matrices.

Original: Given the feature X , the modiﬁed PI takes the coupled iterative update:
Revised: Given the feature X , the modiﬁed PI takes the coupled iterative update:

Original: [Equation]
Revised: [Equation]

Original: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and right singular vectors, respectively.
Revised: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and right singular vectors, respectively.

Original: After certain iterations, the dominant singular value is computed as s 1 = v Tk Xu k .
Revised: After certain iterations, the dominant singular value is computed as s 1 = v Tk Xu k .

Original: As will be illustrated in Sec.
Revised: As will be illustrated in Sec.

Original: 4.3, the approximate solution yielded by PI achieves very competitive performance against the SVD but with much less time overhead.
Revised: 4.3, the approximate solution yielded by PI achieves very competitive performance against the SVD but with much less time overhead.

Original: Combination of multi-scale features.
Revised: Combination of multi-scale features.

Original: Our RankFeat works at various later depths of a model, i.e.,
Revised: Our RankFeat works at various later depths of a model, i.e.,

Original: Block 3 and Block 4.
Revised: Block 3 and Block 4.

Original: Since intermediate features might focus on different semantic information, their decision cues are very likely to be different.
Revised: Since intermediate features might focus on different semantic information, their decision cues are very likely to be different.

Original: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for further performance improvements.
Revised: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for further performance improvements.

Original: Let y (cid:48) and y (cid:48)(cid:48) denote the logit predictions of Block 3 and Block 4 features, respectively.
Revised: Let y (cid:48) and y (cid:48)(cid:48) denote the logit predictions of Block 3 and Block 4 features, respectively.

Original: RankFeat performs the fusion at the logit space and computes the score function as log (cid:80) exp( ( y (cid:48)
Revised: RankFeat performs the fusion at the logit space and computes the score function as log (cid:80) exp( ( y (cid:48)

Original: + y (cid:48)(cid:48) ) / 2 ) .
Revised: + y (cid:48)(cid:48) ) / 2 ) .

Original: Different fusion strategies are explored and discussed in Sec. 4.3.
Revised: Different fusion strategies are explored and discussed in Sec. 4.3.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.
Revised: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.

Original: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper bound of RankFeat score more.
Revised: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Then based on Random Matrix Theory (RMT), we show that removing the rank-1 matrix makes the statistics of OOD features closer to random matrices.
Revised: Then based on Random Matrix Theory (RMT), we show that removing the rank-1 matrix makes the statistics of OOD features closer to random matrices.

Original: Finally, the theoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches work by optimizing the score upper bound determined by s 1 .
Revised: Finally, the theoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches work by optimizing the score upper bound determined by s 1 .

Original: ReAct manually uses a pre-deﬁned threshold to clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.
Revised: ReAct manually uses a pre-deﬁned threshold to clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.

Original: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.
Revised: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.

Original: For our RankFeat score function, we can express its upper bound in an analytical form.
Revised: For our RankFeat score function, we can express its upper bound in an analytical form.

Original: Moreover, the upper bound analysis explicitly indicates that removing the rank-1 matrix with a larger ﬁrst singular value would reduce the upper bound more.
Revised: Moreover, the upper bound analysis explicitly indicates that removing the rank-1 matrix with a larger ﬁrst singular value would reduce the upper bound more.

Original: Speciﬁcally, we have the following proposition.
Revised: Speciﬁcally, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The upper bound of RankFeat score is deﬁned as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni
Revised: The upper bound of RankFeat score is deﬁned as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni

Original: =

Revised: =

Original: s i − s 1 (cid:17) || W || ∞ + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are the weight and bias of the last layer, respectively.
Revised: s i − s 1 (cid:17) || W || ∞ + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are the weight and bias of the last layer, respectively.

Original: A larger s 1 would reduce the upper bound more.
Revised: A larger s 1 would reduce the upper bound more.

Original: Proof.
Revised: Proof.

Original: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation of rank-1 matrices X = (cid:80) s i u
Revised: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation of rank-1 matrices X = (cid:80) s i u

Original: i v Ti .
Revised: i v Ti .

Original: The feature perturbed by RankFeat can be computed as:
Revised: The feature perturbed by RankFeat can be computed as:

Original: [Equation]
Revised: [Equation]

Original: where N denotes the shorter side of the matrix (usually N = HW ).
Revised: where N denotes the shorter side of the matrix (usually N = HW ).

Original: In most deep models [19, 20], usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse the width and height dimensions.
Revised: In most deep models [19, 20], usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse the width and height dimensions.

Original: The GAP layer can be represented by a vector
Revised: The GAP layer can be represented by a vector

Original: [Equation]
Revised: [Equation]

Original: The pooled feature map is calculated as X (cid:48)
Revised: The pooled feature map is calculated as X (cid:48)

Original: m .
Revised: m .

Original: Then the output logits are computed by the matrixvector product with the classiﬁcation head as:
Revised: Then the output logits are computed by the matrixvector product with the classiﬁcation head as:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and
Revised: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and

Original: y (cid:48) ∈ R Q × 1 is the output logits that correspond to the perturbed feature X (cid:48) .
Revised: y (cid:48) ∈ R Q × 1 is the output logits that correspond to the perturbed feature X (cid:48) .

Original: Our RankFeat score is computed as:
Revised: Our RankFeat score is computed as:

Original: [Equation]
Revised: [Equation]

Original: where x is the input image, and Q denotes the number of categories.
Revised: where x is the input image, and Q denotes the number of categories.

Original: Here we choose Energy [40] as the base function due to its theoretical alignment with the input probability density and its strong empirical performance.
Revised: Here we choose Energy [40] as the base function due to its theoretical alignment with the input probability density and its strong empirical performance.

Original: Eq.
Revised: Eq.

Original: (9) can be re-formulated by the Log-Sum-Exp trick
Revised: (9) can be re-formulated by the Log-Sum-Exp trick

Original: [Equation]
Revised: [Equation]

Original: The above equation directly yields the tight bound as:
Revised: The above equation directly yields the tight bound as:

Original: [Equation]
Revised: [Equation]

Original: Since max( y (cid:48) ) ≤ max( | y (cid:48) | )
Revised: Since max( y (cid:48) ) ≤ max( | y (cid:48) | )

Original: =
Revised: =

Original: ||
Revised: ||

Original: y (cid:48)
Revised: y (cid:48)

Original: || ∞ , we have
Revised: || ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||
Revised: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||

Original: + || c
Revised: + || c

Original: || holds for any vectors a and c .
Revised: || holds for any vectors a and c .

Original: Moreover, since both u and v are orthogonal vectors, we have the relation ||
Revised: Moreover, since both u and v are orthogonal vectors, we have the relation ||

Original: u
Revised: u

Original: i || ∞
Revised: i || ∞

Original: ≤and || v i || ∞ ≤ 1 .
Revised: ≤and || v i || ∞ ≤ 1 .

Original: Relying on these two properties, injecting eq.
Revised: Relying on these two properties, injecting eq.

Original: (8) into eq.
Revised: (8) into eq.

Original: (12) leads to
Revised: (12) leads to

Original: RankFeat ( x ) <
Revised: RankFeat ( x ) <

Original: [Equation]
Revised: [Equation]

Original: Since m is a scaled all-ones vector, we have ||
Revised: Since m is a scaled all-ones vector, we have ||

Original: Wm || ∞
Revised: Wm || ∞

Original: =
Revised: =

Original: || W || ∞ / HW .
Revised: || W || ∞ / HW .

Original: The bound is simpliﬁed as:
Revised: The bound is simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.
Revised: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Remark:
Revised: Remark:

Original: Considering that OOD feature usually has a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.
Revised: Considering that OOD feature usually has a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.

Original: Notice that our bound analysis strives to improve the understanding of OOD methods from new perspectives instead of giving a strict guarantee of the score.
Revised: Notice that our bound analysis strives to improve the understanding of OOD methods from new perspectives instead of giving a strict guarantee of the score.

Original: For example, the upper bound can be used to explain the shrinkage and skew of score distributions in Fig.
Revised: For example, the upper bound can be used to explain the shrinkage and skew of score distributions in Fig.

Original: Subtracting s 1 would largely reduce the numerical range of both ID and OOD scores, which could squeeze score distributions.
Revised: Subtracting s 1 would largely reduce the numerical range of both ID and OOD scores, which could squeeze score distributions.

Original: Since the dominant singular value s 1 contributes most to the score, removing s 1 is likely to make many samples have similar scores.
Revised: Since the dominant singular value s 1 contributes most to the score, removing s 1 is likely to make many samples have similar scores.

Original: This would concentrate samples in a smaller region and further skew the distribution.
Revised: This would concentrate samples in a smaller region and further skew the distribution.

Original: Given that the OOD feature tends to have a much larger s 1 , this would have a greater impact on OOD data and skew the OOD score distribution more.
Revised: Given that the OOD feature tends to have a much larger s 1 , this would have a greater impact on OOD data and skew the OOD score distribution more.

Original: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.
Revised: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.

Original: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to random matrices.
Revised: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to random matrices.

Original: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.
Revised: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.

Original: For a random matrix of a given shape, the density of its eigenvalue asymptotically converges to the
Revised: For a random matrix of a given shape, the density of its eigenvalue asymptotically converges to the

Original: Manchenko-Pastur (MP) distribution [41, 50].
Revised: Manchenko-Pastur (MP) distribution [41, 50].

Original: Formally, we have:
Revised: Formally, we have:

Original: Theorem 1 (Manchenko-Pastur Law [41, 50]) .
Revised: Theorem 1 (Manchenko-Pastur Law [41, 50]) .

Original: Let X be a random matrix of shape t × n whose entries are random variables with E ( X ij = 0) and E ( X 2 ij = 1) .
Revised: Let X be a random matrix of shape t × n whose entries are random variables with E ( X ij = 0) and E ( X 2 ij = 1) .

Original: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ
Revised: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: for
Revised: for

Original: [Equation]
Revised: [Equation]

Original: [ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .
Revised: [ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .

Original: This theorem implies the possibility to measure the statistical distance between ID/OOD features and random matrices.
Revised: This theorem implies the possibility to measure the statistical distance between ID/OOD features and random matrices.

Original: To this end, we randomly sample 1 , 000 ID and OOD feature matrices and compute the KL divergence between the actual eigenvalue distribution and the ﬁtted MP distribution.
Revised: To this end, we randomly sample 1 , 000 ID and OOD feature matrices and compute the KL divergence between the actual eigenvalue distribution and the ﬁtted MP distribution.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of
Revised: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of

Original: Block 4 and Block 3 features, respectively.
Revised: Block 4 and Block 3 features, respectively.

Original: For the original feature, the OOD and ID feature matrices exhibit similar behaviors: the distances to the ﬁtted MP distribution are roughly the same ( diff.
Revised: For the original feature, the OOD and ID feature matrices exhibit similar behaviors: the distances to the ﬁtted MP distribution are roughly the same ( diff.

Original: ≈ 0 . 1 ).
Revised: ≈ 0 . 1 ).

Original: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the
Revised: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the

Original: KL divergence.
Revised: KL divergence.

Original: This indicates that removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, i.e., the OOD feature is very likely to become less informative than the ID feature.
Revised: This indicates that removing the rank-1 matrix makes the statistics of OOD feature closer to random matrices, i.e., the OOD feature is very likely to become less informative than the ID feature.

Original: The result partly explains the working mechanism of RankFeat : by removing the feature matrix where OOD data might convey more information than ID data, the two types of distributions have a larger discrepancy and therefore can be better separated.
Revised: The result partly explains the working mechanism of RankFeat : by removing the feature matrix where OOD data might convey more information than ID data, the two types of distributions have a larger discrepancy and therefore can be better separated.

Original: Connection with ReAct [52].
Revised: Connection with ReAct [52].

Original: ReAct clips the activations at the penultimate layer of a model to distinguish ID and OOD samples.
Revised: ReAct clips the activations at the penultimate layer of a model to distinguish ID and OOD samples.

Original: Given the feature X and the pooling layer m , the perturbation can be deﬁned as:
Revised: Given the feature X and the pooling layer m , the perturbation can be deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: (15) where τ is a pre-deﬁned threshold.
Revised: (15) where τ is a pre-deﬁned threshold.

Original: Their method shares some similarity with RankFeat formulation
Revised: Their method shares some similarity with RankFeat formulation

Original: Xm − s 1 u 1 v T 1 m .
Revised: Xm − s 1 u 1 v T 1 m .

Original: Both approaches subtract from the feature a portion of information that is most likely to cause the over-conﬁdence of OOD prediction.
Revised: Both approaches subtract from the feature a portion of information that is most likely to cause the over-conﬁdence of OOD prediction.

Original: ReAct selects the manually-deﬁned threshold τ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrix from the feature itself.
Revised: ReAct selects the manually-deﬁned threshold τ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrix from the feature itself.

Original: Taking a step further, ReAct has the score inequality following eq
Revised: Taking a step further, ReAct has the score inequality following eq

Original: [Equation]
Revised: [Equation]

Original: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .
Revised: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .

Original: Exploiting the vector norm inequality ||
Revised: Exploiting the vector norm inequality ||

Original: X
Revised: X

Original: || F ≥|| X
Revised: || F ≥|| X

Original: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .
Revised: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .

Original: Relying on this property, the above inequality can be re-formulated as:
Revised: Relying on this property, the above inequality can be re-formulated as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .
Revised: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .

Original: In contrast, the upper bound of our RankFeat can be expressed as:
Revised: In contrast, the upper bound of our RankFeat can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The upper bounds of both methods resemble each other with the only different term boxed.
Revised: The upper bounds of both methods resemble each other with the only different term boxed.

Original: From this point of view, both methods distinguish the ID and OOD data by eliminating the impact of the term containing s 1 in the upper bound.
Revised: From this point of view, both methods distinguish the ID and OOD data by eliminating the impact of the term containing s 1 in the upper bound.

Original: ReAct optimizes it by clipping the term with a manually-deﬁned threshold, which is indirect and might be sub-optimal.
Revised: ReAct optimizes it by clipping the term with a manually-deﬁned threshold, which is indirect and might be sub-optimal.

Original: Moreover, the threshold selection requires statistics of the whole ID set.
Revised: Moreover, the threshold selection requires statistics of the whole ID set.

Original: In contrast, our RankFeat does not require any extra data and directly subtracts this underlying term which is likely to cause the over-conﬁdence of OOD samples.
Revised: In contrast, our RankFeat does not require any extra data and directly subtracts this underlying term which is likely to cause the over-conﬁdence of OOD samples.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we ﬁrst discuss the setup in Sec. 4.1, and then present the main experimental results on ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.
Revised: In this section, we ﬁrst discuss the setup in Sec. 4.1, and then present the main experimental results on ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Datasets.
Revised: Datasets.

Original: In line with [26, 52, 27], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].
Revised: In line with [26, 52, 27], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].

Original: The large-scale dataset is more challenging than the traditional CIFAR benchmark [36] because the images are more realistic and diverse ( i.e., 1 .
Revised: The large-scale dataset is more challenging than the traditional CIFAR benchmark [36] because the images are more realistic and diverse ( i.e., 1 .

Original: 28 M images of 1 , 000 classes).
Revised: 28 M images of 1 , 000 classes).

Original: For the OOD datasets, we select four testsets from subsets of iNaturalist [58], SUN [63], Places [70], and Textures [5].
Revised: For the OOD datasets, we select four testsets from subsets of iNaturalist [58], SUN [63], Places [70], and Textures [5].

Original: These datasets are crafted by [26] with non-overlapping categories from ImageNet1k.
Revised: These datasets are crafted by [26] with non-overlapping categories from ImageNet1k.

Original: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of our approach on Species [24] and CIFAR [36] benchmark.
Revised: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of our approach on Species [24] and CIFAR [36] benchmark.

Original: (see Supplementary Material).
Revised: (see Supplementary Material).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with 6 recent post hoc OOD detection methods, namely MSP [22],
Revised: We compare our method with 6 recent post hoc OOD detection methods, namely MSP [22],

Original: ODIN [39], Energy [40], Mahalanobis [38], GradNorm [27], and ReAct [52].
Revised: ODIN [39], Energy [40], Mahalanobis [38], GradNorm [27], and ReAct [52].

Original: The detailed illustration and settings of these methods are kindly referred to Supplementary Material.
Revised: The detailed illustration and settings of these methods are kindly referred to Supplementary Material.

Original: Architectures.
Revised: Architectures.

Original: In line with [27], the main evaluation is done using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 [20].
Revised: In line with [27], the main evaluation is done using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 [20].

Original: We also evaluate the performance on
Revised: We also evaluate the performance on

Original: SqueezeNet [29], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-24 [67], a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.
Revised: SqueezeNet [29], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-24 [67], a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.

Original: For the implementation details and evaluation metrics, please refer to Supplementary Material .
Revised: For the implementation details and evaluation metrics, please refer to Supplementary Material .

Original: [Table]
Revised: [Table]

Original: 4.2 Results
Revised: 4.2 Results

Original: Main results.
Revised: Main results.

Original: Following [27], the main evaluation is conducted using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 architecture [20].
Revised: Following [27], the main evaluation is conducted using Google BiT-S model [35] pretrained on ImageNet-1k with ResNetv2-101 architecture [20].

Original: Table 2 compares the performance of all the post hoc methods.
Revised: Table 2 compares the performance of all the post hoc methods.

Original: For both Block 3 and Block 4 features, our RankFeat achieves the best evaluation results across datasets and metrics.
Revised: For both Block 3 and Block 4 features, our RankFeat achieves the best evaluation results across datasets and metrics.

Original: More speciﬁcally, RankFeat based on the Block 4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block 3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.
Revised: More speciﬁcally, RankFeat based on the Block 4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block 3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.

Original: Their combination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in the average AUROC.
Revised: Their combination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in the average AUROC.

Original: The superior performances at various depths demonstrate the effectiveness and general applicability of RankFeat .
Revised: The superior performances at various depths demonstrate the effectiveness and general applicability of RankFeat .

Original: The Block 3 feature has a higher AUROC but slightly falls behind the Block 4 feature in the FPR95, which can be considered a compromise between the two metrics.
Revised: The Block 3 feature has a higher AUROC but slightly falls behind the Block 4 feature in the FPR95, which can be considered a compromise between the two metrics.

Original: Our RankFeat is also effective on alternative CNN architectures.
Revised: Our RankFeat is also effective on alternative CNN architectures.

Original: Besides the experiment on
Revised: Besides the experiment on

Original: ResNetv2 [20], we also evaluate our method on SqueezeNet [29], an alternative tiny network suitable for mobile devices and on-chip applications.
Revised: ResNetv2 [20], we also evaluate our method on SqueezeNet [29], an alternative tiny network suitable for mobile devices and on-chip applications.

Original: This network is more challenging because the tiny network size makes the model prone to overﬁt the training data, which could increase the difﬁculty to distinguish between ID and OOD samples.
Revised: This network is more challenging because the tiny network size makes the model prone to overﬁt the training data, which could increase the difﬁculty to distinguish between ID and OOD samples.

Original: Table 3 top presents the performance of all the methods.
Revised: Table 3 top presents the performance of all the methods.

Original: Collectively, the performances of RankFeat are very competitive at both depths, as well as the score fusion.
Revised: Collectively, the performances of RankFeat are very competitive at both depths, as well as the score fusion.

Original: Our RankFeat achieves the state-of-the-art performances, outperforming the second-best baseline by 14.22% in FPR95 and by 7.48% in AUROC.
Revised: Our RankFeat achieves the state-of-the-art performances, outperforming the second-best baseline by 14.22% in FPR95 and by 7.48% in AUROC.

Original: Our RankFeat also suits transformer-based architectures.
Revised: Our RankFeat also suits transformer-based architectures.

Original: To further demonstrate the applicability of our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)
Revised: To further demonstrate the applicability of our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)

Original: [67], a popular transformer architecture that can achieve competitive performance against CNNs when trained from scratch.
Revised: [67], a popular transformer architecture that can achieve competitive performance against CNNs when trained from scratch.

Original: Similar to the CNN, RankFeat removes the rank-1 matrix from the ﬁnal token of T2T-ViT before the last normalization layer and the classiﬁcation head.
Revised: Similar to the CNN, RankFeat removes the rank-1 matrix from the ﬁnal token of T2T-ViT before the last normalization layer and the classiﬁcation head.

Original: Table 3 bottom compares the performance on T2T-ViT-24.
Revised: Table 3 bottom compares the performance on T2T-ViT-24.

Original: Our RankFeat outperforms the second-best method by 6.11% in FPR95 and by 2.05% in AUROC.
Revised: Our RankFeat outperforms the second-best method by 6.11% in FPR95 and by 2.05% in AUROC.

Original: Since the transformer models [9, 67] do not have increasing receptive ﬁelds like CNNs, we do not evaluate the performance at alternative network depths.
Revised: Since the transformer models [9, 67] do not have increasing receptive ﬁelds like CNNs, we do not evaluate the performance at alternative network depths.

Original: [Table]
Revised: [Table]

Original: Comparison against training-needed approaches.
Revised: Comparison against training-needed approaches.

Original: Since our method is post hoc , we only compare it with other post hoc baselines.
Revised: Since our method is post hoc , we only compare it with other post hoc baselines.

Original: MOS [26] and KL Matching [24] are not taken into account because
Revised: MOS [26] and KL Matching [24] are not taken into account because

Original: MOS needs extra training processes and KL Matching requires the labeled validation set to compute distributions for each class.
Revised: MOS needs extra training processes and KL Matching requires the labeled validation set to compute distributions for each class.

Original: Nonetheless, we note that our method can still hold an advantage against those approaches.
Revised: Nonetheless, we note that our method can still hold an advantage against those approaches.

Original: Table 4 presents the average FPR95 and AUROC on the ImageNet-1k benchmark.
Revised: Table 4 presents the average FPR95 and AUROC on the ImageNet-1k benchmark.

Original: Our RankFeat achieves the best performance without any extra training or validation set.
Revised: Our RankFeat achieves the best performance without any extra training or validation set.

Original: [Table]
Revised: [Table]

Original: 4.3 Ablation Studies
Revised: 4.3 Ablation Studies

Original: In this subsection
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.
Revised: we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.

Original: Unless explicitly speciﬁed, we apply RankFeat on the Block 4 feature by default.
Revised: Unless explicitly speciﬁed, we apply RankFeat on the Block 4 feature by default.

Original: [Table]
Revised: [Table]

Original: Removing Rank46.
Revised: Removing Rank46.

Original: Removing the rank-1 matrix outperforms keeping only it.
Revised: Removing the rank-1 matrix outperforms keeping only it.

Original: Instead of removing the rank-1 matrix, another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest of the matrix.
Revised: Instead of removing the rank-1 matrix, another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest of the matrix.

Original: Table 5 presents the evaluation results of keeping only the rank-1 matrix.
Revised: Table 5 presents the evaluation results of keeping only the rank-1 matrix.

Original: The performance falls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keeping only the rank-1 feature is inferior to removing it in distinguishing the two distributions.
Revised: The performance falls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keeping only the rank-1 feature is inferior to removing it in distinguishing the two distributions.

Original: Nonetheless, it is worth noting that even keeping only the rank-1 matrix achieves very competitive performance against previous best methods, such as GradNorm [27] and ReAct [52].
Revised: Nonetheless, it is worth noting that even keeping only the rank-1 matrix achieves very competitive performance against previous best methods, such as GradNorm [27] and ReAct [52].

Original: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).
Revised: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).

Original: We evaluate the impact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni
Revised: We evaluate the impact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni

Original: =

Revised: =

Original: s i
Revised: s i

Original: u
Revised: u

Original: i v Ti where n> 1 for the high-level feature X .
Revised: i v Ti where n> 1 for the high-level feature X .

Original: Table 5 compares the performance of removing the rank-2 matrix and rank-3 matrix.
Revised: Table 5 compares the performance of removing the rank-2 matrix and rank-3 matrix.

Original: When the rank of the removed matrix is increased, the average performance degrades accordingly.
Revised: When the rank of the removed matrix is increased, the average performance degrades accordingly.

Original: This demonstrates that removing the rank-1 matrix is the most effective approach to separate ID and OOD data.
Revised: This demonstrates that removing the rank-1 matrix is the most effective approach to separate ID and OOD data.

Original: This result is coherent with the ﬁnding in Fig.
Revised: This result is coherent with the ﬁnding in Fig.

Original: 1(a): only the largest singular value of OOD data is signiﬁcantly different from that of ID data.
Revised: 1(a): only the largest singular value of OOD data is signiﬁcantly different from that of ID data.

Original: Therefore, removing the rank-1 matrix achieves the best performance.
Revised: Therefore, removing the rank-1 matrix achieves the best performance.

Original: [Table]
Revised: [Table]

Original: Block 3 and Block 4 features are the most informative.
Revised: Block 3 and Block 4 features are the most informative.

Original: In addition to exploring the high-level features at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to features at shallow network layers.
Revised: In addition to exploring the high-level features at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to features at shallow network layers.

Original: As shown in Table 6, the performances of RankFeat at the Block 1 and
Revised: As shown in Table 6, the performances of RankFeat at the Block 1 and

Original: Block 2 features are not comparable to those at deeper layers.
Revised: Block 2 features are not comparable to those at deeper layers.

Original: This is mainly because the shallow low-level features do not embed as rich semantic information as the deep features.
Revised: This is mainly because the shallow low-level features do not embed as rich semantic information as the deep features.

Original: Consequently, removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.
Revised: Consequently, removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.

Original: [Table]
Revised: [Table]

Original: The approximate solution by PI yields competitive performances.
Revised: The approximate solution by PI yields competitive performances.

Original: Table 7 compares the time consumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods
Revised: Table 7 compares the time consumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods

Original: ReAct and GradNorm .
Revised: ReAct and GradNorm .

Original: The performance of PI starts to become competitive against that of SVD ( < 0 .
Revised: The performance of PI starts to become competitive against that of SVD ( < 0 .

Original: 1% ) from 20 iterations on with 48.41% time reduction.
Revised: 1% ) from 20 iterations on with 48.41% time reduction.

Original: Compared with ReAct , the PI-based
Revised: Compared with ReAct , the PI-based

Original: RankFeat only requires marginally 4 .
Revised: RankFeat only requires marginally 4 .

Original: 89% more time consumption.
Revised: 89% more time consumption.

Original: GradNorm is not comparable against other baselines in terms of time cost because it does not support the batch mode.
Revised: GradNorm is not comparable against other baselines in terms of time cost because it does not support the batch mode.

Original: Fusion at the logit space achieves the best performance.
Revised: Fusion at the logit space achieves the best performance.

Original: Fig.
Revised: Fig.

Original: 4 displays the performance of different fusion strategies in combining RankFeat at the Block 3 and Block 4 features.
Revised: 4 displays the performance of different fusion strategies in combining RankFeat at the Block 3 and Block 4 features.

Original: As can be observed, averaging logits outperforms other fusion strategies in most datasets and metrics.
Revised: As can be observed, averaging logits outperforms other fusion strategies in most datasets and metrics.

Original: This indicates that the fusing the logits can best coordinate the beneﬁt of both features.
Revised: This indicates that the fusing the logits can best coordinate the beneﬁt of both features.

Original:  Related Work
Revised:  Related Work

Original: Distribution shifts.
Revised: Distribution shifts.

Original: Distribution shifts have been a long-standing problem in the machine learning research community [17, 45, 34, 62].
Revised: Distribution shifts have been a long-standing problem in the machine learning research community [17, 45, 34, 62].

Original: The problem of distributions shifts can be generally categorized as shifts in the input space and shifts in the label space.
Revised: The problem of distributions shifts can be generally categorized as shifts in the input space and shifts in the label space.

Original: Shifts only in the input space are often deemed as covariate shifts [21, 44].
Revised: Shifts only in the input space are often deemed as covariate shifts [21, 44].

Original: In this setting, the inputs are corrupted by perturbations or shifted by domains, but the label space stays the same [25, 53].
Revised: In this setting, the inputs are corrupted by perturbations or shifted by domains, but the label space stays the same [25, 53].

Original: The aim is mainly to improve the robustness and generalization of a model [23].
Revised: The aim is mainly to improve the robustness and generalization of a model [23].

Original: For OOD detection, the labels are disjoint and the main concern is to determine whether a test sample should be predicted by the pre-trained model [39, 25].
Revised: For OOD detection, the labels are disjoint and the main concern is to determine whether a test sample should be predicted by the pre-trained model [39, 25].

Original: Some related sub-ﬁelds also tackle the problem of distribution shifts in the label space, such as novel class discovery [16, 69], open-set recognition [48, 59], and novelty detection [1, 56].
Revised: Some related sub-ﬁelds also tackle the problem of distribution shifts in the label space, such as novel class discovery [16, 69], open-set recognition [48, 59], and novelty detection [1, 56].

Original: These sub-ﬁelds target speciﬁc distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.
Revised: These sub-ﬁelds target speciﬁc distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.

Original: OOD detection with discriminative models.
Revised: OOD detection with discriminative models.

Original: The early work on discriminative OOD detection dates back to the classiﬁcation model with rejection option [4, 12].
Revised: The early work on discriminative OOD detection dates back to the classiﬁcation model with rejection option [4, 12].

Original: The OOD detection methods can be generally divided into training-need methods and post hoc approaches.
Revised: The OOD detection methods can be generally divided into training-need methods and post hoc approaches.

Original: Compared with trainingneeded approaches, post hoc methods do not require any extra training processes and could be directly applied to any pre-trained models.
Revised: Compared with trainingneeded approaches, post hoc methods do not require any extra training processes and could be directly applied to any pre-trained models.

Original: For the wide body of research on OOD detection, please refer to [65] for the comprehensive survey.
Revised: For the wide body of research on OOD detection, please refer to [65] for the comprehensive survey.

Original: Here we only highlight the representative post hoc methods.
Revised: Here we only highlight the representative post hoc methods.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: [43] ﬁrst observed the phenomenon that neural networks easily give over-conﬁdent predictions for OOD samples.
Revised: [43] ﬁrst observed the phenomenon that neural networks easily give over-conﬁdent predictions for OOD samples.

Original: The following researches attempted to improve the OOD uncertainty estimation by proposing ODIN score [39], OpenMax score [2], Mahalanobis distance [38], and
Revised: The following researches attempted to improve the OOD uncertainty estimation by proposing ODIN score [39], OpenMax score [2], Mahalanobis distance [38], and

Original: Energy score [40].
Revised: Energy score [40].

Original: Huang et al.
Revised: Huang et al.

Original: [26] pointed out that the traditional CIFAR benchmark does not extrapolate to real-world settings and proposed a large-scale ImageNet benchmark.
Revised: [26] pointed out that the traditional CIFAR benchmark does not extrapolate to real-world settings and proposed a large-scale ImageNet benchmark.

Original: More recently,
Revised: More recently,

Original: Sun et al.
Revised: Sun et al.

Original: [52] and Huang et al.
Revised: [52] and Huang et al.

Original: [27] proposed to tackle the challenge of OOD detection from the lens of activation abnormality and gradient norm, respectively.
Revised: [27] proposed to tackle the challenge of OOD detection from the lens of activation abnormality and gradient norm, respectively.

Original: In contrast, based on the empirical observation of singular value distributions, we propose a simple yet effective post hoc solution by removing the rank-1 subspace from the high-level features.
Revised: In contrast, based on the empirical observation of singular value distributions, we propose a simple yet effective post hoc solution by removing the rank-1 subspace from the high-level features.

Original: OOD detection with generative models.
Revised: OOD detection with generative models.

Original: Different from discriminative models, generative models detect the OOD samples by estimating the probability density function [32, 55, 47, 57, 8, 28, 3, 30].
Revised: Different from discriminative models, generative models detect the OOD samples by estimating the probability density function [32, 55, 47, 57, 8, 28, 3, 30].

Original: A sample with a low likelihood is deemed as OOD data.
Revised: A sample with a low likelihood is deemed as OOD data.

Original: Recently, a multitude of methods have utilized generative models for OOD detection [46, 51, 61, 64, 33, 49, 31].
Revised: Recently, a multitude of methods have utilized generative models for OOD detection [46, 51, 61, 64, 33, 49, 31].

Original: However, as pointed out in [42], generative models could assign a high likelihood to OOD data.
Revised: However, as pointed out in [42], generative models could assign a high likelihood to OOD data.

Original: Furthermore, generative models can be prohibitively harder to train and optimize than their discriminative counterparts, and the performance is often inferior.
Revised: Furthermore, generative models can be prohibitively harder to train and optimize than their discriminative counterparts, and the performance is often inferior.

Original: This might limit their practical usage.
Revised: This might limit their practical usage.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present RankFeat , a simple yet effective approach for post hoc OOD detection by removing the rank-1 matrix composed by the largest singular value from the high-level feature.
Revised: In this paper, we present RankFeat , a simple yet effective approach for post hoc OOD detection by removing the rank-1 matrix composed by the largest singular value from the high-level feature.

Original: We demonstrate its superior empirical results and the general applicability across architectures, network depths, and benchmarks.
Revised: We demonstrate its superior empirical results and the general applicability across architectures, network depths, and benchmarks.

Original: Extensive ablation studies and comprehensive theoretical analyses are conducted to reveal the important insights and to explain the working mechanism of our method.
Revised: Extensive ablation studies and comprehensive theoretical analyses are conducted to reveal the important insights and to explain the working mechanism of our method.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This research was supported by the EU H2020 projects AI4Media (No. 951911) and SPRING (No.
Revised: This research was supported by the EU H2020 projects AI4Media (No. 951911) and SPRING (No.

Original: We thank our colleague Zhun Zhong for the fruitful discussion and valuable suggestions.
Revised: We thank our colleague Zhun Zhong for the fruitful discussion and valuable suggestions.