Original: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection
Revised: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The task of out-of-distribution (OOD) detection is crucial for deploying machinelearning models in real-world settings.
Revised: The task of out-of-distribution (OOD) detection is crucial for deploying machinelearning models in real-world settings.

Original: In this paper, we observe that the singularvalue distributions of the in-distribution (ID) and OOD features are quite different:the OOD feature matrix tends to have a larger dominant singular value than the
Revised: In this paper, we observe that the singularvalue distributions of the in-distribution (ID) and OOD features are quite different:the OOD feature matrix tends to have a larger dominant singular value than the

Original: ID feature, and the class predictions of OOD samples are largely determined byit.
Revised: ID feature, and the class predictions of OOD samples are largely determined byit.

Original: Based on this observation, we assume that the first singular value of the OODfeature is larger than that of ID feature.
Revised: Based on this observation, we assume that the first singular value of the OODfeature is larger than that of ID feature.

Original: This motivates us to propose RankFeat ,a simple yet effective post hoc approach for OOD detection by removing therank-1 matrix composed of the largest singular value and the associated singularvectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).
Revised: This motivates us to propose RankFeat ,a simple yet effective post hoc approach for OOD detection by removing therank-1 matrix composed of the largest singular value and the associated singularvectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).

Original: RankFeat achieves thestate-of-the-art performance and reduces the average false positive rate (FPR95)
Revised: RankFeat achieves thestate-of-the-art performance and reduces the average false positive rate (FPR95)

Original: by17.90% compared with the previous best method.
Revised: by17.90% compared with the previous best method.

Original: Extensive ablation studies andcomprehensive theoretical analyses are presented to support the empirical results.
Revised: Extensive ablation studies andcomprehensive theoretical analyses are presented to support the empirical results.

Original:  Introduction
Revised:  Introduction

Original: In the real-world applications of deep learning, understanding whether a test sample belongs to thesame distribution of training data is critical to the safe deployment of machine learning models.
Revised: In the real-world applications of deep learning, understanding whether a test sample belongs to thesame distribution of training data is critical to the safe deployment of machine learning models.

Original: Themain challenge stems from the fact that current deep learning models can easily give over-confidentpredictions for out-of-distribution (OOD) data [42].
Revised: Themain challenge stems from the fact that current deep learning models can easily give over-confidentpredictions for out-of-distribution (OOD) data [42].

Original: Recently a rich line of literature has emerged toaddress the challenge of OOD detection [59, 26, 3, 7, 51, 14, 11, 53, 65, 36, 13, 67, 10, 15, 18].
Revised: Recently a rich line of literature has emerged toaddress the challenge of OOD detection [59, 26, 3, 7, 51, 14, 11, 53, 65, 36, 13, 67, 10, 15, 18].

Original: Previous OOD detection approaches either rely on the feature distance [37], activation abnormality [51] or gradient norm [26].
Revised: Previous OOD detection approaches either rely on the feature distance [37], activation abnormality [51] or gradient norm [26].

Original: In this paper, we tackle the problem of OOD detection from anotherperspective: by analyzing the spectrum of the high-level feature matrices ( e.g., the output of Block 3or Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite differentsingular value distributions for the in-distribution (ID) and OOD  (see Fig. 1(a)).
Revised: In this paper, we tackle the problem of OOD detection from anotherperspective: by analyzing the spectrum of the high-level feature matrices ( e.g., the output of Block 3or Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite differentsingular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)).

Original: The OOD featuretends to have a much larger dominant singular value than the ID feature, whereas the magnitudes ofthe rest singular values are very similar.
Revised: The OOD featuretends to have a much larger dominant singular value than the ID feature, whereas the magnitudes ofthe rest singular values are very similar.

Original: This implies that the OOD feature allocates more energyalong with the dominant singular vector direction.
Revised: This implies that the OOD feature allocates more energyalong with the dominant singular vector direction.

Original: The peculiar behavior motivates us to removethe rank-1 matrix composed of the dominant singular value and singular vectors from the feature.
Revised: The peculiar behavior motivates us to removethe rank-1 matrix composed of the dominant singular value and singular vectors from the feature.

Original: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of
Revised: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of

Original: OOD samples; a majority of predictions have been changed.
Revised: OOD samples; a majority of predictions have been changed.

Original: On the contrary, most ID samples haveconsistent classification results before and after removing the subspace.
Revised: On the contrary, most ID samples haveconsistent classification results before and after removing the subspace.

Original: This phenomenon indicatesthat the over-confident prediction of OOD samples might be largely determined by the dominantsingular value and the corresponding singular vectors.
Revised: This phenomenon indicatesthat the over-confident prediction of OOD samples might be largely determined by the dominantsingular value and the corresponding singular vectors.

Original: Based on this observation, we assume that the first singular value of OOD feature tends to be muchlarger than that of ID feature.
Revised: Based on this observation, we assume that the first singular value of OOD feature tends to be muchlarger than that of ID feature.

Original: The intuition behind is that the OOD feature corresponds to a larger
Revised: The intuition behind is that the OOD feature corresponds to a larger

Original: PCA explained variance ratio, and the well-trained network weights might causeand amplify the
Revised: PCA explained variance ratio (less informative), and the well-trained network weights might cause

Original: After removing the rank-1 matrix composed by the dominant singular value and singular vectors, the class predictions of OOD data are severely perturbed, while those of ID data are moderately influenced.
Revised: After removing the rank-1 matrix composed by the dominant singular value and singular vectors the class predictions of OOD data are severely perturbed, while those of ID data are moderately influenced.

Original: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.
Revised: This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.

Original: In light of this finding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In light of this finding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Both observations also hold for other OOD datasets  difference (see Sec.
Revised: Both observations also hold for other OOD datasets and amplify the difference (see Sec.

Original: E of the supplementary for the detailed illustration).
Revised: E of the supplementary for the detailed illustration).

Original: Hence, we conjecturethat leveraging this gap might help to better distinguish ID and OOD samples.
Revised: Hence, weconjecture that leveraging this gap might help to better distinguish ID and OOD samples.

Original: To this end, wepropose RankFeat , a simple but effective post hoc approach for OOD detection.
Revised: To thisend, we propose RankFeat , a simple but effective post hoc approach for OOD detection.

Original: RankFeat perturbsthe high-level feature by removing its rank-1 matrix composed of the dominant singular value andvectors.
Revised: RankFeatperturbs the high-level feature by removing its rank-1 matrix composed of the dominant singularvalue and vectors.

Original: Then the logits derived from the perturbed features are used to compute the OOD scorefunction.
Revised: Then the logits derived from the perturbed features are used to compute the OODscore function.

Original: By removing the rank-1 feature, the over-confidence of OOD samples is mitigated, andconsequently the ID and OOD data can be better distinguished (see Fig. 2).
Revised: By removing the rank-1 feature, the over-confidence of OOD samples is mitigated,and consequently the ID and OOD data can be better distinguished (see Fig. 2).

Original: Our RankFeat establishesthe state-of-the-art performance on the large-scale ImageNet benchmark and a suite of widely used OOD datasets across different network depths and architectures.
Revised: Our RankFeatestablishes the state-of-the-art performance on the large-scale ImageNet benchmark and a suite ofwidely used OOD datasets across different network depths and architectures.

Original: In particular, RankFeat outperformsthe previous best method by 17.90% in the average false positive rate (FPR95) and by 5.44% in thearea under curve (AUROC).
Revised: In particular, RankFeatoutperforms the previous best method by 17.90% in the average false positive rate (FPR95) andby 5.44% in the area under curve (AUROC).

Original: Extensive ablation studies are performed to reveal important insights of RankFeat , and comprehensive theoretical analyses are conducted to explain the working mechanism.
Revised: Extensive ablation studies are performed to revealimportant insights of RankFeat , and comprehensive theoretical analyses are conducted to explainthe working mechanism.

Original: Code will be publicly available.
Revised: Code will be publicly available.

Original: The key results and main contributions are threefold:
Revised: The key results and main contributions are threefold:

Original: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection byremoving the rank-1 matrix from the high-level feature.
Revised: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection byremoving the rank-1 matrix from the high-level feature.

Original: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90%and improving the average AUROC by 5.44% compared to the previous best method. • We perform extensive ablation studies to illustrate the impact of (1) removing or keepingthe rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat atvarious network depths, (4) the number of iterations to iteratively derive the approximaterank-1 matrix for acceleration but without performance degradation, and (5) different fusionstrategies to combine multi-scale features for further performance improvements.
Revised: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90%and improving the average AUROC by 5.44% compared to the previous best method.• We perform extensive ablation studies to illustrate the impact of (1) removing or keepingthe rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat atvarious network depths, (4) the number of iterations to iteratively derive the approximaterank-1 matrix for acceleration but without performance degradation, and (5) different fusionstrategies to combine multi-scale features for further performance improvements.

Original: • Comprehensive theoretical analyses are conducted to explain the working mechanism and to underpin the superior empirical results.
Revised: • Comprehensive theoretical analyses are conducted to explain the working mechanism andto underpin the superior empirical results.

Original: We show that (1) removing the rank-1 matrix reduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the
Revised: We show that (1) removing the rank-1 matrixreduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the

Original: statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [51]work by optimizing the upper bound containing the largest singular value.
Revised: statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [51]work by optimizing the upper bound containing the largest singular value.

Original: ReAct [51]indirectly and manually clips the underlying term, while RankFeat directly subtracts it.
Revised: ReAct [51]indirectly and manually clips the underlying term, while RankFeat directly subtracts it.

Original:  RankFeat: Rank-1 Feature Removal for OOD Detection
Revised:  RankFeat: Rank-1 Feature Removal for OOD Detection

Original: In this section, we introduce the background of OOD detection task and our proposed RankFeat thatperforms the OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In this section, we introduce the background of OOD detection task and our proposed RankFeat thatperforms the OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Preliminary: OOD detection.
Revised: Preliminary: OOD detection.

Original: The OOD detection is often formulated as a binary classificationproblem with the goal to distinguish between ID and OOD data.
Revised: The OOD detection is often formulated as a binary classificationproblem with the goal to distinguish between ID and OOD data.

Original: Let f denote a model trained onsamples from the ID data D in .
Revised: Let f denote a model trained onsamples from the ID data D in .

Original: For the unseen OOD data D out at test time, OOD detection aims todefine a decision function G ( · ) :
Revised: For the unseen OOD data D out at test time, OOD detection aims todefine a decision function G ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function,and γ is a chosen threshold to make a large portion of ID data correctly classified ( e.g., 95% ).
Revised: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function,and γ is a chosen threshold to make a large portion of ID data correctly classified ( e.g., 95% ).

Original: Thedifficulty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the scoredistributions of ID and OOD data overlap as little as possible.
Revised: Thedifficulty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the scoredistributions of ID and OOD data overlap as little as possible.

Original: RankFeat: rank-1 feature removal.
Revised: RankFeat: rank-1 feature removal.

Original: Consider the reshaped high-level feature map X ∈ R C ×
Revised: Consider the reshaped high-level feature map X ∈ R C ×

Original: HWof a deep network (the batch size is omitted for simplicity).
Revised: HWof a deep network (the batch size is omitted for simplicity).

Original: Here ’high-level feature’ denotes thefeature map that carries rich semantics in the deeper layers of a deep model ( e.g., the output of Block3 or Block 4 of a typical deep model like ResNet).
Revised: Here ’high-level feature’ denotes thefeature map that carries rich semantics in the deeper layers of a deep model ( e.g., the output of Block3 or Block 4 of a typical deep model like ResNet).

Original: Our RankFeat first performs the Singular Value
Revised: Our RankFeat first performs the Singular Value

Original: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:
Revised: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and
Revised: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and

Original: V ∈ R
Revised: V ∈ R

Original: HW × HW are left and right orthogonal singular vector matrices, respectively.
Revised: HW × HW are left and right orthogonal singular vector matrices, respectively.

Original: Then RankFeatremoves the rank-1 matrix from the feature as:
Revised: Then RankFeatremoves the rank-1 matrix from the feature as:

Original: [Equation]
Revised: [Equation]

Original: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singularvectors, respectively.
Revised: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singularvectors, respectively.

Original: The perturbed feature is fed into the rest of the network to generate the logitpredictions y ′ .
Revised: The perturbed feature is fed into the rest of the network to generate the logitpredictions y ′ .

Original: Finally, RankFeat computes the energy score of the logits for the input x as:
Revised: Finally, RankFeat computes the energy score of the logits for the input x as:

Original: [Equation]
Revised: [Equation]

Original: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-confidentpredictions of OOD data are largely perturbed.
Revised: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-confidentpredictions of OOD data are largely perturbed.

Original: In contrast, the decisions of ID data are mildlyinfluenced.
Revised: In contrast, the decisions of ID data are mildlyinfluenced.

Original: This could help to separate the ID and OOD data better in the logit space.
Revised: This could help to separate the ID and OOD data better in the logit space.

Original: Acceleration by Power Iteration.
Revised: Acceleration by Power Iteration.

Original: Since RankFeat only involves the dominant singular value andvectors, there is no need to compute the full SVD of the feature matrix.
Revised: Since RankFeat only involves the dominant singular value andvectors, there is no need to compute the full SVD of the feature matrix.

Original: Hence our method can bepotentially accelerated by Power Iteration (PI).
Revised: Hence our method can bepotentially accelerated by Power Iteration (PI).

Original: The PI algorithm is originally used to approximatethe dominant eigenvector of a Hermitian matrix.
Revised: The PI algorithm is originally used to approximatethe dominant eigenvector of a Hermitian matrix.

Original: With a slight modification, it can also be applied togeneral rectangular matrices.
Revised: With a slight modification, it can also be applied togeneral rectangular matrices.

Original: Given the feature X , the modified PI takes the coupled iterative update:
Revised: Given the feature X , the modified PI takes the coupled iterative update:

Original: [Equation]
Revised: [Equation]

Original: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and rightsingular vectors, respectively.
Revised: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and rightsingular vectors, respectively.

Original: After certain iterations, the dominant singular value is computed ass 1 = v Tk Xu k .
Revised: After certain iterations, the dominant singular value is computed ass 1 = v Tk Xu k .

Original: As will be illustrated in Sec.
Revised: As will be illustrated in Sec.

Original: 4.3, the approximate solution yielded by PI achieves verycompetitive performance against the SVD but with less time overhead.
Revised: 4.3, the approximate solution yielded by PI achieves verycompetitive performance against the SVD but with less time overhead.

Original: Combination of multi-scale features.
Revised: Combination of multi-scale features.

Original: Our RankFeat works at various later depths of a model, i.e.,
Revised: Our RankFeat works at various later depths of a model, i.e.,

Original: Block 3 and Block 4.
Revised: Block 3 and Block 4.

Original: Since intermediate features might focus on different semantic information, thedecision cue is very likely to be different.
Revised: Since intermediate features might focus on different semantic information, thedecision cue is very likely to be different.

Original: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for better performance.
Revised: It is thus natural to consider fusing the scores to leverage the distinguishable information of both features for better performance.

Original: Let y ′ and y ′′ denote thelogit predictions of Block 3 and Block 4 features, respectively.
Revised: Let y ′ and y ′′ denote thelogit predictions of Block 3 and Block 4 features, respectively.

Original: RankFeat performs the fusion at thelogit space and computes the score function as log (cid:80) exp( ( y ′ + y ′′ ) / 2 ) .
Revised: RankFeat performs the fusion at thelogit space and computes the score function as log (cid:80) exp( ( y ′ + y ′′ ) / 2 ) .

Original: Different fusion strategies areexplored and discussed in Sec. 4.3.
Revised: Different fusion strategies areexplored and discussed in Sec. 4.3.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.
Revised: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.

Original: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper boundof RankFeat score more.
Revised: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper boundof RankFeat score more.

Original: Then based on Random Matrix Theory (RMT), we show that removingthe rank-1 matrix makes the statistics of OOD features closer to random matrices.
Revised: Then based on Random Matrix Theory (RMT), we show that removingthe rank-1 matrix makes the statistics of OOD features closer to random matrices.

Original: Finally, thetheoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches workby optimizing the score upper bound determined by s 1 .
Revised: Finally, thetheoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches workby optimizing the score upper bound determined by s 1 .

Original: ReAct manually uses a pre-defined thresholdto clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.
Revised: ReAct manually uses a pre-defined thresholdto clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.

Original: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.
Revised: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeat more.

Original: For our RankFeat score function, we can express its upper bound in an analytical form.
Revised: For our RankFeat score function, we can express its upper bound in an analytical form.

Original: Moreover,the upper bound analysis implies that removing the rank-1 matrix with a larger first singular valuewould reduce the upper bound more.
Revised: Moreover,the upper bound analysis implies that removing the rank-1 matrix with a larger first singular valuewould reduce the upper bound more.

Original: Specifically, we have the following proposition.
Revised: Specifically, we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: The upper bound of RankFeat score is defined as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni
Revised: The upper bound of RankFeat score is defined as RankFeat ( x ) < 1 HW (cid:16) (cid:80) Ni

Original: =

Revised: =

Original: s i −s 1 (cid:17)
Revised: s i −s 1 (cid:17)

Original: || W || ∞
Revised: || W || ∞

Original: + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are theweight and bias of the last layer, respectively.
Revised: + || b || ∞ + log( Q ) where Q denotes the number of categories, and W and b are theweight and bias of the last layer, respectively.

Original: A larger s i would reduce the upper bound more.
Revised: A larger s i would reduce the upper bound more.

Original: Proof.
Revised: Proof.

Original: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation ofrank-1 matrices X =
Revised: For the feature X ∈ R C × HW , its SVD USV T = X can be expressed as the summation ofrank-1 matrices X =

Original: (cid:80) s i
Revised: (cid:80) s i

Original: u i v Ti .
Revised: u i v Ti .

Original: The feature perturbed by RankFeat can be computed as:
Revised: The feature perturbed by RankFeat can be computed as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where N denotes the shorter side of the matrix (usually N = HW ).
Revised: where N denotes the shorter side of the matrix (usually N = HW ).

Original: In most deep models [19, 20],usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse thewidth and height dimensions.
Revised: In most deep models [19, 20],usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse thewidth and height dimensions.

Original: The GAP layer can be represented by a vector
Revised: The GAP layer can be represented by a vector

Original: [Equation]
Revised: [Equation]

Original: The pooled feature map is calculated as X ′ m .
Revised: The pooled feature map is calculated as X ′ m .

Original: Then the output logits are computed by the matrixvector product with the classification head as:
Revised: Then the output logits are computed by the matrixvector product with the classification head as:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and
Revised: where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and

Original: y ′ ∈
Revised: y ′ ∈

Original: R Q × 1 isthe output logits that correspond to the perturbed feature X ′ .
Revised: R Q × 1 isthe output logits that correspond to the perturbed feature X ′ .

Original: Our RankFeat score is computed as:
Revised: Our RankFeat score is computed as:

Original: [Equation]
Revised: [Equation]

Original: where x is the input image, and Q denotes the number of categories.
Revised: where x is the input image, and Q denotes the number of categories.

Original: Here we choose Energy [39] asthe base function due to its theoretical alignment with the input probability density and its strongempirical performance.
Revised: Here we choose Energy [39] asthe base function due to its theoretical alignment with the input probability density and its strongempirical performance.

Original: Eq.
Revised: Eq.

Original: (9) can be re-formulated by the Log-Sum-Exp trick
Revised: (9) can be re-formulated by the Log-Sum-Exp trick

Original: [Equation]
Revised: [Equation]

Original: The above equation directly yields the tight bound as:
Revised: The above equation directly yields the tight bound as:

Original: [Equation]
Revised: [Equation]

Original: Since max( y ′ ) ≤ max( | y ′ | )
Revised: Since max( y ′ ) ≤ max( | y ′ | )

Original: =
Revised: =

Original: ||
Revised: ||

Original: y ′
Revised: y ′

Original: || ∞ , we have
Revised: || ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||
Revised: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||

Original: + || c
Revised: + || c

Original: || holds for anyvectors a and c .
Revised: || holds for anyvectors a and c .

Original: Moreover, since both u and v are orthogonal vectors, we have the relation ||
Revised: Moreover, since both u and v are orthogonal vectors, we have the relation ||

Original: u
Revised: u

Original: i || ∞ ≤ 1and || v i || ∞ ≤ 1 .
Revised: i || ∞ ≤ 1and || v i || ∞ ≤ 1 .

Original: Relying on these two properties, injecting eq.
Revised: Relying on these two properties, injecting eq.

Original: (8) into eq.
Revised: (8) into eq.

Original: (12) leads to
Revised: (12) leads to

Original: RankFeat ( x ) <
Revised: RankFeat ( x ) <

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Since m is a scaled all-ones vector, we have ||
Revised: Since m is a scaled all-ones vector, we have ||

Original: Wm || ∞
Revised: Wm || ∞

Original: =
Revised: =

Original: || W || ∞ / HW .
Revised: || W || ∞ / HW .

Original: The bound is simplified as:
Revised: The bound is simplified as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.
Revised: As indicated above, removing a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Our bound explicitly indicates the impact of s 1 on the score.
Revised: Our bound explicitly indicates the impact of s 1 on the score.

Original: Considering that OOD feature usuallyhas a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.
Revised: Considering that OOD feature usuallyhas a much larger s 1 (see Fig. 1(a)), RankFeat would reduce the upper bound of OOD samples more.

Original: Notice that our bound analysis strives to improve the understanding of OOD methods from newperspectives instead of giving a strict guarantee of the score.
Revised: Notice that our bound analysis strives to improve the understanding of OOD methods from newperspectives instead of giving a strict guarantee of the score.

Original: For example, the upper bound can beused to explain the shrinkage and skew of score distributions in Fig.
Revised: For example, the upper bound can beused to explain the shrinkage and skew of score distributions in Fig.

Original: Subtracting s 1 would largelyreduce the numerical range of both ID and OOD scores, which could squeeze score distributions.
Revised: Subtracting s 1 would largelyreduce the numerical range of both ID and OOD scores, which could squeeze score distributions.

Original: Since the dominant singular value s 1 contributes the most to the score, removing s 1 is likely to makemany samples have similar scores.
Revised: Since the dominant singular value s 1 contributes the most to the score, removing s 1 is likely to makemany samples have similar scores.

Original: This would concentrate samples in a smaller region and furtherskew the distribution.
Revised: This would concentrate samples in a smaller region and furtherskew the distribution.

Original: Consider that the OOD feature tends to have a much larger s 1 .
Revised: Consider that the OOD feature tends to have a much larger s 1 .

Original: This wouldhave a greater impact on OOD data and skew the OOD score distribution more.
Revised: This wouldhave a greater impact on OOD data and skew the OOD score distribution more.

Original: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.
Revised: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.

Original: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to randommatrices.
Revised: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to randommatrices.

Original: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.
Revised: Now we turn to use RMT to analyze the statistics of OOD and ID feature matrices.

Original: Fora random matrix of a given shape, the density of its eigenvalue asymptotically converges to the
Revised: Fora random matrix of a given shape, the density of its eigenvalue asymptotically converges to the

Original: Manchenko-Pastur (MP) distribution [40, 49].
Revised: Manchenko-Pastur (MP) distribution [40, 49].

Original: Formally, we have:
Revised: Formally, we have:

Original: Theorem 1 (Manchenko-Pastur Law [40, 49]) .
Revised: Theorem 1 (Manchenko-Pastur Law [40, 49]) .

Original: Let X be a random matrix of shape t × n whose entriesare random variables with E ( X ij = 0) and E ( X 2 ij = 1) .
Revised: Let X be a random matrix of shape t × n whose entriesare random variables with E ( X ij = 0) and E ( X 2 ij = 1) .

Original: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ ) = tn
Revised: Then the eigenvalues of the sample covariance Y = 1 n XX T converges to the probability density function: ρ ( λ ) = tn

Original: [Equation]
Revised: [Equation]

Original: for λ ∈[ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .
Revised: for λ ∈[ λ − , λ + ] where λ − = σ 2 (1 − (cid:112) nt ) 2 and λ + = σ 2 (1 + (cid:112) nt ) 2 .

Original: This theorem indicates the possibility to measure the statistical distance between ID/OOD matrixand random matrices.
Revised: This theorem indicates the possibility to measure the statistical distance between ID/OOD matrixand random matrices.

Original: To this end, we randomly sample 1 , 000 ID and OOD feature matrices andcompute the KL divergence between the actual eigenvalue distribution and the fitted MP distribution.
Revised: To this end, we randomly sample 1 , 000 ID and OOD feature matrices andcompute the KL divergence between the actual eigenvalue distribution and the fitted MP distribution.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of
Revised: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of

Original: Block 4 and Block 3 features, respectively.
Revised: Block 4 and Block 3 features, respectively.

Original: For the original feature, the OOD and ID feature matricesexhibit similar behaviors: the distances to the fitted MP distribution are roughly the same ( diff.
Revised: For the original feature, the OOD and ID feature matricesexhibit similar behaviors: the distances to the fitted MP distribution are roughly the same ( diff.

Original: ≈ 0 . 1 ).
Revised: ≈ 0 . 1 ).

Original: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the
Revised: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the

Original: KL divergence.
Revised: KL divergence.

Original: This indicates that removing the rank-1 matrix makes the statistics of OOD featurecloser to random matrices, i.e., the OOD feature is very likely to become less informative than the IDfeature.
Revised: This indicates that removing the rank-1 matrix makes the statistics of OOD featurecloser to random matrices, i.e., the OOD feature is very likely to become less informative than the IDfeature.

Original: The result partly explains the working mechanism of RankFeat : by removing the featurematrix where OOD data might convey more information than ID data, the two types of distributionshave a larger discrepancy and therefore can be better separated.
Revised: The result partly explains the working mechanism of RankFeat : by removing the featurematrix where OOD data might convey more information than ID data, the two types of distributionshave a larger discrepancy and therefore can be better separated.

Original: Connection with ReAct [51].
Revised: Connection with ReAct [51].

Original: ReAct clips the activations at the penultimate layer of a model todistinguish ID and OOD samples.
Revised: ReAct clips the activations at the penultimate layer of a model todistinguish ID and OOD samples.

Original: Given the feature X and the pooling layer m , the perturbation canbe defined as:
Revised: Given the feature X and the pooling layer m , the perturbation canbe defined as:

Original: [Equation]
Revised: [Equation]

Original: 15) where τ is a pre-defined threshold.
Revised: 15) where τ is a pre-defined threshold.

Original: Their method shares some similarity with RankFeat formulation
Revised: Their method shares some similarity with RankFeat formulation

Original: Xm − s 1 u 1 v T 1 m .
Revised: Xm − s 1 u 1 v T 1 m .

Original: Both approaches subtract from the feature a portion of information that is mostlikely to cause the over-confidence of OOD prediction.
Revised: Both approaches subtract from the feature a portion of information that is mostlikely to cause the over-confidence of OOD prediction.

Original: ReAct selects the manually-defined thresholdτ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrixfrom the feature itself.
Revised: ReAct selects the manually-defined thresholdτ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrixfrom the feature itself.

Original: Taking a step further, ReAct has the score inequality following eq.
Revised: Taking a step further, ReAct has the score inequality following eq.

Original: [Equation]
Revised: [Equation]

Original: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .
Revised: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .

Original: Exploiting the vectornorm inequality ||
Revised: Exploiting the vectornorm inequality ||

Original: X
Revised: X

Original: || F ≥|| X
Revised: || F ≥|| X

Original: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .
Revised: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .

Original: Relying on this property,the above inequality can be re-formulated as:
Revised: Relying on this property,the above inequality can be re-formulated as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .
Revised: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .

Original: Incontrast, the upper bound of our RankFeat can be expressed as:
Revised: Incontrast, the upper bound of our RankFeat can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The upper bounds of both methods resemble each other with the only different term boxed.
Revised: The upper bounds of both methods resemble each other with the only different term boxed.

Original: From thispoint of view, both methods distinguish the ID and OOD data by eliminating the impact of the termcontaining s 1 in the upper bound.
Revised: From thispoint of view, both methods distinguish the ID and OOD data by eliminating the impact of the termcontaining s 1 in the upper bound.

Original: ReAct optimizes it by clipping the term with a manually-definedthreshold, which is indirect and might be sub-optimal.
Revised: ReAct optimizes it by clipping the term with a manually-definedthreshold, which is indirect and might be sub-optimal.

Original: Moreover, the threshold selection requiresstatistics of the whole ID set.
Revised: Moreover, the threshold selection requiresstatistics of the whole ID set.

Original: In contrast, our RankFeat does not require any extra data and directlysubtracts this underlying term which is likely to cause the over-confidence of OOD samples.
Revised: In contrast, our RankFeat does not require any extra data and directlysubtracts this underlying term which is likely to cause the over-confidence of OOD samples.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we first discuss the setup in Sec. 4.1, and then present the main experimental resultson ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.
Revised: In this section, we first discuss the setup in Sec. 4.1, and then present the main experimental resultson ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Datasets.
Revised: Datasets.

Original: In line with [25, 51, 26], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].
Revised: In line with [25, 51, 26], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].

Original: The large-scale dataset is more challenging than the traditional CIFAR benchmark [35] because the images are more realistic and diverse ( i.e., 1 .
Revised: The large-scale dataset is more challenging than the traditional CIFAR benchmark [35] because the images are more realistic and diverse ( i.e., 1 .

Original: 28 M images of 1 , 000 classes).
Revised: 28 M images of 1 , 000 classes).

Original: Forthe OOD datasets, we select four testsets from subsets of iNaturalist [57], SUN [62], Places [69],and Textures [5].
Revised: Forthe OOD datasets, we select four testsets from subsets of iNaturalist [57], SUN [62], Places [69],and Textures [5].

Original: These datasets are crafted by [25] with non-overlapping categories from ImageNet1k.
Revised: These datasets are crafted by [25] with non-overlapping categories from ImageNet1k.

Original: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of ourapproach with the CIFAR dataset as the ID dataset (see Supplementary Material).
Revised: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of ourapproach with the CIFAR dataset as the ID dataset (see Supplementary Material).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with 6 recent post hoc OOD detection methods, namely
Revised: We compare our method with 6 recent post hoc OOD detection methods, namely

Original: MSP [22], ODIN [38], Energy [39], Mahalanobis [37], GradNorm [26], and ReAct [51].
Revised: MSP [22], ODIN [38], Energy [39], Mahalanobis [37], GradNorm [26], and ReAct [51].

Original: Thedetailed illustration and settings of these methods are kindly referred to Supplementary Material.
Revised: Thedetailed illustration and settings of these methods are kindly referred to Supplementary Material.

Original: Architectures.
Revised: Architectures.

Original: In line with [26], the main evaluation is done using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 [20].
Revised: In line with [26], the main evaluation is done using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 [20].

Original: We also evaluate the performance on
Revised: We also evaluate the performance on

Original: SqueezeNet [28], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-

Revised: SqueezeNet [28], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-

Original: [66],a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.
Revised: [66],a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.

Original: For the implementation details and evaluation metrics, please refer to Supplementary Material .
Revised: For the implementation details and evaluation metrics, please refer to Supplementary Material .

Original: [Table]
Revised: [Table]

Original: 4.2 Results
Revised: 4.2 Results

Original: Main results.
Revised: Main results.

Original: Following [26], the main evaluation is conducted using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 architecture [20].
Revised: Following [26], the main evaluation is conducted using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 architecture [20].

Original: Table 2 compares the performanceof all the post hoc methods.
Revised: Table 2 compares the performanceof all the post hoc methods.

Original: For both Block 3 and Block 4 features, our RankFeat achieves thebest evaluation results across datasets and metrics.
Revised: For both Block 3 and Block 4 features, our RankFeat achieves thebest evaluation results across datasets and metrics.

Original: More specifically, RankFeat based on the Block4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.
Revised: More specifically, RankFeat based on the Block4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.

Original: Theircombination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in theaverage AUROC.
Revised: Theircombination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in theaverage AUROC.

Original: The superior performances at various depths demonstrate the effectiveness andgeneral applicability of RankFeat .
Revised: The superior performances at various depths demonstrate the effectiveness andgeneral applicability of RankFeat .

Original: The Block 3 feature has a higher AUROC but slightly falls behindthe
Revised: The Block 3 feature has a higher AUROC but slightly falls behindthe

Original: 4 feature in the FPR95, which can be considered a compromise between the two metrics.
Revised: 4 feature in the FPR95, which can be considered a compromise between the two metrics.

Original: [Table]
Revised: [Table]

Original: Our RankFeat is also effective on alternative CNN architectures.
Revised: Our RankFeat is also effective on alternative CNN architectures.

Original: Besides the experiment on
Revised: Besides the experiment on

Original: ResNetv2 [20], we also evaluate our method on SqueezeNet [28], an alternative tiny network suitablefor mobile devices and on-chip applications.
Revised: ResNetv2 [20], we also evaluate our method on SqueezeNet [28], an alternative tiny network suitablefor mobile devices and on-chip applications.

Original: This network is more challenging because the tinynetwork size makes the model prone to overfit the training data, which could increase the difficultyto distinguish between ID and OOD samples.
Revised: This network is more challenging because the tinynetwork size makes the model prone to overfit the training data, which could increase the difficultyto distinguish between ID and OOD samples.

Original: Table 3 presents the performance of all the methods.
Revised: Table 3 presents the performance of all the methods.

Original: Collectively, the performances of RankFeat are very competitive at both depths, as well as the scorefusion.
Revised: Collectively, the performances of RankFeat are very competitive at both depths, as well as the scorefusion.

Original: Our RankFeat achieves the state-of-the-art performances, outperforming the second-bestbaseline by 14.22% in FPR95 and by 7.48% in AUROC.
Revised: Our RankFeat achieves the state-of-the-art performances, outperforming the second-bestbaseline by 14.22% in FPR95 and by 7.48% in AUROC.

Original: Our RankFeat also suits transformer-based architectures.
Revised: Our RankFeat also suits transformer-based architectures.

Original: To further demonstrate the applicabilityof our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)
Revised: To further demonstrate the applicabilityof our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)

Original: [66],a popular transformer architecture that can achieve competitive performance against CNNs whentrained from scratch.
Revised: [66],a popular transformer architecture that can achieve competitive performance against CNNs whentrained from scratch.

Original: Similar to the CNN, RankFeat removes the rank-1 matrix from the finaltoken of T2T-ViT before the last normalization layer and the classification head.
Revised: Similar to the CNN, RankFeat removes the rank-1 matrix from the finaltoken of T2T-ViT before the last normalization layer and the classification head.

Original: Table 3 comparesthe performance on T2T-ViT-24.
Revised: Table 3 comparesthe performance on T2T-ViT-24.

Original: Our RankFeat outperforms the second-best method by 9.49%in FPR95 and by 3.68% in AUROC.
Revised: Our RankFeat outperforms the second-best method by 9.49%in FPR95 and by 3.68% in AUROC.

Original: Since the transformer models [9, 66] do not have increasingreceptive fields like CNNs, we do not evaluate the performance at alternative network depths.
Revised: Since the transformer models [9, 66] do not have increasingreceptive fields like CNNs, we do not evaluate the performance at alternative network depths.

Original: 4.3 Ablation Studies
Revised: 4.3 Ablation Studies

Original: In this subsection, we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.
Revised: In this subsection, we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.

Original: Unless explicitly specified, we apply RankFeat on the Block 4 feature by default.
Revised: Unless explicitly specified, we apply RankFeat on the Block 4 feature by default.

Original: [Table]
Revised: [Table]

Original: Removing the rank-1 matrix outperforms keeping only it.
Revised: Removing the rank-1 matrix outperforms keeping only it.

Original: Instead of removing the rank-1 matrix,another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest ofthe matrix.
Revised: Instead of removing the rank-1 matrix,another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest ofthe matrix.

Original: Table 4 presents the evaluation results of keeping only the rank-1 matrix.
Revised: Table 4 presents the evaluation results of keeping only the rank-1 matrix.

Original: The performancefalls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keepingonly the rank-1 feature is inferior to removing it in distinguishing the two distributions.
Revised: The performancefalls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keepingonly the rank-1 feature is inferior to removing it in distinguishing the two distributions.

Original: Nonetheless,it is worth noting that even keeping only the rank-1 matrix achieves very competitive performanceagainst previous best methods, such as GradNorm [26] and ReAct [51].
Revised: Nonetheless,it is worth noting that even keeping only the rank-1 matrix achieves very competitive performanceagainst previous best methods, such as GradNorm [26] and ReAct [51].

Original: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).
Revised: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).

Original: We evaluate theimpact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni
Revised: We evaluate theimpact of removing the matrix of a higher rank, i.e., performing X − (cid:80) ni

Original: =

Revised: =

Original: s i
Revised: s i

Original: u
Revised: u

Original: i v Ti where n> 1 forthe high-level feature X .
Revised: i v Ti where n> 1 forthe high-level feature X .

Original: Table 4 compares the performance of removing the rank-2 matrix andrank-3 matrix.
Revised: Table 4 compares the performance of removing the rank-2 matrix andrank-3 matrix.

Original: When the rank of the removed matrix is increased, the average performance degradesaccordingly.
Revised: When the rank of the removed matrix is increased, the average performance degradesaccordingly.

Original: This demonstrates that removing the rank-1 matrix is the most effective approach toseparate ID and OOD data.
Revised: This demonstrates that removing the rank-1 matrix is the most effective approach toseparate ID and OOD data.

Original: This result is coherent with the finding in Fig.
Revised: This result is coherent with the finding in Fig.

Original: 1(a): only the largestsingular value of OOD data is significantly different from that of ID data.
Revised: 1(a): only the largestsingular value of OOD data is significantly different from that of ID data.

Original: Therefore, removing therank-1 matrix achieves the best performance.
Revised: Therefore, removing therank-1 matrix achieves the best performance.

Original: [Table]
Revised: [Table]

Original: Block 3 and Block 4 features are the most informative.
Revised: Block 3 and Block 4 features are the most informative.

Original: In addition to exploring the high-levelfeatures at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to featuresat shallow network layers.
Revised: In addition to exploring the high-levelfeatures at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to featuresat shallow network layers.

Original: As shown in Table 5, the performances of RankFeat at the Block 1 and
Revised: As shown in Table 5, the performances of RankFeat at the Block 1 and

Original: Block 2 features are not comparable to those at deeper layers.
Revised: Block 2 features are not comparable to those at deeper layers.

Original: This is mainly because the shallowlow-level features do not embed as rich semantic information as the deep features.
Revised: This is mainly because the shallowlow-level features do not embed as rich semantic information as the deep features.

Original: Consequently,removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.
Revised: Consequently,removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.

Original: [Table]
Revised: [Table]

Original: The approximate solution by PI yields competitive performances.
Revised: The approximate solution by PI yields competitive performances.

Original: Table 6 compares the timeconsumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods
Revised: Table 6 compares the timeconsumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods

Original: ReAct and GradNorm .
Revised: ReAct and GradNorm .

Original: The performance of PI starts to become competitive against that of SVD
Revised: The performance of PI starts to become competitive against that of SVD

Original: ( < 0 .
Revised: ( < 0 .

Original: 1% ) from 20 iterations on with 48.41% time reduction.
Revised: 1% ) from 20 iterations on with 48.41% time reduction.

Original: Compared with ReAct , the PI-based
Revised: Compared with ReAct , the PI-based

Original: RankFeat only requires marginally 4 .
Revised: RankFeat only requires marginally 4 .

Original: 89% more time consumption.
Revised: 89% more time consumption.

Original: GradNorm is not comparableagainst other baselines in terms of time cost because it does not support the batch mode.
Revised: GradNorm is not comparableagainst other baselines in terms of time cost because it does not support the batch mode.

Original: Fusion at the logit space achieves the best performance.
Revised: Fusion at the logit space achieves the best performance.

Original: Fig.
Revised: Fig.

Original: 4 displays the performance ofdifferent fusion strategies in combining RankFeat at the Block 3 and Block 4 features.
Revised: 4 displays the performance ofdifferent fusion strategies in combining RankFeat at the Block 3 and Block 4 features.

Original: As can beobserved, averaging logits outperforms other fusion strategies in most datasets and metrics.
Revised: As can beobserved, averaging logits outperforms other fusion strategies in most datasets and metrics.

Original: Thisindicates that the fusing the logits can best coordinate the benefit of both features.
Revised: Thisindicates that the fusing the logits can best coordinate the benefit of both features.

Original:  Related Work
Revised:  Related Work

Original: Distribution shifts.
Revised: Distribution shifts.

Original: Distribution shifts have been a long-standing problem in the machine learningresearch community [17, 44, 33, 61].
Revised: Distribution shifts have been a long-standing problem in the machine learningresearch community [17, 44, 33, 61].

Original: The problem of distributions shifts can be generally categorizedas shifts in the input space and shifts in the label space.
Revised: The problem of distributions shifts can be generally categorizedas shifts in the input space and shifts in the label space.

Original: Shifts only in the input space are oftendeemed as covariate shifts [21, 43].
Revised: Shifts only in the input space are oftendeemed as covariate shifts [21, 43].

Original: In this setting, the inputs are corrupted by perturbations or shiftedby domains, but the label space stays the same [24, 52].
Revised: In this setting, the inputs are corrupted by perturbations or shiftedby domains, but the label space stays the same [24, 52].

Original: The aim is mainly to improve the robustnessand generalization of a model [23].
Revised: The aim is mainly to improve the robustnessand generalization of a model [23].

Original: For OOD detection, the labels are disjoint and the main concernis to determine whether a test sample should be predicted by the pre-trained model [38, 24].
Revised: For OOD detection, the labels are disjoint and the main concernis to determine whether a test sample should be predicted by the pre-trained model [38, 24].

Original: Some related sub-fields also tackle the problem of distribution shifts in the label space, such as novelclass discovery [16, 68], open-set recognition [47, 58], and novelty detection [1, 55].
Revised: Some related sub-fields also tackle the problem of distribution shifts in the label space, such as novelclass discovery [16, 68], open-set recognition [47, 58], and novelty detection [1, 55].

Original: These sub-fieldstarget specific distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.
Revised: These sub-fieldstarget specific distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.

Original: OOD detection with discriminative models.
Revised: OOD detection with discriminative models.

Original: The early work on discriminative OOD detection datesback to the classification model with rejection option [4, 12].
Revised: The early work on discriminative OOD detection datesback to the classification model with rejection option [4, 12].

Original: For the wide body of research on OODdetection, please refer to [64] for the comprehensive survey.
Revised: For the wide body of research on OODdetection, please refer to [64] for the comprehensive survey.

Original: Here we highlight the representative posthoc methods that do not require extra training processes and can be directly applied to any pre-trainedmodels.
Revised: Here we highlight the representative posthoc methods that do not require extra training processes and can be directly applied to any pre-trainedmodels.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: [42] first observed the phenomenon that neural networks easily give overconfident predictions for OOD samples.
Revised: [42] first observed the phenomenon that neural networks easily give overconfident predictions for OOD samples.

Original: The following researches attempted to improve the OODuncertainty estimation by proposing ODIN score [38], OpenMax score [2], Mahalanobis distance [37],and Energy score [39].
Revised: The following researches attempted to improve the OODuncertainty estimation by proposing ODIN score [38], OpenMax score [2], Mahalanobis distance [37],and Energy score [39].

Original: Huang et al.
Revised: Huang et al.

Original: [25] pointed out that the traditional CIFAR benchmark does notextrapolate to real-world settings and proposed a large-scale ImageNet benchmark.
Revised: [25] pointed out that the traditional CIFAR benchmark does notextrapolate to real-world settings and proposed a large-scale ImageNet benchmark.

Original: More recently,
Revised: More recently,

Original: Sun et al.
Revised: Sun et al.

Original: [51] and Huang et al.
Revised: [51] and Huang et al.

Original: [26] proposed to tackle the challenge of OOD detection from thelens of activation abnormality and gradient norm, respectively.
Revised: [26] proposed to tackle the challenge of OOD detection from thelens of activation abnormality and gradient norm, respectively.

Original: In contrast, based on the empiricalobservation of singular value distributions, we propose a simple yet effective post hoc solution byremoving the rank-1 subspace from the high-level features.
Revised: In contrast, based on the empiricalobservation of singular value distributions, we propose a simple yet effective post hoc solution byremoving the rank-1 subspace from the high-level features.

Original: OOD detection with generative models.
Revised: OOD detection with generative models.

Original: Different from discriminative models, generative modelsdetect the OOD samples by estimating the probability density function [31, 54, 46, 56, 8, 27, 3, 29].
Revised: Different from discriminative models, generative modelsdetect the OOD samples by estimating the probability density function [31, 54, 46, 56, 8, 27, 3, 29].

Original: A sample with a low likelihood is deemed as OOD data.
Revised: A sample with a low likelihood is deemed as OOD data.

Original: Recently, a multitude of methods haveutilized generative models for OOD detection [45, 50, 60, 63, 32, 48, 30].
Revised: Recently, a multitude of methods haveutilized generative models for OOD detection [45, 50, 60, 63, 32, 48, 30].

Original: However, as pointed outin [41], generative models could assign a high likelihood to OOD data.
Revised: However, as pointed outin [41], generative models could assign a high likelihood to OOD data.

Original: Furthermore, generativemodels can be prohibitively harder to train and optimize than their discriminative counterparts, andthe performance is often inferior.
Revised: Furthermore, generativemodels can be prohibitively harder to train and optimize than their discriminative counterparts, andthe performance is often inferior.

Original: This might limit their practical usage.
Revised: This might limit their practical usage.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present RankFeat , a simple yet effective approach for OOD detection by removingthe rank-1 matrix composed by the largest singular value from the high-level feature.
Revised: In this paper, we present RankFeat , a simple yet effective approach for OOD detection by removingthe rank-1 matrix composed by the largest singular value from the high-level feature.

Original: We demonstratethe superior empirical results and the general applicability of RankFeat across architectures, networkdepths, and benchmarks.
Revised: We demonstratethe superior empirical results and the general applicability of RankFeat across architectures, networkdepths, and benchmarks.

Original: Extensive ablation studies and comprehensive theoretical analyses areconducted to reveal the important insights and to explain the working mechanism of our method.
Revised: Extensive ablation studies and comprehensive theoretical analyses areconducted to reveal the important insights and to explain the working mechanism of our method.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We briefly discuss the slightly
Revised: [Yes] We briefly discuss the slightly

Original: higher time consumption of RankFeat in Sec. 4.3.
Revised: higher time consumption of RankFeat in Sec. 4.3.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do notanticipate any negative societal impacts of our work.
Revised: [No] We do notanticipate any negative societal impacts of our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A] Since our method is post hoc , it will not be influenced
Revised: [N/A] Since our method is post hoc , it will not be influenced

Original: by random seeds.
Revised: by random seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The used data do not contain this kind of
Revised: The used data do not contain this kind of

Original: information.
Revised: information.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]