Original: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection
Revised: RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The task of out-of-distribution (OOD) detection is crucial for deploying machinelearning models in real-world settings.
Revised: The task of out-of-distribution (OOD) detection is crucial for deploying machinelearning models in real-world settings.

Original: In this paper, we observe that the singularvalue distributions of the in-distribution (ID) and OOD features are quite different:the OOD feature matrix tends to have a larger dominant singular value than the IDfeature, and the class predictions of OOD samples are largely determined by thedominant singular value.
Revised: In this paper, we observe that the singularvalue distributions of the in-distribution (ID) and OOD features are quite different:the OOD feature matrix tends to have a larger dominant singular value than the ID feature, and the class predictions of OOD samples are largely determined byit.

Original: Based on this observation, we assume that the first singularvalue of the OOD feature is larger than that of ID feature.
Revised: Based on this observation, we assume that the first singular value of the OODfeature is larger than that of ID feature.

Original: We propose RankFeat ,a simple yet effective post hoc approach for OOD detection by removing therank-1 matrix composed of the largest singular value and the associated singularvectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).
Revised: This motivates us to propose RankFeat ,a simple yet effective post hoc approach for OOD detection by removing therank-1 matrix composed of the largest singular value and the associated singularvectors from the high-level feature ( i.e., X − s 1 u 1 v T 1 ).

Original: RankFeat achieves thestate-of-the-art performance and reduces the average false positive rate (FPR95)
Revised: RankFeat achieves thestate-of-the-art performance and reduces the average false positive rate (FPR95)

Original: by17.90% compared with the previous best method.
Revised: by17.90% compared with the previous best method.

Original: Extensive ablation studies andcomprehensive theoretical analyses are presented to support the empirical results.
Revised: Extensive ablation studies andcomprehensive theoretical analyses are presented to support the empirical results.

Original:  Introduction
Revised:  Introduction

Original: In the real-world applications of deep learning, understanding whether a test sample belongs to thesame distribution of training data is critical to the safe deployment of machine learning models.
Revised: In the real-world applications of deep learning, understanding whether a test sample belongs to thesame distribution of training data is critical to the safe deployment of machine learning models.

Original: Themain challenge stems from the fact that current deep learning models can easily give over-confidentpredictions for out-of-distribution (OOD) data [42].
Revised: Themain challenge stems from the fact that current deep learning models can easily give over-confidentpredictions for out-of-distribution (OOD) data [42].

Original: Recently a rich line of literature has emerged toaddress the challenge of OOD detection [59, 26, 3, 7, 51, 14, 11, 53, 65, 36, 13, 67, 10, 15, 18].
Revised: Recently a rich line of literature has emerged toaddress the challenge of OOD detection [59, 26, 3, 7, 51, 14, 11, 53, 65, 36, 13, 67, 10, 15, 18].

Original: Previous OOD detection approaches either rely on the feature distance [37], activation abnormality [51] or gradient norm [26].
Revised: Previous OOD detection approaches either rely on the feature distance [37], activation abnormality [51] or gradient norm [26].

Original: In this paper, we tackle the problem of OOD detection from anotherperspective: by analyzing the spectrum of the high-level feature matrices ( e.g., the output of Block 3or Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite differentsingular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)).
Revised: In this paper, we tackle the problem of OOD detection from anotherperspective: by analyzing the spectrum of the high-level feature matrices ( e.g., the output of Block 3or Block 4 of a typical ResNet [19] model), we observe that the feature matrices have quite differentsingular value distributions for the in-distribution (ID) and OOD data (see Fig. 1(a)).

Original: The OOD featuretends to have a much larger dominant singular value than the ID feature, whereas the magnitudes ofthe rest singular values are very similar.
Revised: The OOD featuretends to have a much larger dominant singular value than the ID feature, whereas the magnitudes ofthe rest singular values are very similar.

Original: This implies that the OOD feature allocates more energyalong with the dominant singular vector direction.
Revised: This implies that the OOD feature allocates more energyalong with the dominant singular vector direction.

Original: The peculiar behavior motivates us to removethe rank-1 matrix composed of the dominant singular value and singular vectors from the feature.
Revised: The peculiar behavior motivates us to removethe rank-1 matrix composed of the dominant singular value and singular vectors from the feature.

Original: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of
Revised: As displayed in Fig. 1(b), removing the rank-1 feature drastically perturbs the class prediction of

Original: OOD samples; a majority of predictions have been changed.
Revised: OOD samples; a majority of predictions have been changed.

Original: On the contrary, most ID samples haveconsistent classification results before and after removing the subspace.
Revised: On the contrary, most ID samples haveconsistent classification results before and after removing the subspace.

Original: This phenomenon indicatesthat the over-confident prediction of OOD samples might be largely determined by the dominantsingular value and the corresponding singular vectors.
Revised: This phenomenon indicatesthat the over-confident prediction of OOD samples might be largely determined by the dominantsingular value and the corresponding singular vectors.

Original: Enlightened by this finding, we assume that the first singular value of OOD feature is much largerthan that of ID feature.
Revised: Based on this observation, we assume that the first singular value of OOD feature tends to be muchlarger than that of ID feature.

Revised: Hence, we conjecture that leveraging this gap might help to betterdistinguish ID and OOD samples.

Original: Hence, we propose RankFeat , a simple but effective posthoc approach for
Revised: To this end, we propose RankFeat , a simple but effective post

Original: OOD detection.

Original: RankFeat perturbs the high-level feature by removing its rank-1 matrix composed removing the 1 matrix composed by the dominant singular  singular  the class predictions of OOD data are severely perturbed while those of ID data are moderately influenced the decisions of OOD data heavily depend  the  singular value  the corresponding singular vectors of the feature matrix
Revised: After removing the rank-1 matrix composed by the dominant singular value and singular vectors, the class predictions of OOD data are severely perturbed, while those of ID data are moderately influenced. This observation indicates that the decisions of OOD data heavily depend on the dominant singular value and the corresponding singular vectors of the feature matrix.

Original: In light of this finding we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature
Revised: In light of this finding, we get motivated to propose RankFeat for OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Both observations also hold for other OOD datasets of the dominant singular value and vectors.
Revised: Both observations also hold for other OOD datasets hoc approach for OOD detection.

Revised: RankFeat perturbs the high-level feature by removing its rank-1matrix composed of the dominant singular value and vectors.

Original: Then the logits derived from the perturbed features areused to compute the OOD score function.
Revised: Then the logits derived from theperturbed features are used to compute the OOD score function.

Original: By removing the rank-1 feature, the over-confidence of OOD samples is mitigated, and consequently the ID and OOD data can be better distinguished (see Fig. 2).
Revised: By removing the rank-1 feature,the over-confidence of OOD samples is mitigated, and consequently the ID and OOD data can bebetter distinguished (see Fig. 2).

Original: Our RankFeat establishes the state-of-the-art performance on the large-scale ImageNetbenchmark and a suite of widely used OOD datasets across different network depths and architectures.
Revised: Our RankFeat establishes the state-of-the-art performance on thelarge-scale ImageNet benchmark and a suite of widely used OOD datasets across different networkdepths and architectures.

Original: In particular, RankFeat outperforms the previous best method by 17.90% in the average falsepositive rate (FPR95) and by 5.44% in the area under curve (AUROC).
Revised: In particular, RankFeat outperforms the previous best method by 17.90%in the average false positive rate (FPR95) and by 5.44% in the area under curve (AUROC).

Original: Extensive ablation studiesare performed to reveal important insights of RankFeat , and comprehensive theoretical analyses areconducted to explain the working mechanism.
Revised: Extensive ablation studies are performed to reveal important insights of RankFeat , and comprehensivetheoretical analyses are conducted to explain the working mechanism.

Original: Code will be publicly available.
Revised: Code will be publicly available.

Original: The key results and main contributions are threefold:
Revised: The key results and main contributions are threefold:

Original: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection byremoving the rank-1 matrix from the high-level feature.
Revised: • We propose RankFeat , a simple yet effective post hoc approach for OOD detection byremoving the rank-1 matrix from the high-level feature.

Original: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90%and improving the average AUROC by 5.44% compared to the previous best method.• We perform extensive ablation studies to illustrate the impact of (1) removing or keepingthe rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat atvarious network depths, (4) the number of iterations to iteratively derive the approximaterank-1 matrix for acceleration but without performance degradation, and (5) different fusionstrategies to combine multi-scale features for further performance improvements.•
Revised: RankFeat achieves the state-of-theart performance across benchmarks and models, reducing the average FPR95 by 17.90%and improving the average AUROC by 5.44% compared to the previous best method. • We perform extensive ablation studies to illustrate the impact of (1) removing or keeping the rank-1 matrix, (2) removing the rank-n matrix (n>1), (3) applying our RankFeat atvarious network depths, (4) the number of iterations to iteratively derive the approximaterank-1 matrix for acceleration but without performance degradation, and (5) different fusionstrategies to combine multi-scale features for further performance improvements.•

Original: Comprehensive theoretical analyses are conducted to explain the working mechanism andto underpin the superior empirical results.
Revised: Comprehensive theoretical analyses are conducted to explain the working mechanism and to underpin the superior empirical results.

Original: We show that (1) removing the rank-1 matrixreduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the
Revised: We show that (1) removing the rank-1 matrix reduces the upper bound of OOD score more, (2) removing the rank-1 matrix makes the

Original: statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [51]work by optimizing the upper bound containing the largest singular value.
Revised: statistics of OOD feature closer to random matrices, and (3) both RankFeat and ReAct [51]work by optimizing the upper bound containing the largest singular value.

Original: ReAct [51]indirectly and manually clips the underlying term, while RankFeat directly subtracts it.
Revised: ReAct [51]indirectly and manually clips the underlying term, while RankFeat directly subtracts it.

Original:  RankFeat: Rank-1 Feature Removal for OOD Detection
Revised:  RankFeat: Rank-1 Feature Removal for OOD Detection

Original: In this section, we introduce the background of OOD detection task and our proposed RankFeat thatperforms the OOD detection by removing the rank-1 matrix from the high-level feature.
Revised: In this section, we introduce the background of OOD detection task and our proposed RankFeat thatperforms the OOD detection by removing the rank-1 matrix from the high-level feature.

Original: Preliminary: OOD detection.
Revised: Preliminary: OOD detection.

Original: The OOD detection is often formulated as a binary classificationproblem with the goal to distinguish between ID and OOD data.
Revised: The OOD detection is often formulated as a binary classificationproblem with the goal to distinguish between ID and OOD data.

Original: Let f denote a model trained onsamples from the ID data D in .
Revised: Let f denote a model trained onsamples from the ID data D in .

Original: For the unseen OOD data D out at test time, OOD detection aims todefine a decision function G ( · ) :
Revised: For the unseen OOD data D out at test time, OOD detection aims todefine a decision function G ( · ) :

Original: [Equation]
Revised: [Equation]

Original: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function,and γ is a chosen threshold to make a large portion of ID data correctly classified ( e.g., 95% ).
Revised: where x denotes the data encountered at the inference stage, S ( · ) is the seeking scoring function,and γ is a chosen threshold to make a large portion of ID data correctly classified ( e.g., 95% ).

Original: Thedifficulty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the scoredistributions of ID and OOD data overlap as little as possible.
Revised: Thedifficulty of OOD detection lies in designing an appropriate scoring function S ( · ) such that the scoredistributions of ID and OOD data overlap as little as possible.

Original: RankFeat: rank-1 feature removal.
Revised: RankFeat: rank-1 feature removal.

Original: Consider the reshaped high-level feature map X ∈ R C ×
Revised: Consider the reshaped high-level feature map X ∈ R C ×

Original: HWof a deep network (the batch size is omitted for simplicity).
Revised: HWof a deep network (the batch size is omitted for simplicity).

Original: Here ’high-level feature’ denotes thefeature map that carries rich semantics in the deeper layers of a deep model ( e.g., the output of Block3 or Block 4 of a typical deep model like ResNet).
Revised: Here ’high-level feature’ denotes thefeature map that carries rich semantics in the deeper layers of a deep model ( e.g., the output of Block3 or Block 4 of a typical deep model like ResNet).

Original: Our RankFeat first performs the Singular Value
Revised: Our RankFeat first performs the Singular Value

Original: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:
Revised: Decomposition (SVD) on each individual feature matrix in the mini-batch to decompose the feature:

Original: [Equation]
Revised: [Equation]

Original: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and
Revised: where S ∈ R C × HW is the rectangular diagonal singular value matrix, and U ∈ R C × C and

Original: V ∈ R
Revised: V ∈ R

Original: HW × HW are left and right orthogonal singular vector matrices, respectively.
Revised: HW × HW are left and right orthogonal singular vector matrices, respectively.

Original: Then RankFeatremoves the rank-1 matrix from the feature as:
Revised: Then RankFeatremoves the rank-1 matrix from the feature as:

Original: [Equation]
Revised: [Equation]

Original: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singularvectors, respectively.
Revised: where s 1 is the largest singular value, and u 1 and v 1 are the corresponding left and right singularvectors, respectively.

Original: The perturbed feature is fed into the rest of the network to generate the logitpredictions y ′ .
Revised: The perturbed feature is fed into the rest of the network to generate the logitpredictions y ′ .

Original: Finally, RankFeat computes the energy score of the logits for the input x as:
Revised: Finally, RankFeat computes the energy score of the logits for the input x as:

Original: [Equation]
Revised: [Equation]

Original: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-confidentpredictions of OOD data are largely perturbed.
Revised: By removing the rank-1 matrix composed by the dominant singular value s 1 , the over-confidentpredictions of OOD data are largely perturbed.

Original: In contrast, the decisions of ID data are mildlyinfluenced.
Revised: In contrast, the decisions of ID data are mildlyinfluenced.

Original: This could help to separate the ID and OOD data better in the logit space.
Revised: This could help to separate the ID and OOD data better in the logit space.

Original: Acceleration by Power Iteration.
Revised: Acceleration by Power Iteration.

Original: Since RankFeat only involves the dominant singular value andvectors, there is no need to compute the full SVD of the feature matrix.
Revised: Since RankFeat only involves the dominant singular value andvectors, there is no need to compute the full SVD of the feature matrix.

Original: Hence our method can bepotentially accelerated by Power Iteration (PI).
Revised: Hence our method can bepotentially accelerated by Power Iteration (PI).

Original: The PI algorithm is originally used to approximatethe dominant eigenvector of a Hermitian matrix.
Revised: The PI algorithm is originally used to approximatethe dominant eigenvector of a Hermitian matrix.

Original: With a slight modification, it can also be applied togeneral rectangular matrices.
Revised: With a slight modification, it can also be applied togeneral rectangular matrices.

Original: Given the feature X , the modified PI takes the coupled iterative update:
Revised: Given the feature X , the modified PI takes the coupled iterative update:

Original: [Equation]
Revised: [Equation]

Original: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and rightsingular vectors, respectively.
Revised: where u 0 and v 0 are initialized with random orthogonal vectors and converge to the left and rightsingular vectors, respectively.

Original: After certain iterations, the dominant singular value is computed ass 1 = v Tk Xu k .
Revised: After certain iterations, the dominant singular value is computed ass 1 = v Tk Xu k .

Original: As will be illustrated in Sec.
Revised: As will be illustrated in Sec.

Original: 4.3, the approximate solution yielded by PI achieves verycompetitive performance against the SVD but with less time overhead.
Revised: 4.3, the approximate solution yielded by PI achieves verycompetitive performance against the SVD but with less time overhead.

Original: Combination of multi-scale features.
Revised: Combination of multi-scale features.

Original: Our RankFeat works at various later depths of a model, i.e.,
Revised: Our RankFeat works at various later depths of a model, i.e.,

Original: Block 3 and Block 4.
Revised: Block 3 and Block 4.

Original: Since intermediate features might focus on different semantic information, thedecision cue is very likely to be different.
Revised: Since intermediate features might focus on different semantic information, thedecision cue is very likely to be different.

Original: It is thus natural to consider fusing the scores to leveragethe distinguishable information of both features for better performance.
Revised: It is thus natural to consider fusing the scores to leveragethe distinguishable information of both features for better performance.

Original: Let y ′ and y ′′ denote thelogit predictions of Block 3 and Block 4 features, respectively.
Revised: Let y ′ and y ′′ denote thelogit predictions of Block 3 and Block 4 features, respectively.

Original: RankFeat performs the fusion at thelogit space and computes the score function as log (cid:80) exp( ( y ′ + y ′′ ) / 2 ) .
Revised: RankFeat performs the fusion at thelogit space and computes the score function as log (cid:80) exp( ( y ′ + y ′′ ) / 2 ) .

Original: Different fusion strategies areexplored and discussed in Sec. 4.3.
Revised: Different fusion strategies areexplored and discussed in Sec. 4.3.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.
Revised: In this section, we perform some theoretical analyses on RankFeat to support the empirical results.

Original: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper boundof RankFeat score more.
Revised: We start by proving that removing the rank-1 feature with a larger s 1 would reduce the upper boundof RankFeat score more.

Original: Then based on Random Matrix Theory (RMT), we show that removingthe rank-1 matrix makes the statistics of OOD features closer to random matrices.
Revised: Then based on Random Matrix Theory (RMT), we show that removingthe rank-1 matrix makes the statistics of OOD features closer to random matrices.

Original: Finally, thetheoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches workby optimizing the score upper bound determined by s 1 .
Revised: Finally, thetheoretical connection of ReAct and our RankFeat is analyzed and discussed: both approaches workby optimizing the score upper bound determined by s 1 .

Original: ReAct manually uses a pre-defined thresholdto clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.
Revised: ReAct manually uses a pre-defined thresholdto clip the term with s 1 , whereas our RankFeat directly optimizes the bound by subtracting this term.

Original: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeatmore.
Revised: Removing the rank-1 matrix with a larger s 1 would reduce the upper bound of RankFeatmore.

Original: Consider a mini-batch of feature matrices X ∈ R B × C × HW .
Revised: Consider a mini-batch of feature matrices X ∈ R B × C × HW .

Original: The SVD of the batched matrices
Revised: The SVD of the batched matrices

Original: USV T =
Revised: USV T =

Original: X can be expressed as the summation of rank-1 matrices X = (cid:80) s i
Revised: X can be expressed as the summation of rank-1 matrices X = (cid:80) s i

Original: u i v Ti .
Revised: u i v Ti .

Original: The featureperturbed by RankFeat can be computed as:
Revised: The featureperturbed by RankFeat can be computed as:

Original: [Equation]
Revised: [Equation]

Original: where N denotes the shorter side of the matrix (usually N = HW ).
Revised: where N denotes the shorter side of the matrix (usually N = HW ).

Original: In most deep models [19, 20],usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse thewidth and height dimensions.
Revised: In most deep models [19, 20],usually the last feature map needs to pass a Global Average Pooling (GAP) layer to collapse thewidth and height dimensions.

Original: The GAP layer can be represented by a vector
Revised: The GAP layer can be represented by a vector

Original: [Equation]
Revised: [Equation]

Original: The pooled feature map is calculated as X ′ m .
Revised: The pooled feature map is calculated as X ′ m .

Original: Then the output logits are computed by the matrixvector product with the classification head as:
Revised: Then the output logits are computed by the matrixvector product with the classification head as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and
Revised: (8) where W ∈ R W × C denotes the weight matrix, b ∈ R Q × 1 represents the bias vector, and

Original: y ′ ∈
Revised: y ′ ∈

Original: R Q × 1 isthe output logits that correspond to the perturbed feature X ′ .
Revised: R Q × 1 isthe output logits that correspond to the perturbed feature X ′ .

Original: Our RankFeat score is computed as:
Revised: Our RankFeat score is computed as:

Original: [Equation]
Revised: [Equation]

Original: where x is the input image, and Q denotes the number of categories.
Revised: where x is the input image, and Q denotes the number of categories.

Original: Here we choose Energy [39] asthe base function due to its theoretical alignment with the input probability density and its strongempirical performance.
Revised: Here we choose Energy [39] asthe base function due to its theoretical alignment with the input probability density and its strongempirical performance.

Original: Eq.
Revised: Eq.

Original: (9) can be re-formulated by the Log-Sum-Exp trick
Revised: (9) can be re-formulated by the Log-Sum-Exp trick

Original: [Equation]
Revised: [Equation]

Original: The above equation directly yields the tight bound as:
Revised: The above equation directly yields the tight bound as:

Original: [Equation]
Revised: [Equation]

Original: Since max( y ′ ) ≤ max( | y ′ | )
Revised: Since max( y ′ ) ≤ max( | y ′ | )

Original: =
Revised: =

Original: ||
Revised: ||

Original: y ′
Revised: y ′

Original: || ∞ , we have
Revised: || ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||
Revised: The vector norm has the property of triangular inequality, i.e., || a + c || ≤ || a ||

Original: + || c
Revised: + || c

Original: || holds for anyvectors a and c .
Revised: || holds for anyvectors a and c .

Original: Moreover, since both u and v are orthogonal vectors, we have the relation ||
Revised: Moreover, since both u and v are orthogonal vectors, we have the relation ||

Original: u
Revised: u

Original: i || ∞ ≤ 1and || v i || ∞ ≤ 1 .
Revised: i || ∞ ≤ 1and || v i || ∞ ≤ 1 .

Original: Relying on these two properties, injecting eq.
Revised: Relying on these two properties, injecting eq.

Original: (8) into eq.
Revised: (8) into eq.

Original: (12) leads to
Revised: (12) leads to

Original: RankFeat ( x ) <
Revised: RankFeat ( x ) <

Original: [Equation]
Revised: [Equation]

Original: Since m is a scaled all-ones vector, we have ||
Revised: Since m is a scaled all-ones vector, we have ||

Original: Wm || ∞
Revised: Wm || ∞

Original: =
Revised: =

Original: || W || ∞ / HW .
Revised: || W || ∞ / HW .

Original: The bound is simplified as:
Revised: The bound is simplified as:

Original: [Equation]
Revised: [Equation]

Original: Now the data-dependent variables only involve the singular values s i .
Revised: Now the data-dependent variables only involve the singular values s i .

Original: The upper bound is decidedby (cid:80) Ni =
Revised: The upper bound is decidedby (cid:80) Ni =

Original: 1 s i − s 1 .
Revised: 1 s i − s 1 .

Original: For the clean and unperturbed feature, the score bound depends on (cid:80) Ni
Revised: For the clean and unperturbed feature, the score bound depends on (cid:80) Ni

Original: =

Revised: =

Original: s i .
Revised: s i .

Original: Thus,removing the rank-1 feature with a larger s 1 would reduce the upper bound of RankFeat score more.
Revised: Thus,removing the rank-1 feature with a larger s 1 would reduce the upper bound of RankFeat score more.

Original: Considering the fact that OOD feature usually has a much larger s 1 (see Fig. 1(a)), RankFeat islikely to reduce the upper bound for OOD samples more.
Revised: Considering the fact that OOD feature usually has a much larger s 1 (see Fig. 1(a)), RankFeat islikely to reduce the upper bound for OOD samples more.

Original: Notice that our bound analysis strives to improve the understanding of OOD methods from newperspectives instead of giving a strict guarantee of the score.
Revised: Notice that our bound analysis strives to improve the understanding of OOD methods from newperspectives instead of giving a strict guarantee of the score.

Original: For example, the upper bound can beused to explain the shrinkage and skew of score distributions in Fig.
Revised: For example, the upper bound can beused to explain the shrinkage and skew of score distributions in Fig.

Original: Subtracting s 1 would largelyreduce the numerical range of both ID and OOD scores, which could squeeze score distributions.
Revised: Subtracting s 1 would largelyreduce the numerical range of both ID and OOD scores, which could squeeze score distributions.

Original: Since the dominant singular value s 1 contributes the most to the score, removing s 1 is likely to makemany samples have similar scores.
Revised: Since the dominant singular value s 1 contributes the most to the score, removing s 1 is likely to makemany samples have similar scores.

Original: This would concentrate samples in a smaller region and furtherskew the distribution.
Revised: This would concentrate samples in a smaller region and furtherskew the distribution.

Original: Consider that the OOD feature tends to have a much larger s 1 .
Revised: Consider that the OOD feature tends to have a much larger s 1 .

Original: This wouldhave a greater impact on OOD data and skew the OOD score distribution more.
Revised: This wouldhave a greater impact on OOD data and skew the OOD score distribution more.

Original: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.
Revised: After the rank-1 matrix is removed, the lowest bin of OOD feature has a larger reduction and the middle bins gain some growth, making the ODD feature statistics closer to the MP distribution.

Original: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to randommatrices.
Revised: Removing the rank-1 matrix is likely to make the statistics of OOD features closer to randommatrices.

Original: According to Manchenko-Pastur (MP) law [40, 49], given the shape ( t × n ) and variance σ 2of a random matrix, the probability density function (pdf) of its eigenvalues asymptotically convergesto the MP distribution
Revised: According to Manchenko-Pastur (MP) law [40, 49], given the shape ( t × n ) and variance σ 2of a random matrix, the probability density function (pdf) of its eigenvalues asymptotically convergesto the MP distribution

Original: [Equation]
Revised: [Equation]

Original: where ρ ( λ )=0 for λ outsides [ λ − , λ + ] , and the eigenvalues bounds are λ − =
Revised: where ρ ( λ )=0 for λ outsides [ λ − , λ + ] , and the eigenvalues bounds are λ − =

Original: σ 2 (1 − 148 (cid:112) nt ) 2 , λ + = σ 2 (1 + (cid:112) nt ) 2 .
Revised: σ 2 (1 − 149 (cid:112) nt ) 2 , λ + = σ 2 (1 + (cid:112) nt ) 2 .

Original: To evaluate the statistical distance between ID/OOD matrix and randommatrices, we randomly sample 1 , 000 ID and OOD feature matrices.
Revised: To evaluate the statistical distance between ID/OOD matrix and randommatrices, we randomly sample 1 , 000 ID and OOD feature matrices.

Original: Then the KL divergence betweenthe actual eigenvalue distribution and the fitted MP distribution is computed to measure the distance.
Revised: Then the KL divergence betweenthe actual eigenvalue distribution and the fitted MP distribution is computed to measure the distance.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of
Revised: 3 and Table 1 present the exemplary eigenvalue distribution and the average evaluation results of

Original: Block 4 and Block 3 features, respectively.
Revised: Block 4 and Block 3 features, respectively.

Original: For the original feature, the OOD and ID feature matricesexhibit similar behaviors: the distances to the fitted MP distribution are roughly the same ( diff.
Revised: For the original feature, the OOD and ID feature matricesexhibit similar behaviors: the distances to the fitted MP distribution are roughly the same ( diff.

Original: ≈ 0 . 1 ).
Revised: ≈ 0 . 1 ).

Original: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the
Revised: However, when the rank-1 matrix is removed, the OOD feature matrix has a much larger drop in the

Original: KL divergence.
Revised: KL divergence.

Original: This indicates that removing the rank-1 matrix makes the statistics of OOD featurecloser to random matrices, i.e., the OOD feature is very likely to become less informative than the IDfeature.
Revised: This indicates that removing the rank-1 matrix makes the statistics of OOD featurecloser to random matrices, i.e., the OOD feature is very likely to become less informative than the IDfeature.

Original: The result partly explains the working mechanism of RankFeat : by removing the featurematrix where OOD data might convey more information than ID data, the two types of distributionshave a larger discrepancy and therefore can be better separated.
Revised: The result partly explains the working mechanism of RankFeat : by removing the feature matrix where OOD data might convey more information than ID data, the two types of distributionshave a larger discrepancy and therefore can be better separated.

Original: Connection with ReAct [51].
Revised: Connection with ReAct [51].

Original: ReAct clips the activations at the penultimate layer of a model todistinguish ID and OOD samples.
Revised: ReAct clips the activations at the penultimate layer of a model todistinguish ID and OOD samples.

Original: Given the feature X and the pooling layer m , the perturbation canbe defined as:
Revised: Given the feature X and the pooling layer m , the perturbation canbe defined as:

Original: [Equation]
Revised: [Equation]

Original: where τ is a pre-defined threshold.
Revised: where τ is a pre-defined threshold.

Original: Their method shares some similarity with RankFeat formulation
Revised: Their method shares some similarity with RankFeat formulation

Original: Xm − s 1 u 1 v T 1 m .
Revised: Xm − s 1 u 1 v T 1 m .

Original: Both approaches subtract from the feature a portion of information that is mostlikely to cause the over-confidence of OOD prediction.
Revised: Both approaches subtract from the feature a portion of information that is mostlikely to cause the over-confidence of OOD prediction.

Original: ReAct selects the manually-defined thresholdτ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrixfrom the feature itself.
Revised: ReAct selects the manually-defined thresholdτ based on statistics of the whole ID set, while RankFeat generates the structured rank-1 matrixfrom the feature itself.

Original: Taking a step further, ReAct has the score inequality following eq.
Revised: Taking a step further, ReAct has the score inequality following eq.

Original: [Equation]
Revised: [Equation]

Original: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .
Revised: Since X is non-negative (output of ReLU ), we have max( Xm ) ≥ max( X ) / HW .

Original: Exploiting the vectornorm inequality ||
Revised: Exploiting the vectornorm inequality ||

Original: X
Revised: X

Original: || F ≥|| X
Revised: || F ≥|| X

Original: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .
Revised: || 2 leads to the relation max( X ) ≥ s 1 / √ CHW .

Original: Relying on this property,the above inequality can be re-formulated as:
Revised: Relying on this property,the above inequality can be re-formulated as:

Original: [Equation]
Revised: [Equation]

Original: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .
Revised: As indicated above, the upper bound of ReAct is also determined by the largest singular value s 1 .

Original: Incontrast, the upper bound of our RankFeat can be expressed as:
Revised: Incontrast, the upper bound of our RankFeat can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: The upper bounds of both methods resemble each other with the only different term boxed.
Revised: The upper bounds of both methods resemble each other with the only different term boxed.

Original: From thispoint of view, both methods distinguish the ID and OOD data by eliminating the impact of the termcontaining s 1 in the upper bound.
Revised: From thispoint of view, both methods distinguish the ID and OOD data by eliminating the impact of the termcontaining s 1 in the upper bound.

Original: ReAct optimizes it by clipping the term with a manually-definedthreshold, which is indirect and might be sub-optimal.
Revised: ReAct optimizes it by clipping the term with a manually-definedthreshold, which is indirect and might be sub-optimal.

Original: Moreover, the threshold selection requiresstatistics of the whole ID set.
Revised: Moreover, the threshold selection requiresstatistics of the whole ID set.

Original: In contrast, our RankFeat does not require any extra data and directlysubtracts this underlying term which is likely to cause the over-confidence of OOD samples.
Revised: In contrast, our RankFeat does not require any extra data and directlysubtracts this underlying term which is likely to cause the over-confidence of OOD samples.

Original:  Experimental Results
Revised:  Experimental Results

Original: In this section, we first discuss the setup in Sec. 4.1, and then present the main experimental resultson ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.
Revised: In this section, we first discuss the setup in Sec. 4.1, and then present the main experimental resultson ImageNet-1k in Sec. 4.2, followed by the extensive ablation studies in Sec. 4.3.

Original: 4.1 Setup
Revised: 4.1 Setup

Original: Datasets.
Revised: Datasets.

Original: In line with [25, 51, 26], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].
Revised: In line with [25, 51, 26], we mainly evaluate our method on the large-scale ImageNet1k benchmark [6].

Original: The large-scale dataset is more challenging than the traditional CIFAR benchmark [35] because the images are more realistic and diverse ( i.e., 1 .
Revised: The large-scale dataset is more challenging than the traditional CIFAR benchmark [35] because the images are more realistic and diverse ( i.e., 1 .

Original: 28 M images of 1 , 000 classes).
Revised: 28 M images of 1 , 000 classes).

Original: Forthe OOD datasets, we select four testsets from subsets of iNaturalist [57], SUN [62], Places [69],and Textures [5].
Revised: Forthe OOD datasets, we select four testsets from subsets of iNaturalist [57], SUN [62], Places [69],and Textures [5].

Original: These datasets are crafted by [25] with non-overlapping categories from ImageNet1k.
Revised: These datasets are crafted by [25] with non-overlapping categories from ImageNet1k.

Original: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of ourapproach with the CIFAR dataset as the ID dataset (see Supplementary Material).
Revised: Besides the experiment on the large-scale benchmark, we also validate the effectiveness of ourapproach with the CIFAR dataset as the ID dataset (see Supplementary Material).

Original: Baselines.
Revised: Baselines.

Original: We compare our method with 6 recent post hoc OOD detection methods, namely
Revised: We compare our method with 6 recent post hoc OOD detection methods, namely

Original: MSP [22], ODIN [38], Energy [39], Mahalanobis [37], GradNorm [26], and ReAct [51].
Revised: MSP [22], ODIN [38], Energy [39], Mahalanobis [37], GradNorm [26], and ReAct [51].

Original: Thedetailed illustration and settings of these methods are kindly referred to Supplementary Material.
Revised: Thedetailed illustration and settings of these methods are kindly referred to Supplementary Material.

Original: Architectures.
Revised: Architectures.

Original: In line with [26], the main evaluation is done using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 [20].
Revised: In line with [26], the main evaluation is done using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 [20].

Original: We also evaluate the performance on
Revised: We also evaluate the performance on

Original: SqueezeNet [28], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-

Revised: SqueezeNet [28], an alternative tiny architecture suitable for mobile devices and on T2T-ViT-

Original: [66],a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.
Revised: [66],a tokens-to-tokens vision transformer that has impressive performance when trained from scratch.

Original: For the implementation details and evaluation metrics, please refer to Supplementary Material .
Revised: For the implementation details and evaluation metrics, please refer to Supplementary Material .

Original: 4.2 Results
Revised: 4.2 Results

Revised: [Figure]

Original: Main
Revised: Main

Original: results.
Revised: results.

Original: Following [26], the main evaluation is conducted using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 architecture [20].
Revised: Following [26], the main evaluation is conducted using Google BiT-S model [34]pretrained on ImageNet-1k with ResNetv2-101 architecture [20].

Original: Table 2 compares the performanceof all the post hoc methods.
Revised: Table 2 compares the performanceof all the post hoc methods.

Original: For both Block 3 and Block 4 features, our RankFeat achieves thebest evaluation results across datasets and metrics.
Revised: For both Block 3 and Block 4 features, our RankFeat achieves thebest evaluation results across datasets and metrics.

Original: More specifically, RankFeat based on the Block4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block
Revised: More specifically, RankFeat based on the Block4 feature outperforms the second-best baseline by 15.01% in the average FPR95, while the Block

Original: [Table]
Revised: [Table]

Original: 3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.
Revised: 3 feature-based RankFeat beats the second-best method by 4.09% in the average AUROC.

Original: Theircombination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in theaverage AUROC.
Revised: Theircombination further surpasses other methods by 17.90% in the average FPR95 and by 5.44% in theaverage AUROC.

Original: The superior performances at various depths demonstrate the effectiveness andgeneral applicability of RankFeat .
Revised: The superior performances at various depths demonstrate the effectiveness andgeneral applicability of RankFeat .

Original: The Block 3 feature has a higher AUROC but slightly falls behindthe Block 4 feature in the FPR95, which can be considered a compromise between the two metrics.
Revised: The Block 3 feature has a higher AUROC but slightly falls behindthe Block 4 feature in the FPR95, which can be considered a compromise between the two metrics.

Original: [Table]
Revised: [Table]

Original: Our RankFeat is also effective on alternative CNN architectures.
Revised: Our RankFeat is also effective on alternative CNN architectures.

Original: Besides the experiment on
Revised: Besides the experiment on

Original: ResNetv2 [20], we also evaluate our method on SqueezeNet [28], an alternative tiny network suitablefor mobile devices and on-chip applications.
Revised: ResNetv2 [20], we also evaluate our method on SqueezeNet [28], an alternative tiny network suitablefor mobile devices and on-chip applications.

Original: This network is more challenging because the tinynetwork size makes the model prone to overfit the training data, which could increase the difficultyto distinguish between ID and OOD samples.
Revised: This network is more challenging because the tinynetwork size makes the model prone to overfit the training data, which could increase the difficultyto distinguish between ID and OOD samples.

Original: Table 3 presents the performance of all the methods.
Revised: Table 3 presents the performance of all the methods.

Original: Collectively, the performances of RankFeat are very competitive at both depths, as well as the scorefusion.
Revised: Collectively, the performances of RankFeat are very competitive at both depths, as well as the scorefusion.

Original: Our RankFeat achieves the state-of-the-art performances, outperforming the second-bestbaseline by 14.22% in FPR95 and by 7.48% in AUROC.
Revised: Our RankFeat achieves the state-of-the-art performances, outperforming the second-bestbaseline by 14.22% in FPR95 and by 7.48% in AUROC.

Original: Our RankFeat also suits transformer-based architectures.
Revised: Our RankFeat also suits transformer-based architectures.

Original: To further demonstrate the applicabilityof our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)
Revised: To further demonstrate the applicabilityof our method, we evaluate RankFeat on Tokens-to-Tokens Vision Transformer (T2T-ViT)

Original: [66],a popular transformer architecture that can achieve competitive performance against CNNs whentrained from scratch.
Revised: [66],a popular transformer architecture that can achieve competitive performance against CNNs whentrained from scratch.

Original: Similar to the CNN, RankFeat removes the rank-1 matrix from the finaltoken of T2T-ViT before the last normalization layer and the classification head.
Revised: Similar to the CNN, RankFeat removes the rank-1 matrix from the finaltoken of T2T-ViT before the last normalization layer and the classification head.

Original: Table 3 comparesthe performance on T2T-ViT-24.
Revised: Table 3 comparesthe performance on T2T-ViT-24.

Original: Our RankFeat outperforms the second-best method by 9.49%in FPR95 and by 3.68% in AUROC.
Revised: Our RankFeat outperforms the second-best method by 9.49%in FPR95 and by 3.68% in AUROC.

Original: Since the transformer models [9, 66] do not have increasingreceptive fields like CNNs, we do not evaluate the performance at alternative network depths.
Revised: Since the transformer models [9, 66] do not have increasingreceptive fields like CNNs, we do not evaluate the performance at alternative network depths.

Original: 4.3 Ablation Studies
Revised: 4.3 Ablation Studies

Original: In this subsection, we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.
Revised: In this subsection, we conduct several ablation studies based on Google-BiT-S ResNetv2-101 model.

Original: Unless explicitly specified, we apply RankFeat
Revised: Unless explicitly specified, we apply RankFeat

Original: [Table]
Revised: [Table]

Original: Removing the rank-1 matrix outperforms keeping only it.
Revised: Removing the rank-1 matrix outperforms keeping only it.

Original: Instead of removing the rank-1 matrix,another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest ofthe matrix.
Revised: Instead of removing the rank-1 matrix,another seemingly promising approach is keeping only the rank-1 matrix and abandoning the rest ofthe matrix.

Original: Table 4 presents the evaluation results of keeping only the rank-1 matrix.
Revised: Table 4 presents the evaluation results of keeping only the rank-1 matrix.

Original: The performancefalls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keepingonly the rank-1 feature is inferior to removing it in distinguishing the two distributions.
Revised: The performancefalls behind that of removing the rank-1 feature by 18 . 8% in FPR95, which indicates that keepingonly the rank-1 feature is inferior to removing it in distinguishing the two distributions.

Original: Nonetheless,it is worth noting that even keeping only the rank-1 matrix achieves very competitive performanceagainst previous best methods, such as GradNorm [26] and ReAct [51].
Revised: Nonetheless,it is worth noting that even keeping only the rank-1 matrix achieves very competitive performanceagainst previous best methods, such as GradNorm [26] and ReAct [51].

Original: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).
Revised: Removing the rank-1 matrix outperforms removing the rank-n matrix (n>1).

Original: We evaluate theimpact of removing the matrix of a higher rank,
Revised: We evaluate theimpact of removing the matrix of a higher rank,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., performing X − (cid:80) ni
Revised: e., performing X − (cid:80) ni

Original: =

Revised: =

Original: s i
Revised: s i

Original: u
Revised: u

Original: i v Ti where n> 1 forthe high-level feature X .
Revised: i v Ti where n> 1 forthe high-level feature X .

Original: Table 4 compares the performance of removing the rank-2 matrix andrank-3 matrix.
Revised: Table 4 compares the performance of removing the rank-2 matrix andrank-3 matrix.

Original: When the rank of the removed matrix is increased, the average performance degradesaccordingly.
Revised: When the rank of the removed matrix is increased, the average performance degradesaccordingly.

Original: This demonstrates that removing the rank-1 matrix is the most effective approach toseparate ID and OOD data.
Revised: This demonstrates that removing the rank-1 matrix is the most effective approach toseparate ID and OOD data.

Original: This result is coherent with the finding in Fig.
Revised: This result is coherent with the finding in Fig.

Original: 1(a): only the largestsingular value of OOD data is significantly different from that of ID data.
Revised: 1(a): only the largestsingular value of OOD data is significantly different from that of ID data.

Original: Therefore, removing therank-1 matrix achieves the best performance.
Revised: Therefore, removing therank-1 matrix achieves the best performance.

Original: [Table]
Revised: [Table]

Original: Block 3 and Block 4 features are the most informative.
Revised: Block 3 and Block 4 features are the most informative.

Original: In addition to exploring the high-levelfeatures at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to featuresat shallow network layers.
Revised: In addition to exploring the high-levelfeatures at Block 3 and Block 4, we also investigate the possibility of applying RankFeat to featuresat shallow network layers.

Original: As shown in Table 5, the performances of RankFeat at the Block 1 and
Revised: As shown in Table 5, the performances of RankFeat at the Block 1 and

Original: Block 2 features are not comparable to those at deeper layers.
Revised: Block 2 features are not comparable to those at deeper layers.

Original: This is mainly because the shallowlow-level features do not embed as rich semantic information as the deep features.
Revised: This is mainly because the shallowlow-level features do not embed as rich semantic information as the deep features.

Original: Consequently,removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.
Revised: Consequently,removing the rank-1 matrix of shallow features would not help to separate the ID and OOD data.

Original: [Table]
Revised: [Table]

Original: The approximate solution by PI yields competitive performances.
Revised: The approximate solution by PI yields competitive performances.

Original: Table 6 compares the timeconsumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods
Revised: Table 6 compares the timeconsumption and performance of SVD and PI, as well as two recent state-of-the-art OOD methods

Original: ReAct and GradNorm .
Revised: ReAct and GradNorm .

Original: The performance of PI starts to become competitive against that of SVD( < 0 .
Revised: The performance of PI starts to become competitive against that of SVD( < 0 .

Original: 1% ) from 20 iterations on with 48.41% time reduction.
Revised: 1% ) from 20 iterations on with 48.41% time reduction.

Original: Compared with ReAct , the PI-based
Revised: Compared with ReAct , the PI-based

Original: RankFeat only requires marginally 4 .
Revised: RankFeat only requires marginally 4 .

Original: 89% more time consumption.
Revised: 89% more time consumption.

Original: GradNorm is not comparableagainst other baselines in terms of time cost because it does not support the batch mode.
Revised: GradNorm is not comparableagainst other baselines in terms of time cost because it does not support the batch mode.

Original: Fusion at the logit space achieves the best performance.
Revised: Fusion at the logit space achieves the best performance.

Original: Fig.
Revised: Fig.

Original: 4 displays the performance ofdifferent fusion strategies in combining RankFeat at the Block 3 and Block 4 features.
Revised: 4 displays the performance ofdifferent fusion strategies in combining RankFeat at the Block 3 and Block 4 features.

Original: As can beobserved, averaging logits outperforms other fusion strategies in most datasets and metrics.
Revised: As can beobserved, averaging logits outperforms other fusion strategies in most datasets and metrics.

Original: Thisindicates that the fusing the logits can best coordinate the benefit of both features.
Revised: Thisindicates that the fusing the logits can best coordinate the benefit of both features.

Original:  Related Work
Revised:  Related Work

Original: Distribution shifts.
Revised: Distribution shifts.

Original: Distribution shifts have been a long-standing problem in the machine learningresearch community [17, 44, 33, 61].
Revised: Distribution shifts have been a long-standing problem in the machine learningresearch community [17, 44, 33, 61].

Original: The problem of distributions shifts can be generally categorizedas shifts in the input space and shifts in the label space.
Revised: The problem of distributions shifts can be generally categorizedas shifts in the input space and shifts in the label space.

Original: Shifts only in the input space are oftendeemed as covariate shifts [21, 43].
Revised: Shifts only in the input space are oftendeemed as covariate shifts [21, 43].

Original: In this setting, the inputs are corrupted by perturbations or shiftedby domains, but the label space stays the same [24, 52].
Revised: In this setting, the inputs are corrupted by perturbations or shiftedby domains, but the label space stays the same [24, 52].

Original: The aim is mainly to improve the robustnessand generalization of a model [23].
Revised: The aim is mainly to improve the robustnessand generalization of a model [23].

Original: For OOD detection, the labels are disjoint and the main concernis to determine whether a test sample should be predicted by the pre-trained model [38, 24].
Revised: For OOD detection, the labels are disjoint and the main concernis to determine whether a test sample should be predicted by the pre-trained model [38, 24].

Original: Some related sub-fields also tackle the problem of distribution shifts in the label space, such as novelclass discovery [16, 68], open-set recognition [47, 58], and novelty detection [1, 55].
Revised: Some related sub-fields also tackle the problem of distribution shifts in the label space, such as novelclass discovery [16, 68], open-set recognition [47, 58], and novelty detection [1, 55].

Original: These sub-fieldstarget specific distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.
Revised: These sub-fieldstarget specific distribution shifts ( e.g., semantic novelty), while OOD encompasses all forms of shifts.

Original: OOD detection with discriminative models.
Revised: OOD detection with discriminative models.

Original: The early work on discriminative OOD detection datesback to the classification model with rejection option [4, 12].
Revised: The early work on discriminative OOD detection datesback to the classification model with rejection option [4, 12].

Original: For the wide body of research on OODdetection, please refer to [64] for the comprehensive survey.
Revised: For the wide body of research on OODdetection, please refer to [64] for the comprehensive survey.

Original: Here we highlight the representative posthoc methods that do not require extra training processes and can be directly applied to any pre-trainedmodels.
Revised: Here we highlight the representative posthoc methods that do not require extra training processes and can be directly applied to any pre-trainedmodels.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: [42] first observed the phenomenon that neural networks easily give overconfident predictions for OOD samples.
Revised: [42] first observed the phenomenon that neural networks easily give overconfident predictions for OOD samples.

Original: The following researches attempted to improve the OODuncertainty estimation by proposing ODIN score [38], OpenMax score [2], Mahalanobis distance [37],and Energy score [39].
Revised: The following researches attempted to improve the OODuncertainty estimation by proposing ODIN score [38], OpenMax score [2], Mahalanobis distance [37],and Energy score [39].

Original: Huang et al.
Revised: Huang et al.

Original: [25] pointed out that the traditional CIFAR benchmark does notextrapolate to real-world settings and proposed a large-scale ImageNet benchmark.
Revised: [25] pointed out that the traditional CIFAR benchmark does notextrapolate to real-world settings and proposed a large-scale ImageNet benchmark.

Original: More recently,
Revised: More recently,

Original: Sun et al.
Revised: Sun et al.

Original: [51] and Huang et al.
Revised: [51] and Huang et al.

Original: [26] proposed to tackle the challenge of OOD detection from thelens of activation abnormality and gradient norm, respectively.
Revised: [26] proposed to tackle the challenge of OOD detection from thelens of activation abnormality and gradient norm, respectively.

Original: In contrast, based on the empiricalobservation of singular value distributions, we propose a simple yet effective post hoc solution byremoving the rank-1 subspace from the high-level features.
Revised: In contrast, based on the empiricalobservation of singular value distributions, we propose a simple yet effective post hoc solution byremoving the rank-1 subspace from the high-level features.

Original: OOD detection with generative models.
Revised: OOD detection with generative models.

Original: Different from discriminative models, generative modelsdetect the OOD samples by estimating the probability density function [31, 54, 46, 56, 8, 27, 3, 29].
Revised: Different from discriminative models, generative modelsdetect the OOD samples by estimating the probability density function [31, 54, 46, 56, 8, 27, 3, 29].

Original: A sample with a low likelihood is deemed as OOD data.
Revised: A sample with a low likelihood is deemed as OOD data.

Original: Recently, a multitude of methods haveutilized generative models for OOD detection [45, 50, 60, 63, 32, 48, 30].
Revised: Recently, a multitude of methods haveutilized generative models for OOD detection [45, 50, 60, 63, 32, 48, 30].

Original: However, as pointed outin [41], generative models could assign a high likelihood to OOD data.
Revised: However, as pointed outin [41], generative models could assign a high likelihood to OOD data.

Original: Furthermore, generativemodels can be prohibitively harder to train and optimize than their discriminative counterparts, andthe performance is often inferior.
Revised: Furthermore, generativemodels can be prohibitively harder to train and optimize than their discriminative counterparts, andthe performance is often inferior.

Original: This might limit their practical usage.
Revised: This might limit their practical usage.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we present RankFeat , a simple yet effective approach for OOD detection by removingthe rank-1 matrix composed by the largest singular value from the high-level feature.
Revised: In this paper, we present RankFeat , a simple yet effective approach for OOD detection by removingthe rank-1 matrix composed by the largest singular value from the high-level feature.

Original: We demonstratethe superior empirical results and the general applicability of RankFeat across architectures, networkdepths, and benchmarks.
Revised: We demonstratethe superior empirical results and the general applicability of RankFeat across architectures, networkdepths, and benchmarks.

Original: Extensive ablation studies and comprehensive theoretical analyses areconducted to reveal the important insights and to explain the working mechanism of our method.
Revised: Extensive ablation studies and comprehensive theoretical analyses areconducted to reveal the important insights and to explain the working mechanism of our method.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We briefly discuss the slightly
Revised: [Yes] We briefly discuss the slightly

Original: higher time consumption of RankFeat in Sec. 4.3.
Revised: higher time consumption of RankFeat in Sec. 4.3.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We do not
Revised: [No] We do not

Original: anticipate any negative societal impacts of our work.
Revised: anticipate any negative societal impacts of our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] Since our method is post hoc , it will not be influenced
Revised: [N/A] Since our method is post hoc , it will not be influenced

Original: by random seeds.
Revised: by random seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes]
Revised: [Yes]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The used data do not contain this kind of
Revised: The used data do not contain this kind of

Original: information.
Revised: information.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]