Original: E FFICIENT O UT - OF -D ISTRIBUTION D ETECTION BASED ON I N -D ISTRIBUTION D ATA P ATTERNS M EMORIZATION WITH M ODERN H OPFIELD E NERGY
Revised: O UT - OF -D ISTRIBUTION D ETECTION BASED ON I N -D ISTRIBUTION D ATA P ATTERNS M EMORIZATION WITH M ODERN H OPFIELD E NERGY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Out-of-Distribution (OOD) detection is essential for safety-critical applications of deep neural networks.
Revised: Out-of-Distribution (OOD) detection is essential for safety-critical applications of deep neural networks.

Original: OOD detection is challenging since DNN models tend to produce very high logits value even for OOD samples.
Revised: OOD detection is challenging since DNN models may produce very high logits value even for OOD samples.

Original: Hence, it is of great difficulty to discriminate OOD data by directly adopting Softmax on output logits as the confidence score.
Revised: Hence, it is of great difficulty to discriminate OOD data by directly adopting Softmax on output logits as the confidence score.

Original: Differently, we detect the OOD sample with Hopfield energy in a store-then-compare paradigm.
Revised: Differently, we detect the OOD sample with Hopfield energy in a store-then-compare paradigm.

Original: In more detail, penultimate layer outputs on the training set are considered as the representation of in-distribution (ID) data.
Revised: In more detail, penultimate layer outputs on the training set are considered as the representations of in-distribution (ID) data.

Original: Thus they can be transformed into stored patterns that serve as anchors to measure the discrepancy of unseen data for OOD detection.
Revised: Thus they can be transformed into stored patterns that serve as anchors to measure the discrepancy of unseen data for OOD detection.

Original: Starting from an energy function defined in Modern Hopfield Network for the discrepancy score calculation, we derive a simplified version SHE with theoretical analysis.
Revised: Starting from the energy function defined in Modern Hopfield Network for the discrepancy score calculation, we derive a simplified version SHE with theoretical analysis.

Original: In SHE, weonly utilize  one stored pattern to present each class, and these patterns can be obtained by simply averaging the penultimate layer outputs of training samples within this class.
Revised: In SHE, we utilize only one stored pattern to present each class, and these patterns can be obtained by simply averaging the penultimate layer outputs of training samples within this class.

Original: SHE has the advantages of hyperparameter-free and high computational efficiency.
Revised: SHE has the advantages of hyperparameterfree and high computational efficiency.

Original: The evaluations of nine widely-used OOD datasets show the promising performance of such a simple yet effective approach and its superiority over State-of-the-Art models.
Revised: The evaluations of nine widely-used OOD datasets show the promising performance of such a simple yet effective approach and its superiority over State-of-the-Art models.

Revised: Code is available at https://github.com/zjs975584714/SHE ood detection.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Neural Network (DNN) has yielded remarkable achievements in a broad range of fields in recent years (He et al., 2016; Huang et al., 2017; Zagoruyko & Komodakis, 2016), and is extensively deployed in numerous real-world scenarios (Krizhevsky et al., 2017;Ronneberger et al., 2015; Redmon & Farhadi, 2017).
Revised: Deep Neural Network (DNN) has yielded remarkable achievements in a broad range of fields in recent years (He et al., 2016; Huang et al., 2017), and is extensively deployed in numerous real-world scenarios (Krizhevsky et al., 2017; Redmon & Farhadi, 2017).

Original: One of its powerful capabilities lies in the promising generalization ability from training data to unseen in-distribution (ID) samples.
Revised: One of its powerful capabilities lies in the promising generalization ability from training data to unseen in-distribution (ID) data.

Original: However, the finite training data cannot guarantee the completeness of data distribution, so it is inevitable to encounter out-of-distribution (OOD) data.
Revised: However, the finite training data cannot guarantee the completeness of data distribution, so it is inevitable to encounter out-of-distribution (OOD) data.

Original: The Softmax-based prediction allows OOD samples to gain high confidence in specific classes, which is unacceptable in practice, especially for safety-related areas.
Revised: The Softmax-based prediction allows OOD samples to gain high confidence in specific classes, which is unacceptable in practice, especially for safetyrelated areas.

Original: It can lead to erroneous collisions in autonomous driving or irreparably large financial losses.
Revised: It can lead to erroneous collisions in autonomous driving or irreparably large financial losses.

Original: Therefore, OOD detection is critical with respect to AI safety (Amodei et al., 2016).
Revised: Therefore, OOD detection is critical with respect to AI safety (Amodei et al., 2016).

Original: Existing efforts on OOD detection for DNN can be roughly divided into two categories.
Revised: Existing efforts on OOD detection for DNN can be roughly divided into two categories.

Original: The first group of approaches requires designing and retraining new auxiliary networks specifically for OOD detection rather than directly using already trained models (Denouden et al., 2018;
Revised: The first group of approaches requires designing and retraining new auxiliary networks specifically for OOD detection rather than directly using already trained models (Denouden et al., 2018;

Original: DeVries & Taylor, 2018; Yu & Aizawa, 2019; Zhang et al., 2020).
Revised: DeVries & Taylor, 2018; Yu & Aizawa, 2019; Zhang et al., 2020).

Original: The objective should be modified accordingly and OOD samples are sometimes introduced to train the new networks.
Revised: The objective should be modified accordingly and OOD samples are sometimes introduced to train the new networks.

Original: However, it is almost impossible to exhaust all kinds of OOD samples, and retraining can also be cumbersome.
Revised: However, it is almost impossible to exhaust all kinds of OOD samples, and retraining can also be cumbersome.

Original: Methods of the second category elaborate on the confidence design for the network outputs, e.g., the logits, the 
Revised: The methods of the second category elaborate on the confidence design for the network outputs, e.g., the logits, the Softmax probability (Liang et al., 2017;

Original: Softmax probability (Liang et al., 2017; Liu et al., 2020; Sun et al., 2021) or embedding features (Lee et al., 2018; Sehwag et al., 2021; Sun et al., 2022).
Revised: Liu et al., 2020; Sun et al., 2021) or embedding features (Lee et al., 2018; Sehwag et al., 2021; Sun et al., 2022).

Original: By these means, there is no need to modify the backbone model and the objective, which motivates us to explore OOD detection in this manner.
Revised: By these means, there is no need to modify the backbone model and the objective, which motivates us to explore OOD detection in this manner.

Original: In deep learning, the intermediate layer output can be regarded as the representation of input data in the latent space.
Revised: In deep learning, the intermediate layer output can be regarded as the representation of input data in the latent space.

Original: Further, as shown in Figure.
Revised: Further, as shown in Figure.

Original: 1 (left), guided by the training process, such representations of ID samples of the same category tend to present some common patterns for prediction accuracy.
Revised: 1 (left), guided by the training process, such representations of ID samples of the same category tend to present some common patterns for prediction accuracy.

Original: In contrast, these representations of OOD samples should deviate from such commonality since they are not considered during the training process.
Revised: In contrast, these representations of OOD samples should deviate from such commonality since they are not considered during the training process.

Original: Based on this intuition, OOD detection can be formulated as a store-then-compare process : representations of ID samples within each category are maintained during the training procedure as stored patterns, and a test pattern will be compared with the store patterns.
Revised: Based on this intuition, OOD detection can be formulated as a store-then-compare process : representations of ID samples within each category are maintained during the training procedure as stored patterns, and a test pattern will be compared with the store patterns.

Original: If there is a noticeable discrepancy, then it can be judged as an
Revised: If there is a noticeable discrepancy, then it can be judged as an

Original: OOD sample.
Revised: OOD sample.

Original: The critical question is how to measure the discrepancy between the OOD sample and the stored patterns under this setting?
Revised: The critical question is how to measure the discrepancy between the OOD sample and the stored patterns under this setting?

Original: To accomplish this goal, we adopt the critical idea of a classic memory network, Hopfield Network.
Revised: To accomplish this goal, we adopt the key idea of a classic memory network, Hopfield Network.

Original: The Hopfield Network (binary state) was first introduced in (Hopfield, 1982) and (Hopfield, 1984) proposed continuous state version.
Revised: The Hopfield Network (binary state) was first introduced in (Hopfield, 1982) and (Hopfield, 1984) proposed continuous state version.

Original: Modern Hopfield Network (both continuous and binary) was introduced in (Krotov & Hopfield, 2016), and (Ramsauer et al., 2020) proposed a new energy function for continuous state Hopfield networks and point out the relationship with the transformer.
Revised: Modern Hopfield Network (both continuous and binary) was introduced in (Krotov & Hopfield, 2016), and (Ramsauer et al., 2020) proposed a new energy function for continuous state Hopfield networks and point out the relationship with the transformer.

Original: Hopfield Network targets recovering distorted test patterns so that the recovered patterns are as close to the stored patterns as possible.
Revised: Hopfield Network targets recovering distorted test patterns so that the recovered patterns are as close to the stored patterns as possible.

Original: It achieves this goal by specific update rules that minimize the predefined energy function.
Revised: It achieves this goal by specific update rules that minimize the predefined energy function.

Original: The more the recovered pattern resembles the stored one, the lower the energy is.
Revised: The more the recovered pattern resembles the stored pattern, the lower the energy is.

Original: Therefore, the energy function serves a vital role as it indicates the gap between the recovered patterns and the stored patterns.
Revised: Therefore, the energy function serves a vital role as it indicates the gap between the recovered patterns and the stored patterns.

Original: As for OOD detection tasks, the energy function of the
Revised: For OOD detection tasks, the energy function of the

Original: Hopfield Network is well-suited as a desirable measure of the discrepancy between the OOD sample and the stored patterns.
Revised: Hopfield Network is well-suited as a desirable measure of the discrepancy between the OOD sample and the stored patterns.

Original: In this paper, we propose a new OOD detection method HE with memorization of ID data patterns and the Modern H opfield E nergy function (Ramsauer et al., 2020).
Revised: In this paper, we propose a new OOD detection method HE with memorization of ID data patterns and the Modern H opfield E nergy function (Ramsauer et al., 2020).

Original: In more detail, the representations of training ID samples are stored as patterns for each category in advance, and OOD samples are detected under the energy function.
Revised: In more detail, the representations of training ID samples are stored as patterns for each category in advance, and OOD samples are detected under the energy function.

Original: As  intermediate results are more informative than the highlycompressed final output logits, we preserve the outputs of the penultimate layer (i.e., the input of the final output layer) as representations.
Revised: As the intermediate results are more informative than the highly-compressed final output logits, we preserve the outputs of the penultimate layer (i.e., the input of the final output layer) as representations.

Original: Furthermore, to address the challenges of the memory cost of pattern memorization, we derive a S implified H opfield E nergy function-based method SHE .
Revised: Furthermore, to address the challenges of the memory cost of pattern memorization, we derive a S implified H opfield E nergy function-based method SHE .

Original: In SHE , only one pattern is required for each category and there is no hyperparameter to be tuned.
Revised: In SHE , only one pattern is required for each category and there is no hyperparameter to be tuned.

Original: Theoretical analysis proves the effectiveness of our design.
Revised: Theoretical analysis proves the effectiveness of our design.

Original: The remarkable performances on nine widely-used OOD detection datasets on three different networks demonstrate the superiority of our proposed SHE (and HE) over state-of-the-art methods.
Revised: The remarkable performances on nine widely-used OOD detection datasets on three different networks demonstrate the superiority of our proposed SHE (and HE) over state-of-the-art methods.

Original: We summarize the main contributions of our paper as follows:
Revised: We summarize the main contributions of our paper as follows

Original:  • We propose a Modern Hopfield energy-based method HE for out-of-distribution detection.
Revised: : • We propose a Modern Hopfield Energy-based method HE for out-of-distribution detection.

Original: It uses store-then-compare paradigm that compares test samples with pre-stored patterns to measure the discrepancy from in-distribution data according to Hopfield energy.
Revised: It uses store-then-compare paradigm that compares test samples with pre-stored patterns to measure the discrepancy from in-distribution data according to Hopfield energy.

Original: • We derive a simplified version of HE, named as SHE, which greatly reduces the memory and the computation cost.
Revised: • We derive a simplified version of HE, named as SHE, which greatly reduces the memory and the computation cost.

Original: In addition, SHE is hyperparameter-free.
Revised: In addition, SHE is hyperparameter-free.

Original: Theoretical analysis is conducted to illustrate the effectiveness of SHE.
Revised: Theoretical analysis is conducted to illustrate the effectiveness of SHE.

Original: • Extensive experiments on nine OOD detection datasets of three prominent computer vision backbone networks indicate both the effectiveness and the efficiency of our designed methods.
Revised: • Extensive experiments on nine OOD detection datasets of three prominent computer vision backbone networks indicate both the effectiveness and the efficiency of our designed methods.

Original: Experiments on large-scale datasets (e.g., ImageNet-1k) also show the superior of our approach.
Revised: Experiments on large-scale datasets (e.g., ImageNet-1k) also show the superior of our approach.

Original: In-depth analysis and ablation studies are also included to shed light on the mechanism behind it.
Revised: In-depth analysis and ablation studies are also included to shed light on the mechanism behind it.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Network Redesign and Retrain.
Revised: Network Redesign and Retrain.

Original: Naturally, original network architectures are designed for  tasks like classification. One straightforward paradigm  is to elaborate on the network architecture and the objective, and retrain the new network for OOD detection.
Revised: Given original network architectures are designed for target tasks like classification, a straightforward paradigm of OOD detection is to elaborate on the network ar- chitecture and the objective, and retrain a new network for OOD detection.

Original: (Denouden et al., 2018) used ID data to train an AutoEncoder, and OOD samples were supposed to have high reconstruction errors.
Revised: (Denouden et al., 2018) used ID data to train an AutoEncoder, and OOD samples were supposed to have high reconstruction errors.

Original: (DeVries & Taylor, 2018) expanded the original network with an extra branch to predict a confidence score and obtained the final output with this confidence score.
Revised: (DeVries & Taylor, 2018) expanded the original network with an extra branch to predict a confidence score and obtained the final output with this confidence score.

Original: (Zhang et al., 2020) proposed a flowbased model that retrieved the feature map of test samples and fed it into an additional network.
Revised: (Zhang et al., 2020) proposed a flow-based model that retrieved the feature map of test samples and fed it into an additional network.

Original: Such an additional network was designed only for forecasting whether the test sample is ID or not.
Revised: Such an additional network was designed only for forecasting whether the test sample is ID or not.

Original: (Yu & Aizawa, 2019) firstly trained an encoder in a supervised way, and then two classifiers for ID samples and OOD samples were trained with an unsupervised objective, respectively.
Revised: (Yu & Aizawa, 2019) firstly trained an encoder in a supervised way, and then two classifiers for ID samples and OOD samples were trained with an unsupervised objective, respectively.

Original: All these methods aim to redesign or introduce new layers or auxiliary networks with corresponding objectives for OOD detection.
Revised: All these methods aim to redesign or introduce new layers or auxiliary networks with corresponding objectives for OOD detection.

Original: Retraining a network can be extremely time-consuming, especially when the parameter scale is substantially large.
Revised: Retraining a network can be extremely time-consuming, especially when the parameter scale is substantially large.

Original: The modified objective that considers OOD detection in addition to the original task may also have side effects of degrading model performance on the original task.
Revised: The modified objective that considers OOD detection in addition to the original task may also have side effects of degrading model performance on the original task.

Original: In addition, some methods require OOD samples as input, which imposes additional requirements on the datasets.
Revised: In addition, some methods require OOD samples as input, which imposes additional requirements on the datasets.

Original: A potential risk is that the model will achieve poor results on data beyond the distribution of the trained OOD samples.
Revised: A potential risk is that the model will achieve poor results on data beyond the distribution of the trained OOD samples.

Original: Unlike those approaches, our method does not need to make any changes (including both architecture and objective) to the original network and does not need to do any additional training, which is a plausible property for real applications.
Revised: Unlike those approaches, our method does not need to make any changes (including both architecture and objective) to the original network and does not need to do any additional training, which is a plausible property for real applications.

Original: Network Output Transformation.
Revised: Network Output Transformation.

Original: Apart from adjusting network structure or retraining the network with revised objectives, transforming network outputs to obtain the desired measure is the other classic OOD detection paradigm.
Revised: Apart from adjusting network structure or retraining the network with revised objectives, transforming network outputs to obtain the desired measure is the other classic OOD detection paradigm.

Original: The first study on network output transformation was proposed by (Hendrycks & Gimpel, 2016), which used the Maximum Softmax Probability (MSP) to measure the confidence of test samples.
Revised: The first study on network output transformation was proposed by (Hendrycks & Gimpel, 2016), which used the Maximum Softmax Probability (MSP) to measure the confidence of test samples.

Original: Intuitively, ID data is more likely to obtain high confidence from Softmax measure than OOD samples.
Revised: Intuitively, ID data is more likely to obtain high confidence from Softmax measure than OOD samples.

Original: In (Liang et al., 2017), input data were perturbed with ID-sample-friendly perturbations and the Softmax probability was re-scaled by a temperature parameter T ; thus OOD data and ID data are more separable.
Revised: In (Liang et al., 2017), input data were perturbed with ID-sample-friendly perturbations and the Softmax probability was re-scaled by a temperature parameter T ; thus OOD data and ID data are more separable.

Original: (Lee et al., 2018) first generated class-conditional Gaussian distribution from middle layer outputs of the already trained network on training data and then calculated the confidence of test samples under the Mahalanobis distance measure.
Revised: (Lee et al., 2018) first generated class-conditional Gaussian distribution from middle layer outputs of the already trained network on training data and then calculated the confidence of test samples under the Mahalanobis distance measure.

Original: (Serr`a et al., 2019) made assumptions about the complexity of output and input images, and they advocated estimating the complexity of the input image to impact the output for efficient
Revised: (Serr`a et al., 2019) made assumptions about the complexity of output and input images, and they advocated estimating the complexity of the input image to impact the output for efficient

Original: OOD detection.
Revised: OOD detection.

Original: (Liu et al., 2020) detected the OOD samples by an energy-based score function on the final output logits.
Revised: (Liu et al., 2020) detected the OOD samples by an energy-based score function on the final output logits.

Original: Note that such energy function is different from ours as it calculates such score merely based on the output logits of the testing sample instead of comparing with stored patterns.
Revised: Note that such energy function is different from ours as it calculates such score merely based on the output logits of the testing sample instead of comparing with stored patterns.

Original: Based on the observation that the mean activation of the OOD sample had larger variations, (Sun et al., 2021) set a threshold to clip the output of the penultimate layer, thereby reducing the output magnitude of OOD samples in the last layer.
Revised: Based on the observation that the mean activation of the OOD sample had larger variations, (Sun et al., 2021) set a threshold to clip the output of the penultimate layer, thereby reducing the output magnitude of OOD samples in the last layer.

Original: These approaches can be directly applied to
Revised: These approaches can be directly applied to

Original: OOD detection tasks without additional training, which is more practical for real-world applications.
Revised: OOD detection tasks without additional training, which is more practical for real-world applications.

Original: (Sehwag et al., 2021) utilize the advantage of self-supervised training.
Revised: (Sehwag et al., 2021) utilize the advantage of self-supervised training.

Original: (Sun et al., 2022) calculateall the Euclidean distance with each pattern of train sample and use the k-th sorted distance as the metric for OOD detection.
Revised: (Sun et al., 2022) calculate

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we first introduce the preliminaries of the OOD detection task and the Hopfield Network.
Revised: In this section, we first introduce the preliminaries of the OOD detection task and the Hopfield Network.

Original: Then we elaborate on how to leverage the critical concepts of the Hopfield Network on OOD detection.
Revised: Then we elaborate on how to leverage the critical concepts of the Hopfield Network on OOD detection.

Original: More precisely, the energy function defined in the Modern Hopfield Network (Ramsauer et al., 2020) is introduced as the basis of our store-then-compare OOD detection paradigm.
Revised: More precisely, the energy function defined in the Modern Hopfield Network (Ramsauer et al., 2020) is introduced as the basis of our store-then-compare OOD detection paradigm.

Original: A simplified energy function is further proposed to reduce the memory demand, which is of high computational efficiency and free of hyperparameters.
Revised: A simplified energy function is further proposed to reduce the memory demand, which is of high computational efficiency and free of hyperparameters.

Original: Finally, we compare the difference in pattern choice, i.e., patterns derived from the penultimate layer outputs versus final logits.
Revised: Finally, we compare the difference in pattern choice, i.e., patterns derived from the penultimate layer outputs versus final logits.

Original: We also conduct in-depth theoretical analyses of our method.
Revised: We also conduct in-depth theoretical analyses of our method.

Original: 3. O UT - OF -D ISTRIBUTION D ETECTION
Revised: 3. O UT - OF -D ISTRIBUTION D ETECTION

Original: A neural network f aims to learn a mapping function from a training sample x to its corresponding label y as y =
Revised: A neural network f aims to learn a mapping function from a training sample x to its corresponding label y as y =

Original: f
Revised: f

Original: ( x ; θ ) with parameter θ .
Revised: ( x ; θ ) with parameter θ .

Original: Then a testing sample x ′ is fed into the trained network f for the prediction y ′ .
Revised: Then a testing sample x ′ is fed into the trained network f for the prediction y ′ .

Original: When x ′ and training sample x are from the same data distribution, then x ′ is called an ID sample; otherwise, it is regarded as an OOD sample.
Revised: When x ′ and training sample x are from the same data distribution, then x ′ is called an ID sample; otherwise, it is regarded as an OOD sample.

Original: Prediction results for OOD samples in turn fail to be meaningful.
Revised: Prediction results for OOD samples in turn fail to be meaningful.

Original: More severely, blindly classifying OOD samples into any existing class may raise fatal risks in safety-critical scenarios.
Revised: More severely, blindly classifying OOD samples into any existing class may raise fatal risks in safety-critical scenarios.

Original: Thus, the OOD detection task is to design a measure function D ( f ; x ′ ) that allows OOD samples to be as clearly distinguishable from
Revised: Thus, the OOD detection task is to design a measure function D ( f ; x ′ ) that allows OOD samples to be as clearly distinguishable from

Original: ID samples as possible.
Revised: ID samples as possible.

Original: Eventually, OOD detection can be formulated as follows:
Revised: Eventually, OOD detection can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: 3. H OPFIELD N ETWORK
Revised: 3. H OPFIELD N ETWORK

Original: Hopfield Network (Hopfield, 1984; Krotov & Hopfield, 2016; Ramsauer et al., 2020) can store and retrieve continuous patterns.
Revised: Hopfield Network (Hopfield, 1984; Krotov & Hopfield, 2016; Ramsauer et al., 2020) can store and retrieve continuous patterns.

Original: By minimizing the predefined energy function, it can gradually update the input test pattern ξ ∈ R d × 1 to a certain converged pattern that is similar to one of the stored patterns.
Revised: By minimizing the predefined energy function, it can gradually update the input test pattern ξ ∈ R d × 1 to a certain converged pattern that is similar to one of the stored patterns.

Original: We denote all stored patterns as a stored pattern set S ∈ R d × N with each column s j ∈ R d × 1 representing one specific stored pattern, and N is the total number of the stored pattern.
Revised: We denote all stored patterns as a stored pattern set S ∈ R d × N with each column s j ∈ R d × 1 representing one specific stored pattern, and N is the total number of the stored pattern.

Original: Here d is the dimension of patterns.
Revised: Here d is the dimension of patterns.

Original: Thus, the energy function aims to guide the updating procedure in the Modern Hopfield Network can be written as:
Revised: Thus, the energy function aims to guide the updating procedure in the Modern Hopfield Network can be written as:

Original: [Equation]
Revised: [Equation]

Original: where LSE denotes the log-sum-exp function and is defined in Eq.
Revised: where LSE denotes the log-sum-exp function and is defined in Eq.

Original: β and c are two constant.
Revised: β and c are two constant.

Original: The vector e denotes ξ T S , where e j represents the inner product of the input test pattern ξ and the j -th stored pattern s j .
Revised: The vector e denotes ξ T S , where e j represents the inner product of the input test pattern ξ and the j -th stored pattern s j .

Original: The second term ξ T ξ on the right of Eq. 2 serves as a regularization on the magnitude of ξ .
Revised: The second term ξ T ξ on the right of Eq. 2 serves as a regularization on the magnitude of ξ .

Original: Revisiting the energy function, we can find that it is essentially a measure that depicts the similarity between training patterns and testing patterns.
Revised: Revisiting the energy function, we can find that it is essentially a measure that depicts the similarity between training patterns and testing patterns.

Original: 3. HE : OOD DETECTION WITH M ODERN H OPFIELD E NERGY
Revised: 3. HE : OOD DETECTION WITH M ODERN H OPFIELD E NERGY

Original: As described above, it is obvious that the energy function of Modern Hopfield Network can be an appropriate candidate for measuring the discrepancy between OOD instances and ID instances.
Revised: As described above, it is obvious that the energy function of Modern Hopfield Network can be an appropriate candidate for measuring the discrepancy between OOD instances and ID instances.

Original: We denote all stored patterns for class i as a stored pattern set S i ∈ R d × N i with each column s
Revised: We denote all stored patterns for class i as a stored pattern set S i ∈ R d × N i with each column s

Original: ij ∈ R d ×representing one specific stored pattern, and N i is the total number of the stored patterns within class i .
Revised: ij ∈ R d ×representing one specific stored pattern, and N i is the total number of the stored patterns within class i .

Original: More precisely, we preserve the penultimate layer outputs of ID training samples as stored patterns: for each class i , a corresponding stored pattern set S i is derived from the penultimate layer outputs of ID training samples that are correctly classified by the network within this class.
Revised: More precisely, we preserve the penultimate layer outputs of ID training samples as stored patterns: for each class i , a corresponding stored pattern set S i is derived from the penultimate layer outputs of ID training samples that are correctly classified by the network within this class.

Original: When testing a new sample, we can obtain its penultimate layer output ξ as well as its prediction result, e.g., class i , from the trained model.
Revised: When testing a new sample, we can obtain its penultimate layer output ξ as well as its prediction result, e.g., class i , from the trained model.

Original: Then, we just need to conduct the calculation between ξ and the corresponding stored pattern set S i to obtain the similarity score.
Revised: Then, we just need to conduct the calculation between ξ and the corresponding stored pattern set S i to obtain the similarity score.

Original: Notice that, there is the magnitude regularization item ξ T ξ in the energy function Eq. 2.
Revised: Notice that, there is the magnitude regularization item ξ T ξ in the energy function Eq. 2.

Original: Since it is introduced to prevent the input pattern from scaling-up during the pattern updating process of
Revised: Since it is introduced to prevent the input pattern from scaling-up during the pattern updating process of

Original: Modern Hopfield Network, it is not necessary when measuring the discrepancy between the input pattern and the stored patterns.
Revised: Modern Hopfield Network, it is not necessary when measuring the discrepancy between the input pattern and the stored patterns.

Original: Thus we omit such a term as the magnitude of the input pattern ξ can also provide some information for the ID and OOD discrimination.
Revised: Thus we omit such a term as the magnitude of the input pattern ξ can also provide some information for the ID and OOD discrimination.

Original: Also, we omit the constant c in Eq.
Revised: Also, we omit the constant c in Eq.

Original: 2 because the constant in the measure function does not change the OOD detection result.
Revised: 2 because the constant in the measure function does not change the OOD detection result.

Original: In summary, the Hopfield energy-based OOD detection measure can be denoted as ( i denotes the classfication result of ξ by the already trained model):
Revised: In summary, the Hopfield energy-based OOD detection measure can be denoted as ( i denotes the classfication result of ξ by the already trained model):

Original: [Equation]
Revised: [Equation]

Original: Note that, Eq.
Revised: Note that, Eq.

Original: 4 measures the similarity instead of discrepancy, which means the higher score indicates ID data while lower score as OOD data.
Revised: 4 measures the similarity instead of discrepancy, which means the higher score indicates ID data while lower score as OOD data.

Original: It is worth mentioning that, different from most traditional OOD detection methods that only consider the information from the input test sample itself with the trained network, we leverage the information from all training samples of the predicted class to make the comparison for better OOD detection.
Revised: It is worth mentioning that, different from most traditional OOD detection methods that only consider the information from the input test sample itself with the trained network, we leverage the information from all training samples of the predicted class to make the comparison for better OOD detection.

Original: 3. SHE : OOD DETECTION WITH S IMPLIFIED H OPFIELD E NERGY
Revised: 3. SHE : OOD DETECTION WITH S IMPLIFIED H OPFIELD E NERGY

Original: Although HE has the theoretical foundation from the Modern Hopfield network and is proven to be effective through empirical evaluation, the need to store patterns of all correctly classified training samples may prevent it from generalization to real-world applications.
Revised: Although HE has the theoretical foundation from the Modern Hopfield network and is proven to be effective through empirical evaluation, the need to store patterns of all correctly classified training samples may prevent it from generalization to real-world applications.

Original: Particularly when the scale of the dataset is extremely large or the latent representation is ultra-high dimensional, it will impose a considerable burden on the storage and the computation.
Revised: Particularly when the scale of the dataset is extremely large or the latent representation is ultra-high dimensional, it will impose a considerable burden on the storage and the computation.

Original: During the evaluation, we discover that the hyperparameter β in Eq.
Revised: During the evaluation, we discover that the hyperparameter β in Eq.

Original: 4 should be small enough in case that there is any element e ij (denotes the inner product between the testing pattern and the j -th stored pattern of class i ) with extra large value, which will degrade the robustness of OOD detection with large β .
Revised: 4 should be small enough in case that there is any element e ij (denotes the inner product between the testing pattern and the j -th stored pattern of class i ) with extra large value, which will degrade the robustness of OOD detection with large β .

Original: When β is relatively small, we can transform Eq.
Revised: When β is relatively small, we can transform Eq.

Original: 4 with Taylor series (here we use two Taylor series as exp( x ) ≈ 1 + x and log(1 + x ) ≈ x ) by:
Revised: 4 with Taylor series (here we use two Taylor series as exp( x ) ≈ 1 + x and log(1 + x ) ≈ x ) by:

Original: [Equation]
Revised: [Equation]

Original: All test samples that are predicted to be class i share the same stored patterns size N i , so the first term β − 1 log N i remains the same for them and can be regarded as a constant.
Revised: All test samples that are predicted to be class i share the same stored patterns size N i , so the first term β − 1 log N i remains the same for them and can be regarded as a constant.

Original: Thus, we ignore the first term, and the measure function Eq.
Revised: Thus, we ignore the first term, and the measure function Eq.

Original: 4 can be simplified from the LSE function (Eq. 3) to the inner product of ξ and ¯ S i  :
Revised: 4 can be simplified from the LSE function (Eq. 3) to the inner product of ξ and ¯ S i because LSE( β, e i ) is positively related to ξ T ¯ S i :

Original: [Equation]
Revised: [Equation]

Original: Here ¯ S i is defined as:
Revised: Here ¯ S i is defined as:

Original: [Equation]
Revised: [Equation]

Original: We can interpret SHE from another perspective: ¯ S i can be viewed as the average of vectors from the stored pattern set S i .
Revised: We can interpret SHE from another perspective: ¯ S i can be viewed as the average of vectors from the stored pattern set S i .

Original: In other words, a stored pattern set S i is degraded to a representation vector ¯ S i .
Revised: In other words, a stored pattern set S i is degraded to a representation vector ¯ S i .

Original: Considering the redundancy of patterns that frequently appears in deep learning and samples from the same class usually have similar patterns, it is reasonable to prune the stored pattern set into a single average vector, which is also shown in Figure.
Revised: Considering the redundancy of patterns that frequently appears in deep learning and samples from the same class usually have similar patterns, it is reasonable to prune the stored pattern set into a single average vector, which is also shown in Figure.

Original: 1 (right).
Revised: 1 (right).

Original: It enables us to eliminate the memory overhead of storing a large amount of ID data patterns, and further reduces the computational cost as well.
Revised: It enables us to eliminate the memory overhead of storing a large amount of ID data patterns, and further reduces the computational cost as well.

Original: Besides, the only hyperparameter β also disappears, indicating that we do not need to tune any hyperparameter.
Revised: Besides, the only hyperparameter β also disappears, indicating that we do not need to tune any hyperparameter.

Original: In summary, SHE is highly efficient regarding both storage and computation and does not have any hyper parameter to tune.
Revised: In summary, SHE is highly efficient regarding both storage and computation and does not have any hyper parameter to tune.

Original: Such properties can indeed facilitate the deployment of SHE in practice.
Revised: Such properties can indeed facilitate the deployment of SHE in practice.

Original: 3. P ENULTIMATE L AYER VERSUS L OGITS L AYER FOR P ATTERN S TORAGE
Revised: 3. P ENULTIMATE L AYER VERSUS L OGITS L AYER FOR P ATTERN S TORAGE

Original: In this section, we analyze the benefits of choosing outputs from the penultimate layer, compared with the logits layer, as the pattern.
Revised: In this section, we analyze the benefits of choosing outputs from the penultimate layer, compared with the logits layer, as the pattern.

Original: Intuitive Explanation.
Revised: Intuitive Explanation.

Original: Given that most OOD detection methods usually use the output logits, one interesting question is how to choose the layer output as the patterns, the penultimate layer output, or the output logits.
Revised: Given that most OOD detection methods usually use the output logits, one interesting question is how to choose the layer output as the patterns, the penultimate layer output, or the output logits.

Original: Note that, when we calculate the energy function by Eq. 6, the stored pattern ¯
Revised: Note that, when we calculate the energy function by Eq. 6, the stored pattern ¯

Original: S i comes from the same category i as the testing pattern ξ is classified.
Revised: S i comes from the same category i as the testing pattern ξ is classified.

Original: Therefore, when we use the output logits as patterns, no matter ξ comes from an ID or OOD sample, its maximum value position of logits is always the same as the maximum value position of logits of ¯ S i by design, which is the category index of ξ .
Revised: Therefore, when we use the output logits as patterns, no matter ξ comes from an ID or OOD sample, its maximum value position of logits is always the same as the maximum value position of logits of ¯ S i by design, which is the category index of ξ .

Original: We call such alignment of maximum value position of logits as “Peak
Revised: We call such alignment of maximum value position of logits as “Peak

Original: Alignment” which leads to a high energy function score for the OOD pattern more easily.
Revised: Alignment” which leads to a high energy function score for the OOD pattern more easily.

Original: It in turn raises the difficulty of discriminating ID and OOD samples.
Revised: It in turn raises the difficulty of discriminating ID and OOD samples.

Original: However, when we use the penultimate layer output as patterns, there is no such “Peak Alignment” effect between ξ and ¯ S i since the value of the penultimate layer output is not so concentrated distributed, promoting the similarity score calculated from energy function more separable for ID and OOD patterns as shown in Figure.
Revised: However, when we use the penultimate layer output as patterns, there is no such “Peak Alignment” effect between ξ and ¯ S i since the value of the penultimate layer output is not so concentrated distributed, promoting the similarity score calculated from energy function more separable for ID and OOD patterns as shown in Figure.

Original: Moreover, we provide the theoretical analysis in the Appendix B. 
Revised: Moreover, we provide the theoretical analysis in the Appendix B. respectively.

Original: When using penultimate layer, the score can be distinguished more for ID and OOD samples.
Revised: When using penultimate layer, the score can be distinguished more for ID and OOD samples.

Original: The
Revised: The

Original: ID and OOD is CIFAR

Revised: ID and OOD is CIFAR

Original: (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011) respectively, and the backbone network is ResNet18.
Revised: (Krizhevsky et al., 2009) and SVHN (Netzer et al., 2011) respectively, and the backbone network is ResNet18.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct experiments on nine OOD detection datasets with three backbone networks and two ID datasets to evaluate the performance of our methods.
Revised: In this section, we conduct experiments on nine OOD detection datasets with three backbone networks and two ID datasets to evaluate the performance of our methods.

Original: 4. D ATASET
Revised: 4. D ATASET

Original: There are two types of datasets in the experiments: The in-distribution (ID) dataset and the Outof-distribution (OOD) dataset.
Revised: There are two types of datasets in the experiments: The in-distribution (ID) dataset and the Outof-distribution (OOD) dataset.

Original: The former is only utilized during the training procedure, while the latter serves to test models and does not contain any ID dataset sample.
Revised: The former is only utilized during the training procedure, while the latter serves to test models and does not contain any ID dataset sample.

Original: ID Dataset.
Revised: ID Dataset.

Original: CIFAR10, CIFAR100 and ImageNet-1k are three ID datasets in our experiments.
Revised: CIFAR10, CIFAR100 and ImageNet-1k are three ID datasets in our experiments.

Original: CIFAR10 (Krizhevsky et al., 2009) is composed of 60,000 images with 10 categories, each containing 5,000 training images and 1,000 testing images.
Revised: CIFAR10 (Krizhevsky et al., 2009) is composed of 60,000 images with 10 categories, each containing 5,000 training images and 1,000 testing images.

Original: CIFAR

Revised: CIFAR

Original: Krizhevsky et al.
Revised: Krizhevsky et al.

Original: (2009) consists of 60,000 images with 100 categories with 500 training images and 100 testing images for each class.
Revised: (2009) consists of 60,000 images with 100 categories with 500 training images and 100 testing images for each class.

Original: ImageNet-1k (Deng et al., 2009) is composed of 1,350,000 images 1000 different object categories, each containing 1,300 training images and 50 testing images.
Revised: ImageNet-1k (Deng et al., 2009) is composed of 1,350,000 images 1000 different object categories, each containing 1,300 training images and 50 testing images.

Original: OOD Dataset.
Revised: OOD Dataset.

Original: There are nine OOD dataset for evaluation, including SVHN (Netzer et al., 2011),
Revised: There are nine OOD dataset for evaluation, including SVHN (Netzer et al., 2011),

Original: LSUN-C (Yu et al., 2015) (crop) and LSUN-R (Yu et al., 2015) (resize), iSUN (Xu et al., 2015),
Revised: LSUN-C (Yu et al., 2015) (crop) and LSUN-R (Yu et al., 2015) (resize), iSUN (Xu et al., 2015),

Original: Places (Zhou et al., 2017), DTD (Cimpoi et al., 2014), SUN (Xiao et al., 2010), iNaturalist (Van Horn et al., 2018), Tiny-Imagenet (resize)
Revised: Places (Zhou et al., 2017), DTD (Cimpoi et al., 2014), SUN (Xiao et al., 2010), iNaturalist (Van Horn et al., 2018), Tiny-Imagenet (resize)

Original: (Deng et al., 2009).
Revised: (Deng et al., 2009).

Original: Details can be found in the original papers.
Revised: Details can be found in the original papers.

Original: 4. E XPERIMENT S ETTINGS
Revised: 4. E XPERIMENT S ETTINGS

Original: Backbone Network.
Revised: Backbone Network.

Original: We choose ResNet18 (He et al., 2016), ResNet

Revised: We choose ResNet18 (He et al., 2016), ResNet

Original: (He et al., 2016)
Revised: (He et al., 2016)

Original: and
Revised: and

Original: WRN40-2 (Zagoruyko & Komodakis, 2016) as our backbone networks, which are trained on the
Revised: WRN40-2 (Zagoruyko & Komodakis, 2016) as our backbone networks, which are trained on the

Original: ID dataset CIFAR10 and CIFAR100, respectively.
Revised: ID dataset CIFAR10 and CIFAR100, respectively.

Original: For ImageNet, we choose ResNet50 (He et al., 2016) as our backbone network.
Revised: For ImageNet, we choose ResNet50 (He et al., 2016) as our backbone network.

Original: Baseline Methods.
Revised: Baseline Methods.

Original: To evaluate the performance of our proposed design, We also conduct experiments on the Softmax-based approach “MSP” (Hendrycks & Gimpel, 2016) and other excellent methods, “Energy” (Liu et al., 2020) , “ODIN” (Liang et al., 2017), “Mahalanobis” (Lee et al., 2018) and “ReAct” (Sun et al., 2021), the “ReAct” here is combined with Energy as described in (Sun et al., 2021) and is the state-of-the-art method before.
Revised: To evaluate the performance of our proposed design, We also conduct experiments on the Softmax-based approach “MSP” (Hendrycks & Gimpel, 2016) and other excellent methods, “Energy” (Liu et al., 2020) , “ODIN” (Liang et al., 2017), “Mahalanobis” (Lee et al., 2018) and “ReAct” (Sun et al., 2021), the “ReAct” here is combined with Energy as described in (Sun et al., 2021) and is the state-of-the-art method before.

Original: Evaluation Metrics .
Revised: Evaluation Metrics .

Original: The evaluation metrics are: (1) the False Positive Rate (FPR95) of OOD samples when the True Positive Rate of in-distribution samples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC).
Revised: The evaluation metrics are: (1) the False Positive Rate (FPR95) of OOD samples when the True Positive Rate of in-distribution samples is at 95%; (2) the area under the receiver operating characteristic curve (AUROC).

Original: Models with smaller FPR95 and higher AUROC results are regarded as effective.
Revised: Models with smaller FPR95 and higher AUROC results are regarded as effective.

Original: All experimental values are expressed as percentages (%) and the bolded numbers (sometimes colored with gray cell background) denote the best result.
Revised: All experimental values are expressed as percentages (%) and the bolded numbers (sometimes colored with gray cell background) denote the best result.

Original: 4. E XPERIMENTAL R ESULT
Revised: 4. E XPERIMENTAL R ESULT

Original: and Energy (Liu et al., 2020).

Original: 4.3. O VERALL RESULTS OF SHE .
Revised: 4.3. O VERALL RESULTS OF SHE .

Original: In this section, we demonstrate the effectiveness of our method “SHE” through extensive experiments.
Revised: In this section, we demonstrate the effectiveness of our method “SHE” through extensive experiments.

Original: The experimental results on nine OOD datasets are organized in Table 1 taking CIFAR as ID training data.
Revised: The experimental results on nine OOD datasets are organized in Table 1 taking CIFAR as ID training data.

Original: As shown in Table 1, MSP has the worse results, while Energy has a lot of improvement, illustrating the potential of energy-based solutions for OOD detection.
Revised: As shown in Table 1, MSP has the worse results, while Energy has a lot of improvement, illustrating the potential of energy-based solutions for OOD detection.

Original: Our approach
Revised: Our approach

Original: SHE obtains almost all (26/27) of the best performance on nine OOD datasets for three different backbones.
Revised: SHE obtains almost all (26/27) of the best performance on nine OOD datasets for three different backbones.

Original: More precisely, our approach reduces the average FPR95 by 16.81% for ResNet18, 6.29% for ResNet34, and 12.18% for WRN40-2 compared with the best baseline.
Revised: More precisely, our approach reduces the average FPR95 by 16.81% for ResNet18, 6.29% for ResNet34, and 12.18% for WRN40-2 compared with the best baseline.

Original: Our approach performs well when CIFAR100 or ImageNet-1k is choosed as the ID training data. For the space limitation,we put the detailed table 5 (CIFAR100) and table 7 (ImageNet-1k)  in the Appendix A.
Revised: Our approach performs well when CIFAR100 or ImageNet-1k is choosed as the ID training data while with some limitation, the detailed table 5 (CIFAR100) and table 7 (ImageNet-1k) is put in the Appendix A.

Original: We also illustrate Figure 3 to demonstrate model performance directly.
Revised: We also illustrate Figure 3 to demonstrate model performance directly.

Original: Under the peak of ID score distribution (colored red), the less data in blue indicates the better capability in distinguish OOD data and ID data.
Revised: Under the peak of ID score [Table]

Original: It can be seen that SHE can obtain the minimum blue areas under the red peak among three methods.
Revised: [Table]

Original: 4.3. SHE VERSUS HE
Revised: 4.3. SHE VERSUS HE

Original: Notice that SHE is derived from HE that is inspired by Modern Hopfield Network, we make a comparison between them.
Revised: Notice that SHE is derived from HE that is inspired by Modern Hopfield Network, we make a comparison between them.

Original: We can discover from Table 2 that SHE and HE are competitive, indicating[Table] [Table] that the patterns of samples from the same class are similar and contain redundancy for OOD detection.
Revised: We can discover from Table 2 that SHE and HE are competitive, indicating that the patterns of samples from the same class are similar and contain redundancy for OOD detection.

Original: As mentioned in chapter [Table] HE detects OOD samples via the energy function as shown in Eq.
Revised: As mentioned in chapter 3, HE detects OOD samples via the energy function as shown in Eq.

Original: We evaluate the feasibility of HE and show the results in Table [Table] which demonstrates that the energy function (Eq. 4) is able for OOD detection.
Revised: We evaluate the feasibility of HE and show the results in Table 2, which demonstrates that the energy function (Eq. 4) is able for OOD detection.

Original: For SHE, memory cost is reduced to the number of classes instead of the number of samples and the energy-based measure can be simplified into frequently used inner products without any hyperparameter, which is elegant and efficient.
Revised: For SHE, memory cost is reduced to the number of classes instead of the number of samples and the energy-based measure can be simplified into frequently used inner products without any hyperparameter, which is elegant and efficient.

Original: We also provide detailed comparison results on nine OOD detection in Table 8, proving that in most cases
Revised: We also provide detailed comparison results on nine OOD detection in Table 9, proving that in most cases

Original: SHE can achieve similar performance to HE.
Revised: SHE can achieve similar performance to HE.

Original: 4.3. P ERTURBATIONS ON SHE
Revised: 4.3. P ERTURBATIONS ON SHE

Original: Adding perturbations to input samples for OOD detection is proposed by (Liang et al., 2017), which can be formulated as follows:
Revised: Adding perturbations to input samples for OOD detection is proposed by (Liang et al., 2017), which can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original:  where x ′ denotes the testing sample to be detected while ˜ x ′ is the perturbed one.
Revised: (8) where x ′ denotes the testing sample to be detected while ˜ x ′ is the perturbed one.

Original: S ( x ′ ) is the maximum Softmax probability of network outputs, and ε is the perturbation magnitude.
Revised: S ( x ′ ) is the maximum Softmax probability of network outputs, and ε is the perturbation magnitude.

Original: Such perturbation-based methods have been proved to be more beneficial for ID data than OOD data, and are adopted in extensive studies (Lee et al., 2018; Hsu et al., 2020; DeVries & Taylor, 2018).
Revised: Such perturbation-based methods have been proved to be more beneficial for ID data than OOD data, and are adopted in extensive studies (Lee et al., 2018; Hsu et al., 2020; DeVries & Taylor, 2018).

Original: To verify whether perturbation applies to SHE, we also carry out the experiments and organize the results in Table 3.
Revised: To verify whether perturbation applies to SHE, we also carry out the experiments and organize the results in Table 3.

Original: It demonstrates that introducing perturbations can be beneficial to SHE as the performance of SHE could be further improved with perturbations.
Revised: It demonstrates that introducing perturbations can be beneficial to SHE as the performance of SHE could be further improved with perturbations.

Original: Note that, the perturbation requires an additional “forward-backward” procedure to retrieve the gradient, leading to an increase in computational complexity and time.
Revised: Note that, the perturbation requires an additional “forward-backward” procedure to retrieve the gradient, leading to an increase in computational complexity and time.

Original: As a comparison, we record the time overhead before and after adding perturbation to SHE: when we choose ResNet34 as the backbone with CIFAR100 as ID data and TinyImagenet as OOD data, the consuming time raises from 35 .
Revised: As a comparison, we record the time overhead before and after adding perturbation to SHE: when we choose ResNet34 as the backbone with CIFAR100 as ID data and TinyImagenet as OOD data, the consuming time raises from 35 .

Original: 61 s to 105 .
Revised: 61 s to 105 .

Original: 88 s that is around 3 times.
Revised: 88 s that is around 3 times.

Original: Therefore, a balance between efficiency and computational overhead needs to be considered when adopting perturbation.
Revised: Therefore, a balance between efficiency and computational overhead needs to be considered when adopting perturbation.

Original: Nevertheless, SHE is always efficient and effective, which can be combined with perturbations to achieve even higher accuracy.
Revised: Nevertheless, SHE is always efficient and effective, which can be combined with perturbations to achieve even higher accuracy.

Original: [Table]
Revised: [Table]

Original: All values are averaged over the nine OOD datasets described in Section 4.1.
Revised: All values are averaged over the nine OOD datasets described in Section 4.1.

Original: The detailed Table is displayed in the 
Revised: The detailed Tableis displayed in the Appendix

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: P ENULTIMATE LAYER VS . L OGITS LAYER
Revised: P ENULTIMATE LAYER VS . L OGITS LAYER

Original: Note that all the stored patterns or test patterns in our approach are obtained from the penultimate layer output of neural networks.
Revised: Note that all the stored patterns or test patterns in our approach are obtained from the penultimate layer output of neural networks.

Original: But in most methods, logits layer (i.e., the last layer) output is adopted for the confidence computation (Hendrycks & Gimpel, 2016;
Revised: But in most methods, logits layer (i.e., the last layer) output is adopted for the confidence computation (Hendrycks & Gimpel, 2016;

Original: Liang et al., 2017; Liu et al., 2020; Sun et al., 2021).
Revised: Liang et al., 2017; Liu et al., 2020; Sun et al., 2021).

Original: To verify the significance of layer selection, we also provide an experimental evaluation on SHE that chooses patterns from the penultimate layer outputs or final logits, respectively, for comparison.
Revised: To verify the significance of layer selection, we also provide an experimental evaluation on SHE that chooses patterns from the penultimate layer outputs or final logits, respectively, for comparison.

Original: The average results on nine datasets are presented in Table 4.
Revised: The average results on nine datasets are presented in Table 4.

Original: We can conclude that our approach gets much better performance when using the penultimate layer output, instead of final logits, as patterns to apply SHE.
Revised: We can conclude that our approach gets much better performance when using the penultimate layer output, instead of final logits, as patterns to apply SHE.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel approach named HE for OOD detection based on a new “storethen-compare” paradigm.
Revised: In this paper, we propose a novel approach named HE for OOD detection based on a new “storethen-compare” paradigm.

Original: The key idea is to store patterns of ID data and then leverage the energy function defined in the Modern Hopfield Network (Ramsauer et al., 2020) for measuring the similarity between the new test patterns and the stored ID patterns.
Revised: The key idea is to store patterns of ID data and then leverage the energy function defined in the Modern Hopfield Network (Ramsauer et al., 2020) for measuring the similarity between the new test patterns and the stored ID patterns.

Original: To reduce storage and computational overhead, we simplify the energy function with the theoretical analysis by appropriate approximations and obtain the simplified approach named SHE .
Revised: To reduce storage and computational overhead, we simplify the energy function with the theoretical analysis by appropriate approximations and obtain the simplified approach named SHE .

Original: In addition to the great efficiency and effectiveness, SHE does not have any hyperparameters to tune, which is more convenient than most
Revised: In addition to the great efficiency and effectiveness, SHE does not have any hyperparameters to tune, which is more convenient than most

Original: OOD detection methods with cumbersome hyperparameters tuning.
Revised: OOD detection methods with cumbersome hyperparameters tuning.

Original: Also, different from most OOD detection methods focusing on final output logits, we find that the penultimate layer output, rather than the final output logits, is more suitable to be used as patterns in our approach for OOD detection.
Revised: Also, different from most OOD detection methods focusing on final output logits, we find that the penultimate layer output, rather than the final output logits, is more suitable to be used as patterns in our approach for OOD detection.

Original: The conducted evaluations demonstrate the superiority of our proposed simple yet effective approach on nine widely-used OOD datasets.
Revised: The conducted evaluations demonstrate the superiority of our proposed simple yet effective approach on nine widely-used OOD datasets.

Original: A E XPERIMENTAL R ESULTS .
Revised: A E XPERIMENTAL R ESULTS .

Original: [Table]
Revised: [Table]

Original: The ReAct in this table refers to ‘ Energy + ReAct ’ as described in Sun et al.
Revised: The ReAct in this table refers to ‘ Energy + ReAct ’ as described in Sun et al.

Original: (2021), as an auxillary method combined with other SOTA methods, we can also combine the ReAct with SHE (better than
Revised: (2021), as an auxillary method combined with other SOTA methods, we can also combine the ReAct with SHE (better than

Original: Energy ) , which will also outperform SHE itself
Revised: Energy ) , which will also outperform SHE itself

Original: , the results are shown in Table.
Revised: , the results are shown in Table.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: -(cid:26)(cid:12)(cid:3)(cid:9)(cid:10)(cid:11)(cid:12)(cid:13)(cid:14)(cid:15)(cid:13)(cid:16)(cid:4)(cid:17)(cid:18)(cid:19)(cid:20)(cid:21)(cid:14)(cid:14)(cid:18)(cid:2)(cid:22)(cid:14)(cid:23)(cid:24) (cid:4)(cid:3)(cid:4)
Revised: -(cid:26)(cid:12)(cid:3)(cid:9)(cid:10)(cid:11)(cid:12)(cid:13)(cid:14)(cid:15)(cid:13)(cid:16)(cid:4)(cid:17)(cid:18)(cid:19)(cid:20)(cid:21)(cid:14)(cid:14)(cid:18)(cid:2)(cid:22)(cid:14)(cid:23)(cid:24) (cid:4)(cid:3)(cid:4)

Original: Let y :=
Revised: Let y :=

Original: [ y 1 , y 2 , ...y m ] and z :=
Revised: [ y 1 , y 2 , ...y m ] and z :=

Original: [ z 1 , z 2 , ...
Revised: [ z 1 , z 2 , ...

Original: z n ] be the output logits and the penultimate layer output, respectively.
Revised: z n ] be the output logits and the penultimate layer output, respectively.

Original: Here, m is the dimension of logits output (i.e., the number of classes) and n is the dimension of the penultimate layer output.
Revised: Here, m is the dimension of logits output (i.e., the number of classes) and n is the dimension of the penultimate layer output.

Original: We superscript the vector id and ood (e.g., y id denotes logits output derived from an ID sample) to indicate the input type of the vector.
Revised: We superscript the vector id and ood (e.g., y id denotes logits output derived from an ID sample) to indicate the input type of the vector.

Original: For every z oodj in z ood , we assume that they are independent random variables following the same Gaussian distribution, i.e., z oodj ∼ N (cid:0) 0 , σ 2 (cid:1) .
Revised: For every z oodj in z ood , we assume that they are independent random variables following the same Gaussian distribution, i.e., z oodj ∼ N (cid:0) 0 , σ 2 (cid:1) .

Original: Let [ v 1 , v 2 , ... v m ] be the weight matrix of the last layer (linear layer) with each v j denotes the categorical vector for class i .
Revised: Let [ v 1 , v 2 , ... v m ] be the weight matrix of the last layer (linear layer) with each v j denotes the categorical vector for class i .

Original: Thus, we have y =
Revised: Thus, we have y =

Original: [ v 1 , v 2 ..., v m ] T z .
Revised: [ v 1 , v 2 ..., v m ] T z .

Original: For a test sample ξ from OOD data, we assume that it is classified as the category k , and we have the following formula.
Revised: For a test sample ξ from OOD data, we assume that it is classified as the category k , and we have the following formula.

Original: Among them, M represents a distribution of the maximum value of m Gaussian random variables.
Revised: Among them, M represents a distribution of the maximum value of m Gaussian random variables.

Original: [Equation]
Revised: [Equation]

Original: For y idk
Revised: For y idk

Original: [Equation]
Revised: [Equation]

Original: we assume that it follows another distribution I whose expected value is a positive number larger than the expected value of y oodq ̸
Revised: we assume that it follows another distribution I whose expected value is a positive number larger than the expected value of y oodq ̸

Original: = k .
Revised: = k .

Original: When calculated by Eq. 6, the expectation calculated from output logits and penultimate layer outputs are shown as follows ( ∗ denotes inner product):
Revised: When calculated by Eq. 6, the expectation calculated from output logits and penultimate layer outputs are shown as follows ( ∗ denotes inner product):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, compared with using the penultimate layer as the pattern to calculate the energy function, logits output will assign OOD samples with higher scores.
Revised: Therefore, compared with using the penultimate layer as the pattern to calculate the energy function, logits output will assign OOD samples with higher scores.

Original: Therefore, it will be more challenging to distinguish ID samples from OOD samples.
Revised: Therefore, it will be more challenging to distinguish ID samples from OOD samples.

Original: C A BLATION E XPERIMENT .
Revised: C A BLATION E XPERIMENT .

Original: The ablation experiment consists of two parts.
Revised: The ablation experiment consists of two parts.

Original: First, we evaluate different metrics instead of the inner product which is used in our approach to measuring the similarity of the stored pattern and test pattern.
Revised: First, we evaluate different metrics instead of the inner product which is used in our approach to measuring the similarity of the stored pattern and test pattern.

Original: To be specific, we use “Euclidean Distance” and “Cosine Similarity”, the results are shown in Table.
Revised: To be specific, we use “Euclidean Distance” and “Cosine Similarity”, the results are shown in Table.

Original: Second, we use the output from other layers (e.g., layer3 in ResNet) instead of the penultimate layer (layer4 in ResNet) to act as the representation of our approach.
Revised: Second, we use the output from other layers (e.g., layer3 in ResNet) instead of the penultimate layer (layer4 in ResNet) to act as the representation of our approach.

Original: The results are shown in Table.
Revised: The results are shown in Table.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D E XPERIMENTAL D ETAILS .
Revised: D E XPERIMENTAL D ETAILS .

Original: Software and Hardware.
Revised: Software and Hardware.

Original: The experiments are performed on one linux server (Operation system:
Revised: The experiments are performed on one linux server (Operation system:

Original: Ubuntu Linux 18.04.1).
Revised: Ubuntu Linux 18.04.1).

Original: For GPU resource, four NVIDIA GeForce RTX 3090 are used for ResNet and WRN.
Revised: For GPU resource, four NVIDIA GeForce RTX 3090 are used for ResNet and WRN.

Original: The python environment is 3.7 and the libraries we use to implement our experiments is
Revised: The python environment is 3.7 and the libraries we use to implement our experiments is

Original: PyTorch 1.12.1 ,
Revised: PyTorch 1.12.1 ,

Original: Number of Evaluation RunsReported
Revised: Number of Evaluation RunsReported

Original: Following (Liu et al., 2020; Hendrycks et al., 2018), performance for each OOD dataset is averaged over 10 random batches of samples.
Revised: Following (Liu et al., 2020; Hendrycks et al., 2018), performance for each OOD dataset is averaged over 10 random batches of samples.

Original: Training Details.
Revised: Training Details.

Original: During the training procedure, some data augmentation (e.g., rotation, flipping, resizing) and training techniques (e.g., learning rate decay) are adopted to improve the model’s accuracy.
Revised: During the training procedure, some data augmentation (e.g., rotation, flipping, resizing) and training techniques (e.g., learning rate decay) are adopted to improve the model’s accuracy.

Original: In more detail, we set the batch size as 128, the image size as 112 for ResNet, and 64 for
Revised: In more detail, we set the batch size as 128, the image size as 112 for ResNet, and 64 for

Original: WRN, respectively.
Revised: WRN, respectively.

Original: And we use SGD as the optimizer with 0.1 as the initial learning rate and apply the learning rate decay at epochs 50 and 75, respectively.
Revised: And we use SGD as the optimizer with 0.1 as the initial learning rate and apply the learning rate decay at epochs 50 and 75, respectively.