Original: M AXIMUM
Revised: M AXIMUM

Original: E NTROPY
Revised: E NTROPY

Original: I NFORMATION
Revised: I NFORMATION

Original: B OTTLENECK
Revised: B OTTLENECK

Original: FOR C ONFIDENCE
Revised: FOR C ONFIDENCE

Original: - AWARE S TOCHASTIC E MBEDDING
Revised: AWARE S TOCHASTIC E MBEDDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Stochastic embedding has several advantages over deterministic embedding, such as the capability of associating uncertainty with the resulting embedding and robustness to noisy data.
Revised: Stochastic embedding has several advantages over deterministic embedding, such as the capability of associating uncertainty with the resulting embedding and robustness to noisy data.

Original: This is especially useful when the input data has ambiguity (e.g., blurriness or corruption) which often happens with in-the-wild settings.
Revised: This is especially useful when the input data has ambiguity (e.g., blurriness or corruption) which often happens with in-the-wild settings.

Original: Many existing methods for stochastic embedding are limited by the assumption that the embedding follows a standard normal distribution under the variational information bottleneck principle.
Revised: Many existing methods for stochastic embedding are limited by the assumption that the embedding follows a standard normal distribution under the variational information bottleneck principle.

Original: We present a different variational approach to stochastic embedding in which maximum entropy acts as the bottleneck, which we call ”Maximum Entropy Information Bottleneck” or MEIB.
Revised: We present a different variational approach to stochastic embedding in which maximum entropy acts as the bottleneck, which we call ”Maximum Entropy Information Bottleneck” or MEIB.

Original: We show that models trained with the MEIB objective outperform existing methods in terms of regularization, perturbation robustness, probabilistic contrastive learning, and risk-controlled recognition performance.
Revised: We show that models trained with the MEIB objective outperform existing methods in terms of regularization, perturbation robustness, probabilistic contrastive learning, and risk-controlled recognition performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Stochastic embedding is a mapping of an input x to a random variable Z ∼ p ( z | x ) ∈ R D in which the mapped regions of similar inputs are placed nearby.
Revised: Stochastic embedding is a mapping of an input x to a random variable Z ∼ p ( z | x ) ∈ R D in which the mapped regions of similar inputs are placed nearby.

Original: Unlike deterministic embedding, where z = f ( x ) is a point in R D , stochastic embedding can represent the input uncertainty, such as data corruption or ambiguity, by controlling the spread of probability density over a manifold Oh et al.
Revised: Unlike deterministic embedding, where z = f ( x ) is a point in R D , stochastic embedding can represent the input uncertainty, such as data corruption or ambiguity, by controlling the spread of probability density over a manifold Oh et al.

Original: Figure 1 depicts a typical stochastic embedding framework with the neural networks parameterized by θ .
Revised: Figure 1 depicts a typical stochastic embedding framework with the neural networks parameterized by θ .

Original: Input x is mapped to a Gaussian distribution N ( z ; µ, Σ) by a stochastic encoder that consists of a backbone feature extractor
Revised: Input x is mapped to a Gaussian distribution N ( z ; µ , Σ) by a stochastic encoder that consists of a backbone feature extractor

Original: f Bθ followed by two separate branches f µθ and f Σ θ , each of which predicts the µ and Σ . 1 While the covariance matrix Σ , in prior work as well as in this paper, is assumed to be diagonal where f Σ θ outputs a D -dimensional vector, it would be straightforward to extend it to a full covariance matrix, for instance, using a Cholesky decomposition Dorta et al.
Revised: f Bθ followed by two separate branches f µ θ and f Σ θ , each of which predicts the µ and Σ . 1 While the covariance matrix Σ , in prior work as well as in this paper, is assumed to be diagonal where f Σ θ outputs a D -dimensional vector, it would be straightforward to extend it to a full covariance matrix, for instance, using a Cholesky decomposition Dorta et al.

Original: Embeddings sampled from this Gaussian are then consumed by a decoder f Cθ for the downstream task, e.g., classification.
Revised: Embeddings sampled from this Gaussian are then consumed by a decoder f Cθ for the downstream task, e.g., classification.

Original: Majority of leading methods for stochastic embedding Oh et al.
Revised: Majority of leading methods for stochastic embedding Oh et al.

Original: Chang et al.
Revised: Chang et al.

Original: Sun et al.
Revised: Sun et al.

Original: ; Chun et al.
Revised: ; Chun et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021b) are built upon the variational information bottleneck (VIB) principle Alemi et al.
Revised: (2021b) are built upon the variational information bottleneck (VIB) principle Alemi et al.

Original: (2017) where the stochastic encoder p ( z | x ) is regularized by Kullback–Leibler (KL) divergence, KL( p ( z | x ) || r ( z ) ), where p ( z | x ) =
Revised: (2017) where the stochastic encoder p ( z | x ) is regularized by Kullback–Leibler (KL) divergence, KL( p ( z | x ) || r ( z ) ), where p ( z | x ) =

Original: N ( z ; µ, Σ) and r ( z ) =
Revised: N ( z ; µ , Σ) and r ( z ) =

Original: N ( z ; 0 , I ) in general.
Revised: N ( z ; 0 , I ) in general.

Original: This effectively impels the embeddings to be close to a standard normal distribution, which is an explicit assumption that may not always hold true.
Revised: This effectively impels the embeddings to be close to a standard normal distribution, which is an explicit assumption that may not always hold true.

Original: [Figure]
Revised: [Figure]

Original: Furthermore, B¨utepage et al.
Revised: Furthermore, B¨utepage et al.

Original: (2021) showed that the standard variational autoencoder (VAE) trained with KL divergence from the standard normal prior Kingma & Welling (2014) fails to correlate the latent variance with the input uncertainty; the variance decreases with the distance to the latent means of training data, which is contrary to expectation.
Revised: (2021) showed that the standard variational autoencoder (VAE) trained with KL divergence from the standard normal prior Kingma & Welling (2014) fails to correlate the latent variance with the input uncertainty; the variance decreases with the distance to the latent means of training data, which is contrary to expectation.

Original: Since VAE is a special case of an unsupervised variant of VIB, this phenomenon also holds for VIB; our experiments show VIB assigns smaller variance to more uncertain inputs (see the supplemental Section A).
Revised: Since VAE is a special case of an unsupervised variant of VIB, this phenomenon also holds for VIB; our experiments show VIB assigns smaller variance to more uncertain inputs (see the supplemental Section A).

Revised: Motivated by this finding, we explicitly use the variance (entropy) as a confidence indicator rather than a measure of input uncertainties and encourage the model to assign larger variance to more certain inputs.

Original: In this paper, we propose Maximum Entropy Information Bottleneck (MEIB) to lift such constraints on the prior and instead use the conditional entropy of the embedding H ( Z | X ) as the only regularization.
Revised: In this paper, we propose Maximum Entropy Information Bottleneck (MEIB) to lift such constraints of using a fixed prior and instead use the conditional entropy of the embedding H ( Z | X ) as the only regularization.

Original: Based on the maximum entropy principle Jaynes (1957), we postulate that stochastic uncertainty is best represented by the probability distribution with the largest entropy.
Revised: Based on the maximum entropy principle Jaynes (1957), we postulate that stochastic uncertainty is best represented by the probability distribution with the largest entropy.

Original: By maximizing H ( Z | X ) , the embedding distribution is promoted to be more random, pushing for broader coverage in the embedding space, with a trade-off on the expressiveness of Z about target
Revised: By maximizing H ( Z | X ) , the embedding distribution is promoted to be more random, pushing for broader coverage in the embedding space, with a trade-off on the expressiveness of Z about target

Original: Y. The resulting distribution is also the one that makes the fewest assumptions about the true distribution of data Shore & Johnson (1980).
Revised: Y. The resulting distribution is also the one that makes the fewest assumptions about the true distribution of data Shore & Johnson (1980).

Original: Figure 2 depicts our intuition; (a) deterministic encoders would learn embeddings “just enough” to classify the training samples unless any regularization technique, such as a margin loss, is considered.
Revised: Figure 2 depicts our intuition; (a) deterministic encoders would learn embeddings “just enough” to classify the training samples unless any regularization technique, such as a margin loss, is considered.

Original: It would be vulnerable to small changes in test inputs.
Revised: It would be vulnerable to small changes in test inputs.

Original: (b) the embedding distribution by typical stochastic encoders (e.g., VIB) trained with the KL divergence regularization will tend to cover a fixed prior.
Revised: (b) the embedding distribution by typical stochastic encoders (e.g., VIB) trained with the KL divergence regularization will tend to cover a fixed prior.

Original: Note that it is generally difficult to pick a true prior distribution.
Revised: Note that it is generally difficult to pick a true prior distribution.

Original: Also, it is unnecessary to restrict the embedding distribution to be within a specific bound.
Revised: Also, it is unnecessary to restrict the embedding distribution to be within a specific bound.

Original: (c) with MEIB, on the other hand, by maximizing the conditional entropy of the stochastic embeddings, we would have a better regularization effect as it makes the area secured by the embedding distribution for the given input as broad as possible.
Revised: (c) with MEIB, on the other hand, by maximizing the conditional entropy of the stochastic embeddings, we would have a better regularization effect as it makes the area secured by the embedding distribution for the given input as broad as possible.

Original: The key contributions of MEIB to the previous stochastic embedding methods are summarized as follows:
Revised: The key contributions of MEIB to the previous stochastic embedding methods are summarized as follows:

Original: • While it provides a comparable regularization in handwritten digit classification, MEIB outperforms existing approaches in the challenging person re-identification task with three popular datasets.
Revised: • While it provides a comparable regularization in handwritten digit classification, MEIB outperforms existing approaches in the challenging person re-identification task with three popular datasets.

Original: • MEIB shows significantly better perturbation robustness compared to VIB in handwritten digit classification.
Revised: • MEIB shows significantly better perturbation robustness compared to VIB in handwritten digit classification.

Original: • MEIB performs better than VIB when used in a probabilistic contrastive learning framework.
Revised: • MEIB performs better than VIB when used in a probabilistic contrastive learning framework.

Original: • Providing reliable confidence measurements, MEIB shows an outstanding risk-controlled recognition performance in digit classification and person re-identification tasks.
Revised: • Providing reliable confidence measurements, MEIB shows an outstanding risk-controlled recognition performance in digit classification and person re-identification tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Stochastic Embeddings Research on stochastic embeddings has gained popularity in recent years.
Revised: Stochastic Embeddings Research on stochastic embeddings has gained popularity in recent years.

Original: Oh et al.
Revised: Oh et al.

Original: (2019) proposed a similarity-based method based on the pairwise soft contrastive loss and the VIB principle, which has been later applied to human pose embedding Sun et al.
Revised: (2019) proposed a similarity-based method based on the pairwise soft contrastive loss and the VIB principle, which has been later applied to human pose embedding Sun et al.

Original: (2020), cross-modal retrieval Chun et al.
Revised: (2020), cross-modal retrieval Chun et al.

Original: (2021), and ordinal embedding Li et al.
Revised: (2021), and ordinal embedding Li et al.

Original: (2021b).
Revised: (2021b).

Original: Chang et al.
Revised: Chang et al.

Original: (2020) proposed stochastic embedding for face recognition using softmax loss and KL divergence regularization without relying on similarity.
Revised: (2020) proposed stochastic embedding for face recognition using softmax loss and KL divergence regularization without relying on similarity.

Original: All of these methods assume embedding distribution to be unit Gaussian.
Revised: All of these methods assume embedding distribution to be unit Gaussian.

Original: Unlike these approaches, Probabilistic Face Embedding (PFE) Shi & Jain (2019) turns deterministic embeddings into Gaussians with fixed mean by training a post hoc network to maximize the mutual likelihood of same-class embeddings.
Revised: Unlike these approaches, Probabilistic Face Embedding (PFE) Shi & Jain (2019) turns deterministic embeddings into Gaussians with fixed mean by training a post hoc network to maximize the mutual likelihood of same-class embeddings.

Original: Follow-up research on PFE includes its extension to triplets Warburg et al.
Revised: Follow-up research on PFE includes its extension to triplets Warburg et al.

Original: (2021) and for spherical space Li et al.
Revised: (2021) and for spherical space Li et al.

Original: (2021a).
Revised: (2021a).

Original: However, this line of work is limited to fixed embedding mean.
Revised: However, this line of work is limited to fixed embedding mean.

Original: The work most closely related to ours is
Revised: The work most closely related to ours is

Original: DistributionNet Yu et al.
Revised: DistributionNet Yu et al.

Original: (2019) which introduced entropy-based regularization and inspired Yang et al.
Revised: (2019) which introduced entropy-based regularization and inspired Yang et al.

Original: (2021) for their uncertainty-aware loss.
Revised: (2021) for their uncertainty-aware loss.

Original: However, these methods put a margin to bound the total entropy rather than maximizing it as we do.
Revised: However, these methods put a margin to bound the total entropy rather than maximizing it as we do.

Original: Maximum Entropy Maximum entropy is a general principle that has already been widely adopted in designing machine learning models, including supervised and reinforcement learning Zheng et al.
Revised: Maximum Entropy Maximum entropy is a general principle that has already been widely adopted in designing machine learning models, including supervised and reinforcement learning Zheng et al.

Original: Ahmed et al.
Revised: Ahmed et al.

Original: Pereyra et al.
Revised: Pereyra et al.

Original: (2017) used the negative entropy of the class prediction distribution, − H
Revised: (2017) used the negative entropy of the class prediction distribution, − H

Original: ( p θ ( y | x )) , as a regularization term in the loss function to prevent over-confident predictions.
Revised: ( p θ ( y | x )) , as a regularization term in the loss function to prevent over-confident predictions.

Original: In reinforcement learning, the maximum entropy framework encourages diverse explorations in both on-policy and off-policy settings Mei et al.
Revised: In reinforcement learning, the maximum entropy framework encourages diverse explorations in both on-policy and off-policy settings Mei et al.

Original: ; Haarnoja et al.
Revised: ; Haarnoja et al.

Original: ; Han &
Revised: ; Han &

Original: Sung (2021).
Revised: Sung (2021).

Original: However, in most previous work, the entropy regularization has been applied at the decision levels, the distribution of class or action predictions.
Revised: However, in most previous work, the entropy regularization has been applied at the decision levels, the distribution of class or action predictions.

Original: In this work, on the other hand, we focus on the entropy of the stochastic embedding of inputs.
Revised: In this work, on the other hand, we focus on the entropy of the stochastic embedding of inputs.

Original:  M AXIMUM E NTROPY I NFORMATION B OTTLENECK
Revised:  M AXIMUM E NTROPY I NFORMATION B OTTLENECK

Original: MEIB Objective
Revised: MEIB Objective

Original: The maximum entropy principle Jaynes (1957) states that the current state of knowledge about the given system is best represented by the probability distribution with the largest entropy Shore & Johnson (1980).
Revised: The maximum entropy principle Jaynes (1957) states that the current state of knowledge about the given system is best represented by the probability distribution with the largest entropy Shore & Johnson (1980).

Original: By combining this with our hypothesis, the goal is to learn an encoding Z that is maximally expressive about Y while maximizing the expected amount of information (entropy) about Z | X :
Revised: By combining this with our hypothesis, the goal is to learn an encoding Z that is maximally expressive about Y while maximizing the expected amount of information (entropy) about Z | X :

Original: [Equation]
Revised: [Equation]

Original: Introducing a Lagrangian multiplier β , we have the maximization objective:
Revised: Introducing a Lagrangian multiplier β , we have the maximization objective:

Original: [Equation]
Revised: [Equation]

Original: where β ≥ 0 controls the trade-off between the predictiveness and the spread of Z given X .
Revised: where β ≥ 0 controls the trade-off between the predictiveness and the spread of Z given X .

Original: Using the lower bound suggested by Alemi et al.
Revised: Using the lower bound suggested by Alemi et al.

Original: (2017) for the first term of the objective
Revised: (2017) for the first term of the objective

Original: I ( Z, Y ) , we have
Revised: I ( Z, Y ) , we have

Original: [Equation]
Revised: [Equation]

Original: This lower bound L can be computed by approximating the joint distribution p ( x, y )
Revised: This lower bound L can be computed by approximating the joint distribution p ( x, y )

Original: = p ( x ) p ( y | x ) using the empirical data distribution p ( x, y ) =
Revised: = p ( x ) p ( y | x ) using the empirical data distribution p ( x, y ) =

Original: 1 N (cid:80) 
Revised: 1 N (cid:80) Nn

Revised: =1 δ x n ( x ) δ y n ( y ) (Alemi et al., 2017)

Original: [Equation]
Revised: [Equation]

Original: (Alemi et al [Equation] [Equation]
Revised: Consequently

Original: [Equation]
Revised: [Equation]

Original: Consequently, the loss function to be minimized is:
Revised: the loss function to be minimized is:

Original: [Equation]
Revised: [Equation]

Original: where we use the typical reparameterization trick Kingma & Welling (2014) to backpropagate gradients through the sampling of z ∼ p ( z | x n ) .
Revised: where we use the typical reparameterization trick Kingma & Welling (2014) to backpropagate gradients through the sampling of z ∼ p ( z | x n ) .

Original: We use a single sample of z by default unless it is specified.
Revised: We use a single sample of z by default unless it is specified.

Original: Relationship to VIB The minimization loss function of VIB (Alemi et al., 2017) is:
Revised: Relationship to VIB The minimization loss function of VIB (Alemi et al., 2017) is:

Original: [Equation]
Revised: [Equation]

Original: where H ( p ( Z | x n ) , r ( Z )) is the cross-entropy of r ( Z ) relative to the distribution p ( Z | x n )
Revised: where H ( p ( Z | x n ) , r ( Z )) is the cross-entropy of r ( Z ) relative to the distribution p ( Z | x n )

Original: =
Revised: =

Original: N ( Z | f µθ ( x n ) , f Σ θ ( x n )) , which is given by
Revised: N ( Z | f µ θ ( x n ) , f Σ θ ( x n )) , which is given by

Original: [Equation]
Revised: [Equation]

Original: where µ θ = f µθ ( x n ) and Σ θ =  σ 2 θ I =
Revised: where µ θ = f µ θ ( x n ) and Σ θ = Diag ( σ 2 θ ) =

Original: f Σ θ ( x n ) , a diagonal covariance, respectively - a detailed derivation can be found in the supplemental material Section
Revised: f Σ θ ( x n ) , a diagonal covariance, respectively - a detailed derivation can be found in the supplemental material Section

Original: B. Therefore, the VIB loss function is an upper bound of the MEIB loss function with a positive value of β .
Revised: B. Therefore, the VIB loss function is an upper bound of the MEIB loss function with a positive value of β .

Original: Confidence Measure of MEIB MEIB encourages obvious inputs that can be easily classified to take broader embedding areas by assigning larger entropy.
Revised: Confidence Measure of MEIB MEIB encourages obvious inputs that can be easily classified to take broader embedding areas by assigning larger entropy.

Original: On the contrary, the inputs closer to other classes would have smaller entropy to reduce the chance of misclassification according to the loss function.
Revised: On the contrary, the inputs closer to other classes would have smaller entropy to reduce the chance of misclassification according to the loss function.

Original: Therefore, we adopt the conditional entropy H ( Z | x ) as our confidence measure for the input x , which is given by Thomas & Joy (2006):
Revised: Therefore, we adopt the conditional entropy H ( Z | x ) as our confidence measure for the input x , which is given by Cover & Thomas (2006):

Original: [Equation]
Revised: [Equation]

Original: Specifically, we use the dimension-wise average conditional entropy H ( Z | x ) /D to achieve a dimension-agnostic confidence measure.
Revised: Specifically, we use the dimension-wise average conditional entropy H ( Z | x ) /D to achieve a dimension-agnostic confidence measure.

Original:  E XPERIMENTAL R ESULTS AND D ISCUSSION
Revised:  E XPERIMENTAL R ESULTS AND D ISCUSSION

Original: In this section, we show experimental results for various tasks to demonstrate the effectiveness of
Revised: In this section, we show experimental results for various tasks to demonstrate the effectiveness of

Original: MEIB in terms of the regularization, perturbation robustness, and confidence measure.
Revised: MEIB in terms of the regularization, perturbation robustness, and confidence measure.

Original: All computational experiments were implemented using PyTorch Paszke et al.
Revised: All computational experiments were implemented using PyTorch Paszke et al.

Original: (2019) 1.9 with Python 3.7 on a workstation equipped with an NVIDIA ® GeForce ® RTX 2080 Ti graphic card.
Revised: (2019) 1.9 with Python 3.7 on a workstation equipped with an NVIDIA ® GeForce ® RTX 2080 Ti graphic card.

Original: Please refer to the supplemental material Section H for the implementation and training details for each experimental task.
Revised: Please refer to the supplemental material Section H for the implementation and training details for each experimental task.

Original: 4. D IGIT C LASSIFICATION
Revised: 4. D IGIT C LASSIFICATION

Original: First, we evaluate MEIB on a handwritten digit classification as the simplest form of benchmark task.
Revised: First, we evaluate MEIB on a handwritten digit classification as the simplest form of benchmark task.

Original: We use the QMNIST dataset Yadav & Bottou (2019) to utilize its larger set of 60,000 testing data compared to 10,000 of those in the original MNIST dataset, while it has almost identical training data.
Revised: We use the QMNIST dataset Yadav & Bottou (2019) to utilize its larger set of 60,000 testing data compared to 10,000 of those in the original MNIST dataset, while it has almost identical training data.

Original: We adopt the same architecture employed in Alemi et al.
Revised: We adopt the same architecture employed in Alemi et al.

Original: (2017); the backbone encoder f
Revised: (2017); the backbone encoder f

Original: Bθ is a multilayer perceptron (MLP) with two fully-connected (FC) layers of 1024 hidden units with ReLU activations.
Revised: Bθ is a multilayer perceptron (MLP) with two fully-connected (FC) layers of 1024 hidden units with ReLU activations.

Original: Both f µθ and f Σ θ are an FC layer of D hidden units where the exponential (exp) function was applied after f Σ θ .
Revised: Both f µ θ and f Σ θ are an FC layer of D hidden units where the exponential (exp) function was applied after f Σ θ .

Original: We found that applying a batch normalization (BN) layer at the end of f µθ improves the performance of MEIB with a noticeable gap (see the supplementary material Section
Revised: We found that applying a batch normalization (BN) layer at the end of f µ θ improves the performance of MEIB with a noticeable gap (see the supplementary material Section

Original: F).
Revised: F).

Original: The decoder f C θ is an FC layer with the softmax function that outputs p ( y | x ) over ten classes. 
Revised: The decoder f C θ is an FC layer with the softmax function that outputs p ( y | x ) over ten classes. [Table]

Original: Using the same architecture, we compare the following embedding approaches with MEIB: a deterministic baseline, dropout Srivastava et al.
Revised: Using the same architecture, we compare the following embedding approaches with MEIB: a deterministic baseline, dropout Srivastava et al.

Original: (2014), and VIB.
Revised: (2014), and VIB.

Original: The deterministic baseline represents the typical usage of neural network models without stochasticity.
Revised: The deterministic baseline represents the typical usage of neural network models without stochasticity.

Original: By omitting the variance estimator module f Σ θ ( x ) , the embedding of input is deterministically given by z = f µθ ( x ) .
Revised: By omitting the variance estimator module f Σ θ ( x ) , the embedding of input is deterministically given by z = f µ θ ( x ) .

Original: Dropout is one of the most popular regularization methods for neural networks; thus, it is considered the first[Table] benchmark regularization method to compare Alemi et al.
Revised: Dropout is one of the most popular regularization methods for neural networks; thus, it is considered the first benchmark regularization method to compare Alemi et al.

Original: ; Ghiasi et al.
Revised: ; Ghiasi et al.

Original: We use the same deterministic model, but dropout is applied with the probability of 0.5 during the training time.
Revised: We use the same deterministic model, but dropout is applied with the probability of 0.5 during the training time.

Original: Unlike MEIB and VIB, both deterministic models were trained only with the cross-entropy loss.
Revised: Unlike MEIB and VIB, both deterministic models were trained only with the cross-entropy loss.

Original: We set β = α/D where α is equal to 0.01 for VIB, 0.1 for MEIB with D = 2 , and 1 for MEIB with
Revised: We set β = α/D where α is equal to 0.01 for VIB, 0.1 for MEIB with D = 2 , and 1 for MEIB with

Original: D = 256 .
Revised: D = 256 .

Original: Please refer to the supplementary material Section F for the hyperparameter study.
Revised: Please refer to the supplementary material Section F for the hyperparameter study.

Original: All models were trained with five different random seeds, and we report the mean and standard deviation for each performance metric.
Revised: All models were trained with five different random seeds, and we report the mean and standard deviation for each performance metric.

Original: Regularization Effect Table 1 shows the classification results by each method on the QMNIST test set.
Revised: Regularization Effect Table 1 shows the classification results by each method on the QMNIST test set.

Original: Specifically, we compared the methods with two different embedding sizes, D = 2 and 256.
Revised: Specifically, we compared the methods with two different embedding sizes, D = 2 and 256.

Original: With D = 2 , the stochastic methods performed better than the deterministic ones regardless of the usage of dropout.
Revised: With D = 2 , the stochastic methods performed better than the deterministic ones regardless of the usage of dropout.

Original: Using the larger embedding size of D = 256 , on the other hand, the deterministic model trained with dropout performed the best while the others have very similar performance considering the standard deviation.
Revised: Using the larger embedding size of D = 256 , on the other hand, the deterministic model trained with dropout performed the best while the others have very similar performance considering the standard deviation.

Original: However, using 12 Monte Carlo (MC) samples of z , the stochastic methods outperformed the deterministic ones with a larger gap than the single sample case.
Revised: However, using 12 Monte Carlo (MC) samples of z , the stochastic methods outperformed the deterministic ones with a larger gap than the single sample case.

Original: MEIB provides a reasonable amount of regularization comparable to VIB.
Revised: MEIB provides a reasonable amount of regularization comparable to VIB.

Original: Perturbation Robustness
Revised: Perturbation Robustness

Original: The test time noise/perturbation robustness of a deep neural network model is an important aspect, especially when the model is considered to be deployed for a realworld application.
Revised: The test time noise/perturbation robustness of a deep neural network model is an important aspect, especially when the model is considered to be deployed for a realworld application.

Original: We evaluate the robustness of models toward adversarial examples as an alternative form of perturbation robustness evaluation because models that are weak to adversarial examples might be vulnerable to not only intended attacks but also unexpected noise in test time Gilmer et al.
Revised: We evaluate the robustness of models toward adversarial examples as an alternative form of perturbation robustness evaluation because models that are weak to adversarial examples might be vulnerable to not only intended attacks but also unexpected noise in test time Gilmer et al.

Original: Since the primary purpose of MEIB and the other compared methods is not a defense against strong adversarial attacks, we use the Fast Gradient Sign Method (FGSM)
Revised: Since the primary purpose of MEIB and the other compared methods is not a defense against strong adversarial attacks, we use the Fast Gradient Sign Method (FGSM)

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: [Equation]
Revised: [Equation]

Original: [Equation] where (cid:101) x is the crafted adversarial example, ϵ is a scale factor of the perturbations, and J is the target loss function, e.g., cross-entropy loss.
Revised: where (cid:101) x is the crafted adversarial example, ϵ is a scale factor of the perturbations, and J is the target loss function, e.g., cross-entropy loss.

Original: The FGSM is often regarded as a weak adversary but is still widely used as a first benchmark adversary method due to its simplicity.
Revised: The FGSM is often regarded as a weak adversary but is still widely used as a first benchmark adversary method due to its simplicity.

Original: We report evaluation results with stronger adversaries in the supplemental Section G for interested readers.
Revised: We report evaluation results with stronger adversaries in the supplemental Section G for interested readers.

Original: We crafted the adversarial examples from the QMNIST test set using the FGSM with ϵ ∈ [0 . 0 , 0 . 5] with the step size of 0.5 on all models ( D = 256 ) trained with different random seeds for each method.
Revised: We crafted the adversarial examples from the QMNIST test set using the FGSM with ϵ ∈ [0 . 0 , 0 . 5] with the step size of 0.5 on all models ( D = 256 ) trained with different random seeds for each method.

Original: We used 12 MC samples of z for the stochastic methods, MEIB and VIB.
Revised: We used 12 MC samples of z for the stochastic methods, MEIB and VIB.

Original: Figure 3a shows the misclassification rate of each method toward the different strengths of the FGSM perturbations.
Revised: Figure 3a shows the misclassification rate of each method toward the different strengths of the FGSM perturbations.

Original: MEIB is more robust than the other methods with significant gaps.
Revised: MEIB is more robust than the other methods with significant gaps.

Original: Furthermore, the error rate of MEIB increases very slowly with the increasing strength of perturbations until about ϵ = 0 .
Revised: Furthermore, the error rate of MEIB increases very slowly with the increasing strength of perturbations until about ϵ = 0 .

Original: On the other hand, VIB is more vulnerable than both deterministic and dropout baselines, typically with more severe perturbations.
Revised: On the other hand, VIB is more vulnerable than both deterministic and dropout baselines, typically with more severe perturbations.

Original: It might be because we chose the target model with the best performance with the clean dataset, which is a rational choice, for all methods.
Revised: It might be because we chose the target model with the best performance with the clean dataset, which is a rational choice, for all methods.

Original: Thus, a VIB model trained with a different β might yield better robustness, but it still would be difficult to close  the gap with MEIB.
Revised: Thus, a VIB model trained with a different β might yield better robustness, but it still would be difficult to close [Figure] the gap with MEIB.

Original: Please refer to the supplemental Section C for the perturbation robustness with different α values.
Revised: Please refer to the supplemental Section C for the perturbation robustness with different α values.

Original: Risk-controlled Classification
Revised: Risk-controlled Classification

Original: In many real-world application scenarios, it would be favorable to refuse any decision instead of making a false prediction when the model is not confident about the input.
Revised: In many real-world application scenarios, it would be favorable to refuse any decision instead of making a false prediction when the model is not confident about the input.

Original: For doing this, we need a way to correctly estimate the confidence, or the uncertainty, of[Figure] inputs to ML models.
Revised: For doing this, we need a way to correctly estimate the confidence, or the uncertainty, of inputs to ML models.

Original: Using the estimated confidence, we may reject the inputs with insufficient confidence; it is called risk-controlled recognition Shi & Jain (2019).
Revised: Using the estimated confidence, we may reject the inputs with insufficient confidence; it is called risk-controlled recognition Shi & Jain (2019).

Original: We evaluated the risk-controlled classification performance on the QMNIST test set by rejecting the inputs by the confidence estimated by each method.
Revised: We evaluated the risk-controlled classification performance on the QMNIST test set by rejecting the inputs by the confidence estimated by each method.

Original: Similarly to the confidence measured by MEIB (Section 3), we empirically found that the mean of variance vector from f Σ θ ( x ) of VIB is proportional to the confidence.
Revised: Similarly to the confidence measured by MEIB (Section 3), we empirically found that the mean of variance vector from f Σ θ ( x ) of VIB is proportional to the confidence.

Original: For the deterministic and dropout baselines, we use the L 2 -norm of the embedding vector for each input as a proxy measure of confidence Ranjan et al.
Revised: For the deterministic and dropout baselines, we use the L 2 -norm of the embedding vector for each input as a proxy measure of confidence Ranjan et al.

Original: We set the rejection rate from 0% to 95% with a step size of 5% .
Revised: We set the rejection rate from 0% to 95% with a step size of 5% .

Original: A single sample of z was used for MEIB and VIB.
Revised: A single sample of z was used for MEIB and VIB.

Original: Figure 3b shows the risk-controlled classification performance by all methods with D = 256 .
Revised: Figure 3b shows the risk-controlled classification performance by all methods with D = 256 .

Original: MEIB outperformed all other methods across the most range of the input rejection rate.
Revised: MEIB outperformed all other methods across the most range of the input rejection rate.

Original: Specifically, the error rate of MEIB  dropped more than half of the initial value after rejecting 10% of uncertain inputs and reached about 0 .
Revised: Specifically, the error rate of MEIB [Figure] [Table] dropped more than half of the initial value after rejecting 10% of uncertain inputs and reached about 0 .

Original: 1% when half of the inputs were rejected.
Revised: 1% when half of the inputs were rejected.

Original: Embedding Distribution Figure 4 depicts the embedding space learned by VIB and MEIB with
Revised: Embedding Distribution Figure 4 depicts the embedding space learned by VIB and MEIB with

Original: D
Revised: D

Original: = 2 .
Revised: = 2 .

Original: It shows that each embedding distribution by MEIB takes as much area as possible depending on its location from the decision boundaries, which is consistent with our hypothesis.
Revised: It shows that each embedding distribution by MEIB takes as much area as possible depending on its location from the decision boundaries, which is consistent with our hypothesis.

Original: On the other hand, every embedding by VIB has a small standard deviation and thus covers a much smaller [Figure] [Table]
Revised: On the other hand, every embedding by VIB has a small standard deviation and thus covers a much smaller area in both dimensions (axes) than those of MEIB.

Original: [Table] area in both dimensions (axes) than those of MEIB. It would be reasonable to consider increasing σ of the prior distribution N ( 0 , σ 2 I ) used in VIB modeling to achieve a similar effect of entropy maximization by MEIB.
Revised: It would be reasonable to consider increasing σ of the prior distribution N ( 0 , σ 2 I ) used in VIB modeling to achieve a similar effect of entropy maximization by MEIB.

Original: However, even VIB with very large variance priors still performs much worse than MEIB for both aspects of perturbation robustness and risk-controlled classification (see the supplemental Section D).
Revised: However, even VIB with very large variance priors still performs much worse than MEIB for both aspects of perturbation robustness and risk-controlled classification (see the supplemental Section D).

Original: It implies that MEIB increases σ of inputs in an adaptive way while
Revised: It implies that MEIB increases σ of inputs in an adaptive way while

Original: VIB tries to match the given prior distribution
Revised: VIB tries to match the given prior distribution.

Original: [Table]

Original: 4. H EDGED I NSTANCE E MBEDDING
Revised: 4. H EDGED I NSTANCE E MBEDDING

Original: Oh et al.
Revised: Oh et al.

Original: (2019) proposed the hedged instance embedding (HIB), a metric learning method that explicitly models the input uncertainty in the stochastic embedding space.
Revised: (2019) proposed the hedged instance embedding (HIB), a metric learning method that explicitly models the input uncertainty in the stochastic embedding space.

Original: HIB utilizes the VIB objective with the probabilistic contrastive learning framework.
Revised: HIB utilizes the VIB objective with the probabilistic contrastive learning framework.

Original: The authors also proposed a new dataset, called N-digit MNIST, basically images of N adjacent MNIST digits.
Revised: The authors also proposed a new dataset, called N-digit MNIST, basically images of N adjacent MNIST digits.

Original: We examined the
Revised: We examined the

Original: HIB framework with the MEIB objective instead of VIB simply by replacing the KL divergence term in the original HIB loss function with the negative conditional entropy of embeddings, keeping all the other aspects of the neural network model same, including the hyperparameters.
Revised: HIB framework with the MEIB objective instead of VIB simply by replacing the KL divergence term in the original HIB loss function with the negative conditional entropy of embeddings, keeping all the other aspects of the neural network model same, including the hyperparameters.

Original: Please refer to the supplementary material for the architectural details.
Revised: Please refer to the supplementary material for the architectural details.

Original: Table 2 reports the performance of the original HIB with the VIB objective and the MEIB-variant by the average precision (AP) on the test set.
Revised: Table 2 reports the performance of the original HIB with the VIB objective and the MEIB-variant by the average precision (AP) on the test set.

Original: MEIB-variant outperformed the original HIB in every case with the clean test set.
Revised: MEIB-variant outperformed the original HIB in every case with the clean test set, except the very high-dimensional embedding of D = 100 , where both performed well.

Original: For the corrupted test data, on the other hand, the original HIB performed slightly better in lowdimensional embedding D = 2 scenarios, and both performed comparably with D = 3 .
Revised: For the corrupted test data, on the other hand, the original HIB performed slightly better in low-dimensional embedding D = 2 scenarios, and both performed comparably with D = 3 .

Original: However, MEIB outperformed with a higher-dimensional embedding D = 4  in both N = 2 and 3 cases.
Revised: However, MEIB outperformed with a higher-dimensional embedding D = 4 and D = 100 in both N = 2 and 3 cases.

Original: Given the trend, it is reasonable to expect the MEIB-variant to keep performing better with much higher dimensional embeddings.

Original: In most applications, it is very uncommon to use such a small size of embeddings with two or three dimensions, except for direct visualization of data relationships An et al.
Revised: In most applications, it is very uncommon to use such a small size of embeddings with two or three dimensions, except for direct visualization of data relationships An et al.

Original: Consequently, it suggests that the MEIB objective would better fit the contrastive learning framework of HIB than the VIB objective upon using a reasonable size of embedding dimensions.
Revised: Consequently, it suggests that the MEIB objective would better fit the contrastive learning framework of HIB than the VIB objective upon using a reasonable size of embedding dimensions.

Original: 4. P ERSON R E - IDENTIFICATION
Revised: 4. P ERSON R E - IDENTIFICATION

Original: Person re-identification (ReID) is an important computer vision task that is utilized in various applications, including intelligent security and surveillance systems Gong et al.
Revised: Person re-identification (ReID) is an important computer vision task that is utilized in various applications, including intelligent security and surveillance systems Gong et al.

Original: Unlike typical image classification tasks, the objective of the person ReID task is to find the ranked matches of pedestrian images captured across multiple non-overlapping cameras Chen et al.
Revised: Unlike typical image classification tasks, the objective of the person ReID task is to find the ranked matches of pedestrian images captured across multiple non-overlapping cameras Chen et al.

Original: Person
Revised: Person

Original: ReID is challenging due to image-level corruptions and appearance changes, including occlusions 
Revised: ReID is challenging due to image-level corruptions and appearance changes, including occlusions [Table]

Original: Ye et al.
Revised: Ye et al.

Original: Therefore, it is critical to have an embedding method robust to noises and capable of confidence measuring for potential risk-controllability.
Revised: Therefore, it is critical to have an embedding method robust to noises and capable of confidence measuring for potential risk-controllability.

Original: For evaluating those aspects, we used three datasets popularly employed in person ReID literature: Market-1501Zheng et al.
Revised: For evaluating those aspects, we used three datasets popularly employed in person ReID literature: Market-1501Zheng et al.

Original: MSMT17Wei et al.
Revised: MSMT17Wei et al.

Original: (2018), and LPW Song et al.
Revised: (2018), and LPW Song et al.

Original: For LPW, we used a quarter subset of it by selecting every fourth frame of each identity due to the limited computational resource.
Revised: For LPW, we used a quarter subset of it by selecting every fourth frame of each identity due to the limited computational resource.

Original: For all methods compared in ReID experiments, we used ResNet50 He et al.
Revised: For all methods compared in ReID experiments, we used ResNet50 He et al.

Original: as the backbone encoder f Bθ .
Revised: as the backbone encoder f Bθ .

Original: The mean estimator f µθ of an FC layer with 512 hidden units followed by a BN layer was used for all methods.
Revised: The mean estimator f µ θ of an FC layer with 512 hidden units followed by a BN layer was used for all methods.

Original: For the variance estimator f Σ θ , we used an MLP of FC512-BN- [Table] ReLU-FC512-exp for MEIB and followed the original architecture proposed by the authors for each method, except that we used 512 hidden units for their FC layers.
Revised: For the variance estimator f Σ θ , we used an MLP of FC512-BNReLU-FC512-exp for MEIB and followed the original architecture proposed by the authors for each method, except that we used 512 hidden units for their FC layers.

Original: We used α = 1 for MEIB and the recommended hyperparameter values by the authors for each method.
Revised: We used α = 1 for MEIB and the recommended hyperparameter values by the authors for each method.

Original: We compare the performance of MEIB with the following baseline methods not only from person
Revised: We compare the performance of MEIB with the following baseline methods not only from person

Original: ReID literature but also from previous work on face recognition, which shares similar task characteristics:
Revised: ReID literature but also from previous work on face recognition, which shares similar task characteristics:

Original: DistributionNet (DistNet) Yu et al.
Revised: DistributionNet (DistNet) Yu et al.

Original: Probabilistic Face Embedding (PFE) Shi & Jain (2019), and Data Uncertainty Learning (DUL, equivalent to VIB)
Revised: Probabilistic Face Embedding (PFE) Shi & Jain (2019), and Data Uncertainty Learning (DUL, equivalent to VIB)

Original: Chang et al.
Revised: Chang et al.

Original: In addition, we compare the deterministic baseline of the same architecture without the variance estimator f Σ θ .
Revised: In addition, we compare the deterministic baseline of the same architecture without the variance estimator f Σ θ .

Original: We also tried HIB for the ReID task, but we could not achieve meaningful results.
Revised: We also tried HIB for the ReID task, but we could not achieve meaningful results.

Original: All models were trained for the classification task with a softmax classifier of f Cθ to predict the true identity labels.
Revised: All models were trained for the classification task with a softmax classifier of f Cθ to predict the true identity labels.

Original: In testing time, the Euclidean distance between every pair of gallery and query images is calculated using the deterministic embeddings or the mean of stochastic embeddings f µθ ( x ) of them to rank the pairs.
Revised: In testing time, the Euclidean distance between every pair of gallery and query images is calculated using the deterministic embeddings or the mean of stochastic embeddings f µ θ ( x ) of them to rank the pairs.

Original: For PFE, we use the negative mutual likelihood score (MLS) used in the original work of PFE as the distance metric.
Revised: For PFE, we use the negative mutual likelihood score (MLS) used in the original work of PFE as the distance metric.

Original: We first evaluated the methods without considering the input confidence and risk control.
Revised: We first evaluated the methods without considering the input confidence and risk control.

Original: Tablesummarizes the results where we report the mean average precision (mAP)
Revised: Tablesummarizes the results where we report the mean average precision (mAP)

Original: Sch¨utze et al.
Revised: Sch¨utze et al.

Original: (2008) and cumulative matching characteristics (CMC) curve
Revised: (2008) and cumulative matching characteristics (CMC) curve

Original: Gray et al.
Revised: Gray et al.

Original: (2007) at rank-1, rank-5, and rank10.
Revised: (2007) at rank-1, rank-5, and rank10.

Original: MEIB outperforms all the others by from 2 up to 6.6 percentage points of mAP throughout the datasets considered.
Revised: MEIB outperforms all the others by from 2 up to 6.6 percentage points of mAP throughout the datasets considered.

Original: It implies that maximizing the conditional entropy of embeddings in MEIB has better regularization effects than the other methods.
Revised: It implies that maximizing the conditional entropy of embeddings in MEIB has better regularization effects than the other methods.

Original: The importance of risk-controlled recognition is more emphasized in the person ReID task due to the cost of misidentifying a person; it can have a serious societal impact, such as falsely tagging someone as a criminal.
Revised: The importance of risk-controlled recognition is more emphasized in the person ReID task due to the cost of misidentifying a person; it can have a serious societal impact, such as falsely tagging someone as a criminal.

Original: Considering a real-world application of risk-controlled ReID model deployment, a realistic situation is that we can prepare a well-curated clean set of gallery images in advance and expect that arbitrary query/probe images with potential corruptions will be given after model deployment.
Revised: Considering a real-world application of risk-controlled ReID model deployment, a realistic situation is that we can prepare a well-curated clean set of gallery images in advance and expect that arbitrary query/probe images with potential corruptions will be given after model deployment.

Original: Thus, we kept the original test gallery images for each dataset and evaluated the methods with varying amounts of rejected test query images.
Revised: Thus, we kept the original test gallery images for each dataset and evaluated the methods with varying amounts of rejected test query images.

Original: The query images are sorted using the  confidence estimated by each method: MEIB and the deterministic model use the same approaches described in Section 4.1.
Revised: The query images are sorted using the [Figure] confidence estimated by each method: MEIB and the deterministic model use the same approaches described in Section 4.1.

Original: DistNet and DUL use the mean of variance vector from f Σ θ ( x ) , while PFE uses the mean reciprocals of the variance vector elements as their confidence measures.
Revised: DistNet and DUL use the mean of variance vector from f Σ θ ( x ) , while PFE uses the mean reciprocals of the variance vector elements as their confidence measures.

Original: Then, the first R % of low-confidence images are removed from evaluation.
Revised: Then, the first R % of low-confidence images are removed from evaluation.

Original: For each remaining query image, the entire gallery images are ranked by the distance, and the evaluation metrics are calculated.
Revised: For each remaining query image, the entire gallery images are ranked by the distance, and the evaluation metrics are calculated.

Original: Figure 5 shows the risk-controlled identification performance of the methods for each dataset.
Revised: Figure 5 shows the risk-controlled identification performance of the methods for each dataset.

Original: While
Revised: While

Original: MEIB starts with the best performance among all methods, as shown previously, the amount of performance improvement at lower rejection rates is more significant than the other methods in most [Figure]
Revised: MEIB starts with the best performance among all methods, as shown previously, the amount of performance improvement at lower rejection rates is more significant than the other methods in most cases.

Original: cases. Furthermore, the gap between MEIB and the other methods is kept over the most range of the rejection rate.
Revised: Furthermore, the gap between MEIB and the other methods is kept over the most range of the rejection rate.

Original: It confirms that MEIB provides an effective confidence measure and risk-control capability.
Revised: It confirms that MEIB provides an effective confidence measure and risk-control capability.

Original: Figure 6 shows the example query images for each dataset (except MSMT17 prohibited by the license) with the associated confidence (conditional entropy) value.
Revised: Figure 6 shows the example query images for each dataset (except MSMT17 prohibited by the license) with the associated confidence (conditional entropy) value.

Original: There are conspicuous trends: (1) the low-confident images are mostly blurry while the high-confident images are relatively clearer, (2) the persons in the low-confident images are often occluded by another object or person, and (3) the most persons in the low-confident images wear clothes with an achromatic color without patterns while those in the high-confident images wear clothes with vivid colors or patterns such as stripes.
Revised: There are conspicuous trends: (1) the low-confident images are mostly blurry while the high-confident images are relatively clearer, (2) the persons in the low-confident images are often occluded by another object or person, and (3) the most persons in the low-confident images wear clothes with an achromatic color without patterns while those in the high-confident images wear clothes with vivid colors or patterns such as stripes.

Original: This would be another evidence that MEIB provides reasonable confidence measurements. 
Revised: This would be another evidence that MEIB provides reasonable confidence measurements. [Figure] [Table] [Figure]

Original: C ONCLUSION
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original:  In this work, we presented MEIB, a novel framework that produces stochastic embeddings distributed with the maximum entropy.
Revised: [Table] [Figure]  C ONCLUSION In this work we presented MEIB, a novel framework that produces stochastic embeddings distributed with the maximum entropy.

Original: MEIB provides a regularization effect and robustness toward adversarial noise by securing the maximum area in the embedding space, leading to better classification performance.
Revised: MEIB provides a regularization effect and robustness toward adversarial noise by securing the maximum area in the embedding space, leading to better classification performance.

Original: Moreover, the experimental results in digit classification and person ReID tasks showed that MEIB enables an effective risk-controlled recognition by providing reliable confidence measurements.
Revised: Moreover, the experimental results in digit classification and person ReID tasks showed that MEIB enables an effective risk-controlled recognition by providing reliable confidence measurements.

Original: While the experiments in this work utilized somewhat elementary neural network backbone encoders, it would be straightforward to combine MEIB with more sophisticated architectures to yield further classification performance improvement. [Figure] [Table] [Figure]
Revised: While the experiments in this work utilized somewhat elementary neural network backbone encoders, it would be straightforward to combine MEIB with more sophisticated architectures to yield further classification performance improvement

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: A U NCERTAINTY E STIMATION OF VIB
Revised: A U NCERTAINTY E STIMATION OF VIB

Original: Figure 7 shows the QMNIST test set samples with the lowest and highest variance averaged over feature dimension, i.e., 1 D (cid:80) Dd =1 σ 2 θ,d , estimated by VIB.
Revised: Figure 7 shows the QMNIST test set samples with the lowest and highest variance averaged over feature dimension, i.e., 1 D (cid:80) Dd =1 σ 2 θ,d , estimated by VIB.

Original: It shows that VIB assigns relatively smaller variances for uncertain inputs and higher variances for clear inputs.
Revised: It shows that VIB assigns relatively smaller variances for uncertain inputs and higher variances for clear inputs.

Original: Moreover, Figure 8 shows the risk-controlled classification (see Section 4.1 in the main paper) performance of VIB with the ascending and descending order of the average variance.
Revised: Moreover, Figure 8 shows the risk-controlled classification (see Section 4.1 in the main paper) performance of VIB with the ascending and descending order of the average variance.

Original: In the ascending order, the samples with small variances are rejected first.
Revised: In the ascending order, the samples with small variances are rejected first.

Original: Similarly, the high variance samples are rejected first in the descending order case.
Revised: Similarly, the high variance samples are rejected first in the descending order case.

Original: If we assume that the variance proportionally represents the ‘uncertainty’ associated with the inputs, we will reject the samples with the higher variance first.
Revised: If we assume that the variance proportionally represents the ‘uncertainty’ associated with the inputs, we will reject the samples with the higher variance first.

Original: In this case of the descending order rejection, the risk-controlled classification did not work as expected; the more samples were rejected, the worse performance was achieved.
Revised: In this case of the descending order rejection, the risk-controlled classification did not work as expected; the more samples were rejected, the worse performance was achieved.

Original: On the contrary, the risk-controlled classification worked correctly using the ascending order, regardless of its performance.
Revised: On the contrary, the risk-controlled classification worked correctly using the ascending order, regardless of its performance.

Original: Therefore, we can infer from these results that it is more rational to utilize the variance σ 2 θ in VIB to represent a ‘confident’ area/interval rather than the ‘uncertainty’.
Revised: Therefore, we can infer from these results that it is more rational to utilize the variance σ 2 θ in VIB to represent a ‘confident’ area/interval rather than the ‘uncertainty’.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: B C ROSS - ENTROPY H ( p θ ( z | x n ) , r ( z ))

Original: Let two D -dimensional multivariate Gaussian distributions p θ ( z | x n ) =
Revised: Let two D -dimensional multivariate Gaussian distributions p θ ( z | x n ) =

Original: N ( z | µ θ , Σ θ ) and r ( z 
Revised: N ( z | µ θ , Σ θ ) and r ( z ) =

Original: [Equation]

Original: = N ( z ; µ 0 , Σ I ) where µ θ = f µθ ( x n ) , Σ θ = σ 2 θ I =
Revised: N ( z ; µ 0 , Σ I ) where µ θ = f µ θ ( x n ) , Σ θ = σ 2 θ I =

Original: f Σ θ ( x n ) , a diagonal covariance, µ 0 = 0 , a zero mean vector, and
Revised: f Σ θ ( x n ) , a diagonal covariance, µ 0 = 0 , a zero mean vector, and

Original: Σ I
Revised: Σ I

Original: =
Revised: =

Original: I , an identity covariance matrix, respectively.
Revised: I , an identity covariance matrix, respectively.

Original: Then, H ( p ( z | x n ) , r ( z )) , the cross-entropy of r ( z ) relative to p ( z | x n ) , is given by
Revised: Then, H ( p ( z | x n ) , r ( z )) , the cross-entropy of r ( z ) relative to p ( z | x n ) , is given by

Original: [Equation]
Revised: [Equation]

Original: 20) where each of z d , µ θ,d , and σ 2 θ,d is the d -th dimension of the corresponding vector.
Revised: where each of z d , µ θ,d , and σ 2 θ,d is the d -th dimension of the corresponding vector.

Original: Note again that Σ θ = σ 2 θ I =
Revised: Note again that Σ θ = σ 2 θ I =

Original: f Σ θ ( x n ) , a diagonal covariance, i.e., each dimension of z is independent to each other.
Revised: f Σ θ ( x n ) , a diagonal covariance, i.e., each dimension of z is independent to each other.

Original: C P ERTURBATION R OBUSTNESS BY D IFFERENT α
Revised: C P ERTURBATION R OBUSTNESS BY D IFFERENT α

Original: Figure 9a shows the misclassification rates against the FGSM perturbations for MEIB trained with different α values.
Revised: Figure 9a shows the misclassification rates against the FGSM perturbations for MEIB trained with different α values.

Original: It shows a more consistent trend that a larger α provides more robustness for the given perturbation strength, unlike the VIB case that shows saddle-shaped curves in Figure 9b.
Revised: It shows a more consistent trend that a larger α provides more robustness for the given perturbation strength, unlike the VIB case that shows saddle-shaped curves in Figure 9b.

Original: It is reasonable because a larger α (thus, a larger β ) value encourages the model to assign larger entropy and thus secure a larger area for the distribution of z given input x , which leads to larger margins from the decision boundary.
Revised: It is reasonable because a larger α (thus, a larger β ) value encourages the model to assign larger entropy and thus secure a larger area for the distribution of z given input x , which leads to larger margins from the decision boundary.

Original: [Figure]
Revised: [Figure]

Original: D VIB WITH L ARGE - VARIANCE P RIORS
Revised: D VIB WITH L ARGE - VARIANCE P RIORS

Original: We trained and evaluated VIB models with different values of σ in their prior distribution.
Revised: We trained and evaluated VIB models with different values of σ in their prior distribution.

Original: Figureshows the embeddings of VIBs trained with 2-dimensional bottleneck and different values of σ .
Revised: Figureshows the embeddings of VIBs trained with 2-dimensional bottleneck and different values of σ .

Original: By visual inspection, it may seem getting similar to those of MEIB shown in Figure 4b as σ increases.
Revised: By visual inspection, it may seem getting similar to those of MEIB shown in Figure 4b as σ increases.

Original: On the other hand, Figure 11 presents the perturbation robustness (toward FGSM attacks) and riskcontrolled classification performance by VIB trained with the different values of σ while keeping the other configurations same as in Section 4, including the 256-dimensional bottleneck.
Revised: On the other hand, Figure 11 presents the perturbation robustness (toward FGSM attacks) and riskcontrolled classification performance by VIB trained with the different values of σ while keeping the other configurations same as in Section 4, including the 256-dimensional bottleneck.

Original: It shows that the adversarial robustness is improved with increasing σ , until σ = 10 , but becomes worse again with much larger σ = 100 , while all of them were still worse than MEIB with a significant gap.
Revised: It shows that the adversarial robustness is improved with increasing σ , until σ = 10 , but becomes worse again with much larger σ = 100 , while all of them were still worse than MEIB with a significant gap.

Original: Moreover, the risk-controlled classification performance has little difference with different σ values.
Revised: Moreover, the risk-controlled classification performance has little difference with different σ values.

Original: On the other hand, the magnitude of σ s estimated by MEIB for QMNIST dataset are in a range below than those by VIB with σ = 5 , as shown in Table 4.
Revised: On the other hand, the magnitude of σ s estimated by MEIB for QMNIST dataset are in a range below than those by VIB with σ = 5 , as shown in Table 4.

Original: It provides an insight that MEIB increase σ of inputs in a more effective way.
Revised: It provides an insight that MEIB increase σ of inputs in a more effective way.

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Figure]

Original: E R ANK -1 A CCURACY FOR R ISK - CONTROLLED P ERSON R E ID
Revised: E R ANK -1 A CCURACY FOR R ISK - CONTROLLED P ERSON R E ID

Original: Figure 12 shows the risk-controlled CMC rank-1 accuracy for each method on each ReID dataset.
Revised: Figure 12 shows the risk-controlled CMC rank-1 accuracy for each method on each ReID dataset.

Original: The results were obtained from the same experiments described in Section 4.3, in addition to the risk-controlled mAP in Figure 5 of the main paper.
Revised: The results were obtained from the same experiments described in Section 4.3, in addition to the risk-controlled mAP in Figure 5 of the main paper.

Original: It confirms again that MEIB provides the most effective confidence measure and risk-control capability compared to the other methods over the most range of the rejection rate. 
Revised: It confirms again that MEIB provides the most effective confidence measure and risk-control capability compared to the other methods over the most range of the rejection rate. [Figure]

Original: [Figure]
Revised: [Figure]

Original: F H YPERPARAMETER S TUDY
Revised: F H YPERPARAMETER S TUDY

Original: Figure 13 shows the performance of VIB and MEIB for different values of α and the usage of the
Revised: Figure 13 shows the performance of VIB and MEIB for different values of α and the usage of the

Original: BN layer at the end of the f µθ branch in the QMNIST digit classification task with D = 256 .
Revised: BN layer at the end of the f µ θ branch in the QMNIST digit classification task with D = 256 .

Original: For
Revised: For

Original: VIB, the BN layer is beneficial only for the small values of α ( 0 ∼ 10 − 4 ), and the best result was achieved without using the BN layer at α = 0 .
Revised: VIB, the BN layer is beneficial only for the small values of α ( 0 ∼ 10 − 4 ), and the best result was achieved without using the BN layer at α = 0 .

Original: For MEIB, on the other hand, it is a common phenomenon that using the BN layer at the end of f µ θ noticeably improves the performance across  all α values, while the best performance was obtained with α = 1 using the BN layer.
Revised: For MEIB, on the other hand, it is a common phenomenon that using the BN layer at the end of f µ θ noticeably improves the performance across [Figure] all α values, while the best performance was obtained with α = 1 using the BN layer.

Original: Furthermore,
Revised: Furthermore,

Original: Figure 14 shows that using the BN layer at the end of the f µθ branch also significantly improves the performance of MEIB on all datasets considered in the person ReID task, commonly with the best performance at α = 1 .
Revised: Figure 14 shows that using the BN layer at the end of the f µ θ branch also significantly improves the performance of MEIB on all datasets considered in the person ReID task, commonly with the best performance at α = 1 .

Original: While we need a more concrete study about the effect of BN in MEIB, which we leave as our future work, it can be assumed that α = 1 with a BN layer at the end of f µθ is a reasonable default configuration for these tasks.
Revised: While we need a more concrete study about the effect of BN in MEIB, which we leave as our future work, it can be assumed that α = 1 with a BN layer at the end of f µ θ is a reasonable default configuration for these tasks.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: G S TRONGER A DVERSARIAL A TTACKS E XPERIMENTS
Revised: G S TRONGER A DVERSARIAL A TTACKS E XPERIMENTS

Original: While the FGSM evaluated in Section 4 is a popular first-choice baseline method, it is often ineffective compared to multi-step attack methods.
Revised: While the FGSM evaluated in Section 4 is a popular first-choice baseline method, it is often ineffective compared to multi-step attack methods.

Original: Although the main contribution of this paper is not an adversarial defensive method, we evaluated MEIB and other baseline methods on the following stronger adversaries for a more concrete adversarial robustness analysis: the multi-step Projected
Revised: Although the main contribution of this paper is not an adversarial defensive method, we evaluated MEIB and other baseline methods on the following stronger adversaries for a more concrete adversarial robustness analysis: the multi-step Projected

Original: Gradient Descent (PGD)
Revised: Gradient Descent (PGD)

Original: Madry et al.
Revised: Madry et al.

Original: (2018), two variants of Auto-PGD (APGD), one based on the cross-entropy loss (APGD CE ) and one with the difference of logits ratio (DLR) loss (APGD DLR ), and AutoAttack (AA), which is an ensemble of those two APGDs Croce & Hein (2020).
Revised: (2018), two variants of Auto-PGD (APGD), one based on the cross-entropy loss (APGD CE ) and one with the difference of logits ratio (DLR) loss (APGD DLR ), and AutoAttack (AA), which is an ensemble of those two APGDs Croce & Hein (2020).

Original: We used the projection on the L ∞ -ball of radius ϵ for all attacks where we set ϵ = 0 .
Revised: We used the projection on the L ∞ -ball of radius ϵ for all attacks where we set ϵ = 0 .

Original: For PGD, we used 1000 steps (PGD-1000) with the step size ϵ step = 0 .
Revised: For PGD, we used 1000 steps (PGD-1000) with the step size ϵ step = 0 .

Original: 01 and 100 random restarts.
Revised: 01 and 100 random restarts.

Original: For APGDs and
Revised: For APGDs and

Original: AA, Expectation over Transformation (EOT) Athalye et al.
Revised: AA, Expectation over Transformation (EOT) Athalye et al.

Original: (2018) was used with an average over 20 times of the gradient computations for more effective attacks on the stochastic models, MEIB and VIB.
Revised: (2018) was used with an average over 20 times of the gradient computations for more effective attacks on the stochastic models, MEIB and VIB.

Original: We used Adversarial Robustness Toolbox (ART)
Revised: We used Adversarial Robustness Toolbox (ART)

Original: Nicolae et al.
Revised: Nicolae et al.

Original: (2018) to leverage its PGD implementation and the AA implementation provided by the authors 2 , including APGDs.
Revised: (2018) to leverage its PGD implementation and the AA implementation provided by the authors 2 , including APGDs.

Original: Table 5 shows the evaluation results on class-balanced 10 , 000 samples of the QMNIST test set.
Revised: Table 5 shows the evaluation results on class-balanced 10 , 000 samples of the QMNIST test set.

Original: MEIB outperformed all the other baseline methods across all different types of adversarial attacks.
Revised: MEIB outperformed all the other baseline methods across all different types of adversarial attacks.

Original: Although MEIB was also effectively fooled by these multi-step and adaptive PGD attacks, it performs better than other non-adversarially trained models such as Mixup evaluated with the PGD attacks on the original MNIST datasetAddepalli et al.
Revised: Although MEIB was also effectively fooled by these multi-step and adaptive PGD attacks, it performs better than other non-adversarially trained models such as Mixup evaluated with the PGD attacks on the original MNIST datasetAddepalli et al.

Original: It would be possible to achieve better robustness once MEIB is trained with adversarial training methods.
Revised: It would be possible to achieve better robustness once MEIB is trained with adversarial training methods.

Revised: with different adversarial attacks on the QMNIST test samples

Original: [Table]
Revised: [Table]

Original: H I MPLEMENTATION AND T RAINING D ETAILS
Revised: H I MPLEMENTATION AND T RAINING D ETAILS

Original: QMNIST Digit Classification All models were trained for 200 epochs with a batch size of 100.
Revised: QMNIST Digit Classification All models were trained for 200 epochs with a batch size of 100.

Original: Adam optimizer Kingma & Ba (2015) was used with β 1 = 0 .
Revised: Adam optimizer Kingma & Ba (2015) was used with β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 , and the learning rate of 0.0001, which is decayed by the factor of 0.97 every other epoch.
Revised: 999 , and the learning rate of 0.0001, which is decayed by the factor of 0.97 every other epoch.

Original: The exponential moving average of the model parameters with a decay factor of 0.999 was tracked during the training [Table] and the final averaged parameters were used at test time.
Revised: The exponential moving average of the model parameters with a decay factor of 0.999 was tracked during the training, and the final averaged parameters were used at test time.

Original: We trained and evaluated all methods with five different random seed values.
Revised: We trained and evaluated all methods with five different random seed values.

Original: All pixel values of input digit images were rescaled to [ − 1 , 1] .
Revised: All pixel values of input digit images were rescaled to [ − 1 , 1] .

Original: The embedding size of D = 256 was used by default for all experiments unless it was specified.
Revised: The embedding size of D = 256 was used by default for all experiments unless it was specified.

Original: For the stochastic methods, MEIB and VIB, we used 12 MC samples of z for the adversarial robustness experiment and a single sample for the risk-controlled classification experiment.
Revised: For the stochastic methods, MEIB and VIB, we used 12 MC samples of z for the adversarial robustness experiment and a single sample for the risk-controlled classification experiment.

Original: Hedged Instance Embedding We used the same architecture in the original HIB work
Revised: Hedged Instance Embedding We used the same architecture in the original HIB work

Original: Oh et al.
Revised: Oh et al.

Original: (2019), except for some parts not mentioned by the authors.
Revised: (2019), except for some parts not mentioned by the authors.

Original: The backbone encoder f
Revised: The backbone encoder f

Original: Bθ consists of two convolutional layers, 32 and 64 filters of 5 × 5 kernels, each followed by ReLU activation and a max-pooling with 2 × 2 kernels.
Revised: Bθ consists of two convolutional layers, 32 and 64 filters of 5 × 5 kernels, each followed by ReLU activation and a max-pooling with 2 × 2 kernels.

Original: Each f µθ and f Σ θ is an FC layer of D hidden units, and a BN layer is attached at the end of f µθ .
Revised: Each f µ θ and f Σ θ is an FC layer of D hidden units, and a BN layer is attached at the end of f µ θ .

Original: Both MEIB and VIB variants of the HIB models were trained over 500,000 iterations with a batch size of 128.
Revised: Both MEIB and VIB variants of the HIB models were trained over 500,000 iterations with a batch size of 128.

Original: We followed the same batching strategy used in Oh et al.
Revised: We followed the same batching strategy used in Oh et al.

Original: (2019) to ensure we had enough number of both positive and negative pairs of images.
Revised: (2019) to ensure we had enough number of both positive and negative pairs of images.

Original: Adam optimizer Kingma & Ba (2015) was used with β 1 = 0 .
Revised: Adam optimizer Kingma & Ba (2015) was used with β 1 = 0 .

Original: 5 , β 2 = 0 .
Revised: 5 , β 2 = 0 .

Original: 999 and the learning rate of 0.0001.
Revised: 999 and the learning rate of 0.0001.

Original: Person Re-identification We utilized the Torchreid framework Zhou & Xiang (2019) to implement the methods and conduct the experiments.
Revised: Person Re-identification We utilized the Torchreid framework Zhou & Xiang (2019) to implement the methods and conduct the experiments.

Original: Except for PFE and DistNet, all other models were initialized with the ResNet50 parameters pre-trained on the ImageNet dataset Deng et al.
Revised: Except for PFE and DistNet, all other models were initialized with the ResNet50 parameters pre-trained on the ImageNet dataset Deng et al.

Original: and trained for 60 epochs with a batch size of 32.
Revised: and trained for 60 epochs with a batch size of 32.

Original: Specifically, we adopted the two-stepped transfer learning strategy Geng et al.
Revised: Specifically, we adopted the two-stepped transfer learning strategy Geng et al.

Original: (2016), where the backbone encoders f
Revised: (2016), where the backbone encoders f

Original: Bθ were frozen for the first 5 epochs.
Revised: Bθ were frozen for the first 5 epochs.

Original: AMSGrad optimizer Reddi et al.
Revised: AMSGrad optimizer Reddi et al.

Original: (2018) was used with the learning rate of 0.0003, which was reduced by the factor of 10 every 20 epochs.
Revised: (2018) was used with the learning rate of 0.0003, which was reduced by the factor of 10 every 20 epochs.

Original: PFE and DistNet were initialized from the fully-trained deterministic baseline model and fine-tuned over another 60 epochs only for the parts specified in their original work.
Revised: PFE and DistNet were initialized from the fully-trained deterministic baseline model and fine-tuned over another 60 epochs only for the parts specified in their original work.

Original: While the same optimizer was used, the initial learning rate was set as 0.0001, which was also reduced by the factor of 10 every 20 epochs.
Revised: While the same optimizer was used, the initial learning rate was set as 0.0001, which was also reduced by the factor of 10 every 20 epochs.

Original: For all methods, each batch consists of 4 random identities and 8 random images for each identity.
Revised: For all methods, each batch consists of 4 random identities and 8 random images for each identity.

Original: All input images were rescaled to 128 × 256 , and random horizontal flipping with the probability of 0.5 was applied.
Revised: All input images were rescaled to 128 × 256 , and random horizontal flipping with the probability of 0.5 was applied.

Original: We reported the mean and standard deviation of the performance for all models trained with five different random seeds.
Revised: We reported the mean and standard deviation of the performance for all models trained with five different random seeds.

Revised: I M ISCLASSIFIED QMNIST S AMPLES

Revised: Figure 15 shows a subset of QMNIST test images misclassified by MEIB after filter out 80% of the test set in the risk-controlled classification experiment in Section 4.1.

Revised: Although MEIB was overconfident, i.e., assigned inappropriately high entropy values, for these uncertain images, each image has a plausible shape for the predicted class also.

Revised: [Figure]

Revised: J L2 N ORMALIZATION FOR MEIB AND VIB

Revised: L 2 normalization of feature embedding Ranjan et al.

Revised: (2017) is widely used with the softmax loss

Revised: Wang et al.

Revised: (2017), angular margin-based loss Deng et al.

Revised: (2019), and contrastive loss Chen et al.

Revised: We also trained and tested MEIB and VIB with L 2 -normalized z samples to see its benefits on stochastic embeddings.

Revised: Figure 16 shows the perturbation robustness and risk-controlled classification performance, similar to Figure 3, of the MEIB and VIB models with and without L 2 normalization of embeddings.

Revised: MEIB performs much worse with the L 2 normalization in terms of both perturbation robustness and risk-controlled classification.

Revised: While VIB shows a slightly improved robustness with L 2 normalization, its risk-controlled classification performance become worse than without the normalization shortly after about 30% of rejection rate.

Revised: It could be difficult for stochastic embeddings, especially MEIB, to utilize their advantage in angular space since their magnitudes are not considered anymore.

Revised: For example, the main intuition of MEIB is spreading out each input embedding as wide as possible to take more space within a decision region.

Revised: However, after applying

Revised: L 2 normalization, only the angle of each feature vector contributes to the decision regardless of their magnitude in the embedding space.

Revised: [Figure]