M ODELING U NCERTAINTY WITH H EDGED I NSTANCE
E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Instance embeddings are an efﬁcient and versatile image representation that facilitates applications like recognition, veriﬁcation, retrieval, and clustering.
Many metric learning methods represent the input as a single point in the embedding space.
Often the distance between points is used as a proxy for match conﬁdence.
However, this can fail to represent uncertainty which can arise when the input is ambiguous, e.g., due to occlusion or blurriness.
This work addresses this issue and explicitly models the uncertainty by “hedging” the location of each input in the embedding space.
We introduce the hedged instance embedding ( HIB ) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle (Alemi et al., 2016; Achille & Soatto, 2018).
Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of “hedging its bets” across the embedding space upon encountering ambiguous inputs.
This results in improved performance for image matching and classiﬁcation tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure which is correlated with downstream performance.
I NTRODUCTION
An instance embedding is a mapping f from an input x , such as an image, to a vector representation, z ∈ R D , such that “similar” inputs are mapped to nearby points in space.
Embeddings are a versatile representation that support various downstream tasks, including image retrieval (Babenko et al., 2014) and face recognition (Schroff et al., 2015).
Instance embeddings are often treated deterministically, i.e. , z = f ( x ) is a point in R D .
We refer to this approach as a point embedding .
One drawback of this representation is the difﬁculty of modeling aleatoric uncertainty (Kendall & Gal, 2017), i.e. uncertainty induced by the input.
In the case of images this can be caused by occlusion, blurriness, low-contrast and other factors.
To illustrate this, consider the example in Figure 1a.
On the left, we show an image composed of two adjacent MNIST digits, the ﬁrst of which is highly occluded.
The right digit is clearly a 7, but the left digit could be a 1, or a 4.
One way to express this uncertainty about which choice to make is to map the input to a region of space, representing the inherent uncertainty of “where it belongs”.
We propose a new method, called hedged instance embedding ( HIB ), which achieves this goal.
Each embedding is represented as a random variable, Z ∼ p ( z | x ) ∈ R D .
The embedding effectively spreads probability mass across locations in space, depending on the level of uncertainty.
For example in Figure 1b, the corrupted image is mapped to a two-component mixture of Gaussians covering both the “17” and “47” clusters.
We propose a training scheme for the HIB with a learnable-margin contrastive loss and the variational information bottleneck (VIB) principle (Alemi et al., 2016; Achille & Soatto, 2018).
To evaluate our method, we propose a novel dataset, N-digit MNIST, which we will open source.
Using this dataset, we show that HIB exhibits several desirable properties compared to point embeddings: (1) downstream task performance ( e.g. recognition and veriﬁcation) improves for uncertain inputs; (2) the embedding space exhibits enhanced structural regularity; and (3) a per-exemplar uncertainty measure that predicts when the output of the system is reliable.
[Figure]
When both “17” and “47” are plausible, our 2-component Gaussian mixture embedding has the power to spread probability mass on clusters with clean “17” and “47” images.
By contrast, the point embedding will choose to be close to one or the other of these points (or somewhere between).
M ETHODS
In this section, we describe our method in detail.
2. P OINT EMBEDDINGS
Standard point embedding methods try to compute embeddings such that z 1 = f ( x 1 ) and z 2 = f ( x 2 ) are “close” in the embedding space if x 1 and x 2 are “similar” in the ambient space.
To obtain such a mapping, we must decide on the deﬁnition of “closeness” as well as a training objective, as we explain below.
Contrastive loss Contrastive loss (Hadsell et al., 2006) is designed to encourage a small Euclidean distance between a similar pair, and large distance of margin M > 0 for a dissimilar pair.
The loss is
[Equation]
where z i = f ( x i ) .
The hyperparameter M is usually set heuristically or based on validation-set performance.
Soft contrastive loss A probabilistic alternative to contrastive loss, which we will use in our experiments is deﬁned here.
It represents the probability that a pair of points is matching:
[Equation]
with scalar parameters a > 0 and b ∈ R , and the sigmoid function σ ( t ) = 1 1+ e − t .
This formulation calibrates Euclidean distances into a probabilistic expression for similarity.
Instead of setting a hard threshold like M , a and b together comprise a soft threshold on the Euclidean distance.
We will later let a and b be trained from data.
Having deﬁned the match probability p ( m | z 1 , z 2 ) , we formulate the contrastive loss as a binary classiﬁcation loss based on the softmax cross-entropy (negative log-likelihood loss).
More precisely, for an embedding pair ( z 1 , z 2 ) the loss is deﬁned as
[Equation]
where ˆ m is the indicator function with value 1 for ground-truth match and 0 otherwise.
Although some prior work has explored this soft contrastive loss ( e.g. Bertinetto et al.
Orekondy et al.
, it does not seem to be widely used.
However, in our experiments, it performs strictly better than the hard margin version, as explained in Appendix B.
2. S TOCHASTIC EMBEDDINGS
In HIB, we treat embeddings as stochastic mappings x
(cid:55)→ Z , and write the distribution as Z ∼ p ( z | x ) .
In the sections below, we show how to learn and use this mapping.
[Figure] Match probability for probabilistic embeddings The probability of two inputs matching, given in Equation 2, can easily be extended to stochastic embeddings, as follows:
[Equation]
We approximate this integral via Monte-Carlo sampling from z ( k 1 )∼ p ( z
[Equation]
|
[Equation]
and z
[Equation]
In practice, we get good results using K = 8 samples per input image.
Now we discuss the computation of p ( z | x ) .
Single Gaussian embedding The simplest setting is to let p ( z | x ) be a D -dimensional Gaussian with mean µ ( x ) and diagonal covariance Σ( x ) , where µ and Σ are computed via a deep neural network with a shared “body” and 2 D total outputs.
Given a Gaussian representation, we can draw K samples z (1) , · · · , z ( K ) iid ∼ p ( z | x ) , which we can use to approximate the match probability.
Furthermore, we can use the reparametrization trick (Kingma & Welling, 2013) to rewrite the samples as z ( k ) =
diag (cid:16)(cid:112) Σ( x ) (cid:17) · (cid:15) ( k ) + µ ( x ) , where (cid:15) (1) , · · · , (cid:15) ( K ) iid ∼ N (0 , I ) .
This enables easy backpropagation during training.
Mixture of Gaussians (MoG) embedding We can obtain a more ﬂexible representation of uncertainty by using a mixture of C Gaussians to represent our embeddings, i.e. p ( z | x ) =
(cid:80)
Cc =1 N ( z ; µ ( x, c ) , Σ( x, c )) .
To enhance computational efﬁciency, the 2 C mappings { ( µ ( x, c ) , Σ( x, c ))
} Cc =1 share a common CNN stump and are branched with one linear layer per branch.
When approximating Equation 5, we use stratiﬁed sampling, i.e. we sample the same number of samples from each Gaussian component.
Computational considerations The overall pipeline for computing the match probability is shown in Figure 2.
If we use a single Gaussian embedding, the cost (time complexity) of computing the stochastic representation is essentially the same as for point embedding methods, due to the use of a shared network for computing µ ( x ) and Σ( x ) .
Also, the space requirement is only 2 × more.
(This is an important consideration for many embedding-based methods.)
2. VIB TRAINING OBJECTIVE
For training our stochastic embedding, we combine two ingredients: soft contrastive loss in Equation 3 and the VIB principle Alemi et al.
; Achille & Soatto (2018).
We start with a summary of the original VIB formulation, and then describe its extension to our setting.
Variational Information Bottleneck (VIB)
A discriminative model p ( y | x ) is trained under the information bottleneck principle (Tishby et al., 1999) by maximizing the following objective:
[Equation]
where I is the mutual information, and β > 0 is a hyperparameter which controls the tradeoff between the sufﬁciency of z for predicting y , and the minimality (size) of the representation.
Intuitively, this objective lets the latent encoding z capture the salient parts of x (salient for predicting y ), while disallowing it to “memorise” other parts of the input which are irrelevant.
Computing the mutual information is generally computationally intractable, but it is possible to use a tractable variational approximation as shown in Alemi et al.
; Achille & Soatto (2018).
In particular, under the Markov assumption that p ( z | x, y ) =
p ( z | x ) we arrive at a lower bound on
Equation 6 for every training data point ( x, y ) as follows:
[Equation]
where p ( z | x ) is the latent distribution for x , q ( y | z ) is the decoder (classiﬁer), and r ( z ) is an approximate marginal term that is typically set to the unit Gaussian N (0 , I ) .
In Alemi et al.
(2016), this approach was shown (experimentally) to be more robust to adversarial image perturbations than deterministic classiﬁers.
It has also been shown to provide a useful way to detect out-of-domain inputs (Alemi et al., 2018).
Hence we use it as the foundation for our approach.
VIB for learning stochastic embeddings We now apply the above method to learn our stochastic embedding.
In particular, we train a discriminative model based on matching or mismatching pairs of inputs ( x 1 , x 2 ) , by minimizing the following loss:
[Equation]
where the ﬁrst term is given by the negative log likelihood loss with respect to the ground truth match ˆ m (this is identical to Equation 3, the soft contrastive loss), and the second term is the KL regularization term, r ( z ) =
N ( z ; 0 , I ) .
The full derivation is in appendix F. We optimize this loss with respect to the embedding function ( µ ( x ) , Σ( x )) , as well as with respect to the a and b terms in the match probability in Equation 2.
Note that most pairs are not matching, so the m = 1 class is rare.
To handle this, we encourage a balance of m = 0 and m =
1 pair samples within each SGD minibatch by using two streams of input sample images.
One samples images from the training set at random and the other selects images from speciﬁc class labels, and then these are randomly shufﬂed to produce the ﬁnal batch.
As a result, each minibatch has plenty of positive pairs even when there are a large number of classes.
2. U NCERTAINTY MEASURE
One useful property of our method is that the embedding is a distribution and encodes the level of uncertainty for given inputs. As a scalar uncertainty measure, we propose the self-mismatch probability as follows:
[Equation]
Intuitively, the embedding for an ambiguous input will span diverse semantic classes (as in Figure 1b). η ( x ) quantiﬁes this by measuring the chance two samples of the embedding z 1 , z 2 iid ∼ p ( z | x ) belong to different semantic classes (i.e., the event m = 0 happens).
We compute η ( x ) using the Monte-Carlo estimation in Equation 5.
Prior works (Vilnis & McCallum, 2014; Bojchevski & G¨unnemann, 2018) have computed uncertainty for Gaussian embeddings based on volumetric measures like trace or determinant of covariance matrix.
Unlike those measures, η ( x ) can be computed for any distribution from which one can sample, including multi-modal distributions like mixture of Gaussians.
R ELATED W ORK
In this section, we mention the most closely related work from the ﬁelds of deep learning and probabilistic modeling.
Probabilistic DNNs Several works have considered the problem of estimating the uncertainty of a regression or classiﬁcation model, p ( y | x ) , when given ambiguous inputs.
One of the simplest and most widely used techniques is known as Monte Carlo dropout (Gal & Ghahramani, 2016).
In this approach, different random components of the hidden activations are “masked out” and a distribution over the outputs f ( x ) is computed.
By contrast, we compute a parametric representation of the uncertainty and use Monte Carlo to approximate the probability of two points matching.
Monte Carlo dropout is not directly applicable in our setting as the randomness is attached to model parameters and is independent of input; it is designed to measure model uncertainty (epistemic uncertainty).
On the other hand, we measure input uncertainty where the embedding distribution is conditioned on the input.
Our model is designed to measure input uncertainty (aleatoric uncertainty).
VAEs and VIB A variational autoencoder (VAE, Kingma & Welling (2013)) is a latent variable model of the form p ( x, z )
= p ( z ) p ( x | z ) , in which the generative decoder p ( x | z ) and an encoder network, q ( z | x ) are trained jointly so as to maximize the evidence lower bound.
By contrast, we compute a discriminative model on pairs of inputs to maximize a lower bound on the match probability.
The variational information bottleneck (VIB) method (Alemi et al., 2016; Achille & Soatto, 2018) uses a variational approximation similar to the VAE to approximate the information bottleneck objective (Tishby et al., 1999).
We build on this as explained in 2.3.
Point embeddings Instance embeddings are often trained with metric learning objectives, such as contrastive (Hadsell et al., 2006) and triplet (Schroff et al., 2015) losses.
Although these methods work well, they require careful sampling schemes (Wu et al., 2017; Movshovitz-Attias et al., 2017).
Many other alternatives have attempted to decouple the dependency on sampling, including softmax cross-entropy loss coupled with the centre loss (Wan et al., 2018), or a clustering-based loss (Song et al., 2017), and have improved the embedding quality.
In HIB, we use a soft contrastive loss, as explained in section 2.1.
Probabilistic embeddings The idea of probabilistic embeddings is not new.
For example, Vilnis & McCallum (2014) proposed Gaussian embeddings to represent levels of speciﬁcity of word embeddings ( e.g. “Bach” is more speciﬁc than “composer”).
The closeness of the two Gaussians is based on their KL-divergence, and uncertainty is computed from the spread of Gaussian (determinant of covariance matrix).
See also Karaletsos et al.
Bojchevski & G ¨ unnemann (2018) for related work.
Neelakantan et al.
(2014) proposed to represent each word using multiple prototypes, using a “best of K ” loss when training.
HIB, on the other hand, measures closeness based on a quantity related to the expected Euclidean distance, and measures uncertainty using the selfmismatch probability.
E XPERIMENTS
In this section, we report our experimental results, where we compare our stochastic embeddings to point embeddings.
We consider two main tasks: the veriﬁcation task ( i.e. , determining if two input images correspond to the same class or not), and the identiﬁcation task ( i.e. , predicting the label for an input image).
For the latter, we use a K-nearest neighbors approach with K = 5 .
We compare performance of three methods: a baseline deterministic embedding method, our stochastic embedding method with a Gaussian embedding, and our stochastic embedding method with a mixture of
Gaussians embedding.
We also conduct a qualitative comparison of the embeddings of each method.
4. E XPERIMENTAL DETAILS
We conduct all our experiments on a new dataset we created called N-digit MNIST, which consists of images composed of N adjacent MNIST digits, which may be randomly occluded (partially or fully).
See appendix A for details.
During training, we occlude 20% of the digits independently.
A single image can have multiple corrupted digits.
During testing, we consider both clean (unoccluded) and corrupted (occluded) images, and report results separately.
We use images with N = 2 and N =digits.
We will open source the data to ensure reproducibility.
Since our dataset is fairly simple, we use a shallow CNN model to compute the embedding function.
Speciﬁcally, it consists of 2 convolutional layers, with 5 × 5 ﬁlters, each followed by max pooling, [Figure] and then a fully connected layer mapping to D dimensions.
We focus on the case where D = 2 or
D = 3 .
When we use more dimensions, we ﬁnd that all methods (both stochastic and deterministic) perform almost perfectly (upper 90%), so there are no interesting differences to report.
We leave exploration of more challenging datasets and higher dimensional embeddings, to future work.
Our networks are built with TensorFlow (Abadi et al., 2015).
For each task, the input is an image of size 28 × ( N × 28) , where N is the number of concatenated digit images.
We use a batch size ofand 500k training iterations.
Each model is trained from scratch with random weight initialization.
The KL-divergence hyperparameter β is set to 10 − 4 throughout the experiments.
We report effects of changing β in appendix E.
[Table]
[Table]
4. Q UALITATIVE EVALUATION OF THE REPRESENTATION
Figure 3 shows HIB 2D Gaussian embeddings for the clean and corrupt subsets of the test set.
We can easily see that the corrupt images generally have larger ( i.e. , less certain) embeddings.
In the
Appendix, Figure 7 shows a similar result when using a 2D MoG representation, and Figure 8 shows a similar result for 3D Gaussian embeddings.
Figure 4 illustrates the embeddings for several test set images, overlaid with an indication of each class’ centroid.
Hedged embeddings capture the uncertainty that may exist across complex subsets of the class label space, by learning a layout of the embedding space such that classes that may be confused are able to receive density from the underlying hedged embedding distribution.
We observe enhanced spatial regularity when using HIB.
Classes with a common least or most signiﬁcant digit roughly align parallel to the x or y axis.
This is because of the diagonal structure of the embedding covariance matrix.
By controlling the parametrization of the covariance matrix, one may apply varying degrees and types of structures over the embedding space ( e.g. diagonally aligned embeddings).
See appendix D for more analysis of the learned latent space.
4. Q UANTITATIVE EVALUATION OF THE BENEFITS OF STOCHASTIC EMBEDDING
We ﬁrst measure performance on the veriﬁcation task, where the network is used to compute p ( m | x 1 , x 2 ) for 10k pairs of test images, half of which are matches and half are not.
The average precision (AP) of this task is reported in the top half of Table 1.
HIB shows improved performance, especially for corrupted test images.
For example, in the N = 2 digit case, when using D =dimensions, point embeddings achieve 88.0% AP on corrupt test images, while hedged instance embeddings improves to 90.7% with C = 1 Gaussian, and 91.2% with C = 2 Gaussians.
We next measure performance on the KNN identiﬁcation task.
The bottom half of Table 1 reports the results.
Again, proposed stochastic embeddings generally outperform point embeddings, with the greatest advantage for the corrupted input samples.
For example, in the N = 2 digit case, when using D = 2 dimensions, point embeddings achieve 58.3% AP on corrupt test images, while HIB improves to 76.0% with C = 1 Gaussian, and 75.7% with C = 2 Gaussians. [Figure]
4. K NOWN UNKNOWNS
In this section, we address the task of estimating when an input can be reliably recognized or not, which has important practical applications.
To do this, we use the measure of uncertainty η
( x ) deﬁned in Equation 9.
We measure the utility of η ( x ) for the identiﬁcation task as follows.
For the test set, we sort all test input examples according to η ( x ) , and bin examples into 20 bins ranging from the lowest to highest range of uncertainty.
We then measure the KNN classiﬁcation accuracy for the examples falling in each bin.
To measure the utility of η ( x ) for the veriﬁcation task, we take random pairs of samples, ( x 1 , x 2 ) , and compute the mean of their uncertainties, η ( x 1 , x 2 )
=
12 ( η ( x 1 ) + η ( x 2 )) .
We then distribute the test pairs to 20 equal-sized bins according to their uncertainty levels, and compute the probability of a match for each pair.
To cope with the severe class imbalance (most pairs don’t match), we measure performance for each bin using average precision (AP).
Then, again, the Kendall’s tau is applied to measure the uncertainty-performance correlation.
Figure 5 plots the AP and KNN accuracy vs the uncertainty bin index, for both clean and corrupted inputs.
We see that when the performance drops off, the model’s uncertainty measure increases, as desired.
To quantify this, we compute the correlation between the performance metric and the uncertainty metric.
Instead of the standard linear correlations (Pearson correlation coefﬁcient), we use Kendall’s tau correlation (Kendall, 1938) that measures the degree of monotonicity between the performance and the uncertainty level (bin index), inverting the sign so that positive correlation aligns with our goal.
The results of different models are shown in table 2.
In general, the measure η ( x ) correlates with the task performance.
As a baseline for point embeddings in KNN, we explored using the distance to the nearest neighbor as a proxy for uncertainty, but found that it performed poorly.
The
HIB uncertainty metric correlates with task accuracy even in within the subset of clean (uncorrupted) input images having no corrupted digits, indicating that HIB’s understanding of uncertainty goes beyond simply detecting which images are corrupted.
D ISCUSSION AND F UTURE W ORK
Hedged instance embedding is a stochastic embedding that captures the uncertainty of the mapping of an image to a latent embedding space, by spreading density across plausible locations.
This results in improved performance on various tasks, such as veriﬁcation and identiﬁcation, especially for ambiguous corrupted input.
It also allows for a simple way to estimate the uncertainty of the embedding that is correlated with performance on downstream tasks.
There are many possible directions for future work, including experimenting with higherdimensional embeddings, and harder datasets.
It would also be interesting to consider the “open world” (or “unknown unknowns”) scenario, in which the test set may contain examples of novel classes, such as digit combinations that were not in the training set (see e.g., Lakkaraju et al.
G¨unther et al.
This is likely to result in uncertainty about where to embed the input which is different from the uncertainty induced by occlusion, since uncertainty due to open world is epistemic (due to lack of knowledge of a class), whereas uncertainty due to occlusion is aleatoric (intrinsic, due to lack of information in the input), as explained in Kendall & Gal (2017).
Preliminary experiments suggest that η ( x ) correlates well with detecting occluded inputs, but does not work as well for novel classes.
We leave more detailed modeling of epistemic uncertainty as future work
Appendix A N- DIGIT MNIST D ATASET
We present N-digit MNIST, a new dataset based upon MNIST (LeCun, 1998) that has an exponentially large number of classes on the number of digits N , for which embedding-style classiﬁcation methods are well-suited.
The dataset is created by horizontally concatenating N MNIST digit images.
While constructing new classes, we respect the training and test splits.
For example, a test image from 2-digit MNIST of a “54” does not have its “5” or its “4” shared with any image from the training set (in all positions).
[Table]
We employ 2- and 3-digit MNIST (Table 3) in our experiments.
This dataset is meant to provide a test bed for easier and more efﬁcient evaluation of embedding algorithms than with larger and more realistic datasets.
N-digit MNIST is more suitable for embedding evaluation than other synthetic datasets due to the exponentially increasing number of classes as well as the factorizability aspect: each digit position corresponds to e.g. a face attribute for face datasets.
We inject uncertainty into the underlying tasks by randomly occluding (with black ﬁll-in) regions of images at training and test time.
Speciﬁcally, the corruption operation is done independently on each digit of number samples in the dataset.
A random-sized square patch is identiﬁed from a random location of each 28 × 28 digit image.
The patch side length is ﬁrst sampled L ∼ Unif (0 , 28) , and then the top left patch corner coordinates are sampled ( TL x , TL y ) iid ∼ Unif (0 , 28 − L ) , so that the occluded square size is always L 2 .
During training, we set independent binary random ﬂags B for every digit determining whether to perform the occlusion at all; the occlusion chance is set to 20% .
For testing, we prepare twin datasets, clean and corrupt, with digit images that are either not corrupted with occlusion at all or always occluded, respectively.
B S OFT C ONTRASTIVE L OSS VERSUS C ONTRASTIVE L OSS
[Figure]
As a building block for the HIB, soft contrastive loss has been proposed in § 2.1.
Soft contrastive loss has a conceptual advantage over the vanilla contrastive loss that the margin hyperparameter M does not have to be hand-tuned.
Here we verify that soft contrastive loss outperforms the vanilla version over a range of M values.
Figure 6 shows the veriﬁcation (average precision) and identiﬁcation (KNN accuracy) performance of embedding 2-digit MNIST samples.
In both evaluations, soft contrastive loss performance is upper bounding the vanilla contrastive case.
This new formulation removes one hyperparameter from the learning process, while not sacriﬁcing performance.
[Figure]
C E XTRA R ESULTS
In this section, we include some extra results which would not ﬁt in the main paper.
Figure 7 shows some 2D embeddings of 2-digit images using a MoG representation, with C = 2 or C = 4 clusters per embedding.
Figure 8 shows some 3D embeddings of 3-digit images using a single Gaussian.
[Figure]
D O RGANIZATION OF THE L ATENT E MBEDDING S PACE
As hedged instance embedding training progresses, it is advantageous for any subset of classes that may be confused with one another to be situated in the embedding space such that a given input image’s embedding can strategically place probability mass.
We observe this impacts the organization of the underlying space.
For example, in the 2D embeddings shown in Figure 9, the class centers of mass for each class are roughly axis-aligned so that classes that share a tens’ digit vary by x-coordinate, and classes that share a least signiﬁcant (ones) digit vary by y-coordinate.
To further explore this idea, we embed 2-digit MNIST into a single dimension, to see how the classes get embedded along the number line.
For hedged instance embedding, a single Gaussian embedding was chosen as the representation.
We conjectured that because hedged instance embedding reduces objective loss by placing groups of confusing categories nearby one another, the resulting embedding space would be organized to encourage classes that share a tens or ones digit to be nearby.
Figureshows an example embedding learned by the two methods.
We assess the embedding space as follows.
First the centroids for each of the 100 classes are derived from the test set embeddings.
After sorting the classes, a count is made of adjacent class pairs that share a ones or tens digit, with the maximum possible being 99.
The hedged embeddings outscored the point embeddings on each of the the four trials, with scores ranging from 76 to 80 versus scores of 42 to 74.
Similarly, consider a run as a series of consecutive class pairs that share a ones or tens digit.
The average run contains 4.6 classes with from hedged embeddings, and only 3.0 for point embeddings, as illustrated in Figure 10.
E KL D IVERGENCE R EGULARIZATION
The training objective (Equation 8) contains a regularization hyperparameter β ≥ 0 controlling the weight of the KL divergence regularization that controls, from information theoretic perspective, bits of information encoded in the latent representation. In the main experiments, we consistently [Figure]
[Figure]
use β = 10 − 4 .
In this Appendix, we explore the effect of this regularization by considering other values of β .
See Figure 11 for visualization of embeddings according to β .
We observe that increasing KL term weight induces overall increase in variances of embeddings.
For quantitative analysis on the impact of β on main task performance and uncertainty quality, see Table 4 where we compare β =and β =
10 − 4 cases.
We observe mild improvements in main task performances when the KL divergence regularization is used.
For example, KNN accuracy improves from 0.685 to 0.730 by including the KL term for N = 3 , D = 3 case.
Improvement in uncertainty quality, measured in terms of Kendall’s tau correlation, is more pronounced.
For example, KNN accuracy and uncertainty are nearly uncorrelated when β
= 0 under the N = 3 , D = 3 setting (-0.080 for clean and 0. for corrupt inputs), while they are well-correlated when β
= 10 − 4 (0.685 for clean and 0.549 for corrupt inputs).
KL divergence helps generalization, as seen by the main task performance boost, and improves the uncertainty measure by increasing overall variances of embeddings, as seen in
Figure 11.
F D ERIVATION OF VIB O BJECTIVE FOR S TOCHASTIC E MBEDDING
Our goal is to train a discriminative model for match prediction on a pair of variables, p ( m | f ( x 1 ) , f ( x 2 )) , as opposed to predicting a class label, p ( y | x ) .
Our VIB loss (Equation 8) follows straightforwardly from the original VIB, with two additional independence assumptions.
In particular, we assume that the samples in the pair are independent, so p ( x 1 , x 2 ) = p ( x 1 ) p ( x 2 ) .
We also assume the embeddings do not depend on the other input in the pair, p ( z 1 , z 2 | x 1 , x 2 ) =
p ( z 1 | x 1 ) p ( z 2 | x 2 ) .
With these two assumptions, the VIB objective is given by the following:
[Equation]
We variationally bound the ﬁrst term using the approximation q ( m | z 1 , z 2 ) of p ( m | z 1 , z 2 ) as follows I
[Equation]
[Equation]
[Equation]
The inequalities follow from the non-negativity of KL-divergence KL ( · ) and entropy H ( · ) .
The ﬁnal equality follows from our assumptions above.
The second term is variationally bounded using approximation r ( z i ) of p ( z i | x i ) as follows: I (( z 1 , z 2 ) , ( x
[Equation] [Equation]
The inequality, again, follows from the non-negativity of KL-divergence
[Equation]
and the last equality follows from our additional independence assumptions
[Equation] Combining the two bounds, the VIB objective (Equation 10) for a ﬁxed input pair ( x 1 , x 2 ) is bounded from below by
[Equation]
The negative of the above expression is identical to the loss L VIBEmb in Equation 8.