Paper:
DEEP VARIATIONAL CANONICAL CORRELATION ANALYSIS
We present deep variational canonical correlation analysis (VCCA), a deep multiview learning model that extends the latent variable model interpretation of linear CCA (Bach and Jordan, 2005) to nonlinear observation models parameterized by deep neural networks (DNNs). Computing the marginal data likelihood, as well as inference of the latent variables, are intractable under this model. We derive a variational lower bound of the data likelihood by parameterizing the posterior density of the latent variables with another DNN, and approximate the lower bound via Monte Carlo sampling. Interestingly, the resulting model resembles that of multiview autoencoders (Ngiam et al., 2011), with the key distinction of an additional sampling procedure at the bottleneck layer. We also propose a variant of VCCA called VCCA-private which can, in addition to the “common variables” underlying both views, extract the “private variables” within each view. We demonstrate that VCCA-private is able to disentangle the shared and private information for multi-view data without hard supervision.
1 INTRODUCTION
In the multi-view representation learning setting, we have multiple views/measurements of the same underlying signal, and the goal is to learn useful features of each view using complementary information contained in the views. The intuition underlying this setting is that the learned features can help uncover the common sources of variation in the views, which can be helpful for exploratory analysis or for downstream tasks.
A classical approach in this setting is canonical correlation analysis (CCA, Hotelling, 1936) and its nonlinear extensions, including the kernel extension (Lai and Fyfe, 2000; Akaho, 2001; Melzer et al., 2001; Bach and Jordan, 2002) and the deep neural network (DNN) extension (Andrew et al., 2013; Wang et al., 2015b). CCA projects two random vectors x ∈ Rdx and y ∈ Rdy into a lowerdimensional subspace so that the projections are maximally correlated. There is a probabilistic latent variable model interpretation of linear CCA (Bach and Jordan, 2005) as shown in Figure 1 (left). Assume that x and y are linear functions of some lower-dimensional random variable z ∈ Rdz , where dz ≤ min(dx, dy). When the prior distribution of the latent variable p(z) and the conditional distributions p(x|z) and p(y|z) are Gaussian, Bach and Jordan (2005) showed that E[z|x] (resp. E[z|y]) lives in the same space as the linear CCA projection for x (resp. y). This generative interpretation of CCA is often lost in nonlinear extensions of CCA. For example, in deep CCA (DCCA, (Andrew et al., 2013)), to extend CCA to nonlinear mappings with greater representation power, one extracts nonlinear features from the original inputs of each view using two DNNs, f for x and g for y, so that the canonical correlation of the DNN outputs (measured by a linear CCA with projection matrices U and V) is maximized. Formally, given a dataset of N pairs of observations (x1,y1), . . . , (xN ,yN ) of the random vectors (x,y), DCCA optimizes
max Wf ,Wg
U,V
tr ( U>f(X)g(Y)>V ) s.t. U> ( f(X)f(X)> ) U = V> ( g(Y)g(Y)> ) V = NI, (1)
where f(X) = [f(x1), . . . , f(xN )] and g(Y) = [g(y1), . . . ,g(yN )], and Wf denotes all weight parameters of the DNN f (and similarly for g).
DCCA has achieved good performance in the multi-view representation learning setting across different domains (Wang et al., 2015b,a; Lu et al., 2015; Yan and Mikolajczyk, 2015). However, a disadvantage of DCCA is that it directly looks for DNNs that can map inputs into the low-dimensional space, without a model for generating samples from the latent space. Although Wang et al. (2015b)’s deep canonically correlated autoencoders (DCCAE) model optimizes the combination of the autoencoder objective (reconstruction errors) and the canonical correlation objective, the authors found that in practice, the canonical correlation term tends to dominate the reconstruction error terms in the DCCAE objective when tuning performance for a downstream task (especially when the inputs are noisy), and as a result the inputs are not reconstructed well. At the same time, optimization of the DCCA and DCCAE objectives is challenging due to the constraints that couple all training samples.
The main contribution of this paper is the proposal of a new deep multi-view learning model named deep variational CCA (VCCA), which extends the latent variable model interpretation of linear CCA to nonlinear observation models parameterized by DNNs. Computing the marginal data likelihood, as well as inference of the latent variables, are intractable under this model. Inspired by variational autoencoders (VAE, Kingma and Welling, 2014), we parameterize the posterior distribution of the latent variables with another DNN, and derive a variational lower bound of the data likelihood as the objective of VCCA, which is further approximated by Monte Carlo sampling. With the reparameterization trick, sampling for the Monte Carlo approximation is trivial and all DNN weights in VCCA can be optimized jointly via stochastic gradient descent, using unbiased gradient estimates from small minibatches. Interestingly, VCCA is related to multi-view autoencoders (Ngiam et al., 2011), with the key distinctions of additional regularization on the posterior distribution and the sampling procedure at the bottleneck layer.
We also propose a variant of VCCA called VCCA-private that can, in addition to the “common variables” underlying both views, extract the “private variables” within each view. We demonstrate that VCCA-private is able to disentangle the shared and private information for multi-view data without hard supervision. Last but not least, as generative models, VCCA and VCCA-private enable us to obtain high-quality samples for the input of each view.
2 VARIATIONAL CCA
The probabilistic latent variable model of CCA (Bach and Jordan, 2005) defines the following joint distribution over the random variables (x,y):
p(x,y, z) = p(z)p(x|z)p(y|z), p(x,y) = ∫ p(x,y, z)dz. (2)
The assumption underlying this model is that, conditioned on the latent variables z ∈ Rdz , the two views x and y are independent. However, linear observation models (p(x|z) and p(y|z) as shown in Figure 1 (left)) have limited representation power. In this paper, we consider nonlinear
observation models pθ(x|z;θx) and pθ(y|z;θy), parameterized by θx and θy respectively, which can be the collections of weights of DNNs. In this case, the marginal likelihood pθ(x,y) does not have a closed form. In addition, the inference problem pθ(z|x)—the problem of inferring the latent variables given one of the views—is also intractable.
Inspired by Kingma and Welling (2014)’s work on variational autoencoders (VAE), we approximate pθ(z|x) with the conditional density qφ(z|x;φz), where φz is the collection of parameters of another DNN.1 We can derive a lower bound on the marginal data likelihood using qφ(z|x):
log pθ(x,y) = log pθ(x,y) ∫ qφ(z|x)dz = ∫ log pθ(x,y)qφ(z|x)dz
= ∫ qφ(z|x) ( log
qφ(z|x) pθ(z|x,y) + log pθ(x,y, z) qφ(z|x)
) dz
= DKL(qφ(z|x)||pθ(z|x,y)) + Eqφ(z|x) [ log pθ(x,y, z)
qφ(z|x) ] ≥ Eqφ(z|x) [ log pθ(x,y, z)
qφ(z|x)
] =: L(x,y;θ,φ) (3)
where we used the fact that KL divergence is nonnegative in the last step. As a result, L(x,y;θ,φ) is a lower bound on the data log-likelihood logθ p(x,y). Substituting (2) into (3), we have
L(x,y;θ,φ) = ∫ qφ(z|x) ( log p(z)
qφ(z|x) + log pθ(x|z) + log pθ(y|z)
) dz
= −DKL(qφ(z|x)||p(z)) + Eqφ(z|x) [log pθ(x|z) + log pθ(y|z)] . (4) VCCA maximizes this variational lower bound on the data likelihood on the training set:
max θ,φ
1
N N∑ i=1 L(xi,yi;θ,φ). (5)
The first term in (4) measures the KL divergence between the approximate posterior distribution and the prior distribution of the latent variables z. When the parameterization qφ(z|x) is chosen properly, this term can be computed exactly in closed form. As a concrete example, let the variational approximate posterior be a multivariate Gaussian with diagonal covariance. That is, for a sample pair (xi,yi), we have
log qφ(zi|xi) = logN (zi;µi,Σi), Σi = diag ( σ2i1, . . . , σ 2 idz ) , (6)
where the mean µi and covariance Σi are outputs of an encoding DNN f (and thus [µi,Σi] = f(xi;φz) are deterministic nonlinear functions of xi). In this case, we have
DKL(qφ(zi|xi)||p(zi)) = − 1
2 dz∑ j=1 ( 1 + log σ2ij − σ2ij − µ2ij ) .
The second term of (4) corresponds to the expected complete data likelihood under the approximate posterior distribution. Though still intractable, this term can be approximated by Monte Carlo sampling. In particular, we draw L samples z(l)i ∼ qφ(zi|xi):
z (l) i = µi + Σi (l), where (l) ∼ N (0, I), for l = 1, . . . , L, (7) and have
Eqφ(zi|xi) [log pθ(xi|zi) + log pθ(yi|zi)] ≈ 1
L L∑ l=1 log pθ ( xi|z(l)i ) + log pθ ( yi|z(l)i ) . (8)
Notice that we parameterized qφ(zi|xi) above to obtain the VCCA objective; this is useful when the first view is available for downstream tasks, in which case we can directly apply qφ(zi|xi) to obtain its projection (as features). One could also derive likelihood lower bounds by parameterizing the approximate posteriors qφ(zi|yi) and qφ(zi|xi,yi), and optimize their convex combinations for training. We give a sketch of VCCA in Figure 1 (right).
1For notational simplicity, we denote by θ the collection of parameters associated with the model probabilities pθ(·), and φ the collection of parameters associated with the variational approximate probabilities qφ(·), and often omit specific parameters inside the probabilities.
Connection to multi-view autoencoder (MVAE) If we use the Gaussian observation models
log pθ(x|z) = logN (gx(z;θx), I), log pθ(y|z) = logN (gy(z;θy), I), we observe that log pθ ( xi|z(l)i ) and log pθ ( yi|z(l)i ) measure the reconstruction errors of each
view’s inputs from samples z(l)i using the two DNNs gx and gy respectively. In this case, maximizing L(x,y;θ,φ) is equivalent to
min θ,φ
1
N N∑ i=1 DKL(qφ(zi|xi)||p(zi)) + 1 2NL N∑ i=1 L∑ l=1 ∥∥∥xi − gx (z(l)i ;θx)∥∥∥2 + ∥∥∥yi − gy (z(l)i ;θy)∥∥∥2 (9)
s.t. z(l)i = µi + Σi (l), where (l) ∼ N (0, I), l = 1, . . . , L.
Now, consider the case of Σi → 0, for i = 1, . . . , N , and we have z(l)i → µi which is a deterministic function of x (and there is no need for sampling). In the limit, the second term of (9) reduces to
1
2N N∑ i=1 ‖xi − gx(f(xi;φz);θx)‖ 2 + ‖yi − gy(f(xi;φz);θy)‖ 2 , (10)
which is the objective of the multi-view autoencoder (MVAE, Ngiam et al., 2011). Note, however, that Σi → 0 is prevented by the VCCA objective as it results in a large penalty in DKL(qφ(zi|xi)||p(zi)). Compared with the MVAE objective, in the VCCA objective we are creating L different “noisy” versions of the latent representation and enforce that these versions reconstruct the original inputs well. The “noise” distribution (the variances Σi) are also learned and regularized by the KL divergence DKL(qφ(zi|xi)||p(zi)). Using the VCCA objective, we expect to learn different representations from those of MVAE, due to these regularization effects.
2.1 EXTRACTING PRIVATE VARIABLES
So far, VCCA aims at extracting only the latent variables z that are common to both views. A potential disadvantage of this model is that it assumes the common variables are sufficient by themselves to generate the views, which can be too restrictive in practice. Consider the example of audio and articulatory measurements as two views for speech. Although the transcription is a common variable behind the views, it combines with the physical environment and the vocal tract anatomy to generate the individual views. In other words, there might be large variations in the input space that can not be explained by the common variables, making the objective (4) hard to optimize. It may then be beneficial to explicitly model the private variables within each view.
We therefore propose a new probabilistic graphical model, shown in Figure 2, that we refer to as VCCA-private. We introduce two sets of hidden variables hx ∈ Rdhx and hy ∈ Rdhy to explain the aspects of x and y not captured by the common variables z. Under this model, the data likelihood
is defined by
pθ(x,y, z,hx,hy) = p(z)p(hx)p(hy)pθ(x|z,hx;θx)pθ(y|z,hy;θy), (11)
pθ(x,y) = ∫ ∫ ∫ pθ(x,y, z,hx,hy)dz dhx dhy.
To obtain tractable inference, we introduce the following factored variational posterior
qφ(z,hx,hy|x,y) = qφ(z|x;φz)qφ(hx|x;φx)qφ(hy|y;φy), (12)
where each factor is parameterized by a different DNN. Similarly to VCCA, we can derive a variational lower bound on the data likelihood for VCCA-private as
log pθ(x,y) ≥ ∫ ∫ ∫ qφ(z,hx,hy|x,y) log pθ(x,y, z,hx,hy)
qφ(z,hx,hy|x,y) dz dhx dhy
= ∫ ∫ ∫ qφ(z,hx,hy|x,y) [ log p(z)
qφ(z|x) + log
p(hx)
qφ(hx|x) + log
p(hy)
qφ(hy|y) + log pθ(x|z,hx) + log pθ(y|z,hy) ] dz dhx dhy
= −DKL(qφ(z|x)||p(z))−DKL(qφ(hx|x)||p(hx))−DKL(qφ(hy|y)||p(hy))
+ ∫ ∫ qφ(z|x)qφ(hx|x) log pθ(x|z,hx)dz dhx + ∫ ∫ qφ(z|x)qφ(hy|y) log pθ(y|z,hy)dz dhy
=: Lprivate(x,y;θ,φ). (13)
As in VCCA, the last two terms of (14) can be approximated by Monte Carlo sampling. For example, we draw samples of z and hx from their corresponding approximate posteriors, and concatenate their samples as inputs to the DNN parameterizing pθ(x|z,hx). In this paper, we use simple Gaussian prior distributions for the private variables, i.e., hx ∼ N (0, I) and hy ∼ N (0, I). We leave to future work to examine the effect of more sophisticated prior distributions for the latent variables.
VCCA-private maximizes this lower bound on the training set, i.e.,
max θ,φ
1
N N∑ i=1 Lprivate(xi,yi;θ,φ). (14)
Optimization The objectives (5) and (14) decouple over the training samples and can be trained efficiently using stochastic gradient descent. Enabled by the reparameterization trick, unbiased gradient estimates are obtained by Monte Carlo sampling and the standard backpropagation procedure on minibatches of training samples. We apply the ADAM algorithm (Kingma and Ba, 2015) for optimizing our objectives.
3 RELATED WORK
Recently, there has been much interest in unsupervised deep generative models (Kingma and Welling, 2014; Rezende et al., 2014; Goodfellow et al., 2014; Gregor et al., 2015; Makhzani et al., 2016; Burda et al., 2016; Alain et al., 2016). A common motivation behind these models is that, with the expressive power of DNNs, the generative models can capture distributions for complex inputs. Additionally, if we are able to generate realistic samples from the learned distribution, we can infer that we have discovered the underlying structure of the data, which may allow us to reduce the sample complexity for learning for downstream tasks. These previous models have mostly focused on single-view data. Here we focus on the multi-view setting where multiple views of the data are present for feature extraction but only one view is available at test time (in downstream tasks).
Some recent work has explored deep generative models for (semi-)supervised learning. Kingma et al. (2014) built a generative model based on variational autoencoders (VAEs) for semi-supervised classification, where the authors model the input distribution with two set of latent variables: the class label (if it is missing) and another set that models the intra-class variabilities (styles). Sohn
et al. (2015) proposed a conditional generative model for structured output prediction, where the authors explicitly model the uncertainty in the input/output using Gaussian latent variables. While there are two set of observations (input and output labels) in these work, their graphical models are different from that of VCCA.
Our work is also related to the deep multi-view probabilistic models based on restricted Boltzmann machines (Srivastava and Salakhutdinov, 2014; Sohn et al., 2014). We note that these are undirected graphical models for which both inference and learning are difficult, and one typically resorts to carefully designed variational approximation and Gibbs sampling procedures for training such models. In contrast, our models only require sampling from simple, standard distributions (such as Gaussians), and all parameters can be learned end-to-end by standard stochastic gradient methods. Therefore, our models are more scalable than the previous multi-view probabilistic models.
On the other hand, there is a rich literature in modeling multi-view data using the same or similar graphical models behind VCCA/VCCA-private (Wang, 2007; Jia et al., 2010; Salzmann et al., 2010; Virtanen et al., 2011; Memisevic et al., 2012; Klami et al., 2013). Our methods differ from previous work in parameterizing the probability distributions using DNNs. This makes the model more powerful, while still having tractable objectives and efficient end-to-end training using the local reparameterization technique. We note that, unlike earlier work on probabilistic models of linear CCA (Bach and Jordan, 2005), VCCA does not optimize the same criterion, nor produce the same solution, as any linear or nonlinear CCA. However, we retain the terminology in order to clarify the connection with earlier work on probabilistic models for CCA, which we are extending with DNN models for the observations and for the variational posterior distribution approximation.
4 EXPERIMENTAL RESULTS
In this section, we compare different multi-view representation learning algorithms on three tasks involving several domains: image-image, speech-articulation, and image-text. The algorithms we choose to compare below are closely related to the proposed model or have been shown to have strong empirical performance under similar settings.
• Linear CCA: its probabilistic interpretation motivates this work.
• Deep CCA (DCCA) (Andrew et al., 2013): see its objective in (1).
• Deep canonically correlated autoencoders (DCCAE) (Wang et al., 2015b): combination of the DCCA objective and the reconstruction errors of each view.
• Multi-view autoencoder (MVAE) (Ngiam et al., 2011): see its objective in (10).
• Multi-view contrastive loss (Hermann and Blunsom, 2014): based on the intuition that the distance between embeddings of paired examples x+ and y+ should be smaller than the distance between embeddings of x+ and an unmatched negative example y− by a margin:
min f,g Lcontrast :=
1
N N∑ i max ( 0, m+ dis ( f(x+i ), g(y + i ) ) − dis ( f(x+i ), g(y − i ) )) ,
where y−i is a randomly sampled view 2 example, and m is a margin hyperparameter. We use the cosine distance dis (a,b) = 1− 〈
a ‖a‖ , b ‖b‖
〉 .
4.1 NOISY MNIST DATASET
We first demonstrate our algorithms on the noisy MNIST dataset used by Wang et al. (2015b). The dataset is generated using the MNIST dataset (LeCun et al., 1998), which consists of 28 × 28 grayscale digit images, with 60K/10K images for training/testing. We first linearly rescale the pixel values to the range [0, 1]. Then, we randomly rotate the images at angles uniformly sampled from [−π/4, π/4] and the resulting images are used as view 1 inputs. For each view 1 image, we randomly select an image of the same identity (0-9) from the original dataset, add independent random noise uniformly sampled from [0, 1] to each pixel, and truncate the pixel final values to [0, 1] to obtain the corresponding view 2 sample. Selection of input images are given in Figure 3 (left). The original
training set is further split into training/tuning sets of size 50K/10K. The data generation process ensures that the digit identity is the only common variable underlying both views.
To evaluate the amount of class information extracted by different methods, after unsupervised learning of latent representations, we reveal the labels and train a linear SVM on the projected view 1 training data (using the one-versus-all scheme), and use it to classify the projected test set. This experiment simulates the typical usage of multi-view learning methods, which is to extract useful representations for downstream discriminative tasks.
Note that this synthetic dataset perfectly satisfies the multi-view assumption that the two views are independent given the class label, so the latent representation should contain precisely the class information. This is indeed achieved by CCA-based and contrastive loss-based multi-view approaches. In Figure 3 (right), we show 2D t-SNE (van der Maaten and Hinton, 2008) visualizations of the original view 1 inputs and view 1 projections by various deep multi-view methods.
We use DNNs with 3 hidden layers of 1024 rectified linear units (ReLUs, Nair and Hinton, 2010) each to parameterize the distributions: qφ(z|x), pθ(x|z), pθ(y|z) in VCCA, and additionally qφ(hx|x) and qφ(hy|y) in VCCA-private. The capacities of these networks are the same as those of their counterparts in DCCA and DCCAE from Wang et al. (2015b). The reconstruction networks pθ(x|z) or pθ(x|z,hx) model each pixel of x as an independent Bernoulli variable and parameterize its mean (using a sigmoid activation); pθ(y|z) and pθ(y|z,hy) model y with diagonal Gaussians and parameterize the mean (using a sigmoid activation) and standard deviation for each pixel dimension. We tune the dimensionality dz over {10, 20, 30, 40, 50}, and fix dhx = dhy = 30 for VCCA-private. We select the hyperparameter combination that yields the best SVM classification accuracy on the projected tuning set, and report the corresponding accuracy on the projected test set.
The effect of dropout We add dropout (Srivastava et al., 2014) to all intermediate layers and the input layers and find it to be very useful in our models, with most of the gain coming from dropout applied to the samples of z, hx and hy . This is because dropout encourages each latent dimension to reconstruct the inputs well in the absence of other dimensions, and therefore avoids learning coadapted features. Intuitively, in VCCA-private dropout also helps to prevent the degenerate situation where the pathways x → hx → x and y → hy → y achieve good reconstruction while ignoring z (e.g., by setting it to a constant). We use the same dropout rate for all layers and tune it over {0, 0.1, 0.2, 0.3, 0.4}. We show the 2D t-SNE embeddings of the common variables z learned by VCCA and VCCA-private on test set in Figure 4. We observe that in general, VCCA/VCCA-private tend to separate the classes
in the projection well; dropout significantly improves the performance of both VCCA and VCCAprivate, with the latter slightly outperforming the former. While such class separation can also be achieved by DCCA/contrastive loss as well, these methods can not naturally generate samples in the input space. On the other hand, such separation is not achieved by multi-view autoencoders.
The effect of private variables on reconstructions We show sample reconstructions (mean and standard deviation) by VCCA for the view 2 images from the test set in Figure 5 (columns 2 and 3). We observe that for each input, the mean reconstruction of yi by VCCA is a prototypical image of the same digit, regardless of the individual style in yi. This is to be expected, as yi contains an arbitrary image of the same digit as xi, and the variation in background noise in yi does not appear in xi and can not be reflected in qφ(z|x); thus the best way for pθ(y|z) to model yi is to output a prototypical image of that class to achieve on average small reconstruction error. On the other hand, since yi contains little rotation of the digits, this variation is suppressed to a large extent in qφ(z|x) (it is no longer the major variation in z as in the original inputs).
We show sample reconstructions by VCCA-private for the same set of view 2 images in Figure 5 (columns 4 and 5). With the help of private variables hy (as part of the input to pθ(y|z,hy)), the model does a much better job in reconstructing the styles of y. And by disentangling the private variables from the shared variables, qφ(z|x) achieves even better class separation than VCCA does.
We also note that the standard deviation of the reconstruction is low within the digit and high outside the digit, implying that pθ(y|z,hy) is able to separate the background noise from the digit image.
Disentanglement of private/shared variables In Figure 6 (in Appendix) we provide the 2D tSNE embeddings of the shared variables z (top row) and the private variables hx (bottom row) learned by VCCA-private. In the embedding of hx, digits with different identities but the same rotation are mapped close together, and the rotation varies smoothly from left to right, confirming that the private variables contain little class information but mainly style information.
Finally, we give the test error rates of linear SVMs applied to the features learned with different models in Table 1. VCCA-private is comparable in performance to the best previous approach (DCCAE), while having the advantage that it can also generate.
4.2 XRMB SPEECH-ARTICULATION DATASET
We now consider the task of learning acoustic features for speech recognition. We use data from the Wisconsin X-ray microbeam (XRMB) corpus (Westbury, 1994), which contains simultaneously recorded speech and articulatory measurements from 47 American English speakers. We follow the setup of Wang et al. (2015a,b) and use the learned features for speaker-independent phonetic recognition.2 The two input views are standard 39D acoustic features (13 mel frequency cepstral coefficients (MFCCs) and their first and second derivatives) and 16D articulatory features (horizontal/vertical displacement of 8 pellets attached to several parts of the vocal tract), each then concatenated over a 7-frame window around each frame to incorporate context. The speakers are split into disjoint sets of 35/8/2/2 speakers for feature learning/recognizer training/tuning/testing. The 35 speakers for feature learning are fixed; the remaining 12 are used in a 6-fold experiment (recognizer training on 8 speakers, tuning on 2 speakers, and testing on the remaining 2 speakers). Each speaker has roughly 50K frames. We remove the per-speaker mean and variance of the articulatory measurements for each training speaker, and remove the mean of the acoustic measurements for each utterance. All learned feature types are used in a “tandem” speech recognizer (Hermansky et al., 2000), i.e., they are appended to the original 39D features and used in a standard hidden Markov model (HMM)-based recognizer with Gaussian mixture observation distributions.
Each algorithm uses up to 3 ReLU hidden layers, each of 1500 units, for the projection and reconstruction mappings. For VCCA/VCCA-private, we use Gaussian observation models as the inputs are real-valued. In contrast to the MNIST experiments, we do not learn the standard deviations of each output dimension on training data, as this leads to poor downstream task performance. Instead, we use isotropic covariances for each view, and tune the standard deviations by grid search. The best model uses a smaller standard deviation (0.1) for the view 2 than for view 1 (1.0), effectively putting more emphasis on the reconstruction of articulatory measurements. Our best performing VCCA model uses dz = 70, while the best performing VCCA-private model uses dz = 70 and dhx = dhy = 10.
2As in Wang and Livescu (2016), we use the Kaldi toolkit (Povey et al., 2011) for feature extraction and recognition with hidden Markov models. Our results do not match Wang et al. (2015a,b) (who instead used the HTK toolkit (Young et al., 1999)) for the same types of features, but the relative results are consistent.
The mean phone error rates (PER) over 6 folds obtained by different algorithms are given in Table 1. Our methods achieve competitive performance in comparison to previous deep multi-view methods.
4.3 MIR-FLICKR DATASET
Finally, we consider the task of learning cross-modality features for topic classification on the MIRFlickr database (Huiskes and Lew, 2008). The Flickr database contains 1 million images accompanied by user tags, among which 25000 images are labeled with 38 topic classes (each image may be categorized as multiple topics). We use the same image and text features as in previous work (Srivastava and Salakhutdinov, 2014; Sohn et al., 2014): the image feature is 3857 dimensional real-valued vector, composed of Pyramid Histogram of Words (PHOW) (Bosch et al., 2007), GIST (Oliva and Torralba, 2001), and MPEG-7 descriptors (Manjunath et al., 2001), while the text feature is a 2000-dimensional binary vector of frequent tags.
Following the same protocol as Sohn et al. (2014), we train multi-view representations using the unlabelled data,3 and use projected image features of the labeled data (further divided into splits of 10000/5000/10000 samples for training/tuning/testing) for training and evaluating a classifier that predicts the topic labels, corresponding to the unimodal query task in Srivastava and Salakhutdinov (2014); Sohn et al. (2014). For each algorithm, we select the model which achieves the highest mean average precision (mAP) on the validation set, and report its performance on the test set.
Each algorithm uses up to 4 ReLU hidden layers, each of 1024 units, for the projection and reconstruction mappings. For VCCA/VCCA-private, we use Gaussian observation models with isotropic covariance for image features, with standard deviation tuned by grid search, and a Bernoulli model for text features. In this experiment, we also found it helpful to tune an additional trade-off parameter for the text-view likelihood (cross-entropy); the best VCCA/VCCA-private models prefer a large trade-off parameter of the level 104, emphasizing the reconstruction of the sparse text-view inputs. Our best performing VCCA model uses dz = 1024, while the best performing VCCA-private model uses dz = 1024 and dhx = dhy = 16.
As shown in Table 1, VCCA/VCCA-private achieve significantly higher mAPs than other methods considered here. Being much easier to train, the performance of our methods are competitive with the previous state-of-the-art mAP result of 0.607 achieved by the multi-view RBMs of Sohn et al. (2014) under the same setting.
5 CONCLUSIONS
We have proposed variational canonical correlation analysis (VCCA), a deep generative method for multi-view representation learning. Our method embodies a natural idea for multi-view learning: the multiple views can be generated from a small set of shared latent variables. VCCA is parameterized by DNNs and can be trained efficiently by backpropagation, and is therefore scalable. We have also shown that, by modeling the private variables that are specific to each view, the VCCA-private variant can disentangle shared/private variables and provide higher-quality reconstructions.
In the future, we will explore other prior distributions such as mixtures of Gaussians or discrete random variables, which may enforce clustering in the latent space and in turn work better for discriminative tasks. We will also explore other observation models, including replacing the autoencoder objective with that of adversarial networks (Goodfellow et al., 2014; Makhzani et al., 2016; Chen et al., 2016). Another direction is to explicitly incorporate the structure of the inputs, such as the sequence structure of speech and text and the spatial structure of images.
ACKOWLEDGEMENTS
This research was supported by NSF grant IIS-1321015. The opinions expressed in this work are those of the authors and do not necessarily reflect the views of the funding agency. This research used GPUs donated by NVIDIA Corporation.
3As in Sohn et al. (2014), we exclude about 250000 samples which contain fewer than two tags.
A ADDITIONAL T-SNE VISUALIZATION OF NOISY MNIST

Review:
This paper considers the case where multiple views of data are learned through a probabilistic deep neural network formulation. This makes the model non-linear (unlike e.g. CCA) but makes inference difficult. Therefore, the VAE framework is invoked for inference.
In [Ref 1] the authors show that maximum likelihood estimation based on their linear latent model leads to the canonical correlation directions. But in the non-linear case with DNNs it's not clear (at least with the present analysis) what the solution is wrt to the canonical directions. There's no such analysis in the paper, hence I find it a stretch to refer to this model as a CCA type of model. In contrast, e.g. DCCA / DCCAE are taking the canonical correlation between features into account inside the objective and provide interpretations.
[Ref 1] F. R. Bach and M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, 2005.
There is also a significant body of very related work on non-linear multi-view models which is not discussed in this paper. For example, there's been probabilistic non-linear multi-view models [Ref 2, 3], also extended to the Bayesian case with common/private spaces [Ref 4] and the variational / deep learning case [Ref 5].
[Ref 2] Ek et al. Gaussian process latent variable models for human pose estimation. MLMI, 2007.
[Ref 3] Shon et al. Learning shared latent structure for image synthesis and robotic imitation. NIPS, 2006.
[Ref 4] Damianou et al. Manifold relevance determination. ICML, 2012.
[Ref 5] Damianou and Lawrence. Deep Gaussian processes. AISTATS, 2013.
I can see the utility of this model as bringing together two elements: multi-view modeling and VAEs. This seems like an obvious idea but to the best of my knowledge it hasn't been done before and is actually a potentially very useful model.
However, the question is, what is the proper way of extending VAE to multiple views? The paper didn't convince me that VAE can work well with multiple views using the shown straightforward construction. Specifically, VCCA doesn't seem to promote the state of the art in terms of results (it actually is overall below the SOA), while the VCCA-private seems a quite ill-posed model: the dimensionalities d have to be manually tuned with exhaustive search; further, the actual model does not provide a consinstent way of encouraging the private and common variables to avoid learning redundant information. Relying only on dropout for this seems a quite ad-hoc solution (in fact, from Fig. 4 (ver2) it seems that the dropout rate is quite crucial). Perhaps good performance might be achieved with a lot of tuning (which might be why the FLICKR results got better in ver2 without changing the model), but it seems quite difficult to optimize for the above reasons. From a purely experimental point of view, VCCA-private doesn't seem to promote the SOA either. Of course one wouldn't expect any new published paper to beat all previous baselines, but it seems that extension of VAE to multiple views is a very interesting idea which deserves some more investigation of how to do it efficiently.
Another issue is the approximate posterior being parameterized only from one of the views. This makes the model less useful as a generic multi-view model, since it will misbehave in tasks other than classification. But if classification is the main objective, then one should compare to a proper classification model, e.g. a feedforward neural network.
The plots of Fig. 8 are very nice. Overall, the paper convinced me that there is merit in attaching multiple views to VAE. However, it didn't convince me a) that the proposed way to achieve this is practical b) that there is a connection to CCA (other than being a method for multiple views). The bottom line is that, although the paper is interesting, it needs a little more work.