Original: D EEP M ULTI -V IEW L EARNING VIA T ASK -O PTIMAL CCA
Revised: D EEP M ULTI -V IEW L EARNING VIA T ASK -O PTIMAL CCA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original:  Canonical Correlation Analysis (CCA) is widely used for multimodal data analysis and, more recently,  fordiscriminative tasks such as multi-view learning; however, it makes no use of class labels.
Revised: Multi-view learning seeks to form better models by making use of multiple feature sets representing the same samples. Exploiting feature correlations during training can enable better models. The traditional method for computing correlations between feature sets is Canonical Correlation Analysis (CCA), which ﬁnds linear projections that maximize correlation between feature vectors, essentially computing a shared embedding for two views of data. More recently, CCA has been used for multi-view discriminative tasks; however, CCA makes no use of class labels.

Original: Recent CCA methods have started to address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only.
Revised: Recent CCA methods have started to address this weakness but are limited in that they do not simultaneously optimize the CCA projection for discrimination and the CCA projection itself, or they are linear only.

Original: We address these deﬁciencies by simultaneously optimizing a CCAbased and a task objective in an end-to-end manner.
Revised: We address these deﬁciencies by simultaneously optimizing a CCA-based and a task objective in an end-to-end manner.

Original: Together, these two objectives learn a non-linear CCA projection to a shared latent space that is highly correlated and discriminative.
Revised: Together, these two objectives learn a non-linear CCA projection to a shared latent space that is highly correlated and discriminative.

Original: Our method shows a signiﬁcant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classiﬁcation (8.5% increase), regularization with a second view during training when only one view is available at test time (2.2-3.2%), and semi-supervised learning (15%) on real data.
Revised: Our method shows a signiﬁcant improvement over previous state-of-the-art (including deep supervised approaches) for cross-view classiﬁcation (8.5% increase), regularization with a second view during training when only one view is available at test time (2.2-3.2%), and semi-supervised learning (15%) on real data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Revised: Parallel modalities of data are increasingly common in a variety of applications, including images and text, audio and video, parallel texts of different languages, and a variety of medical imaging and omics modalities for each patient.

Revised: Each view provides essential information for classiﬁcation and, when used together, can form a more accurate model.

Original: CCA is  a popular data analysis technique that projects two  data sources into a space in which they are maximally correlated (Hotelling, 1936; Bie et al., 2005).It was initially used for unsupervised data analysis to gain insights into components shared by the two sources (Andrew et al., 2013; Wang et al., 2015a; 2016).
Revised: This is especially important for difﬁcult discriminative tasks such as those with a small training set size. Canonical Correlation Analysis (CCA) is the most common method for computing a shared representation from two views of data by computing a space in which they are maximally correlated (Hotelling, 1936; Bie et al., 2005).

Original:  CCA is also used to compute a shared latent space for cross-view classiﬁcation (Kan et al., 2015; Wang et al., 2015a; Chandar et al., 2016; Chang et al., 2018), for representation learning on multiple views that are then joined for prediction (Sargin et al., 2007; Dorfer et al., 2016b), and for classiﬁcation from a single view when a second view is available during training (Arora & Livescu, 2012).
Revised: In this paper we will demonstrate that, through optimizing for both discriminative features and correlation between views, we can improve classiﬁcation accuracy for three real world scenarios. CCA is an unsupervised method but has been applied to many discriminative tasks (Kan et al., 2015; Sargin et al., 2007; Arora & Livescu, 2012).

Original: While some of the correlated CCA features are useful for discriminative tasks, many represent properties that are of no use for classiﬁcation and obscure correlated information that is beneﬁcial.
Revised: While some of the correlated CCA features are useful for discriminative tasks, many represent properties that are of no use for classiﬁcation and obscure correlated information that is beneﬁcial.

Original: This problem is magniﬁed with recent non-linear extensions of CCA, implemented via neural networks (NNs), that  make signiﬁcant strides in improving correlation (Andrew et al., 2013;
Revised: This problem is magniﬁed with recent non-linear extensions of CCA that use deep learning to make signiﬁcant strides in improving correlation (Andrew et al., 2013;

Original: Wang et al., 2015a; 2016; Chang et al., 2018) but often at the expense of discriminative capability (cf. § 4.1).
Revised: Wang et al., 2015a; 2016; Chang et al., 2018) but often at the expense of discriminative capability (cf. § 5.1).

Original: Therefore, we present  a new deep learning technique to project the data from two views to a shared space that is also discriminative.
Revised: Therefore, we present Task-Optimal CCA (TOCCA), a new deep learning technique to project the data from two views to a shared space that is also discriminative (Fig. 1).

Revised: Implementing a task-optimal variant of CCA required a fundamental change in formulation.

Revised: We show that the CCA objective can equivalently be expressed as an (cid:96) 2 distance minimization in the shared space plus an orthogonality constraint.

Revised: Orthogonality constraints help regularize neural networks (NNs)

Revised: (Huang et al., 2018); we present three techniques to accomplish this.

Revised: While our method is derived from CCA, by manipulating the orthogonality constraints, we obtain deep CCA approaches that compute a shared latent space that is also discriminative.

Original: Most prior work that boosts the discriminative capability of  CCA  is linear only (Lee et al., 2015;
Revised: Our family of solutions for supervised CCA required a crucial and non-trivial change in formulation. We demonstrate the effectiveness and versatility of our model for three different tasks: 1) crossview classiﬁcation on a variation of MNIST (LeCun, 1998), 2) regularization when two views are [Figure] Goal : shared embedding that is also discriminative

Revised: [Figure]

Revised: [Figure]

Revised: available for training but only one at test time on a cancer imaging and genomic data set with only 1,000 samples, and 3) semi-supervised representation learning to improve speech recognition.

Revised: All experiments showed a signiﬁcant improvement in accuracy over previous state-of-the-art.

Revised: In addition, our approach is more robust in the small sample size regime than alternative methods.

Revised: Overall, our experiments on real data show the effectiveness of our method in learning a shared space that is more discriminative than previous methods for a variety of practical problems.

Revised:  R ELATED W ORK

Revised: CCA was initially used for unsupervised data analysis to gain insights into components shared by two sources (Andrew et al., 2013; Wang et al., 2015a; 2016).

Revised: CCA has also been used to compute a shared latent space for cross-view classiﬁcation (Kan et al., 2015; Wang et al., 2015a; Chandar et al., 2016; Chang et al., 2018), for representation learning on multiple views that are then joined for prediction (Sargin et al., 2007; Dorfer et al., 2016b), and for classiﬁcation from a single view when a second view is available during training (Arora & Livescu, 2012).

Revised: Recent non-linear extensions of CCA implemented via NNs make signiﬁcant improvements in correlation (Andrew et al., 2013;

Revised: Wang et al., 2015a; 2016; Chang et al., 2018)

Revised: but with little focus on discriminative capability.

Revised: Most prior work that boosts the discriminative capability of CCA is linear only (Lee et al., 2015;

Original: Singanamalli et al., 2014; Duan et al., 2016).
Revised: Singanamalli et al., 2014; Duan et al., 2016).

Original: More recent work using NNs still remains limited in that it optimizes discriminative capability for an intermediate representation rather than the ﬁnal
Revised: More recent work using NNs still remains limited in that it optimizes discriminative capability for an intermediate representation rather than the ﬁnal

Original: CCA projection (Dorfer et al., 2016b), or optimizes the CCA objective only during pre-training, not while training the task objective (Dorfer et al., 2018).
Revised: CCA projection (Dorfer et al., 2016b), or optimizes the CCA objective only during pre-training, not while training the task objective (Dorfer et al., 2018).

Original: We advocate to jointly optimize CCA and a discriminative objective by computing the CCA projection within a network layer while applying a task-driven operation such as classiﬁcation.
Revised: We advocate to jointly optimize CCA and a discriminative objective by computing the CCA projection within a network layer while applying a task-driven operation such as classiﬁcation.

Original: Experimental results show that our method signiﬁcantly improves upon previous work (Dorfer et al., 2016b; 2018) due to its focus on both the shared latent space and a task-driven objective.
Revised: Experimental results show that our method signiﬁcantly improves upon previous work (Dorfer et al., 2016b; 2018) due to its focus on both the shared latent space and a task-driven objective.

Original: The latter is particularly important on small training set sizes.
Revised: The latter is particularly important on small training set sizes.

Original: While alternative approaches to multi-view learning via CCA exist, they typically focus on a reconstruction objective.
Revised: While alternative approaches to multi-view learning via CCA exist, they typically focus on a reconstruction objective.

Original: That is, they transform the input into a shared space such that the input could be reconstructed – either individually or reconstructing one view from the other.
Revised: That is, they transform the input into a shared space such that the input could be reconstructed – either individually or reconstructing one view from the other.

Original: Variations of coupled dictionary learning (Shekhar et al., 2014; Xu et al., 2015; Cha et al., 2015; Bahrampour et al., 2015) and autoencoders (Wang et al., 2015a; Bhatt et al., 2017) have been used in this context.
Revised: Variations of coupled dictionary learning (Shekhar et al., 2014; Xu et al., 2015; Cha et al., 2015; Bahrampour et al., 2015) and autoencoders (Wang et al., 2015a; Bhatt et al., 2017) have been used in this context.

Original: CCA-based objectives, such as the model used in this work, instead learn a transformation to a shared space without the need for reconstructing the input.
Revised: CCA-based objectives, such as the model used in this work, instead learn a transformation to a shared space without the need for reconstructing the input.

Original: This task may be easier and sufﬁcient in producing a representation for multi-view classiﬁcation (Wang et al., 2015a).
Revised: This task may be easier and sufﬁcient in producing a representation for multi-view classiﬁcation (Wang et al., 2015a).

Original: Implementing a task-optimal variant of CCA required a fundamental change in formulation.

Original: We show that the CCA objective can equivalently be expressed as an (cid:96) 2 distance minimization in the shared space plus an orthogonality constraint.

Original: Orthogonality constraints help regularize NNs (Huang et al., 2018); we present three techniques to accomplish this.

Original: While our method is derived from

Original: CCA, by manipulating the orthogonality constraints, we obtain deep CCA approaches that compute a shared latent space that is also discriminative.

Original: Our family of solutions for supervised CCA required a crucial and non-trivial change in formulation.

Original: We demonstrate the effectiveness and versatility of our model for three different tasks: 1) crossview classiﬁcation on a variation of MNIST (LeCun, 1998), 2) regularization when two views are available for training but only one at test time on a cancer imaging and genomic data set with only 1,000 samples, and 3) semi-supervised representation learning to improve speech recognition.

Original: All experiments showed a signiﬁcant improvement in accuracy over previous state-of-the-art.

Original: In addition, our approach is more robust in the small sample size regime than alternative methods.

Original: Overall, our experiments on real data show the effectiveness of our method in learning a shared space that is more discriminative than previous methods for a variety of practical problems.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst introduce CCA and present our task-driven approach in § 3.
Revised: We ﬁrst introduce CCA and present our task-driven approach in § 4.

Original: Linear and non-linear CCA are unsupervised and ﬁnd the shared signal between a pair of data sources, by maximizing the sum correlation between corresponding projections.
Revised: Linear and non-linear CCA are unsupervised and ﬁnd the shared signal between a pair of data sources, by maximizing the sum correlation between corresponding projections.

Original: Let X 1 ∈ R d 1 × n and X 2 ∈ R
Revised: Let X 1 ∈ R d 1 × n and X 2 ∈ R

Original: d
Revised: d

Original: 2 × n be meancentered input data from two different views with n samples and d 1 , d 2 features, respectively.
Revised: 2 × n be meancentered input data from two different views with n samples and d 1 , d 2 features, respectively.

Original: CCA.
Revised: CCA.

Original: The objective is to maximize the correlation between a 1 = w (cid:62) 1 X 1 and a 2 = w (cid:62) 2 X 2 , where w 1 and w 2 are projection vectors (Hotelling, 1936).
Revised: The objective is to maximize the correlation between a 1 = w (cid:62) 1 X 1 and a 2 = w (cid:62) 2 X 2 , where w 1 and w 2 are projection vectors (Hotelling, 1936).

Original: The ﬁrst canonical directions are found via
Revised: The ﬁrst canonical directions are found via

Original: [Equation]
Revised: [Equation]

Original: and subsequent projections are found by maximizing the same correlation but in orthogonal directions.
Revised: and subsequent projections are found by maximizing the same correlation but in orthogonal directions.

Original: Combining the projection vectors into matrices W 1 = [ w (1)1 , . .
Revised: Combining the projection vectors into matrices W 1 = [ w (1)1 , . .

Original: , w ( k ) 1 ] and W 2 =
Revised: , w ( k ) 1 ] and W 2 =

Original: [ w (1)2 , . . .
Revised: [ w (1)2 , . . .

Original: , w ( k ) 2 ] ( k ≤ min ( d 1 , d 2 ) ), CCA can be reformulated as a trace maximization under orthonormality constraints on the projections, i.e.,
Revised: , w ( k ) 2 ] ( k ≤ min ( d 1 , d 2 ) ), CCA can be reformulated as a trace maximization under orthonormality constraints on the projections, i.e.,

Original: [Equation]
Revised: [Equation]

Original: for covariance matrices Σ 1 = X 1 X T 1 , Σ 2 = X 2 X T 2 , and cross-covariance matrix Σ 

Revised: for covariance matrices Σ 1 = X 1 X T 1 , Σ 2 = X 2 X T 2 , and cross-covariance matrix Σ

Original: = X 1 X T 2 .
Revised: = X 1 X T 2 .

Original: Let T =
Revised: Let T =

Original: Σ − 1 / 2Σ 12 Σ − 1 / 2and its singular value decomposition (SVD) be T
Revised: Σ − 1 / 2Σ 12 Σ − 1 / 2and its singular value decomposition (SVD) be T

Original: = U 1 diag ( σ ) U (cid:62)with singular values σ = [ σ 1 , . . .
Revised: = U 1 diag ( σ ) U (cid:62)with singular values σ = [ σ 1 , . . .

Original: , σ min ( d 1 ,d 2 ) ] in descending order.
Revised: , σ min ( d 1 ,d 2 ) ] in descending order.

Original: W 1 and W 2 are computed from the top k singular vectors of T as W 1 = Σ − 1 / 

Revised: W 1 and W 2 are computed from the top k singular vectors of T as W 1 = Σ − 1 /

Original: U (1: k )and W 2 = Σ − 1 / 

Revised: U (1: k )and W 2 = Σ − 1 /

Original: U (1: k )where U (1: k ) denotes the k ﬁrst columns of matrix U .
Revised: U (1: k )where U (1: k ) denotes the k ﬁrst columns of matrix U .

Original: The sum correlation in the projection space is equivalent tok
Revised: The sum correlation in the projection space is equivalent to

Original: [Equation]
Revised: [Equation]

Original: i.e., the sum of the top k singular values.
Revised: i.e., the sum of the top k singular values.

Original: A regularized variation of CCA (RCCA) ensures that the covariance matrices are positive deﬁnite by computing the covariance matrices as ˆ Σ 1 =n − 1 X 1 X (cid:62) 1 + r I
Revised: A regularized variation of CCA (RCCA) ensures that the covariance matrices are positive deﬁnite by computing the covariance matrices as ˆ Σ 1 =n − 1 X 1 X (cid:62) 1 + r I

Original: and ˆ Σ 2 = 1 n − 1 X 2 X (cid:62) 

Revised: and ˆ Σ 2 = 1 n − 1 X 2 X (cid:62)

Original: + r I , for regularization parameter r > 0 and identity matrix
Revised: + r I , for regularization parameter r > 0 and identity matrix

Original: I (Bilenko & Gallant, 2016).
Revised: I (Bilenko & Gallant, 2016).

Original: DCCA .
Revised: DCCA .

Original: Deep CCA adds non-linear projections to CCA by non-linearly mapping the input via a multilayer perceptron (MLP).
Revised: Deep CCA adds non-linear projections to CCA by non-linearly mapping the input via a multilayer perceptron (MLP).

Original: In particular, inputs X 1 and X 2 are mapped via non-linear functions fand f 2 , parameterized by θ 1 and θ 2 , resulting in activations A 1 = f 1 ( X 1 ; θ 1 ) and A 2 = f 2 ( X 2 ; θ 2 ) (assumed to be mean centered)
Revised: In particular, inputs X 1 and X 2 are mapped via non-linear functions fand f 2 , parameterized by θ 1 and θ 2 , resulting in activations A 1 = f 1 ( X 1 ; θ 1 ) and A 2 = f 2 ( X 2 ; θ 2 ) (assumed to be mean centered)

Original: (Andrew et al., 2013).
Revised: (Andrew et al., 2013).

Original: When implemented by a NN, A 1 and Aare the output activations of the ﬁnal layer with d o features.
Revised: When implemented by a NN, A 1 and Aare the output activations of the ﬁnal layer with d o features.

Original: Fig.
Revised: Fig.

Original: 1(a) shows the network structure.
Revised: 2(a) shows the network structure.

Original: DCCA optimizes the same objective as CCA (equation 1) but using activations A 1 and A 2 .
Revised: DCCA optimizes the same objective as CCA (equation 1) but using activations A 1 and A 2 .

Original: Regularized covariance matrices are computed accordingly and the solution for W 1 and W 2 can be computed using SVD just as with linear CCA.
Revised: Regularized covariance matrices are computed accordingly and the solution for W 1 and W 2 can be computed using SVD just as with linear CCA.

Original: When k =
Revised: When k =

Original: d o (i.e., the number of CCA components 
Revised: d o (i.e., the number of CCA components is equal to the number of features in A 1 and A 2 ), optimizing the sum correlation in the projection space (equation 2) is equivalent to optimizing the following matrix trace norm objective (TNO) [Equation] where T = Σ − 1 / 2Σ 12 Σ − 1 / 2as in CCA (Andrew et al., 2013). DCCA optimizes this objective directly, without a need to compute the CCA projection within the network.

Original: MLP
Revised: The TNO is optimized ﬁrst, followed by a linear CCA operation before downstream tasks like classiﬁcation are performed. This formulation does not allow for combining directly with a supervised term. SoftCCA. While DCCA enforces orthogonality constraints on projections W (cid:62) 1 A 1 and W (cid:62) 2 A 2 ,

Revised: SoftCCA relaxes them using regularization (Chang et al., 2018).

Revised: Final projection matrices Wand W 2 are integrated into f 1 and f 2 as the top network layer.

Revised: The trace objective for DCCA in equation 1 can be rewritten as minimizing the (cid:96) 2 distance between the projections when each feature in A 1 and A 2 is normalized to a unit variance (Li et al., 2003), leading to 1 L (cid:96) 2 dist ( A 1 , A 2 )

Revised: = (cid:107) A 1 −

Revised: A 2 (cid:107) 2 F .

Revised: Regularization in SoftCCA penalizes the off-diagonal elements of the covariance matrix

Original: MLP
Revised: MLP

Revised: MLP

Original: TNO
Revised: TNO

Original: X
Revised: X

Original: Xf 2 f
Revised: Xf 2 f

Original: A 2 A
Revised: A 2 A

Original: MLP
Revised: MLP

Original: MLP
Revised: MLP

Original: X
Revised: X

Original: Xf 2 f
Revised: Xf 2 f

Original: A 2 A
Revised: A 2 A

Original: Decorr.
Revised: Decorr.

Original: Decorr.
Revised: Decorr.

Original: MLP MLP
Revised: MLP MLP

Original: X
Revised: X

Original: Xf 2 f
Revised: Xf 2 f

Original: A 2 A  Whitening
Revised: A 2 A 1 Whitening

Original: Whitening (cid:96) 2 Loss
Revised: Whitening (cid:96) 2 Loss

Original: MLP
Revised: MLP

Original: Task loss
Revised: Task loss

Original: B 2 B
Revised: B 2 B

Original: Y
Revised: Y

Original: MLP MLP
Revised: MLP MLP

Original: X
Revised: X

Original: Xf 2 f
Revised: Xf 2 f

Original: MLP
Revised: MLP

Original: Task loss
Revised: Task loss

Original: Y
Revised: Y

Original: Decorr.
Revised: Decorr.

Original: Decorr. f task f task (cid:96) 2 Loss
Revised: Decorr. f task f task (cid:96) 2 Loss

Original: A A
Revised: A A

Original: Improve discriminability 1 Max.
Revised: Improve discriminability 1 Max.

Original: sum correlation (equiv.
Revised: sum correlation (equiv.

Original: to l2 loss) 2 Such that projections are orthogonal
Revised: to l

Original: [Figure]
Revised: [Figure]

Original: is equal to the number of features in A 1 and A 2 ), optimizing the sum correlation in the projection space (equation 2) is equivalent to optimizing the following matrix trace norm objective (TNO) [Equation] where T = Σ − 1 / 2Σ 12 Σ − 1 / 2as in CCA (Andrew et al., 2013).
Revised: ) 2 Such that projections are orthogonal

Original: DCCA optimizes this objective directly, without a need to compute the CCA projection within the network. The TNO is optimized ﬁrst, followed by a linear CCA operation before downstream tasks like classiﬁcation are performed. This formulation does not allow for combining directly with a supervised term. SoftCCA.
Revised: [Figure]

Original: While DCCA enforces orthogonality constraints on projections W (cid:62) 1 A 1 and W (cid:62) 2 A 2 ,

Original: SoftCCA relaxes them using regularization (Chang et al., 2018).

Original: Final projection matrices Wand W 2 are integrated into f 1 and f 2 as the top network layer.

Original: The trace objective for DCCA in equation 1 can be rewritten as minimizing the (cid:96) 2 distance between the projections when each feature in A 1 and A 2 is normalized to a unit variance (Li et al., 2003), leading to 1 L (cid:96) 2 dist ( A 1 , A 2 )

Original: = (cid:107) A 1 − A 2 (cid:107) 2 F . Regularization in SoftCCA penalizes the off-diagonal elements of the covariance matrix Σ , using a running average computed over batches as ˆ Σ and a loss of L Decorr ( A ) =
Revised: Σ , using a running average computed over batches as ˆ Σ and a loss of L Decorr ( A ) =

Original: (cid:80) d
Revised: (cid:80) d

Original: o i (cid:54) =
Revised: o i (cid:54) =

Original: i | ˆ Σ i,j | .
Revised: i | ˆΣ i,j | .

Original: Overall, the SoftCCA loss takes the form
Revised: Overall, the SoftCCA loss takes the form

Original: [Equation]
Revised: [Equation]

Revised: L Decorr ( A 2 ) (cid:1) .

Original: Supervised CCA methods.
Revised: Supervised CCA methods.

Original: CCA, DCCA, and SoftCCA are all unsupervised methods to learn a projection to a shared space in which the data is maximally correlated.
Revised: CCA, DCCA, and SoftCCA are all unsupervised methods to learn a projection to a shared space in which the data is maximally correlated.

Original: Although these methods have shown utility for discriminative tasks, a CCA decomposition may not be optimal for classiﬁcation because features that are correlated may not be discriminative.
Revised: Although these methods have shown utility for discriminative tasks, a CCA decomposition may not be optimal for classiﬁcation because features that are correlated may not be discriminative.

Original: Our experiments will show that maximizing the correlation objective too much can degrade performance on discriminative tasks.
Revised: Our experiments will show that maximizing the correlation objective too much can degrade performance on discriminative tasks.

Original: CCA has previously been extended to supervised settings in three ways: 1) with methods that are linear only (Singanamalli et al., 2014; Lee et al., 2015; Kan et al., 2015; Duan et al., 2016), 2) by maximizing the total correlation between each view and the training labels in addition to each pair of views (Lee et al., 2015; Singanamalli et al., 2014), and 3) with Linear Discriminant Analysis (LDA)-style approaches to encourage class separation (Kan et al., 2015; Dorfer et al., 2016b; El- madany et al., 2016).
Revised: CCA has previously been extended to supervised settings in three ways: 1) with methods that are linear only (Singanamalli et al., 2014; Lee et al., 2015; Kan et al., 2015; Duan et al., 2016), 2) by maximizing the total correlation between each view and the training labels in addition to each pair of views (Lee et al., 2015; Singanamalli et al., 2014), and 3) with Linear Discriminant Analysis (LDA)-style approaches to encourage class separation (Kan et al., 2015; Dorfer et al., 2016b; Elmadany et al., 2016).

Original: 2 LDA approaches to supervision are generative rather than discriminative.
Revised: 2 LDA approaches to supervision are generative rather than discriminative.

Original: Importantly, we will show in § 4.3 that encouraging class separation with an LDA-style objective performs signiﬁcantly inferior to a softmax.
Revised: Importantly, we will show in § 5.3 that encouraging class separation with an LDA-style objective performs signiﬁcantly inferior to a softmax.

Original: Further, Dorfer et al.
Revised: Further, Dorfer et al.

Original: (2016b) did not apply LDA to the shared space itself but to the NN layer below it, and Elmadany et al.
Revised: (2016b) did not apply LDA to the shared space itself but to the NN layer below it, and Elmadany et al.

Original: (2016) did not validate the shared space created, only its use in multi-view classiﬁcation using both views for training and test.
Revised: (2016) did not validate the shared space created, only its use in multi-view classiﬁcation using both views for training and test.

Original: Dorfer et.
Revised: Dorfer et.

Original: al’s CCA Layer (CCAL) is the closest to our method.
Revised: al’s CCA Layer (CCAL) is the closest to our method.

Original: It optimizes a task loss operating on a CCA projection; however, the CCA objective itself is only optimized during pre-training, not in an end-to-end manner (Dorfer et al., 2018).
Revised: It optimizes a task loss operating on a CCA projection; however, the CCA objective itself is only optimized during pre-training, not in an end-to-end manner (Dorfer et al., 2018).

Original: Further, their goal is retrieval with a pairwise rank loss, not classiﬁcation.
Revised: Further, their goal is retrieval with a pairwise rank loss, not classiﬁcation.

Original: Instead of computing the CCA projection explicitly within the network, we optimize the non-linear mapping into the shared space together with the task objective, requiring a fundamental change in formulation.
Revised: Instead of computing the CCA projection explicitly within the network, we optimize the non-linear mapping into the shared space together with the task objective, requiring a fundamental change in formulation.

Original: We optimize for the shared space with the (cid:96) 2 distance between activations (similar to SoftCCA) and propose three different ways to apply the orthogonality constraints of CCA.
Revised: We optimize for the shared space with the (cid:96) 2 distance between activations (similar to SoftCCA) and propose three different ways to apply the orthogonality constraints of CCA.

Original:  T ASK -O PTIMAL CCA (TOCCA)
Revised:  T ASK -O PTIMAL CCA (TOCCA)

Original: To compute a shared latent space that is also discriminative, we reformulate DCCA to add a taskdriven term to the optimization objective.
Revised: To compute a shared latent space that is also discriminative, we reformulate DCCA to add a taskdriven term to the optimization objective.

Original: The CCA component ﬁnds features that are correlated between views, while the task component ensures that they are also discriminative.
Revised: The CCA component ﬁnds features that are correlated between views, while the task component ensures that they are also discriminative.

Original: This model can be used for representation learning on multiple views before joining representations for prediction (Sargin et al., 2007; Dorfer et al., 2016b) and for classiﬁcation when two views are available for training but only one at test time (Arora & Livescu, 2012).
Revised: This model can be used for representation learning on multiple views before joining representations for prediction (Sargin et al., 2007; Dorfer et al., 2016b) and for classiﬁcation when two views are available for training but only one at test time (Arora & Livescu, 2012).

Original: In § 4, we demonstrate both use cases on real data.
Revised: In § 5, we demonstrate both use cases on real data.

Original: Our methods and related NN models from the literature are summarized in Tab.
Revised: Our methods and related NN models from the literature are summarized in Tab.

Original: A2; Fig.shows schematic diagrams.
Revised: A2; Fig.shows schematic diagrams.

Original: Challenges and solutions.
Revised: Challenges and solutions.

Original: While DCCA optimizes the sum correlation with an equivalent loss function (TNO), the CCA projection itself is computed only after optimization.
Revised: While DCCA optimizes the sum correlation with an equivalent loss function (TNO), the CCA projection itself is computed only after optimization.

Original: Hence, the projections cannot be used to optimize another task simultaneously.
Revised: Hence, the projections cannot be used to optimize another task simultaneously.

Original: The main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network – a necessary step to enable simultaneous training of both objectives.
Revised: The main challenge in developing a task-optimal form of deep CCA that discriminates based on the CCA projection is in computing this projection within the network – a necessary step to enable simultaneous training of both objectives.

Original: We tackle this by focusing on the two components of DCCA: maximizing the sum correlation between activations A 1 and A 2 and enforcing orthonormality constraints within A 1 and A 2 .
Revised: We tackle this by focusing on the two components of DCCA: maximizing the sum correlation between activations A 1 and A 2 and enforcing orthonormality constraints within A 1 and A 2 .

Original: We achieve both by transforming the CCA objective and present three methods that progressively relax the orthogonality constraints.
Revised: We achieve both by transforming the CCA objective and present three methods that progressively relax the orthogonality constraints.

Original: We further improve upon DCCA by enabling mini-batch computations for improved ﬂexibility and test performance.
Revised: We further improve upon DCCA by enabling mini-batch computations for improved ﬂexibility and test performance.

Original: DCCA was developed for large batches because correlation is not separable across batches.
Revised: DCCA was developed for large batches because correlation is not separable across batches.

Original: While large batch implementations of stochastic gradient optimization can increase computational efﬁciency via parallelism, small batch training provides more up-to-date gradient calculations, allowing a wider range of learning rates and improving test accuracy (Masters & Luschi, 2018).
Revised: While large batch implementations of stochastic gradient optimization can increase computational efﬁciency via parallelism, small batch training provides more up-to-date gradient calculations, allowing a wider range of learning rates and improving test accuracy (Masters & Luschi, 2018).

Original: We reformulate the correlation objective as the (cid:96) 2 distance (following SoftCCA), enabling separability across batches.
Revised: We reformulate the correlation objective as the (cid:96) 2 distance (following SoftCCA), enabling separability across batches.

Original: We ensure a normalization to one via batch normalization without the scale and shift parameters (Ioffe & Szegedy, 2015).
Revised: We ensure a normalization to one via batch normalization without the scale and shift parameters (Ioffe & Szegedy, 2015).

Original: Wang et al.
Revised: Wang et al.

Original: (2016) also developed a stochastic mini-batch solution to DCCA but handled the orthonormality constraints in a different way (discussed below).
Revised: (2016) also developed a stochastic mini-batch solution to DCCA but handled the orthonormality constraints in a different way (discussed below).

Original: Task-driven objective.
Revised: Task-driven objective.

Original: First, we apply non-linear functions f 1 and f 2 with parameters θ (via MLPs) to each view X 1 and X 2 , i.e., A 1 = f 1 ( X 1 ; θ 1 ) and A 2 = f 2 ( X 2 ; θ 2 ) .
Revised: First, we apply non-linear functions f 1 and f 2 with parameters θ (via MLPs) to each view X 1 and X 2 , i.e., A 1 = f 1 ( X 1 ; θ 1 ) and A 2 = f 2 ( X 2 ; θ 2 ) .

Original: Second, a task-speciﬁc function f task (
Revised: Second, a task-speciﬁc function f task (

Original: A ; θ task ) operates on the outputs A 1 and A 2 .
Revised: A ; θ task ) operates on the outputs A 1 and A 2 .

Original: In particular, f 1 and f 2 are optimized so that the (cid:96) 2 distance between A 1 and A 2 is minimized; therefore, f task can be trained to operate on both inputs A 1 and A 2 .
Revised: In particular, f 1 and f 2 are optimized so that the (cid:96) 2 distance between A 1 and A 2 is minimized; therefore, f task can be trained to operate on both inputs A 1 and A 2 .

Original: We combine CCA and task-driven objectives as a weighted sum with a hyperparameter for tuning.
Revised: We combine CCA and task-driven objectives as a weighted sum with a hyperparameter for tuning.

Original: This model is ﬂexible, in that the task-driven goal can be used for classiﬁcation (Krizhevsky et al., 2012; Dorfer et al., 2016a), regression (Katzman et al., 2016), clustering (Caron et al., 2018), or any other task.
Revised: This model is ﬂexible, in that the task-driven goal can be used for classiﬁcation (Krizhevsky et al., 2012; Dorfer et al., 2016a), regression (Katzman et al., 2016), clustering (Caron et al., 2018), or any other task.

Original: Other prior attempts to integrate a classiﬁer into deep CCA only used LDA (Kan et al., 2015; Dorfer et al., 2016b; Elmadany et al., 2016).
Revised: Other prior attempts to integrate a classiﬁer into deep CCA only used LDA (Kan et al., 2015; Dorfer et al., 2016b; Elmadany et al., 2016).

Original: See
Revised: See

Original: Tab. A2 for an overview.
Revised: Tab. A2 for an overview.

Original: Orthogonality constraints
Revised: Orthogonality constraints.

Original: The remaining complications for mini-batch optimization are the orthogonality constraints, for which we propose three solutions, each handling the orthogonality constraints of CCA in a different way: whitening, soft decorrelation, and no decorrelation.
Revised: The remaining complications for mini-batch optimization are the orthogonality constraints, for which we propose three solutions, each handling the orthogonality constraints of CCA in a different way: whitening, soft decorrelation, and no decorrelation.

Original: 1) Whitening ( TOCCA-W ).
Revised: 1) Whitening ( TOCCA-W ).

Original: CCA applies orthogonality constraints to A 1 and A 2 .
Revised: CCA applies orthogonality constraints to A 1 and A 2 .

Original: We accomplish this with a linear whitening transformation that transforms the activations such that their covariance becomes the identity matrix, i.e., features are uncorrelated.
Revised: We accomplish this with a linear whitening transformation that transforms the activations such that their covariance becomes the identity matrix, i.e., features are uncorrelated.

Original: Decorrelated Batch Normalization (DBN) has previously been used to regularize deep models by decorrelating features (Huang et al., 2018) and inspired our solution.
Revised: Decorrelated Batch Normalization (DBN) has previously been used to regularize deep models by decorrelating features (Huang et al., 2018) and inspired our solution.

Original: In particular, we apply a transformation B = UA to make B orthonormal, i.e., BB (cid:62)
Revised: In particular, we apply a transformation B = UA to make B orthonormal, i.e., BB (cid:62)

Original: =
Revised: =

Original: I .
Revised: I .

Original: We use a Zero-phase Component Analysis (ZCA) whitening transform composed of three steps: rotate the data to decorrelate it, rescale each axis, and rotate back to the original space.
Revised: We use a Zero-phase Component Analysis (ZCA) whitening transform composed of three steps: rotate the data to decorrelate it, rescale each axis, and rotate back to the original space.

Original: Each transformation is learned from the data.
Revised: Each transformation is learned from the data.

Original: Any matrix U (cid:15) R
Revised: Any matrix U (cid:15) R

Original: d o × d o satisfying U (cid:62) U
Revised: d o × d o satisfying U (cid:62) U

Original: = Σ − 1 whitens the data, where Σ denotes the covariance matrix of A .
Revised: = Σ − 1 whitens the data, where Σ denotes the covariance matrix of A .

Original: As U is only deﬁned up to a rotation, it is not unique.
Revised: As U is only deﬁned up to a rotation, it is not unique.

Original: PCA whitening follows the ﬁrst two steps and uses the eigendecomposition of Σ :
Revised: PCA whitening follows the ﬁrst two steps and uses the eigendecomposition of Σ :

Original: U PCA =
Revised: U PCA =

Original: Λ − 1 / 2 V (cid:62) for Λ =
Revised: Λ − 1 / 2 V (cid:62) for Λ =

Original: diag
Revised: diag

Original: ( λ 1 , . . .
Revised: ( λ 1 , . . .

Original: , λ d o ) and V =
Revised: , λ d o ) and V =

Original: [ v 1 , . .
Revised: [ v 1 , . .

Original: . , v d o ] , where ( λ i , v i ) are the eigenvalue, eigenvector pairs of Σ .
Revised: . , v d o ] , where ( λ i , v i ) are the eigenvalue, eigenvector pairs of Σ .

Original: As PCA whitening suffers from stochastic axis swapping, neurons are not stable between batches (Huang et al., 2018).
Revised: As PCA whitening suffers from stochastic axis swapping, neurons are not stable between batches (Huang et al., 2018).

Original: ZCA whitening uses the transformation
Revised: ZCA whitening uses the transformation

Original: U ZCA = VΛ − 1 / 2 V T in which PCA whitening is ﬁrst applied, followed by a rotation back to the original space.
Revised: U ZCA = VΛ − 1 / 2 V T in which PCA whitening is ﬁrst applied, followed by a rotation back to the original space.

Original: Adding the rotation V brings the whitened data B as close as possible to the original data A (Kessy et al., 2015).
Revised: Adding the rotation V brings the whitened data B as close as possible to the original data A (Kessy et al., 2015).

Original: Computation of U ZCA is clearly depend on Σ .
Revised: Computation of U ZCA is clearly depend on Σ .

Original: While Huang et al.
Revised: While Huang et al.

Original: (2018) used a running average of U ZCA over batches, we apply this stochastic approximation to Σ for each view using the update Σ ( k ) =
Revised: (2018) used a running average of U ZCA over batches, we apply this stochastic approximation to Σ for each view using the update Σ ( k ) =

Original: α Σ ( k − 1) +(1 − α ) Σ b for batch k where Σ b is the covariance matrix for the current batch and α ∈ (0 , 1) is the momentum.
Revised: α Σ ( k − 1) +(1 − α ) Σ b for batch k where Σ b is the covariance matrix for the current batch and α ∈ (0 , 1) is the momentum.

Original: We then compute the ZCA transformation from Σ ( k ) to do whitening as B = f ZCA ( A ) =
Revised: We then compute the ZCA transformation from Σ ( k ) to do whitening as B = f ZCA ( A ) =

Original: U ( k ) ZCA A .
Revised: U ( k ) ZCA A .

Original: At test time, U ( k ) from the last training batch is used.
Revised: At test time, U ( k ) from the last training batch is used.

Original: Algorithm A describes ZCA whitening in greater detail.
Revised: Algorithm A describes ZCA whitening in greater detail.

Original: In summary, TOCCA-W integrates both the correlation and task-driven objectives, with decorrelation performed by whitening, into
Revised: In summary, TOCCA-W integrates both the correlation and task-driven objectives, with decorrelation performed by whitening, into

Original: [Equation]
Revised: [Equation]

Original: where B 1 and B 2 are whitened outputs of A 1 and A 2 , respectively, and Y is the class labels.
Revised: where B 1 and B 2 are whitened outputs of A 1 and A 2 , respectively, and Y is the class labels.

Original: This is a novel approach to integrating the orthogonality constraints of CCA into a NN as it is the ﬁrst to use ZCA whitening in this manner.
Revised: This is a novel approach to integrating the orthogonality constraints of CCA into a NN as it is the ﬁrst to use ZCA whitening in this manner.

Original: Wang et al.
Revised: Wang et al.

Original: (2016)’s stochastic mini-batch solution to DCCA used nonlinear orthogonal iterations and does not state what type of whitening operation was used.
Revised: (2016)’s stochastic mini-batch solution to DCCA used nonlinear orthogonal iterations and does not state what type of whitening operation was used.

Original: 2) Soft decorrelation ( TOCCA-SD ).
Revised: 2) Soft decorrelation ( TOCCA-SD ).

Original: While fully independent components may be beneﬁcial in regularizing NNs on some data sets, a softer decorrelation may be more suitable on others.
Revised: While fully independent components may be beneﬁcial in regularizing NNs on some data sets, a softer decorrelation may be more suitable on others.

Original: In this second formulation we relax the orthogonality constraints using regularization, following the Decorr loss of SoftCCA (Chang et al., 2018).
Revised: In this second formulation we relax the orthogonality constraints using regularization, following the Decorr loss of SoftCCA (Chang et al., 2018).

Original: The loss function for this formulation is
Revised: The loss function for this formulation is

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: 2 ) (cid:1) . While this solution is based on SoftCCA, our experiments ( § 4) will demonstrate that the task component is essential when using the model for classiﬁcation.
Revised: While this solution is based on SoftCCA, our experiments ( § 5) will demonstrate that the task component is essential when using the model for classiﬁcation.

Original: 3) No decorrelation ( TOCCA-ND ).
Revised: 3) No decorrelation ( TOCCA-ND ).

Original: When CCA is used in an unsupervised manner, some form of orthogonality constraint or decorrelation is necessary to ensure that f 1 and f 2 do not simply produce multiple copies of the same feature.
Revised: When CCA is used in an unsupervised manner, some form of orthogonality constraint or decorrelation is necessary to ensure that f 1 and f 2 do not simply produce multiple copies of the same feature.

Original: While this result could maximize the sum correlation, it is not helpful in capturing useful projections.
Revised: While this result could maximize the sum correlation, it is not helpful in capturing useful projections.

Original: In the task-driven setting, the discriminative term ensures that the features in f 1 and f 2 are not replicates of the same information.
Revised: In the task-driven setting, the discriminative term ensures that the features in f 1 and f 2 are not replicates of the same information.

Original: TOCCA-ND therefore removes the decorrelation term entirely, forming the simpler objective
Revised: TOCCA-ND therefore removes the decorrelation term entirely, forming the simpler objective

Original: [Equation]
Revised: [Equation]

Original: These three models allow testing whether whitening or decorrelation beneﬁt a task-driven model.
Revised: These three models allow testing whether whitening or decorrelation beneﬁt a task-driven model.

Original: Computational complexity.
Revised: Computational complexity.

Original: Due to the eigendecomposition, TOCCA-W has a complexity of O ( d 3 o ) compared to O
Revised: Due to the eigendecomposition, TOCCA-W has a complexity of O ( d 3 o ) compared to O

Original: ( d 2 o ) for TOCCA-SD , with respect to output dimension d o .
Revised: ( d 2 o ) for TOCCA-SD , with respect to output dimension d o .

Original: However, d o is typically small ( ≤ 100 ) and this extra computation is only performed once per batch.
Revised: However, d o is typically small ( ≤ 100 ) and this extra computation is only performed once per batch.

Original: The difference in runtime is less than 6.5% for a batch size of 100 or 9.4% for a batch size of 30 (Tab. A4).
Revised: The difference in runtime is less than 6.5% for a batch size of 100 or 9.4% for a batch size of 30 (Tab. A4).

Original: Summary.
Revised: Summary.

Original: All three variants are motivated by adding a task-driven component to deep CCA.
Revised: All three variants are motivated by adding a task-driven component to deep CCA.

Original: TOCCA-ND is the most relaxed and directly attempts to obtain identical latent representations.
Revised: TOCCA-ND is the most relaxed and directly attempts to obtain identical latent representations.

Original: Experiments will show that whitening ( TOCCA-W ) and soft decorrelation ( TOCCA-SD ) provide a beneﬁcial regularization.
Revised: Experiments will show that whitening ( TOCCA-W ) and soft decorrelation ( TOCCA-SD ) provide a beneﬁcial regularization.

Original: Further, since the (cid:96) 2 distance that we optimize was shown to be equivalent to the sum correlation (cf.
Revised: Further, since the (cid:96) 2 distance that we optimize was shown to be equivalent to the sum correlation (cf.

Original: § 2 SoftCCA paragraph), all three TOCCA models maintain the goals of
Revised: § 3 SoftCCA paragraph), all three TOCCA models maintain the goals of

Original: CCA, just with different relaxations of the orthogonality constraints.
Revised: CCA, just with different relaxations of the orthogonality constraints.

Original: Our method is the ﬁrst to simultaneously optimize for CCA and a discriminative task with end-to-end training.
Revised: Our method is the ﬁrst to simultaneously optimize for CCA and a discriminative task with end-to-end training.

Original: See Tab.
Revised: See Tab.

Original: A2 for an overview.
Revised: A2 for an overview.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We validated our methods on three different data sets: MNIST handwritten digits, the Carolina
Revised: We validated our methods on three different data sets: MNIST handwritten digits, the Carolina

Original: Breast Cancer Study (CBCS) using imaging and genomic features, and speech data from the Wisconsin X-ray Microbeam Database (XRMB).
Revised: Breast Cancer Study (CBCS) using imaging and genomic features, and speech data from the Wisconsin X-ray Microbeam Database (XRMB).

Original: Our experiments show the utility of our methods for 1) cross-view classiﬁcation, 2) regularization with a second view during training when only one view is available at test time, and 3) representation learning on multiple views that are joined for prediction.
Revised: Our experiments show the utility of our methods for 1) cross-view classiﬁcation, 2) regularization with a second view during training when only one view is available at test time, and 3) representation learning on multiple views that are joined for prediction.

Original: Implementation.
Revised: Implementation.

Original: Each layer of our network consists of a fully connected layer, followed by a
Revised: Each layer of our network consists of a fully connected layer, followed by a

Original: ReLU activation and batch normalization (Ioffe & Szegedy, 2015).
Revised: ReLU activation and batch normalization (Ioffe & Szegedy, 2015).

Original: Our implementations of DCCA,
Revised: Our implementations of DCCA,

Original: SoftCCA, and Joint DCCA/DeepLDA (Dorfer et al., 2016b) also use ReLU activation and batch normalization.
Revised: SoftCCA, and Joint DCCA/DeepLDA (Dorfer et al., 2016b) also use ReLU activation and batch normalization.

Original: We modiﬁed CCAL-
Revised: We modiﬁed CCAL-

Original: L rank (Dorfer et al., 2018) to use a softmax function and crossentropy loss for classiﬁcation, instead of a pairwise ranking loss for retrieval, referring to this modiﬁcation as CCAL- L ce .
Revised: L rank (Dorfer et al., 2018) to use a softmax function and crossentropy loss for classiﬁcation, instead of a pairwise ranking loss for retrieval, referring to this modiﬁcation as CCAL- L ce .

Original: We used the Nadam optimizer and tuned hyperparameters on a validation set via random search; settings and ranges are speciﬁed in Tab. A3.
Revised: We used the Nadam optimizer and tuned hyperparameters on a validation set via random search; settings and ranges are speciﬁed in Tab. A3.

Original: The same hyperparameter tuning procedure was used for our methods and those we compare with.
Revised: The same hyperparameter tuning procedure was used for our methods and those we compare with.

Original: We used Keras with the Theano backend and an Nvidia GeForce GTX 1080 Ti.
Revised: We used Keras with the Theano backend and an Nvidia GeForce GTX 1080 Ti.

Original: The following experiments compare our methods with two linear methods (CCA and RCCA), two unsupervised deep methods (DCCA and SoftCCA), and two supervised deep methods (Joint
Revised: The following experiments compare our methods with two linear methods (CCA and RCCA), two unsupervised deep methods (DCCA and SoftCCA), and two supervised deep methods (Joint

Original: DCCA/DeepLDA and CCAL-
Revised: DCCA/DeepLDA and CCAL-

Original: L ce ).
Revised: L ce ).

Original: Many other variants exist ( § 2), but the ones we selected are the current state-of-the-art in each of these classes.
Revised: Many other variants exist ( § 3), but the ones we selected are the current state-of-the-art in each of these classes.

Original: We did not run a direct comparison with Wang et al.
Revised: We did not run a direct comparison with Wang et al.

Original: (2015a)
Revised: (2015a)

Original: as Chang et al.
Revised: as Chang et al.

Original: (2018) already showed that SoftCCA is superior.
Revised: (2018) already showed that SoftCCA is superior.

Original: We chose Joint
Revised: We chose Joint

Original: DCCA/DeepLDA to represent supervised LDA-style CCA methods rather than comparing with all methods in this group (Kan et al., 2015; Elmadany et al., 2016) 4 .
Revised: DCCA/DeepLDA to represent supervised LDA-style CCA methods rather than comparing with all methods in this group (Kan et al., 2015; Elmadany et al., 2016) 4 .

Original: 4. C ROSS - VIEW CLASSIFICATION ON MNIST DIGITS
Revised: 5. C ROSS - VIEW CLASSIFICATION ON MNIST DIGITS

Original: We formed a multi-view data set from the MNIST handwritten digit data set (LeCun, 1998).
Revised: We formed a multi-view data set from the MNIST handwritten digit data set (LeCun, 1998).

Original: Following Andrew et al.
Revised: Following Andrew et al.

Original: (2013), we split each 28 × 28 image in half horizontally, creating left and right views that are each 14 × 28 pixels.
Revised: (2013), we split each 28 × 28 image in half horizontally, creating left and right views that are each 14 × 28 pixels.

Original: All images were ﬂattened into a vector with 392 features.
Revised: All images were ﬂattened into a vector with 392 features.

Original: The full data set consists of 60k training images and 10k test images.
Revised: The full data set consists of 60k training images and 10k test images.

Original: We used a random set of up to 50k for training and the remaining training images for validation.
Revised: We used a random set of up to 50k for training and the remaining training images for validation.

Original: We used the full 10k image test set.
Revised: We used the full 10k image test set.

Original: In order to validate both the discriminativeness of the embedding and the success in ﬁnding a shared space, we studied performance on cross-view classiﬁcation.
Revised: In order to validate both the discriminativeness of the embedding and the success in ﬁnding a shared space, we studied performance on cross-view classiﬁcation.

Original: We evaluated cross-view classiﬁcation accuracy by ﬁrst computing the projection for each view, then we trained a linear SVM on one view’s projection, and ﬁnally we used the other view’s projection at test time.
Revised: We evaluated cross-view classiﬁcation accuracy by ﬁrst computing the projection for each view, then we trained a linear SVM on one view’s projection, and ﬁnally we used the other view’s projection at test time.

Original: While the task-driven methods presented in this work learn a classiﬁer within the model, this test setup enables a fair comparison with the unsupervised CCA variants and validates the discriminativity of the features learned.
Revised: While the task-driven methods presented in this work learn a classiﬁer within the model, this test setup enables a fair comparison with the unsupervised CCA variants and validates the discriminativity of the features learned.

Original: It is also the standard method in the literature to test CCA methods for classiﬁcation.
Revised: It is also the standard method in the literature to test CCA methods for classiﬁcation.

Original: Notably, using the built-in softmax classiﬁer (not shown) performed similarly to the SVM, as much of the power of our methods comes from the representation learning part.
Revised: Notably, using the built-in softmax classiﬁer (not shown) performed similarly to the SVM, as much of the power of our methods comes from the representation learning part.

Original: We do not compare with a simple supervised NN because this setup does not learn the shared space necessary for cross-view classiﬁcation.
Revised: We do not compare with a simple supervised NN because this setup does not learn the shared space necessary for cross-view classiﬁcation.

Original: We report results averaged over ﬁve randomly selected training/validation sets; the test set always remained the same.[Figure]
Revised: We report results averaged over ﬁve randomly selected training/validation sets; the test set always remained the same.

Original: Correlation vs. classiﬁcation accuracy
Revised: Correlation vs. classiﬁcation accuracy

Original: We ﬁrst demonstrate the importance of adding a taskdriven component to DCCA by showing that maximizing the sum correlation between views is not sufﬁcient.
Revised: We ﬁrst demonstrate the importance of adding a taskdriven component to DCCA by showing that maximizing the sum correlation between views is not sufﬁcient.

Original: Fig.
Revised: Fig.

Original: 2 ( left ) shows the sum correlation vs. cross-view classiﬁcation accuracy across many different hyperparameter settings for DCCA (Andrew et al., 2013), SoftCCA (Chang et al., 2018), and TOCCA .
Revised: 3 ( left ) shows the sum correlation vs. cross-view classiﬁcation accuracy across many different hyperparameter settings for DCCA (Andrew et al., 2013), SoftCCA (Chang et al., 2018), and TOCCA .

Original: We used 50 components for each; thus, the maximum sum correlation was 50.
Revised: We used 50 components for each; thus, the maximum sum correlation was 50.

Original: The sum correlation was measured after applying linear CCA to ensure that components were independent.
Revised: The sum correlation was measured after applying linear CCA to ensure that components were independent.

Original: With DCCA a larger correlation tended to produce a larger classiﬁcation accuracy, but there was still a large variance in classiﬁcation accuracy amongst hyperparameter settings that produced a similar sum correlation.
Revised: With DCCA a larger correlation tended to produce a larger classiﬁcation accuracy, but there was still a large variance in classiﬁcation accuracy amongst hyperparameter settings that produced a similar sum correlation.

Original: For example, with the two farthest right points in the plot (colored red), their classiﬁcation accuracy differs by 10%, and they are not even the points with the best classiﬁcation accuracy (colored purple).
Revised: For example, with the two farthest right points in the plot (colored red), their classiﬁcation accuracy differs by 10%, and they are not even the points with the best classiﬁcation accuracy (colored purple).

Original: The pattern is different for SoftCCA.
Revised: The pattern is different for SoftCCA.

Original: There was an increase in classiﬁcation accuracy as sum correlation increased but only up to a point.
Revised: There was an increase in classiﬁcation accuracy as sum correlation increased but only up to a point.

Original: For higher sum correlations, the classiﬁcation accuracy varied even more from 20% to 80%.
Revised: For higher sum correlations, the classiﬁcation accuracy varied even more from 20% to 80%.

Original: Further experiments (not shown) have indicated that when the sole objective is correlation, some of the projection directions are simply not discriminative, particularly when there are a large number of classes.
Revised: Further experiments (not shown) have indicated that when the sole objective is correlation, some of the projection directions are simply not discriminative, particularly when there are a large number of classes.

Original: Hence, optimizing for sum correlation alone does not guarantee a discriminative model.
Revised: Hence, optimizing for sum correlation alone does not guarantee a discriminative model.

Original: TOCCA-W and TOCCA-SD show a much greater classiﬁcation accuracy across a wide range of correlations and, overall, the best accuracy when correlation is greatest.
Revised: TOCCA-W and TOCCA-SD show a much greater classiﬁcation accuracy across a wide range of correlations and, overall, the best accuracy when correlation is greatest.

Original: Effect of batch size.
Revised: Effect of batch size.

Original: Fig.
Revised: Fig.

Original: 2 ( right ) plots the batch size vs. classiﬁcation accuracy for a training set size of 10 , 000 .
Revised: 3 ( right ) plots the batch size vs. classiﬁcation accuracy for a training set size of 10 , 000 .

Original: We tested batch sizes from 10 to 10 , 000 ; a batch size of 10 or 30 was best for all  three variations of TOCCA .
Revised: We tested batch sizes from 10 to 10 , 000 ; a batch size of 10 or 30 was best for all [Figure] [Figure] three variations of TOCCA .

Original: This is in line with previous work that found the best performance with a batch size between 2 and 32 (Masters & Luschi, 2018).
Revised: This is in line with previous work that found the best performance with a batch size between 2 and 32 (Masters & Luschi, 2018).

Original: We used a batch size of 32 in the remaining experiments on MNIST.
Revised: We used a batch size of 32 in the remaining experiments on MNIST.

Original: Effect of training set size.
Revised: Effect of training set size.

Original: We manipulated the training set size in order to study the robustness of our methods.
Revised: We manipulated the training set size in order to study the robustness of our methods.

Original: In particular, Fig.
Revised: In particular, Fig.

Original: 2 ( right ) shows the cross-view classiﬁcation accuracy for training set sizes from n = 300 to 50 , 000 .
Revised: 3 ( right ) shows the cross-view classiﬁcation accuracy for training set sizes from n = 300 to 50 , 000 .

Original: While we expected that performance would decrease for smaller training set sizes, some methods were more susceptible to this degradation than others.
Revised: While we expected that performance would decrease for smaller training set sizes, some methods were more susceptible to this degradation than others.

Original: The classiﬁcation accuracy with CCA dropped signiﬁcantly for n = 300 and 1 , 000 , due to overﬁtting and instability issues related to the covariance and cross-covariance matrices.
Revised: The classiﬁcation accuracy with CCA dropped signiﬁcantly for n = 300 and 1 , 000 , due to overﬁtting and instability issues related to the covariance and cross-covariance matrices.

Original: SoftCCA shows similar behavior (prior work (Chang et al., 2018) on this method did not test such small training set sizes).
Revised: SoftCCA shows similar behavior (prior work (Chang et al., 2018) on this method did not test such small training set sizes).

Original: Across all training set sizes, our TOCCA variations consistently exhibited good performance, e.g., increasing classiﬁcation accuracy from 78.3% to 86.7% for n = 1 , 000 and from 86.1% to 94.6% for n = 50 , 000 with TOCCA-SD .
Revised: Across all training set sizes, our TOCCA variations consistently exhibited good performance, e.g., increasing classiﬁcation accuracy from 78.3% to 86.7% for n = 1 , 000 and from 86.1% to 94.6% for n = 50 , 000 with TOCCA-SD .

Original: Increases in accuracy over TOCCA-ND were small, indicating that the[Figure] [Table] different decorrelation schemes have only a small effect on this data set; the task-driven component is the main reason for the success of our method.
Revised: Increases in accuracy over TOCCA-ND were small, indicating that the different decorrelation schemes have only a small effect on this data set; the task-driven component is the main reason for the success of our method.

Original: In particular, the classiﬁcation accuracy with n = 1 , 000 did better than the unsupervised DCCA method on n = 10 , 000 .
Revised: In particular, the classiﬁcation accuracy with n = 1 , 000 did better than the unsupervised DCCA method on n = 10 , 000 .

Original: Further, TOCCA with n = 300 did better than linear methods on n = 50 , 000 , clearly showing the beneﬁts of the proposed formulation.
Revised: Further, TOCCA with n = 300 did better than linear methods on n = 50 , 000 , clearly showing the beneﬁts of the proposed formulation.

Original: We also examined the CCA projections qualitatively via a 2D t -SNE embedding (Van
Revised: We also examined the CCA projections qualitatively via a 2D t -SNE embedding (Van

Original: Der Maaten & Hinton, 2008).
Revised: Der Maaten & Hinton, 2008).

Original: Fig.
Revised: Fig.

Original: 3 shows the CCA projection of the left view for each method.
Revised: 4 shows the CCA projection of the left view for each method.

Original: As expected, the task-driven variant produced more clearly separated classes. 
Revised: As expected, the task-driven variant produced more clearly separated classes. [Table]

Original: 4. R EGULARIZATION FOR CANCER CLASSIFICATION
Revised: 5. R EGULARIZATION FOR CANCER CLASSIFICATION

Original: In this experiment, we address the following question: Given two views available for training but only one at test time, does the additional view help to regularize the model?
Revised: In this experiment [Table] we address the following question: Given two views available for training but only one at test time, does the additional view help to regularize the model?

Original: We study this question using 1,003 patient samples with image and genomic data from CBCS(Troester et al., 2018).
Revised: We study this question using 1,003 patient samples with image and genomic data from CBCS(Troester et al., 2018).

Original: Images consisted of four cores per patient from a tissue microarray that was stained with hematoxylin and eosin.
Revised: Images consisted of four cores per patient from a tissue microarray that was stained with hematoxylin and eosin.

Original: Image features were extracted using a VGG16 backbone (Simonyan & Zisserman, 2015), pre-trained on ImageNet, by taking the mean of the 512D output of the fourth set of conv.
Revised: Image features were extracted using a VGG16 backbone (Simonyan & Zisserman, 2015), pre-trained on ImageNet, by taking the mean of the 512D output of the fourth set of conv.

Original: layers across the tissue region and further averaging across all core images for the same patient.
Revised: layers across the tissue region and further averaging across all core images for the same patient.

Original: For gene expression (GE), we used the set of 50 genes in the PAM50 array (Parker et al., 2009).
Revised: For gene expression (GE), we used the set of 50 genes in the PAM50 array (Parker et al., 2009).

Original: The data set was randomly split into half for training and one quarter for validation/testing; we report the mean over eight cross-validation runs.
Revised: The data set was randomly split into half for training and one quarter for validation/testing; we report the mean over eight cross-validation runs.

Original: Classiﬁcation tasks included predicting 1) Basal vs. non-Basal genomic subtype using images, which is typically done from GE, and 2) predicting grade 1 vs. 3 from GE, typically done from images.
Revised: Classiﬁcation tasks included predicting 1) Basal vs. non-Basal genomic subtype using images, which is typically done from GE, and 2) predicting grade 1 vs. 3 from GE, typically done from images.

Original: This is not a multi-task classiﬁcation setup; it is a means for one view to stabilize the representation of the other.
Revised: This is not a multi-task classiﬁcation setup; it is a means for one view to stabilize the representation of the other.

Original: The ﬁrst task is also a valuable clinical use case in which expensive and time-consuming genomic analysiscan help regularize during training even if it is  not available for test patients.
Revised: The ﬁrst task is also a valuable clinical use case. Genomic analysis is expensive and not routinely performed, while histologic imaging is standard practice by pathologists for detecting cancer and assessing its aggressiveness.

Revised: In working with our clinical collaborators, our goal has been to predict tumor subtypes from images - something that is too complex for pathologists.

Revised: We hope that this will one day make tumor subtypes accessible to more patients and improve treatment decisions.

Revised: This experiment demonstrates that the second view of data can help regularize during training even if it is not available for test patients.

Original: We tested different classiﬁer training methods when only one view was available at test time: a) a linear SVM trained on one view, b) a deep NN trained on one view using the same architecture as the lower layers of TOCCA , c) CCAL-
Revised: We tested different classiﬁer training methods when only one view was available at test time: a) a linear SVM trained on one view, b) a deep NN trained on one view using the same architecture as the lower layers of TOCCA , c) CCAL-

Original: L ce trained on both views, d) TOCCA trained on both views.
Revised: L ce trained on both views, d) TOCCA trained on both views.

Original: Tab. 1 lists the classiﬁcation accuracy for each method and task.
Revised: Tab. 1 lists the classiﬁcation accuracy for each method and task.

Original: When predicting genomic subtype
Revised: When predicting genomic subtype

Original: Basal from images, all our methods showed an improvement in classiﬁcation accuracy; the best result was with TOCCA-W , which produced a 2.2% improvement.
Revised: Basal from images, all our methods showed an improvement in classiﬁcation accuracy; the best result was with TOCCA-W , which produced a 2.2% improvement.

Original: For predicting grade from GE, all our methods again improved the accuracy – by up to 3.2% with TOCCA-W .
Revised: For predicting grade from GE, all our methods again improved the accuracy – by up to 3.2% with TOCCA-W .

Original: These results show that having additional information during training can boost performance at test time.
Revised: These results show that having additional information during training can boost performance at test time.

Original: Notably, this experiment used a static set of pre-trained VGG16 image features in order to assess the utility of the method.
Revised: Notably, this experiment used a static set of pre-trained VGG16 image features in order to assess the utility of the method.

Original: The network itself could be ﬁne-tuned end-to-end with our TOCCA model, providing an easy opportunity for data augmentation and likely further improvements in classiﬁcation accuracy.
Revised: The network itself could be ﬁne-tuned end-to-end with our TOCCA model, providing an easy opportunity for data augmentation and likely further improvements in classiﬁcation accuracy.

Original: 4. S EMI - SUPERVISED LEARNING FOR SPEECH RECOGNITION
Revised: 5. S EMI - SUPERVISED LEARNING FOR SPEECH RECOGNITION

Original: Our ﬁnal experiments use speech data from XRMB, consisting of simultaneously recorded acoustic and articulatory measurements.
Revised: Our ﬁnal experiments use speech data from XRMB, consisting of simultaneously recorded acoustic and articulatory measurements.

Original: Prior work has shown that CCA-based algorithms can improve phonetic recognition (Wang et al., 2015b;a; 2016;
Revised: Prior work has shown that CCA-based algorithms can improve phonetic recognition (Wang et al., 2015b;a; 2016;

Original: Dorfer et al., 2016b).
Revised: Dorfer et al., 2016b).

Original: The 45 speakers were split into 35 for training, 2 for validation, and 8 for testing – a total of 1,429,236 samples for training, 85,297 for validation, and 111,314 for testing.
Revised: The 45 speakers were split into 35 for training, 2 for validation, and 8 for testing – a total of 1,429,236 samples for training, 85,297 for validation, and 111,314 for testing.

Original: The acoustic features are 112D and the articulatory ones are 273D.
Revised: The acoustic features are 112D and the articulatory ones are 273D.

Original: We removed the per-speaker mean & variance for both views.
Revised: We removed the per-speaker mean & variance for both views.

Original: Samples are annotated with one of 38 phonetic labels.
Revised: Samples are annotated with one of 38 phonetic labels.

Original: [Table]
Revised: [Table]

Original: Our task on this data set was representation learning for multi-view prediction – that is, using both views of data to learn a shared discriminative representation.
Revised: Our task on this data set was representation learning for multi-view prediction – that is, using both views of data to learn a shared discriminative representation.

Original: We trained each model using both views and their labels.
Revised: We trained each model using both views and their labels.

Original: To test each
Revised: To test each

Original: CCA model, we followed prior work and concatenated the original input features from both views with the projections from both views.
Revised: CCA model, we followed prior work and concatenated the original input features from both views with the projections from both views.

Original: Due to the large training set size, we used a Linear Discriminant Analysis (LDA) classiﬁer for efﬁciency.
Revised: Due to the large training set size, we used a Linear Discriminant Analysis (LDA) classiﬁer for efﬁciency.

Original: The same construction was used at test time.
Revised: The same construction was used at test time.

Original: This setup was used to assess whether a task-optimal
Revised: This setup was used to assess whether a task-optimal

Original: DCCA model can improve discriminative power.
Revised: DCCA model can improve discriminative power.

Original: We tested TOCCA with a task-driven loss of LDA (Dorfer et al., 2016a) or softmax to demonstrate the ﬂexibility of our model.
Revised: We tested TOCCA with a task-driven loss of LDA (Dorfer et al., 2016a) or softmax to demonstrate the ﬂexibility of our model.

Original: [Table]
Revised: [Table]

Original: We compared the discriminability of a variety of methods to learn a shared latent representation.
Revised: We compared the discriminability of a variety of methods to learn a shared latent representation.

Original: Tab.
Revised: Tab.

Original: 4 lists the classiﬁcation results with a baseline that used only the original input features for
Revised: 5 lists the classiﬁcation results with a baseline that used only the original input features for

Original: LDA.
Revised: LDA.

Original: Although deep methods, i.e., DCCA and SoftCCA, improved upon the linear methods, all TOCCA variations signiﬁcantly outperformed previous state-of-the-art techniques.
Revised: Although deep methods, i [Table] e., DCCA and SoftCCA, improved upon the linear methods, all TOCCA variations signiﬁcantly outperformed previous state-of-the-art techniques.

Original: Using softmax consistently beat LDA by a large margin.
Revised: Using softmax consistently beat LDA by a large margin.

Original: TOCCA-SD and TOCCA-ND produced equivalent results as a weight of 0 on the decorrelation term performed best.
Revised: TOCCA-SD and TOCCA-ND produced equivalent results as a weight of 0 on the decorrelation term performed best.

Original: However, TOCCA-W showed the best result with an improvement of 15% over the best alternative method.
Revised: However, TOCCA-W showed the best result with an improvement of 15% over the best alternative method.

Original: TOCCA can also be used in a semi-supervised manner when labels are available for only some samples.
Revised: TOCCA can also be used in a semi-supervised manner when labels are available for only some samples.

Original: Tab.
Revised: Tab.

Original: 5 lists the results for TOCCA-W in this setting.
Revised: 6 lists the results for TOCCA-W in this setting.

Original: With 0% labeled data, the result would be similar to DCCA.
Revised: With 0% labeled data, the result would be similar to DCCA.

Original: Notably, a large improvement over the unsupervised results in Tab. 4 is seen even with labels for only 10% of the training samples.
Revised: Notably, a large improvement over the unsupervised results in Tab. 5 is seen even with labels for only 10% of the training samples.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We proposed a method to ﬁnd a shared latent space that is also discriminative by adding a taskdriven component to deep CCA while enabling end-to-end training.
Revised: We proposed a method to ﬁnd a shared latent space that is also discriminative by adding a taskdriven component to deep CCA while enabling end-to-end training.

Original: This required a fundamental change in formulation because Deep CCA does not compute the embeddings directly as it optimizes an equivalent objective; therefore, we could not simply add an additional term.
Revised: This required a fundamental change in formulation because Deep CCA does not compute the embeddings directly as it optimizes an equivalent objective; therefore, we could not simply add an additional term.

Original: Instead, we found an alternative formulation by replacing the CCA projection with (cid:96) 2 distance minimization and orthogonality constraints on the activations, and we implemented this in three different ways.
Revised: Instead, we found an alternative formulation by replacing the CCA projection with (cid:96) 2 distance minimization and orthogonality constraints on the activations, and we implemented this in three different ways.

Original: TOCCA -W or
Revised: TOCCA -W or

Original: TOCCA-SD performed the best, dependent on the data set – both of which include some means of decorrelation to provide a regularizing effect to the model and thereby outperforming TOCCA-ND .
Revised: TOCCA-SD performed the best, dependent on the data set – both of which include some means of decorrelation to provide a regularizing effect to the model and thereby outperforming TOCCA-ND .

Original: TOCCA showed large improvements over state-of-the-art in cross-view classiﬁcation accuracy on
Revised: TOCCA showed large improvements over state-of-the-art in cross-view classiﬁcation accuracy on

Original: MNIST and signiﬁcantly increased robustness when the training set size was small.
Revised: MNIST and signiﬁcantly increased robustness to a small training set size.

Original: On CBCS, TOCCA provided a regularizing effect when both views were available for training but only one at test time.
Revised: On CBCS, TOCCA provided a regularizing effect when both views were available for training but only one at test time.

Original: TOCCA also produced a large increase over state-of-the-art for multi-view representation learning on a much larger data set, XRMB.
Revised: TOCCA also produced a large increase over state-of-the-art for multi-view representation learning on a much larger data set, XRMB.

Original: On this data set we also demonstrated a semi-supervised approach to get a large increase in classiﬁcation accuracy with only a small proportion of the labels.
Revised: On this data set we also demonstrated a semi-supervised approach to get a large increase in classiﬁcation accuracy with only a small proportion of the labels.

Original: Using a similar technique, our method could be applied when some samples are missing a second view.
Revised: Using a similar technique, our method could be applied when some samples are missing a second view.

Original: Classiﬁcation tasks using a softmax operation or LDA were explored in this work; however, the formulation presented can also be used with other tasks such as regression or clustering.
Revised: Classiﬁcation tasks using a softmax operation or LDA were explored in this work; however, the formulation presented can also be used with other tasks such as regression or clustering.

Original: Another possible avenue for future work entails extracting components shared by both views as well as individual components.
Revised: Another possible avenue for future work entails extracting components shared by both views as well as individual components.

Original: This approach has been developed for dictionary learning (Lock et al., 2013;
Revised: This approach has been developed for dictionary learning (Lock et al., 2013;

Original: Ray et al., 2014; Feng et al., 2018) but could be extended to deep CCA-based methods.
Revised: Ray et al., 2014; Feng et al., 2018) but could be extended to deep CCA-based methods.

Original: Finally, we have yet to apply data augmentation to the proposed framework; this could provide a signiﬁcant beneﬁt for small training sets.
Revised: Finally, we have yet to apply data augmentation to the proposed framework; this could provide a signiﬁcant beneﬁt for small training sets.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: This appendix includes additional details on our TOCCA algorithm and experiments, including 1) a comparison of our formulation with other related CCA approaches, 2) pseudocode for the ZCA whitening algorithm used by TOCCA-W , 3) details on hyperparameter selection, and 4) training runtime experiments.
Revised: This appendix includes additional details on our TOCCA algorithm and experiments, including 1) a comparison of our formulation with other related CCA approaches, 2) pseudocode for the ZCA whitening algorithm used by TOCCA-W , 3) details on hyperparameter selection, and 4) training runtime experiments.

Original: A. C OMPARISON OF TOCCA WITH RELATED ALGORITHMS
Revised: A. C OMPARISON OF TOCCA WITH RELATED ALGORITHMS

Original: Our TOCCA methods ﬁnds a shared latent space that is also discriminative by changing the CCA formulation in order to add a task-driven component.
Revised: Our TOCCA methods ﬁnds a shared latent space that is also discriminative by changing the CCA formulation in order to add a task-driven component.

Original: Tab.
Revised: Tab.

Original: A2 compares our three TOCCA formulations with other related methods (discussed in § 2).
Revised: A2 compares our three TOCCA formulations with other related methods (discussed in § 3).

Original: CCA is the baseline linear method with a goal of maximizing the correlation between a set of orthogonal linear projections on two views of data.
Revised: CCA is the baseline linear method with a goal of maximizing the correlation between a set of orthogonal linear projections on two views of data.

Original: DCCA and SoftCCA are unsupervised deep methods.
Revised: DCCA and SoftCCA are unsupervised deep methods.

Original: DCCA optimizes an equivalent objective to CCA but uses non-linear projections implemented with a NN; however, the projections are not computed in the network, only after optimization is complete.
Revised: DCCA optimizes an equivalent objective to CCA but uses non-linear projections implemented with a NN; however, the projections are not computed in the network, only after optimization is complete.

Original: SoftCCA changes the correlation objective to, equivalently, minimize the (cid:96) 2 distance between projections and relaxes the orthogonality constraints by using regularization.
Revised: SoftCCA changes the correlation objective to, equivalently, minimize the (cid:96) 2 distance between projections and relaxes the orthogonality constraints by using regularization.

Original: CCAL-
Revised: CCAL-

Original: L rank does compute the CCA projections in the network but does not optimize the ﬁnal NN for correlation; it instead focuses on a pairwise ranking loss for use in retrieval.
Revised: L rank does compute the CCA projections in the network but does not optimize the ﬁnal NN for correlation; it instead focuses on a pairwise ranking loss for use in retrieval.

Original: Our family of TOCCA methods were detailed in § 3.
Revised: Our family of TOCCA methods were detailed in § 4.

Original: In this supervised formulation, we use the same (cid:96) 2 distance as SoftCCA and simultaneously optimize a task-driven objective.
Revised: In this supervised formulation, we use the same (cid:96) 2 distance as SoftCCA and simultaneously optimize a task-driven objective.

Original: We handle the orthogonality constraints in three different ways: with whitening ( TOCCA-W ), with regularization ( TOCCA-SD ) as was used in SoftCCA, and with no explicit decorrelation ( TOCCA-ND ).
Revised: We handle the orthogonality constraints in three different ways: with whitening ( TOCCA-W ), with regularization ( TOCCA-SD ) as was used in SoftCCA, and with no explicit decorrelation ( TOCCA-ND ).

Original: and non for image retrieval – not
Revised: and non for image retrieval – not

Original: A are
Revised: A are

Original: [Table]
Revised: [Table]

Original: is is function
Revised: is is function

Original: [Table]
Revised: [Table]

Original: g
Revised: g

Original: [Table]
Revised: [Table]

Original: A. A LGORITHM FOR WHITENING
Revised: A. A LGORITHM FOR WHITENING

Original: TOCCA-W uses whitening to achieve orthogonality (see § 3 for details).
Revised: TOCCA-W uses whitening to achieve orthogonality (see § 4 for details).

Original: The goal is to transform the activations such that their covariance becomes the identity matrix.
Revised: The goal is to transform the activations such that their covariance becomes the identity matrix.

Original: We use ZCA whitening which ﬁrst applies PCA whitening to decorrelate the data and rescale each axis, followed by a rotation back to the original space.
Revised: We use ZCA whitening which ﬁrst applies PCA whitening to decorrelate the data and rescale each axis, followed by a rotation back to the original space.

Original: The ﬁnal rotation reduces the stochastic axis swapping problems of PCA whitening (Huang et al., 2018).
Revised: The ﬁnal rotation reduces the stochastic axis swapping problems of PCA whitening (Huang et al., 2018).

Original: Pseudocode for ZCA whitening is shown in Algorithm A1.
Revised: Pseudocode for ZCA whitening is shown in Algorithm A1.

Original: Algorithm A1 Whitening layer for orthogonality.
Revised: Algorithm A1 Whitening layer for orthogonality.

Original: Input: activations A(cid:15)
Revised: Input: activations A(cid:15)

Original: R d o × n
Revised: R d o × n

Original: Hyperparameters: batch size m , momentum α
Revised: Hyperparameters: batch size m , momentum α

Original: Parameters of layer: mean µ , covariance Σ if training then
Revised: Parameters of layer: mean µ , covariance Σ if training then

Original: [Equation]
Revised: [Equation]

Original: Mean center data } Σ ← α Σ
Revised: Mean center data } Σ ← α Σ

Original: + (1 − α ) 1 m − 1 ¯
Revised: + (1 − α ) 1 m − 1 ¯

Original: A 1 ¯ A T 2 { Update covariance } ˆΣ ← Σ + (cid:15)I { Add (cid:15)I for numerical stability } Λ , V ← eig (ˆΣ) { Compute eigendecomposition }
Revised: A 1 ¯ A T 2 { Update covariance } ˆΣ ← Σ + (cid:15)I { Add (cid:15)I for numerical stability } Λ , V ← eig (ˆΣ) { Compute eigendecomposition }

Original: U ← V Λ − 1 / 2 V T { Compute transformation matrix } else ¯ A ← A − µ { Mean center data } end if
Revised: U ← V Λ − 1 / 2 V T { Compute transformation matrix } else ¯ A ← A − µ { Mean center data } end if

Original: B ← U ¯ A { Apply ZCA whitening transform } return B
Revised: B ← U ¯ A { Apply ZCA whitening transform } return B

Original: A. I MPLEMENTATION DETAILS : HYPERPARAMETERS
Revised: A. I MPLEMENTATION DETAILS : HYPERPARAMETERS

Original: A random search over hyperparameters was used to train our methods and those that we compare with.
Revised: A random search over hyperparameters was used to train our methods and those that we compare with.

Original: The hyperparameter settings and ranges for each data set are provided in Tab. A3.
Revised: The hyperparameter settings and ranges for each data set are provided in Tab. A3.

Original: Random search in these intervals was performed 100 times for MNIST and CBCS.
Revised: Random search in these intervals was performed 100 times for MNIST and CBCS.

Original: Fewer tries were done for
Revised: Fewer tries were done for

Original: XRMB because of the much greater runtime on this large data set.
Revised: XRMB because of the much greater runtime on this large data set.

Original: A larger batch size was used for
Revised: A larger batch size was used for

Original: XRMB to improve runtime.
Revised: XRMB to improve runtime.

Original: The hyperparameter ranges were initially set as an educated guess and, in some cases, were widened for a particular data set (for all methods) after observing results.
Revised: The hyperparameter ranges were initially set as an educated guess and, in some cases, were widened for a particular data set (for all methods) after observing results.

Original: [Table]
Revised: [Table]

Original: A. R UNTIME EXPERIMENTS
Revised: A. R UNTIME EXPERIMENTS

Original: The computational complexity of TOCCA-W is greater than that of TOCCA-SD due to the eigendecomposition operation (see the end of § 3); however, this extra computation is only carried out once per batch.
Revised: The computational complexity of TOCCA-W is greater than that of TOCCA-SD due to the eigendecomposition operation (see the end of § 4); however, this extra computation is only carried out once per batch.

Original: A runtime comparison of the two methods on all three data sets is provided in Tab. A4.
Revised: A runtime comparison of the two methods on all three data sets is provided in Tab. A4.

Original: The difference in runtime was less than 6.5% for a batch size of 100 or 9.4% for a batch size of
Revised: The difference in runtime was less than 6.5% for a batch size of 100 or 9.4% for a batch size of

Original: [Table] [Table]
Revised: [Table] [Table]