Original: L EARNING RELEVANT FEATURES FOR STATISTICAL INFERENCE
Revised: L EARNING RELEVANT FEATURES FOR STATISTICAL INFERENCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original:  We introduce an new technique to learn correlations between two types of data.
Revised: Given two views of data, we consider the problem of ﬁnding the features of one view which can be most faithfully inferred from the other. We ﬁnd that these are also the most correlated variables in the sense of deep canonical correlation analysis (DCCA). Moreover, we show that these variables can be used to construct a non-parametric representation of the implied joint probability distribution.

Original: The learned representation can be used todirectly compute the expectations of functions over one type of data conditioned on the other, such as Bayesian estimators and their standard deviations.
Revised: This representation can be used to compute the expectations of functions over one view of data conditioned on the other, such as Bayesian estimators and their standard deviations.

Original: Speciﬁcally, our loss function teaches two neural nets to extract features representing the singular probability vectors of highest singular value for the stochastic map (set of conditional probabilities) implied by the joint dataset, relative to the inner product deﬁned by the Fisher information metrics evaluated at the marginals.

Original: We test the approach usinga synthetic dataset, analytical calculations, and inference on occluded MNIST images.
Revised: We test the approach using inference on occluded MNIST images, and show that our representation contains multiple modes.

Original: Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective.
Revised: Surprisingly, when applied to supervised learning (one dataset consists of labels), this approach automatically provides regularization and faster convergence compared to the cross-entropy objective.

Original: We also explore using this approach to discover salient independent features of a single dataset.
Revised: We also explore using this approach to discover salient independent variables of a single dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Further progress in artiﬁcial intelligence requires algorithms which can learn to model unlabeled data (unsupervised learning).
Revised: Given samples ( x 1 , y 1 ) , . . . , ( x n , y n ) from an unknown joint probability distribution p ( x, y ) , we want to construct a useful representation of the conditional probabilities p ( x | y ) and p ( y | x ) , so that we that we can infer one view from the other on new data.

Original: Here we assume that the data naturally comes in two parts, such as past and future histories, visual and auditory inputs, actions and their effects, etc.
Revised: For instance, x and y could be past and future histories of dynamical data, visual and auditory inputs, actions and their effects, etc.

Original: Our goal isto produce a useful model of  the correlations which exist between both variables.
Revised: Following the approach introduced in B ´ eny & Osborne (2013; 2015b) in the context of quantum information theory, we look at the problem as follows: the conditional distributions p ( y | x ) can be thought of as representing a noisy communication channel (stochastic map). This channel is a linear map between spaces of typically ludicrously large dimensions (the spaces of all probability distributions over x or y ). We want a pair of small subspaces which best represent the channel. Speciﬁcally, we look for those vectors representing probability distributions over x which lose least distinguishability under the channel, where the distinguishability is measured by the χ 2 divergence.

Original: By contrast, existing techniques perform well in assymetric situation, to predict one variable that has a very small number of possible values (such as labels in supervised learning) or that depends almost deterministically on the other.
Revised: We show in Section 3 that this is equivalent to performing a certain singular value decomposition of the channel (seen as an operator in Hilbert space) and keep only the components with the largest singular values. Moreover, the full singular value decomposition is equivalent to the decomposition in terms of canonical variables introduced in Lancaster (1958), namely, [Equation] where u i and u j are real non-linear functions such that E ( u i u j )

Revised: =

Original: An existing option would be to use an autoencoder with a probabilistic decoder, such as a variational autoencoder (VAE) (Kingma & Welling, 2013) or further reﬁnements of  the concept (Higgins et al., 2017; Chen et al., 2018), trained to produce one variable given the other instead of  the same variable (such as in Iten et al. 
Revised: δ ij , E ( v i v j ) = δ ij , and 0 < η D ≤ · · · ≤ η 1 ≤ η 0 = 1 are the singular values. The practical advantage of this representation for inference (or prediction) is that it reduces the evaluation of conditional expectations to that of empirical averages over the (unconditional) marginal p ( y ) . As observed in Michaeli et al. (2016), the span of the ﬁrst k canonical variables u i , v j is what is learned by the deep canonical correlation analysis (DCCA) (Andrew et al., 2013).

Original: Instead, we proposehere a new approach which has very different characteristics.
Revised: Indeed, these variables are those which maximize the correlations E ( u i v i ) subject to the same constraints as above. (This reduces to CCA (Hotelling, 1936) when u i , v j are linear maps). In this work, besides establishing this new information-theoretical interpretation of canonical variables and DCCA, we experiment with using this representation for performing inference on new data. Moreover, we propose a strategy for extracting disentangled variables from the canonical variables, inspired by analytical solutions.

Original: We will refer to it as relevant feature analysis (RFA).
Revised: This general problem (of building an effective representation of the conditional probability distributions implied by joint samples) covers many existing approaches in different contexts. For instance, if the variables y has few possible states, then it reduces to a classiﬁcation problem, usually solved by minimizing the crossentropy between a predicted distribution and the one-hot encoding of the classes. When y has a large number of states, or is fundamentally continuous, existing approaches usually do not model the whole conditional distribution, but either provide the average (regression), or approximately sample from it.

Original: Some of  the design advantages over a VAE are: (i) there is no tunable variable in  the objective function, (ii) it does not require parametric models of the conditional probability distributions, (iii) it allows for a direct evaluation of expectation values over the conditional distributions (but doesn’t directly enable sampling from it).
Revised: Two main classes of methods can be used in principle to sample from the conditional distributions: the “variational” methods in which deterministic neural networks produce the parameters of analytical classes of probabilities inspired by variational autoencoders Kingma & Welling (2013), such as Iten et al. ; Wang et al.

Original: Moreover, (iv) the gradient need not be propagated through latent variables, which ought to ease training.
Revised: (2016), or non-parametric methods based on generative adversarial training Goodfellow et al. (2014), such as possibly a conditional Mirza & Osindero (2014) version of an energy-based GAN Zhao et al.

Original: This last point results from the fact that we do not need to learn a generative model (decoder).
Revised: By contrast, our approach doesn’t require the training of a generative model.

Original: Instead, we use two deterministic encoders, one for each variable. The individual output neurons of these feed-forwards networks represent functions over the data which we call features . Our learning objective, or loss function, effectively maximizes the amount of correlations between these features.
Revised: Instead, conditional expectations are constructed as linear combinations of unconditional empirical averages over the training data. Previous approaches to equipping CCA or DCCA with an information-theoretic interpretation have explored different directions.

Original: The resulting relevant features can then be used to do inference in both directions. Speciﬁcally, they allow one to directly evaluate the expectation of any of the relevant features of one variable conditioned on the other variable taking a speciﬁc value.
Revised: For instance, in Wang et al. (2016), the authors generalize a probabilistic interpretation for CCA in terms of gaussian distributions, which leads to a variational approach. In Painsky et al.

Original: Moreover, if thenumber of learned features is sufﬁciently large for the problem, then one can estimate the conditional expectation of any “smooth” 1 function over the conditional probabilities.
Revised: , additional constraints on the mutual information between the data and the learned variables are added to the optimizations. A previous attempt at designing a numerical solution for our singular value problem can be found in B ´ eny (2018b). In that work, the relevant variables were represented through a PCA kernel produced by Monte Carlo sampling, but wasn’t practical.

Original:  T HEORY
Revised:  T HEORY

Original: Our approach is based on a theory introduced in B´eny & Osborne (2013; 2015b) for understanding the role of information in effective quantum ﬁeld theories, but restricted here to the classical nonquantum setting.

Original: Relations to deconvolution B´eny (2018b) and kernel PCA B´eny (2018c) were also previously explored.

Original: We formalize the problem by assuming that our data was sampled from an unknown joint distribution p ( x, y ) over two random variables X and Y .
Revised: We formalize the problem by assuming that our data was sampled from an unknown joint distribution p ( x, y ) over two random variables X and Y .

Original: Let V X and V Y denotes the linear spaces spanned by the probability distributions over X and Y respectively.
Revised: Let V X and V Y denote the linear spaces spanned by all probability distributions over X and Y respectively.

Revised: Here we assume that X and Y take ﬁnitely many values for simplicity, but this formalism can be straightforwardly extended to inﬁnite-dimensional vector spaces.

Original: The joint distribution p (x, y ) yields conditionals p 
Revised: We will need inner products on V X and V Y , to make them into real Hilbert spaces. We use the Fisher information metrics evaluated at the points p ( x ) and p ( y ) respectively (marginals of p ( x, y ) ), that is, [Equation] for any vectors µ, µ (cid:48) ∈ V X and ν, ν (cid:48)

Original:  (y | x ) and p ( x| y ), which can be understood as the components of stochastic matrices, deﬁning linear maps N and N ∗ between V X and V Y .
Revised: ∈ V Y . Below we also call the marginals p X and p Y respectively when omitting their arguments ( p X ( x ) ≡ p ( x ) and p Y ( y ) ≡ p ( y ) ). These inner products allow us to deﬁne the χ 2 divergence: [Equation]

Original: If µ ∈ V X and ν ∈ V Y , then theimages N ( µ ) ∈
Revised: which a measures of statistical distinguishability between q and p X Speciﬁcally, it quantiﬁes how easy it is to reject the null hypothesis that the state is p X when it is actually q , based on the empirical distribution obtained from independent samples. It is also the lowest order approximation of the

Original:  V Y and N ∗ ( ν
Revised: Kullback-Leibler divergence. The joint distribution p ( x, y ) yields conditional distributions p ( y | x ) and p ( x | y ) . These can be understood as the components (or kernels) of stochastic maps N : V X → V Y and N ∗ : V X → V Y respectively. Explicitely, if µ ∈ V X and ν ∈ V Y , then the images N ( µ ) ∈ V Y and N ∗ ( ν

Original: ) ∈ V X are deﬁned by
Revised: ) ∈ V X are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: These stochastic maps N and N ∗ perform inference of one variable given some (possibly imperfect) knowledge about the other, with priors given by the marginals p ( x ) or p ( y ) of p ( x, y ) depending on the direction of the inference.
Revised: These stochastic maps N and N ∗ perform inference of one variable given some (possibly imperfect) knowledge about the other, with priors given by the marginals p ( x ) or p ( y ) of p ( x, y ) depending on the direction of the inference.

Original: We want to approximate N and N ∗ by setting their smaller singular values to zero. However, in order to deﬁned thesingular value decomposition, we need inner products on V X and V Y . For that purpose, we use the Fisher information metrics evaluated at the points p ( x ) and p ( y ) respectively (marginals of p ( x, y ) ), that is,
Revised: Importantly, N ∗ is the transpose of N in terms of the inner products deﬁned above:

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: We now have the tools to address the problem mentioned in the introduction.

Original: [Equation] for any probability distributions µ, µ (cid:48) ∈ V X and [-ν, ν (cid:
-]
Revised: The distinguishability between q ∈ V X and [-ν, ν (cid:
-]p X after the action of the channel N is χ 2 ( N ( q ) , p Y ) since p Y =

Original:  )∈ V Y .
Revised: N ( p X ) .

Original: A beauty of this choice is that it makes N ∗ the transpose of N :
Revised: Hence we want to ﬁnd the distributions p which maximize the relevance B ´ eny & Osborne (2013).

Original: [Equation]
Revised: [Equation]

Original: It follows that N and N ∗ have thesame set of singular values, which are  the square roots of the eigenvalues of N ∗ N .
Revised: where µ = q − p X . The inner-product formulation makes it clear that this amounts to ﬁnding the eigenvector with largest eigenvalue for the symmetric map N ∗ N , which is also the singular vector with largest singular value for N . On can then go on to ﬁnd the eigenvector with next largest eigenvalue and so on, which are automatically orthogonal. In practice, the inner products are more tractable to compute if we express elements µ ∈ V X and ν ∈ V Y in terms of variables f

Original: The singular vectors of N and N ∗ are respectively the left- and right- eigenvectors of N ∗ N .
Revised: and g as µ = p X f and ν = p Y g , or µ [Equation] p

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: and ν

Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: y ) for all x, y .

Original: Moreover, the singular values are all in  the interval [0 , 1] because the Fisher metric contracts under any stochastic map.
Revised: Indeed, this yields simply [Equation] We are now in measure to make the connection with DCCA Andrew et al. Indeed, the aims of DCCA is to maximize the correlations corr ( f, g ) =

Original: Another useful fact is the following: if we write µ ( x ) = p ( x )  f ( x ) and ν ( y ) 
Revised: E ( fg ) over function f ( x ) and g ( y ) such that E ( f 2 ) = E ( g 2 ) = 1 . But, using µ = p X f and ν = p Y g , we have [Equation]

Revised: which is maximized by the left- and right- singular vectors µ and ν of N with largest singular value.

Original:  = 
Revised: Given all the singular vectors µ i = p X f i and ν i = p

Original: p ( y ) g ( y ) for all x, y (we call the function f andg features ), then the inner products become simple covariances:
Revised: Y g i with singular values η i , we obtain the representation [Equation] X , which, using a more standard notation and the Kronecker delta δ x , yields Eq.

Original: [Equation]
Revised: [Equation]

Revised: where µ i = p

Revised: X u i

Revised: and ν

Revised: i = p

Revised: Y v i .

Revised: 3. N ON - DIAGONAL FORM AND RELEVANT VARIABLES

Original: Moreover, it is easy to check that
Revised: For the purpose of the optimization and inference, we do not need to full diagonal decomposition, but just functions f i = µ i /p X and g j = ν j /p Y , i, j = 1 , . . . , k 0 , which have the same span as the canonical variables u i and v j respectively for i, j = 1 , . . .

Original: [Equation]
Revised: , k 0 (assuming that η 1 , . .

Original: These three covariances can be approximated straightforwardly by averages on the data. If we have neural network producing some arbitrary functions f 1 ( x ) , . . , f k ( x ) and g 1 ( y ) , . . . , g k ( y ) , then we can use the above three equations to compute the components of N in the 
Revised: , η k 0 are the largest singular vector).

Original: In subspaces spanned by the corresponding functions µ i ( x ) = p ( x ) f i( x ) and ν j ( y )
Revised: Below we refer to f i and g j as the k 0 most relevant variables . Because these functions may not be orthogonal, we need the covariance matrices

Original: =
Revised: [Equation]

Original: p ( y ) g j ( y ) , i.e., the numbers N ij such that N ( µ j ) = (cid:80) j N ij ν i .
Revised: If N ij denote the components of N in the sense that N ( p X f j ) = (cid:80) i N ij p Y g

Original: Indeed, using A ij= (cid:104) ν i ,  N ( µ j ) (cid:105) Y and
Revised: i , then, using our inner products to isolate N ij , we obtain N =

Original: Lij = (cid:104) ν i , ν j (cid:105) Y , we have N = L − 1 A (using the matrix inverse and matrix product).
Revised: L − 1 A .

Original: Similarly we can deﬁne the componentsN ∗ ij of N ∗, which aregiven by N ∗  =
Revised: Similarly, the components of N ∗ are N ∗ ij =

Original: K − 1 A (cid:62)where K ij = (cid:104) µ i , µ j (cid:105) X .
Revised: K − 1 A (cid:62) .

Original: We observe thatTr ( N ∗ N ) is the sum of the square of the singular values of N restricted to the span of the functions µ i and ν i .
Revised: This implies that the sum of the square of the singular values of N restricted to the spans of the vectors p X f i and p Y

Original: Maximizing Tr ( N ∗ N ) over the parameters of the networks then yields the subspaces of V X and V Y spanned by the k singular vectors of N with largest singular values.
Revised: g j for all i, j , which is what we want to maximize, is just given by [Equation] This is the DCCA objective. Below we use the objective function C = k 0 − Tr ( N ∗ N ) , for the cosmetic reason that its optimal value is zero.

Original: We refer to the corresponding features as the k most relevant features .
Revised: Moreover, the corresponding truncated representation of the conditional distribution is

Original: Since we know explicitly the components of N and N ∗ with respect to these subspaces, we can also perform any inference using these projected forms of the channels.
Revised: [Equation] where we used the fact that the components of δ x are δ j = (cid:80) i K − 1 ji f i ( x ) .

Original: Of course, This approach can produce a faithful representation of the correlations only if N is actually close to being of rank k {+
+} (see Appendix B for a more precise statement).
Revised: Of course, This approach can produce a faithful representation of the correlations only if N is actually close to being of rank k {+
+} (see Appendix B for a more precise statement).

Original: If we interpret the relevant subspace as a space of probability over latent variable, this means that our latent variables have at most k  discrete states.
Revised: If we interpret the relevant subspace as a space of probability over latent variable, this means that our latent variables have at most k 0 discrete states.

Original: However, even if the rank- k projection on N is a not a good approximation, this strategy allows us to  do exact inference on certain random variables, namely those which are in the span of the relevant features! Indeed, N maps any of the most k relevant functions µ i ( x ) to the span of the most k relevant functions ν i ( y ) , i.e., for any i = 1 , . . . , k , N ( µ i ) =
Revised: However, even if the rank k 0 corner of N is a not a good approximation, this strategy allows us to nevertheless do the correct inference on certain random variables, namely those which are in the span of the canonical variables!

Original: (cid:80) kj = 1 c j ν j . Hence the  expectationvalue of the feature f i ( x ) = 1 p ( x ) µ i ( x ) conditioned on Y = y ∗ is given by
Revised: Indeed, the exact conditional expectation of g k is (assuming D is the actual rank of N ),

Original: [Equation]
Revised: [Equation]

Original: The role of the variables X and Y can be exchanged in this result by the swaps N ↔ N ∗ , f i ↔ g i .
Revised: where the last truncation is exact if k ≤ k 0 due to the assumption that the basis f i and g j have the same span as the k 0 largest right and left singular vectors of N respectively.

Original: For instance, if p ( x, y ) is Gaussian, the singular vectors can be computed analytically following
Revised: For instance, if p ( x, y ) is Gaussian, the canonical variables can be computed analytically, as in

Original: B ´ eny & Osborne (2013); B ´ eny (2018a) (Appendix A).
Revised: Lancaster (1958) or B´eny (2018a) in the multivariate case. Solutions for other distributions were also computed in Eagleson (1964).

Original: Notably, for any two dimensional Gaussian, the space of k most relevant features is simply spanned by the moments f n ( x ) = x n and g n ( y ) =
Revised: Notably, for any two dimensional Gaussian, the space of k most relevant variables is simply spanned by the moments f n ( x ) = x n and g n ( y ) =

Original: y n for n = 0 , . . .
Revised: y n for n = 0 , . . .

Original: , k − 1 .
Revised: , k − 1 .

Original: Hence, in this case the ﬁrst k moments can be inferred exactly using only  k + 1 -dimensional subspaces of features (See Appendix C).
Revised: Hence, in this case the ﬁrst k moments can be inferred exactly using only the k + 1 most relevant variables (See Appendix C).

Original: RFA  A LGORITHM
Revised:  A LGORITHM

Original: Let us explicit the algorithm resulting from the above analysis.
Revised: Let us explicit the algorithm resulting from the above analysis.

Original: We assume that we are given independent samples ( x 1 , y 1 ) , ( x 1 , y 2 ) , . . .
Revised: We assume that we are given independent samples ( x 1 , y 1 ) , ( x 1 , y 2 ) , . . .

Original: from the otherwise unknown joint distribution p ( x, y ) .
Revised: from the otherwise unknown joint distribution p ( x, y ) .

Revised: We ﬁrst perform DCCA (Andrew et al., 2013).

Original: We need two independent deterministic feed-forward neural networks.
Revised: That is, we need two independent deterministic feedforward neural networks.

Original: The ﬁrst maps x to a set of k 0 real-valued features f 1 ( x ) , . . .
Revised: The ﬁrst maps x to a set of k 0 real-valued variables f 1 ( x ) , . . .

Original: , f k 0 ( x ) .
Revised: , f k 0 ( x ) .

Original: The second maps y to a different set of k 0 features g 1 ( y ) , . .
Revised: The second maps y to a different set of k 0 variables g 1 ( y ) , . .

Original: , g k 0 ( y ) .
Revised: , g k 0 ( y ) .

Original: The parameters of the neural networks are to be set to minimize the positive loss function
Revised: The parameters of the neural networks are to be set to minimize the objective function

Original: [Equation]
Revised: [Equation]

Original: where the matrices K, L, A can be approximated over a mini-batch ( x n , y n ) , n = 1 , . .
Revised: where the matrices K, L, A can be approximated over a mini-batch ( x n , y n ) , n = 1 , . .

Original: , N via
Revised: , N via

Original: K ij

Original: [Equation]
Revised: [Equation]

Original: N
Revised: We found that, provided the batch size is sufﬁciently large compared to k 0 (about 10 times in our experience), this can be minimized using ADAM or direct gradient descent.

Original: [Equation]
Revised: However, to guarantee stability when using large k 0 , we needed to explicit the gradient of the objective function in order to force the use of the Moore-Penrose pseudo-inverses for K − 1 and L − 1 in both the forward and backward passes, in addition to using 64 bits ﬂoats in these computations.

Original: Once the features have been learned, we still need to use the training data in a second step.
Revised: Once the relevant variables have been learned, we still need to use the training data in a second step.

Original: Indeed, suppose that we wish to use our model to infer the value of some function Θ( x ) , i.e., to compute it’s approximate expectation value in terms of the conditional distribution x
Revised: Indeed, suppose that we wish to use our model to infer the value of some function Θ( x ) , i.e., to compute its approximate expectation value in terms of the conditional distribution x

Original: (cid:55)→ p ( x | y ) .
Revised: (cid:55)→ p ( x | y ) .

Original: Then we need to store, for each feature j = 1 , . ..
Revised: Then we need to store, for each variable j = 1 , . .

Original:  , k 0 , the quantities
Revised: . , k 0 , the quantities

Original: [Equation]
Revised: [Equation]

Original: where the average is to be taken on the full training batch (of size N full ).
Revised: where the average is to be taken on the full training batch (of size N full ).

Original: The same can be done exchanging x with y and f j with g j for the reverse inference.
Revised: The same can be done exchanging x with y and f j with g j for the reverse inference.

Original: For instance, if  we are interested in theBayesian estimator for an l 2 distance, then we needat least the expectation values of thereal components Θ( x ) =
Revised: For instance, if a data point x is composed of real components x a —such as pixel color components for an image—and we are interested in the estimator which minimize the expected l 2 distance to the predicted values of these components, then we need the expectation values of the components Θ( x ) =

Original: x of the data, and possibly higher moments to gain more knowledge about the shape of the posterior distribution, such as the second moments Θ (cid:48)
Revised: x a for all a , and possibly higher moments to gain more knowledge about the shape of the posterior distribution, such as the second moments Θ (cid:48)

Original: ( x ) = x 2 , etc.
Revised: ( x ) = x 2 , etc.

Original: Inference can then be performed with new data using
Revised: Inference can then be performed with new data using

Original: [Equation]
Revised: [Equation]

Original: Moreover, the accuracy of these predictions does not depend on the rank k 0 if Θ is taken in the span of the relevant features, i.e., Θ( x ) = (cid:80)
Revised: Moreover, the accuracy of these predictions does not depend on the rank k 0 if Θ is taken in the span of the relevant variables, i.e., Θ( x ) = (cid:80)

Original: ki =1 c
Revised: ki = [Equation] c

Original: i f i ( x ) , for which Θ j = (cid:80)
Revised: i f i ( x ) , for which Θ j = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: The reverse inference formulas are obtained simply by the exchanges K ↔ L , A ↔ A (cid:62) , and g i ↔ f i .
Revised: The reverse inference formulas are obtained simply by the exchanges K ↔ L , A ↔ A (cid:62) , and g i ↔ f i .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: For all our experiments, we used the  Flux package (Innes, 2018) for Julia, and the built-in ADAM optimizer with default parameters.
Revised: In all our experiments, we used the ADAM optimizer with learning rate 0 . We used the Flux package (Innes, 2018) for Julia, as well as Tensorﬂow.

Original: As usual the data is divided into a training set and a testing set.
Revised: As usual the data is divided into a training set and a testing set.

Original: No aspect of the testing set is used during training.
Revised: No aspect of the testing set is used during training.

Original: The loss function refers to Eq.
Revised: The loss function refers to Eq.

Original: In order to monitor overﬁtting, we compute a “test loss” and a “training loss”.
Revised: In order to monitor overﬁtting, we compute a “test loss” and a “training loss”.

Original: The test loss is computed from the trained features using only the test data, and accordingly, the training loss is computed purely using the training data.
Revised: The test loss is computed from the trained variables using only the test data, and accordingly, the training loss is computed purely using the training data.

Original: Moreover, when performing inference on test data using Eq.
Revised: Moreover, when performing inference on test data using Eq.

Original: (10), we use the covariances A, L, K and expectations Θ j (Equ. (9))
Revised: (10), we use the covariances A, L, K and expectations Θ j (Equ. (9))

Original: built from the training data only.
Revised: built from the training data only.

Revised: Inaccuracy on MNIST (%)

Revised: [Figure]

Original: 4. S UPERVISED LEARNING 
Revised: 5. S UPERVISED LEARNING : DCCI

Original: In the context of a supervised classiﬁcation task, one of the dataset (the labels) is of sufﬁciently low dimensionality that we can use a complete basis over its probability space as our features.
Revised: In the context of a supervised classiﬁcation task, one of the dataset (the labels) is of sufﬁciently low dimensionality that we can use a complete basis over its probability space as our relevant variables, such as the standard one-hot encoding of labels.

Original: This serves as a good ﬁrst sanity test for our approach.
Revised: This serves as a good ﬁrst sanity test for our approach.

Original: Surprisingly, we ﬁnd that RFA converges faster than standard approaches, and without the need for regularization.
Revised: Surprisingly, we ﬁnd that it converges faster than standard approaches, and without the need for regularization.

Original: Let the variable Y stands for the labels, with values in { 1 , . . . , n } .
Revised: Let the variable Y stands for the labels, with values in { 1 , . . . , n } .

Original: The probability space consists of vectors with n real components.
Revised: The probability space consists of vectors with n real components.

Original: The canonical basis corresponds to the one-hot encoding g
Revised: The canonical basis corresponds to the one-hot encoding g

Original: i ( j ) =
Revised: i ( j ) =

Original: δ ij (Kronecker delta).
Revised: δ ij (Kronecker delta).

Original: All we need is a neural network to encode n features f 1 , . . .
Revised: All we need is a neural network to encode n variables f 1 , . . .

Original: , f n on X .
Revised: , f n on X .

Original: After learning the most relevant features f i . We apply the reverse of Eq.
Revised: After learning the most relevant variables f i , we apply the reverse of Eq.

Original: (10) for function Θ( y ) =
Revised: (10) for function Θ( y ) =

Original: y , and use the maximum component of expected value y to infer the labels from the data.
Revised: y , and use the maximum component of expected value y to infer the labels from the data.

Revised: Let us refer to this procedure as DCCI (Deep Canonical Correlations based Inference).

Original: We tested this approach on the MNIST and CIFAR10 datasets, and compared the results to the standard cross-entropy objective (Fig. 1).
Revised: We tested this approach on the MNIST and CIFAR10 datasets, and compared the results to the standard cross-entropy objective (Fig. 1).

Revised: We plotted the accuracy as function of the epoch rather than clock time which would depend on many factors.

Revised: But the time per epoch is roughly the same for each approaches in the above experiments.

Revised: Indeed, the training time is dominated by the forward and backward evaluations of the neural networks which are identical.

Revised: (However, the time it takes to evaluate our objective can become signiﬁcant for much larger number of labels k , since it involves the inversion of matrices of dimension k .

Revised: This is in addition to the fact that a greater dimension would require also larger batches.)

Original: We found that, without regularization, simply changing the objective from cross-entropy to RFA provided a large improvement both of convergence speed and ﬁnal accuracy for both models.
Revised: We found that, without regularization, simply changing the objective from cross-entropy to DCCI provided a large improvement both of convergence speed and ﬁnal accuracy for both models.

Original: On MNIST, RFA alone also outperformed cross-entropy with dropout.
Revised: On MNIST, DCCI alone also outperformed cross-entropy with dropout.

Original: (Dropout did not yield any improvement in conjunction with RFA).
Revised: (Dropout did not yield any improvement in conjunction with DCCI).

Original: However, adding batch-normalization layers on the CIFAR example, erased any distinction between RFA and cross-entropy.
Revised: However, adding batch-normalization layers on the CIFAR example, erased any distinction between DCCI and cross-entropy.

Original: The code containing all parameters used can be found in (B´eny), including a Tensorﬂow/Keras implementation.

Original: [Figure] 4. I NFERENCE 
Revised: 5. I NFERENCE ON OCCLUDED MNIST

Original: In this next experiment, we used the left and right halves of the MNIST digit images as correlated variables X and Y .
Revised: In this next experiment, we used the left and right halves of the MNIST digit images as correlated variables X and Y .

Original: The goal is to obtain the expected left halves given the right halves, or vice versa.
Revised: The goal is to obtain the expected left halves given the right halves, or vice versa.

Revised: The training set was augmented by random small rotations and displacements to make the task more ambiguous, as we want to explore the uncertainty in the prediction.

Original: The features were represented by two identical convolutional neural networks with the same architecture as in the previous section.
Revised: The relevant variables were represented by two identical convolutional neural networks with the same architecture as in the previous section.

Original: After training the features, we used the training dataset to also compute the expected pixel gray value as well as their covariance for each feature using Eq.
Revised: After training, we used the training dataset to also compute the expected pixel gray value as well as their covariance for each relevant variable using Eq.

Original: These were used into Eq.
Revised: These were used into Eq.

Original: (10) to compute the mean pixel gray values and their covariances over the conditional probability of X given Y on test data.
Revised: (10) to compute the mean pixel gray values and their covariances over the conditional probability of X given Y on test data.

Original: This mean is the Bayesian estimator for the ldistance between half images, i.e., it should minimize the expected distance d l 2 over the conditional distribution, where d 2 l 2 ( x, y ) =
Revised: This mean is the Bayesian estimator for the ldistance between half images, i.e., it should minimize the expected distance d l 2 over the conditional distribution, where d 2 l 2 ( x, y ) =

Original: (cid:80) i
Revised: (cid:80) i

Original: ( x i − y i ) 2 , where x i ∈ [0 , 1] is the value of the i th pixel.
Revised: ( x i − y i ) 2 , where x i ∈ [0 , 1] is the value of the i th pixel.

Original: (This is equivalent to minimizing the mean square error).
Revised: (This is equivalent to minimizing the mean square error).

Original: The results are shown in Fig.
Revised: The results on a randomly selected subset of test digits is shown in Fig.

Original: For each example, we also computed the images obtained by adding plus or minus one standard deviation along the direction of greatest variance in the space of features.
Revised: For each example, we also computed the images obtained by adding plus or minus one standard deviation along the direction of greatest variance in the space of relevant variables.

Original: This reveals the main ambiguities (such as between 8 and 3 or 7 and 9 which share a similar right half).
Revised: This reveals the main ambiguities (such as between 8 and 3 or 7 and 9 which share a similar right half).

Original: The graph of the singular values shows that the rank cutoff of 120 is too low to capture all of the relevant features, but the results are reasonable nevertheless. An IPython notebook reproducing this result and more can be found in (B´eny).
Revised: The graph of the singular values shows that the rank cutoff of 200 is too low to capture all of the relevant variables (the sudden drop at the end is not robust to an increase in the cutoff), but the results are reasonable nevertheless. This shows that our representation of the conditional distributions contains valuable information besides the simple mean.

Original: 4. F EATURE DISCOVERY
Revised: 5. S TRUCTURE OF THE RELEVANT VARIABLES

Original: We mentioned in Section 2 that if p ( x, y ) is a two-dimensional Gaussian distribution with zero mean, then the n most relevant features of X are the ﬁrst n powers of X itself, independently of the covariance matrix.
Revised: We mentioned in Section 3 that if p ( x, y ) is a two-dimensional Gaussian distribution with zero mean, then the n most relevant variables of X are the ﬁrst n powers of X itself, independently of the covariance matrix.

Original: This implies that the singular features are the Hermite polynomials in X (which results from applying the Gram-Schmidt procedure to the basis { 1 , x, x 2 , . . . } ).[Figure]
Revised: This implies that the canonical variables are the Hermite polynomials in X (which results from applying the Gram-Schmidt procedure to the basis { 1 , x, x 2 , . . . } ).

Original: A similar property holds for multivariate Gaussians, namely, the less relevant singular values are polynomials in the more relevant ones.
Revised: A similar property holds for multivariate Gaussians, namely, the less relevant singular values are polynomials in the more relevant ones.

Original: If this is true more generally, it should be possible to further compress and organize the latent space extracted with RFA by ﬁnding a minimal set of generators, which ought to also be in the span of the most relevant features.
Revised: If this is true more generally, it should be possible to further compress and organize the latent space extracted with DCCA by ﬁnding a minimal set of generators, which ought to also be in the span of the most relevant variables.

Original: We applied RFA to a synthetic dataset to explore this idea, shown in Fig.
Revised: We applied DCCA to a synthetic dataset to explore this idea, shown in Fig.

Revised: In this case, we actually performed a ﬁnal SVD to obtain the unique uncorrelated canonical variables, and ordered them by decreasing relevance (their respective singular values).

Original: Here, X consists of two real numbers, distributed uniformly within a ring and a disk.
Revised: Here, X consists of two real numbers, distributed uniformly within a ring and a disk.

Original: The variable Y is obtained by adding a random Gaussian shift to X with a small standard deviation.
Revised: The variable Y is obtained by adding a random Gaussian shift to X with a small standard deviation.

Original: The more relevant features ought to be those which are more robust to such small random displacement.
Revised: The more relevant variables ought to be those which are more robust to such small random displacement.

Original: This formalizes the idea that we are interested in extracting “large-scale” features.
Revised: This formalizes the idea that we are interested in extracting “large-scale” variables B´eny (2018b).

Original: We would expect the relevant independent variables to be: the binary variable indicating whether the point is in the disk or the ring and the angle around the ring, followed by the radial component in the ring, and ﬁnally the Cartesian coordinates inside the disk.
Revised: We would expect the relevant independent variables to be: the binary variable indicating whether the point is in the disk or the ring and the angle around the ring, followed by the radial component in the ring, and ﬁnally the Cartesian coordinates inside the disk.

Original: This is precisely what we see in
Revised: This is precisely what we see in

Original: Fig.
Revised: Fig.

Original: Indeed—if we put aside for now the fact that the angle itself is not directly represented—besides the constant function, the two most relevant variables are the sine and cosine of the angle, followed by the binary variable separating the disk from the ring.
Revised: Indeed—if we put aside for now the fact that the angle itself is not directly represented—besides the constant function, the two most relevant variables are the sine and cosine of the angle, followed by the binary variable separating the disk from the ring.

Original: But these features ought to span the space of probabilities over the relevant variables, not just the variables themselves.
Revised: But these variables ought to span the space of probabilities over the relevant variables, not just the variables themselves.

Original: Hence the next six features are sines and cosines of smaller wavelength, which can encode probability distributions which are increasingly more precisely localized, down to a precision (wavelength) comparable with the diameter of the inner disk.
Revised: Hence the next six variables are sines and cosines of smaller wavelength, which can encode probability distributions which are increasingly more precisely localized, down to [Figure] reconstruction error vs latent dimension [Figure]

Revised: a precision (wavelength) comparable with the diameter of the inner disk.

Original: Accordingly, the next two most relevant features are the Cartesian coordinates inside the disk.
Revised: Accordingly, the next two most relevant variables are the Cartesian coordinates inside the disk.

Original: This is followed by additional moments of the angle, down to a wavelength equal to the ring’s thickness, at which point we see the radius in the ring appear.
Revised: This is followed by additional moments of the angle, down to a wavelength equal to the ring’s thickness, at which point we see the radius in the ring appear.

Original: As mentioned, we see that the angle itself is not represented, likely because it is discontinuous.
Revised: As mentioned, we see that the angle itself is not represented, likely because it is discontinuous.

Original: However, as shown also in Fig.
Revised: However, as shown also in Fig.

Original: 3, creating a gap in the ring allows for the angle to emerge as most relevant variable.
Revised: 3, creating a gap in the ring allows for the angle to emerge as most relevant variable.

Original: This suggests that this approach may be able to automatically learn intrinsic coordinates of the latent variable manifold.
Revised: This suggests that this approach may be able to automatically learn intrinsic coordinates of the latent variable manifold.

Original: 4. D ISENTANGLED FEATURES AND GENERATIVE MODEL
Revised: 5. I NDEPENDENT VARIABLES AND GENERATIVE MODEL

Original: If we postulate that the independent (or disentangled) relevant latent variables can be found in the linear span of the relevant features, we can attempt to extract them by optimizing a neural network composed of two parts.
Revised: If we postulate that the independent (or disentangled) relevant latent variables can be found in the linear span of the relevant variables, we can attempt to extract them by optimizing a neural network composed of two parts.

Original: Firstly, a linear layer maps the relevant features to a small number of outputs (equal to the latent dimension ).
Revised: Firstly, a linear layer maps the relevant variables to a small number of outputs (equal to the latent dimension ).

Original: The purpose of this linear layer is to ﬁnd the independent variables.
Revised: The purpose of this linear layer is to ﬁnd the independent variables.

Original: These latent variables are then processed by an arbitrarily complex generative network to produce a possible value of the variable X .
Revised: These latent variables are then processed by an arbitrarily complex generative network to produce a possible value of the variable X .

Original: As objective function, we may us an appropriate measure of similarity between the output and the data element from which the features were obtained.
Revised: As objective function, we may us an appropriate measure of similarity between the output and the data element from which the variables were obtained.

Original: We tested this idea as follows.
Revised: We tested this idea as follows.

Original: We took X to consist of the MNIST digits, and produced Y by randomly permuting neighboring pixels in the image, until the mean displacement per pixel is of order 1 .
Revised: We took X to consist of the MNIST digits, and produced Y by randomly permuting neighboring pixels in the image, until the mean displacement per pixel is of order 1 .

Original: In addition, we added independent Gaussian noise to the pixel values.
Revised: In addition, we added independent Gaussian noise to the pixel values.

Original: (Hence the noise map N simulates the coarse-graining channel introduced in B´eny & Osborne (2015a)).
Revised: (Hence the noise map N simulates the coarse-graining channel introduced in B´eny & Osborne (2015a)).

Original: As in the previous experiment, we do so to implement our intuition that the more relevant features ought to be the ones which are of larger scale, or more robust to local perturbations.
Revised: As in the previous experiment, we do so to implement our intuition that the more relevant variables ought to be the ones which are of larger scale, or more robust to local perturbations.

Original: The features of the clean images were produced by the same convolutional neural network as in Section 4.1, while the features of the coarse-grained images were extracted by a network of the same geometry, but with half the number of ﬁlters and neurons.
Revised: The relevant variables of the clean images were produced by the same convolutional neural network as in Section 5.1, while the variables of the coarse-grained images were extracted by a network of the same geometry, but with half the number of ﬁlters and neurons.

Original: We extracted the 1000 most relevant out of 1200 learned features in this way.
Revised: We extracted the 1000 most relevant out of 1200 learned variables in this way.

Original: (The least relevant features in this system happen to be highly dependent on the total number of features and hence cannot be trusted to be correct).
Revised: (The least relevant variables in this system happen to be highly dependent on the total number of variables and hence cannot be trusted to be correct).

Original: As a second step, we trained a linear layer coupled to a network composed of 5 fully-connected layers of 800 hidden neurons each.
Revised: As a second step, we trained a linear layer coupled to a network composed of 5 fully-connected layers of 800 hidden neurons each.

Original: We refer to the number of output neurons in the ﬁrst linear layer as the latent dimension .
Revised: We refer to the number of output neurons in the ﬁrst linear layer as the latent dimension .

Original: [Figure] As input, this network received the features extracted from MNIST images using the above convolutional neural net (after it was fully trained using RFA), and was trained to minimize the mean square error between its output and the original MNIST digit.
Revised: As input, this network received the variables extracted from MNIST images using the above convolutional neural net (after it was fully trained using DCCA), and was trained to minimize the mean square error between its output and the original MNIST digit.

Original: The resulting best mean square errors are shown in Fig.
Revised: The resulting best mean square errors are shown in Fig.

Original: 4, as function of the latent dimension.
Revised: 4, as function of the latent dimension.

Original: Here we see a distinct change of polynomial scaling law at dimension 19 .
Revised: Here we see a distinct change of polynomial scaling law at dimension 19 .

Original: Increasing the dimension further provides no improvement.
Revised: Increasing the dimension further provides no improvement.

Original: This behaviour is compatible with our hypothesis that the extra features are just functions of those ﬁrst twenty features (functions which are effectively re-implemented by the generative network).
Revised: This behaviour is compatible with our hypothesis that the extra variables are just functions of those ﬁrst twenty variables (functions which are effectively re-implemented by the generative network).

Original: Images generated by sampling from a Gaussian approximation of the latent distribution for different latent dimensions are shown in Fig.
Revised: Images generated by sampling from a Gaussian approximation of the latent distribution for different latent dimensions are shown in Fig.

Original: Below dimenson 20 , most generated image can be recognized as a speciﬁc digit.
Revised: Below dimension 20 , most generated image can be recognized as a speciﬁc digit.

Original:  O UTLOOK
Revised:  O UTLOOK

Original: We only brushed a few potential applications of RFA.
Revised: We studied the classical (non-quantum) form of the theory introduced in B´eny & Osborne (2013), and found that the relevant observables of that theory are just the most correlated canonical variables in the sense of DCCA Andrew et al. (2013), and can be learned effectively using standard machine learning methods. This point of views on DCCA provided us with several new insights. The ﬁrst is that the learned relevant variables provide a useful representation of a joint probability distribution.

Original: More detailed studies will be needed to see if it can be used to obtain state-of-the art results for inference, disentangled feature extraction or for data generation.
Revised: We showed that performing inference using this representation can outperform crossentropy in predicting classes. Our experiments on halves of MNIST also show that the conditional distribution we obtain can effectively represent the uncertainty in the prediction of high-dimensional data. A second insight relates to the interpretation of the canonical variables as spanning directions in the space of probability distributions. As suggested by the gaussian solutions and our experiment on synthetic data, we postulate that the canonical variables are functions of a small number of independent generators contained in their span.

Original: The regularizing effect for supervised learning also warrants further study.
Revised: This hypothesis is supported by our experiment on MNIST, but further work is required to ﬁnd a way to cleanly extract these variables.

Original: We have not explored potential applications of one of the salient aspect of RFA, namely the fact thatit provides the functions which can be most reliably predicted.
Revised: We have yet to explore the potential applications of one of the salient aspect of this approach to inference, the fact that the canonical variables learned using DCCA are also those which can be most reliably predicted, irrespective of the value of the cutoff.

Original: To see why this is potentially signiﬁcant, we observe that a central feature of scientiﬁc exploration is that the properties of a system which can be predicted and understood are not  given a priori, but must be discovered.
Revised: To see why this is potentially signiﬁcant, we observe that a central feature of scientiﬁc exploration is that we are not so concerned with making predictions about some given variables, as much as we are with discovering variables which can be predicted.

Original: Another important feature of RFA is the fact that the resulting model allows for the direct evaluation of the expectation values in the posterior distribution without sampling.
Revised: Another important feature of this approach is the fact that the resulting model allows for the direct evaluation of the expectation values in the posterior distribution without sampling.

Original: In particular this allows for the evaluation of credible intervals.
Revised: In particular this allows for the evaluation of credible intervals.

Original: Hence it should be especially suited to scientiﬁc applications where the ability to quantify uncertainty is essential.
Revised: Hence it should be especially suited to scientiﬁc applications where the ability to quantify uncertainty is essential.

Original: The fact that RFA is grounded in a consistent information-theoretical analysis which comes with a large class of analytically solvable examples (namely all Gaussian joint distributions), presents exciting potentials for further reﬁnements and developments of the approach. Another interesting aspect of this approach is thatthe theory it is based on has a complete quantumtheoretical formulation. Hence it would be natural to ﬁnd extensions of the present work to contexts where the data is quantum, or results from quantum measurements.
Revised: Finally, the relationship that we established with theory of quantum origin points towards a potential quantum generalization of DCCA that would apply to quantum data, or classical measurements of quantum systems.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We would like to thank Jo¨el B´eny and Raban Iten for helpful suggestions.
Revised: We would like to thank Jo¨el B´eny and Raban Iten for helpful suggestions.

Revised: We are also indebted to an anonymous ICLR2020 referee for pointing out the connection between our approach and DCCA.

Original: This work was supported by the National Research Foundation of Korea (NRF-2018R1D1A1A02048436).
Revised: This work was supported by the National Research Foundation of Korea (NRF2018R1D1A1A02048436).

Original: A E XTRA INFORMATION ABOUT THE ALGORITHM A. A LTERNATIVE INTERPRETATION OF THE LOSS
Revised: A E XTRA INFORMATION ABOUT THE ALGORITHM A. A LTERNATIVE INTERPRETATION OF THE OBJECTIVE

Original: If we write F ij := f j ( x i ) and G ij := g j ( y i ) for the value of the features on the dataset then
Revised: If we write F ij := f j ( x i ) and G ij := g j ( y i ) for the value of our variables on the dataset then

Original: K = 1 N F (cid:62) F , L = 1 N G (cid:62) G and A = 1 N G (cid:62) F .
Revised: K = 1 N F (cid:62) F , L = 1 N G (cid:62) G and A = 1 N G (cid:62) F .

Original: The relevance can then be written as
Revised: The DCCA objective can then be written as

Original: [Equation]
Revised: [Equation]

Original: where P = F ( F (cid:62) F ) − 1 F (cid:62) and Q = G ( G (cid:62) G )
Revised: where P = F ( F (cid:62) F ) − 1 F (cid:62) and Q = G ( G (cid:62) G )

Original: − 1 G (cid:62) are the projectors on the ranges of F and
Revised: − 1 G (cid:62) are the projectors on the ranges of F and

Original: G respectively.
Revised: G respectively.

Original: Hence we are maximizing the overlap between those ranges (which represents possible linear combinations of datapoints, respectively determined from features of one or the other correlated variables
Revised: Hence, we are maximizing the overlap between those ranges (which represent possible linear combinations of datapoints, respectively determined from variables of one or the other correlated views.)

Original: A. H EURISTIC
Revised: A. H EURISTIC

Original: Batch size —In our experiments, we observed that the batch size during training needs to be an order of magnitude larger than the number of features (rank cutoff).
Revised: Batch size —In our experiments, we observed that the batch size during training needs to be an order of magnitude larger than the number of variables (rank cutoff).

Original: When the batch size was too small, learning seemed to converge normally in terms of training and test loss, but resulted in features which yield dramatically different losses when evaluated on larger batches, and yield spurious predictions.
Revised: When the batch size was too small, learning seemed to converge normally in terms of training and test loss, but resulted in variables which yield dramatically different losses when evaluated on larger batches, and yield spurious predictions.

Original: Constant features —The loss function C takes value between 0 and k 0 − 1 because the constant feature always has relevance 1 .
Revised: Constant variables —The loss function C takes value between 0 and k 0 − 1 because the constant variable always has relevance 1 .

Original: The constant feature could be enforced a priori rather than learned, which, due to the objective, automatically forces the learned features to have zero expectation values (be orthogonal to the constant feature).
Revised: The constant variable could be enforced a priori rather than learned, which, due to the objective, automatically forces the learned variables to have zero expectation values (be orthogonal to the constant variable).

Original: This might have advantages in certain circumstances, but in our experiments we found that this sometime hindered convergence.
Revised: This might have advantages in certain circumstances, but in our experiments we found that this sometime hindered convergence.

Original: Invertibility issues —The covariance matrices K and L can be ill-conditioned, potentially causing the gradient to “explode” because of the inverses K − 1 et L − 1 involved in the loss function.
Revised: Invertibility issues —The covariance matrices K and L can be ill-conditioned, potentially causing the gradient to “explode” because of the inverses K − 1 et L − 1 involved in the loss function.

Original: This can be avoided either by using the Moore-Penrose pseudo-inverse, or by replacing K − 1 by ( K + (cid:15) 1 ) −in the loss for some small positive number (cid:15) , and likewise for L − 1 .
Revised: This can be avoided either by using the Moore-Penrose pseudo-inverse, or by replacing K − 1 by ( K + (cid:15) 1 ) −in the loss for some small positive number (cid:15) , and likewise for L − 1 .

Original: Symmetries in the loss function —The loss C only depends on the span of the features f i and g j , hence it has a very large group of symmetries.
Revised: Symmetries in the loss function —The loss C only depends on the span of the variables f i and g j , hence it has a very large group of symmetries.

Original: In particular, it is invariant under a change of the norm of each features independently from each other.
Revised: In particular, it is invariant under a change of the norm of each variable independently from each other.

Original: Because of that, it is preferable not to have a linear last layer.
Revised: Because of that, it is preferable not to have a linear last layer.

Original: Using a hyperbolic tangent as last nonlinearity worked in our experiments.
Revised: Using a hyperbolic tangent as last nonlinearity worked in our experiments.

Original: Regularization —In all our tests, dropout had no beneﬁcial effect.
Revised: Regularization —In all our tests, dropout had no beneﬁcial effect.

Original: In fact, our objective seems to already provide a form of regularization, as shown in Section 4.1.
Revised: In fact, our objective seems to already provide a form of regularization, as shown in Section 5.1.

Original: B T HEORY IN MORE DETAILS
Revised: B T HEORY IN MORE DETAILS

Original: We consider two correlated random variables X and Y with a joint probability distribution p ( x, y ) .
Revised: We consider two correlated random variables X and Y with a joint probability distribution p ( x, y ) .

Original: We assume that we are able to numerically evaluate expectations with respect to this distribution, for instance because we can sample from it.
Revised: We assume that we are able to numerically evaluate expectations with respect to this distribution, for instance because we can sample from it.

Original: We want to use this ability in order to compute expectations with respect to the conditional distributions p X | Y ( x | y ) =
Revised: We want to use this ability in order to compute expectations with respect to the conditional distributions p X | Y ( x | y ) =

Original: p ( x, y ) /p
Revised: p ( x, y ) /p

Original: X ( x ) and p Y | X ( y | x ) =
Revised: X ( x ) and p Y | X ( y | x ) =

Original: p ( x, y ) /p
Revised: p ( x, y ) /p

Original: Y ( y ) , where p X ( x ) =
Revised: Y ( y ) , where p X ( x ) =

Original: (cid:80) y p ( x, y ) and p Y ( y )
Revised: (cid:80) y p ( x, y ) and p Y ( y )

Original: =
Revised: =

Original: (cid:80) x p ( x, y ) are the marginals of p .
Revised: (cid:80) x p ( x, y ) are the marginals of p .

Original: Below we sometime remove the subscripts X , X | Y or Y | X if there is no ambiguity.
Revised: Below we sometime remove the subscripts X , X | Y or Y | X if there is no ambiguity.

Original: For instance, suppose we generated samples of y given x , through explicit knowledge of p Y | X .
Revised: For instance, suppose we generated samples of y given x , through explicit knowledge of p Y | X .

Original: Then the evaluation of expectations with respect to p X | Y is the subject of Bayesian inference.
Revised: Then the evaluation of expectations with respect to p X | Y is the subject of Bayesian inference.

Original: However, this is generally done in a context where the variable X has low dimensionality and parameterizes a hand-crafted model.
Revised: However, this is generally done in a context where the variable X has low dimensionality and parameterizes a hand-crafted model.

Original: Our approach, however, is free of such a model and the variable X can be of very high dimensionality.
Revised: Our approach, however, is free of such a model and the variable X can be of very high dimensionality.

Original: B. I NNER PRODUCT ON PROBABILITY VECTORS
Revised: B. I NNER PRODUCT ON PROBABILITY VECTORS

Original: In order to deﬁne our strategy, we need to equip the spaces of probability distributions for X and Y with an inner product structure.
Revised: In order to deﬁne our strategy, we need to equip the spaces of probability distributions for X and Y with an inner product structure.

Original: Let us focus on X , and assume that it takes discrete values to avoid unnecessary technicalities.
Revised: Let us focus on X , and assume that it takes discrete values to avoid unnecessary technicalities.

Original: The set of probability vectors is a convex subset of the real linear space
Revised: The set of probability vectors is a convex subset of the real linear space

Original: V X =
Revised: V X =

Original: R n .
Revised: R n .

Original: Let us equip this space with the product
Revised: Let us equip this space with the product

Original: [Equation]
Revised: [Equation]

Original: for any µ, µ (cid:48) ∈ V X .
Revised: for any µ, µ (cid:48) ∈ V X .

Original: We also write (cid:107) µ (cid:107) 2 X
Revised: We also write (cid:107) µ (cid:107) 2 X

Original: =
Revised: =

Original: (cid:104) µ, µ (cid:105) X .
Revised: (cid:104) µ, µ (cid:105) X .

Original: Importantly, this depends explicitly on the ﬁxed probability
Revised: Importantly, this depends explicitly on the ﬁxed probability

Original: vector p X
Revised: vector p X

Original: ( x ) , which we took to be the marginal of p ( x, y ) .
Revised: ( x ) , which we took to be the marginal of p ( x, y ) .

Original: If p X has full support, this makes V X into a real inner product space.
Revised: If p X has full support, this makes V X into a real inner product space.

Original: The same can be done for the variable Y , yielding the inner product (cid:104) ν, ν (cid:48)
Revised: The same can be done for the variable Y , yielding the inner product (cid:104) ν, ν (cid:48)

Original: (cid:105) Y for ν, ν (cid:48)
Revised: (cid:105) Y for ν, ν (cid:48)

Original: ∈ V Y .
Revised: ∈ V Y .

Original: Had we interpreted µ and µ (cid:48) as tangent vectors to V X , considered as a manifold, this would be the
Revised: Had we interpreted µ and µ (cid:48) as tangent vectors to V X , considered as a manifold, this would be the

Original: Fisher information (Riemannian) metric, as in B´eny & Osborne (2015b).
Revised: Fisher information (Riemannian) metric, as in B´eny & Osborne (2015b).

Original: But this quantity is also meaningful for ﬁnite vectors: the induced norm distance between p X and any probability vector q is the χ 2 -divergence:
Revised: But this quantity is also meaningful for ﬁnite vectors: the induced norm distance between p X and any probability vector q is the χ 2 -divergence:

Original: [Equation]
Revised: [Equation]

Original: It measures how distinguishable q is from p X in the contexts of the Pearson χ 2 -test.

Original: Speciﬁcally, it quantiﬁes how easy it is to reject the null hypothesis that the state is p X when it is actually q , based on the empirical distribution obtained from independent samples.

Original: The set of conditional probability distributions p Y | X form a stochastic map, i.e., a linear map N :
Revised: The set of conditional probability distributions p Y | X form a stochastic map, i.e., a linear map N :

Original: V X → V Y , µ (cid:55)→ N ( µ ) , where
Revised: V X → V Y , µ (cid:55)→ N ( µ ) , where

Original: [Equation]
Revised: [Equation]

Original: ) for any µ ∈ V X .
Revised: 13) for any µ ∈ V X .

Original: It is straightforward to check that the stochastic map N ∗ deﬁned by
Revised: It is straightforward to check that the stochastic map N ∗ deﬁned by

Original: [Equation]
Revised: [Equation]

Original: ) is the transpose N ∗ of N with respect to the inner products we deﬁned (Ohya & Petz, 2004), i.e., for all ν ∈ V Y and µ ∈ V X ,
Revised: 14) is the transpose N ∗ of N with respect to the inner products we deﬁned (Ohya & Petz, 2004), i.e., for all ν ∈ V Y and µ ∈ V X ,

Original: [Equation]
Revised: [Equation]

Original: Also, we observe that N ( p X ) =
Revised: Also, we observe that N ( p X ) =

Original: p Y and N ∗ ( p Y )
Revised: p Y and N ∗ ( p Y )

Original: =
Revised: =

Original: p X .
Revised: p X .

Original: B. E IGEN - RELEVANCE DECOMPOSITION
Revised: B. E IGEN - RELEVANCE DECOMPOSITION

Original: We can use the inner products on V X and V Y to deﬁne a singular value decomposition of the stochastic map N .
Revised: We can use the inner products on V X and V Y to deﬁne a singular value decomposition of the stochastic map N .

Original: That is, there is an orthonormal family u 1 , . . .
Revised: That is, there is an orthonormal family u 1 , . . .

Original: , u k of V X and an orthonormal family v 1 , . . .
Revised: , u k of V X and an orthonormal family v 1 , . . .

Original: , v k of V Y , such that
Revised: , v k of V Y , such that

Original: [Equation]
Revised: [Equation]

Original:  for j = 1 , . ..
Revised: ) for j = 1 , . .

Original: , k .
Revised: k .

Original: For each j , η j is a singular value of N , whose square we call the relevance of the vector v j .
Revised: For each j , η j is a singular value of N , whose square we call the relevance of the vector v j .

Original: Moreover η j ∈ [0 , 1] since the χ 2 divergence is contractive under any stochastic map.
Revised: Moreover η j ∈ [0 , 1] since the χ 2 divergence is contractive under any stochastic map.

Original: Given that N ∗ is the transpose of N :
Revised: Given that N ∗ is the transpose of N :

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Equivalently, u j is an eigenvector of N ∗ ◦ N and v j is an eigenvectors of N ◦ N ∗ , both with eigenvalue η 2 j .
Revised: Equivalently, u j is an eigenvector of N ∗ ◦ N and v j is an eigenvectors of N ◦ N ∗ , both with eigenvalue η 2 j .

Original: Because N maps p X to p Y , we always have the dual eigenvectors u 0 = p X
Revised: Because N maps p X to p Y , we always have the dual eigenvectors u 0 = p X

Original: and v 0 =
Revised: and v 0 =

Original: p Y with eigenvalue 1 .
Revised: p Y with eigenvalue 1 .

Original: B. L OW - RANK APPROXIMATION
Revised: B. L OW - RANK APPROXIMATION

Original: Typically, the dimension k of the space of probabilities is more than astronomically large.
Revised: Typically, the dimension k of the space of probabilities is more than astronomically large.

Original: For instance, if the values of X consists of small 256 gray level images of 28 × 28 pixels, then k = 256 28 2 (cid:39) 10 1888 .
Revised: For instance, if the values of X consists of small 256 gray level images of 28 × 28 pixels, then k = 256 28 2 (cid:39) 10 1888 .

Original: However, in many case, only very few of these dimensions may be relevant for the purpose of inferring other variables.
Revised: However, in many case, only very few of these dimensions may be relevant for the purpose of inferring other variables.

Original: The core of our approach is to approximate N and N ∗ by restricting them to the span of the ﬁrst keigenvectors u j and v j with largest singular values η j .
Revised: The core of our approach is to approximate N and N ∗ by restricting them to the span of the ﬁrst keigenvectors u j and v j with largest singular values η j .

Original: That is, if we order the singular values η j , j = 1 , . . .
Revised: That is, if we order the singular values η j , j = 1 , . . .

Original: , k in decreasing order, we propose to use the approximations
Revised: , k in decreasing order, we propose to use the approximations

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] to N and N ∗ respectively, for some k 0 typically much smaller than k , and any µ ∈ V X , ν ∈ V Y .
Revised: to N and N ∗ respectively, for some k 0 typically much smaller than k , and any µ ∈ V X , ν ∈ V Y .

Original: We denote the components of N 0 and N ∗ 0 by q ( y | x ) and q ( x | y ) , e.g.,
Revised: We denote the components of N 0 and N ∗ 0 by q ( y | x ) and q ( x | y ) , e.g.,

Original: [Equation]
Revised: [Equation]

Original: Since N 0 and N ∗ 0 are adjoint, we can deﬁne q ( x, y )
Revised: Since N 0 and N ∗ 0 are adjoint, we can deﬁne q ( x, y )

Original: =
Revised: =

Original: q ( x | y ) p Y
Revised: q ( x | y ) p Y

Original: ( y ) =
Revised: ( y ) =

Original: q ( y | x ) p X ( x ) .
Revised: q ( y | x ) p X ( x ) .

Original: Although the marginals of q ( x, y ) are the probability distributions p X and p Y , the numbers q ( x, y ) are not necessarily positive.
Revised: Although the marginals of q ( x, y ) are the probability distributions p X and p Y , the numbers q ( x, y ) are not necessarily positive.

Original: The quality of this approximation for a given k 0 does not directly depend on the dimensionality of
Revised: The quality of this approximation for a given k 0 does not directly depend on the dimensionality of

Original: X and Y , but only on the amount of correlations between the two variables.
Revised: X and Y , but only on the amount of correlations between the two variables.

Original: Our aim is to use a ksmall enough that the components of N 0 and N ∗ 0 can be computed explicitly.
Revised: Our aim is to use a ksmall enough that the components of N 0 and N ∗ 0 can be computed explicitly.

Original: Theorem 1.
Revised: Theorem 1.

Original: N 0 is the map of rank k 0 which minimizes the average distance
Revised: N 0 is the map of rank k 0 which minimizes the average distance

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The low rank approximation N 0 minimizes the distance (cid:107)N 0 − N(cid:107) F where
Revised: The low rank approximation N 0 minimizes the distance (cid:107)N 0 − N(cid:107) F where

Original: [Equation]
Revised: [Equation]

Original: 23) is the Hilbert-Schmidt (or Frobenius) norm (Eckart & Young, 1936).
Revised: 23) is the Hilbert-Schmidt (or Frobenius) norm (Eckart & Young, 1936).

Original: This follows from the fact that this is also the l 2 -norm of the vector of singular values of M .
Revised: This follows from the fact that this is also the l 2 -norm of the vector of singular values of M .

Original: Let us ﬁnd the explicit form of the trace.
Revised: Let us ﬁnd the explicit form of the trace.

Original: Each possible value x of the variable X is associated with a probability distribution δ x ( y )
Revised: Each possible value x of the variable X is associated with a probability distribution δ x ( y )

Original: = 1 when x = y and zero otherwise.
Revised: = 1 when x = y and zero otherwise.

Original: These distributions form an orthogonal basis of V X , and have norms (cid:104) δ x , δ x (cid:105)
Revised: These distributions form an orthogonal basis of V X , and have norms (cid:104) δ x , δ x (cid:105)

Original: =
Revised: =

Original: 1 /p
Revised: 1 /p

Original: X ( x ) .
Revised: X ( x ) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: B. F EATURES
Revised: B. R ELEVANT VARIABLES

Original: We express the elements µ ∈ V X and ν ∈ V Y in terms of the marginals p X and p Y as simple products:[Equation]
Revised: We express the elements µ ∈ V X and ν ∈ V Y in terms of the marginals p X and p Y as simple products:

Original: [Equation]
Revised: [Equation]

Original: for all x, y , where f and g are real functions of x and ywhich we call features .
Revised: for all x, y , where f and g are real functions of x and y .

Original: The inner products then simply become correlations among features.
Revised: The inner products then simply become correlations among variables.

Original: Using also µ (cid:48)
Revised: Using also µ (cid:48)

Original: =
Revised: =

Original: p X f (cid:48) and ν (cid:48)
Revised: p X f (cid:48) and ν (cid:48)

Original: =
Revised: =

Original: p Y g (cid:48) , we obtain
Revised: p Y g (cid:48) , we obtain

Original: [Equation]
Revised: [Equation]

Original: These are simple expectation values with respect to p , which we assumed is the type of quantity we can evaluate for arbitrary functions f, f (cid:48) , g, g (cid:48) .
Revised: These are simple expectation values with respect to p , which we assumed is the type of quantity we can evaluate for arbitrary functions f, f (cid:48) , g, g (cid:48) .

Original: Since N ∗ N is self-adjoint in terms of this inner product, its eigenvectors
Revised: Since N ∗ N is self-adjoint in terms of this inner product, its eigenvectors

Original: u i are orthogonal, and hence the corresponding features a i deﬁned by u i ( x ) =
Revised: u i are orthogonal, and hence the corresponding variables a i deﬁned by u i ( x ) =

Original: p X ( x ) a i ( x ) are uncorrelated.
Revised: p X ( x ) a i ( x ) are uncorrelated.

Original: Indeed,
Revised: Indeed,

Original: [Equation]
Revised: [Equation]

Original: 27) for all i, j .
Revised: 27) for all i, j .

Original: Moreover, accounting for the eigenvector u 0 = p X (corresponding to the constant feature
Revised: Moreover, accounting for the eigenvector u 0 = p X (corresponding to the constant feature

Original: a 0 ( x ) = 1 for all x ),
Revised: a 0 ( x ) = 1 for all x ),

Original: [Equation]
Revised: [Equation]

Original: for all i (cid:54) = 0 .
Revised: for all i (cid:54) = 0 .

Original: Hence we trivially have
Revised: Hence we trivially have

Original: [Equation]
Revised: [Equation]

Original: (29) for all i, j (cid:54) = 0 .
Revised: (29) for all i, j (cid:54) = 0 .

Original: Likewise for the eigenvectors of NN ∗ .
Revised: Likewise for the eigenvectors of NN ∗ .

Original: If v i ( y ) =
Revised: If v i ( y ) =

Original: p Y ( y ) b i ( y ) :
Revised: p Y ( y ) b i ( y ) :

Original: [Equation]
Revised: [Equation]

Original: (30) for all i, j (cid:54) = 0 .
Revised: (30) for all i, j (cid:54) = 0 .

Original: Importantly, this does not mean that the features u 1 , u 2 , . . .
Revised: Importantly, this does not mean that the variables u 1 , u 2 , . . .

Original: nor v 1 , v 2 , . .
Revised: nor v 1 , v 2 , . .

Original: . are “disentangled”, i.e., they are not statistically independent.
Revised: . are “disentangled”, i.e., they are not statistically independent.

Original: These features represent components in the space of probability vectors, rather than the “sample” space.
Revised: These variables represent components in the space of probability vectors, rather than the “sample” space.

Original: They should be understood as spanning a subspace of the space of functions over the relevant independent variables.
Revised: They should be understood as spanning a subspace of the space of functions over the relevant independent variables.

Original: We discuss this in more detail in
Revised: We discuss this in more detail in

Original: Section 4.3.
Revised: Section 5.3.

Original: B. C ORNERS OF N AND LOSS FUNCTION
Revised: B. C ORNERS OF N AND LOSS FUNCTION

Original: The ﬁnal piece of puzzle we need, is the ability to express the components (corners) of N and N ∗ in the span of possible non-orthogonal families of features.
Revised: The ﬁnal piece of puzzle we need, is the ability to express the components (corners) of N and N ∗ in the span of possible non-orthogonal families of variables.

Original: Let us therefore consider two arbitrary families f 1 , . . .
Revised: Let us therefore consider two arbitrary families f 1 , . . .

Original: , f k 0 and g 1 , . .
Revised: , f k 0 and g 1 , . .

Original: , g k 0 of features, which respectively represent the vectors p X f j ∈ V X and p Y g j ∈ V Y .
Revised: , g k 0 of variables, which respectively represent the vectors p X f j ∈ V X and p Y g j ∈ V Y .

Original: Firstly, we need matrices representing the components of the inner products on V X and V Y .
Revised: Firstly, we need matrices representing the components of the inner products on V X and V Y .

Original: Those are the symmetric matrices
Revised: Those are the symmetric matrices

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The components N ij of N are deﬁned by
Revised: The components N ij of N are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Taking the inner product with p Y g k , we obtain
Revised: Taking the inner product with p Y g k , we obtain

Original: [Equation]
Revised: [Equation]

Original: The left-hand side can be computed using Equ. 13.
Revised: The left-hand side can be computed using Equ. 13.

Original: It is the matrix
Revised: It is the matrix

Original: [Equation]
Revised: [Equation]

Original: Therefore, in matrix notation, Equ. (34) is A = LN , or
Revised: Therefore, in matrix notation, Equ. (34) is A = LN , or

Original: [Equation]
Revised: [Equation]

Original: The components N ∗ ij of N ∗ are obtained by just swapping X and Y , yielding
Revised: The components N ∗ ij of N ∗ are obtained by just swapping X and Y , yielding

Original: [Equation]
Revised: [Equation]

Original: Hence the singular values of the corner of N deﬁned by the features f j and g j are just the squareroot of the eigenvalues of the matrix N ∗ N = K − 1 A (cid:62) L − 1 A .
Revised: Hence the singular values of the corner of N deﬁned by the variables f j and g j are just the squareroot of the eigenvalues of the matrix N ∗ N = K − 1 A (cid:62) L − 1 A .

Original: In order to ﬁnd the features f j and g j with the same span as the ﬁrst k 0 eigenvectors u j , v j , we just need to maximize all the eigenvalues of N ∗ N .
Revised: In order to ﬁnd the variables f j and g j with the same span as the ﬁrst k 0 eigenvectors u j , v j , we just need to maximize all the eigenvalues of N ∗ N .

Original: A simple way to do this is to use (minus) the trace of N ∗ N as loss function, since it is the sum of the square of the singular values.
Revised: A simple way to do this is to use (minus) the trace of N ∗ N as loss function, since it is the sum of the square of the singular values.

Original: We call Tr ( N ∗ N ) the relevance of the subspaces deﬁnes by the features f j ad g i for all i, j .
Revised: We call Tr ( N ∗ N ) the relevance of the subspaces deﬁnes by the variables f j ad g i for all i, j .

Original: This yields the loss/cost function:
Revised: This yields the loss/cost function:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Once optimal features have been found, one can obtain the components of the eigenvectors in the span of f 1 , . . .
Revised: Once optimal variables have been found, one can obtain the components of the eigenvectors in the span of f 1 , . . .

Original: , f k 0 through standard numerical diagonalization of N ∗ N .
Revised: , f k 0 through standard numerical diagonalization of N ∗ N .

Original: B. I NFERENCE
Revised: B. I NFERENCE

Original: The features minimizing C can be used to infer one variable from the other.
Revised: The variables minimizing C can be used to infer one variable from the other.

Original: For instance, given y , the inferred probability distribution over x is given by p X | Y ( x | y )
Revised: For instance, given y , the inferred probability distribution over x is given by p X | Y ( x | y )

Original: =
Revised: =

Original: N ∗ ( δ y )( x ) , where δ y ( y (cid:48) ) is 1 when y = y (cid:48) and zero otherwise.
Revised: N ∗ ( δ y )( x ) , where δ y ( y (cid:48) ) is 1 when y = y (cid:48) and zero otherwise.

Original: In order to compute this, we ﬁrst need the components of the distribution δ y in terms of the family p Y g 1 , . . .
Revised: In order to compute this, we ﬁrst need the components of the distribution δ y in terms of the family p Y g 1 , . . .

Original: , p Y g k 0 , i.e., the real numbers ( δ y )
Revised: , p Y g k 0 , i.e., the real numbers ( δ y )

Original: j such that
Revised: j such that

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) r, p Y δ i (cid:105) Y = 

Revised: where (cid:104) r, p Y δ i (cid:105) Y =

Original: for all i .
Revised: for all i .

Original: Taking the inner product with p Y g j , we obtain
Revised: Taking the inner product with p Y g j , we obtain

Original: [Equation]
Revised: [Equation]

Revised: (40) where the left hand side is also just (cid:104)

Original: [Equation]
Revised: [Equation]

Original: where the left hand side is also just (cid:104)
Revised: = g [Equation] ( y )

Original: [Equation]
Revised: [Equation]

Original: Therefore the components of δ y are explicitly
Revised: Therefore the components of δ y are explicitly

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Then, for instance, the expected inferred value of X is
Revised: Then, for instance, the expected inferred value of X is

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: For the inference of Y from x , we have
Revised: For the inference of Y from x , we have

Original: [Equation]
Revised: [Equation]

Original: C A NALYTICAL EXAMPLE
Revised: C A NALYTICAL EXAMPLE

Original: When p ( x, y ) is any multivariate Gaussian distribution, everything can be computed analytically.
Revised: When p ( x, y ) is any multivariate Gaussian distribution, everything can be computed analytically.

Original: Let us consider here the one-dimensional case.
Revised: Let us consider here the one-dimensional case.

Original: We use p ( x ) ∝ exp (cid:0) − x 2 / 2 τ 2 (cid:1) , and the conditional p ( y | x ) ∝ exp (cid:0) − ( y − x ) 2 / 2 σ 2 (cid:1) .
Revised: We use p ( x ) ∝ exp (cid:0) − x 2 / 2 τ 2 (cid:1) , and the conditional p ( y | x ) ∝ exp (cid:0) − ( y − x ) 2 / 2 σ 2 (cid:1) .

Original: That is, y is equal to x but with some added Gaussian noise.
Revised: That is, y is equal to x but with some added Gaussian noise.

Original: This gives
Revised: This gives

Original: [Equation]
Revised: [Equation]

Original: It was show in B´eny & Osborne (2013), that the most relevant subspace of dimension k 0 on the variable X is simply spanned by the features
Revised: It was show in Lancaster (1958), that the most relevant subspace of dimension k 0 on the variable X is simply spanned by the variables

Original: [Equation]
Revised: [Equation]

Revised: n = 0 , . .

Revised: k 0 − 1 .

Original: Similarly for Y
Revised: Similarly for Y

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This independence of the relevant features on the detailed parameters of p is a general property of
Revised: This independence of the relevant variables on the detailed parameters of p is a general property of

Original: Gaussian joint distributions.
Revised: Gaussian joint distributions.

Original: This means, for instance, that the most relevant feature ( n = 1 ) for predicting the value of X given
Revised: This means, for instance, that the most relevant feature ( n = 1 ) for predicting the value of X given

Original: Y =
Revised: Y =

Original: y is simply Y itself.
Revised: y is simply Y itself.

Original: The higher order features have to do with inferring extra aspects of the probability distribution over X .
Revised: The higher order variables have to do with inferring extra aspects of the probability distribution over X .

Original: A set of orthogonal features can be obtain from the Gram-Schmidt procedure, which, if done from small to large n much necessarily yield the eigenvectors u n and v n .
Revised: A set of orthogonal variables can be obtain from the Gram-Schmidt procedure, which, if done from small to large n much necessarily yield the eigenvectors u n and v n .

Original: For illustration purpose, let us work with the non-orthogonal vectors f n and g n , keeping only the ﬁrst k 0 = 3 vectors.
Revised: For illustration purpose, let us work with the non-orthogonal vectors f n and g n , keeping only the ﬁrst k 0 = 3 vectors.

Original: The three matrices (correlators) we need can be easily computed:
Revised: The three matrices (correlators) we need can be easily computed:

Original: [Equation]
Revised: [Equation]

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: The eigenvalues of M can be read on the diagonal, and the corresponding eigenvectors are (1 , 0 , 0) , (0 , 1 , 0) and ( − τ 2 , 0 , 1) , which means that the eigen-features are in order u 0 ( x ) = 1 , u 1 ( x ) = x and u 2 ( x ) =
Revised: The eigenvalues of M can be read on the diagonal, and the corresponding eigenvectors are (1 , 0 , 0) , (0 , 1 , 0) and ( − τ 2 , 0 , 1) , which means that the eigenfunctions are in order u 0 ( x ) = 1 , u 1 ( x ) = x and u 2 ( x ) =

Original: x 2 − τ 2 .
Revised: x 2 − τ 2 .

Original: Because we are working with continuous variables, the true rank of N is inﬁnite, even for any ﬁnite cutoff on the singular values.
Revised: Because we are working with continuous variables, the true rank of N is inﬁnite, even for any ﬁnite cutoff on the singular values.

Original: Nevertheless, it is instructive to see how the approximate inference fares for rank k 0 = 3 .
Revised: Nevertheless, it is instructive to see how the approximate inference fares for rank k 0 = 3 .

Original: Given the value y for Y , the inferred distribution over X is
Revised: Given the value y for Y , the inferred distribution over X is

Original: [Equation]
Revised: [Equation]

Original: The approximately inferred ﬁrst and second moments of X is given by integrating the above times x (resp.
Revised: The approximately inferred ﬁrst and second moments of X is given by integrating the above times x (resp.

Original: x 2 ) over x .
Revised: x 2 ) over x .

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: which are actually exact: they are equal to the ﬁrst two moments of X over p X | Y as given in Eq.
Revised: which are actually exact: they are equal to the ﬁrst two moments of X over p X | Y as given in Eq.

Original: In fact, it is easy to see that this would be true for the ﬁrst k 0 − 1 moments had we kept the k 0 most relevant features.
Revised: In fact, it is easy to see that this would be true for the ﬁrst k 0 − 1 moments had we kept the k 0 most relevant variables.