A N
I NFORMATION
T HEORETIC A PPROACH TO D IS -
TRIBUTED R EPRESENTATION
L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The problem of distributed representation learning is one in which multiple sources of information X 1 , . . .
, X K are processed separately so as to extract useful information about some statistically correlated ground truth Y .
We investigate this problem from informationtheoretic grounds.
For both discrete memoryless (DM) and memoryless vector Gaussian models, we establish fundamental limits of learning in terms of optimal tradeoffs between relevance and complexity .
We also develop a variational bound on the optimal tradeoff that generalizes the evidence lower bound (ELBO) to the distributed setting.
Furthermore, we provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent.
Experimental results on synthetic and real datasets are provided to support the efﬁciency of the approaches and algorithms which we develop in this paper.
I NTRODUCTION
Let a measurable variable X ∈ X and a target variable Y ∈ Y with unknown joint distribution P X,Y be given.
In the classic problem of statistical learning, one wishes to infer an accurate predictor of the target variable Y ∈ Y based on observed realizations of X ∈ X .
That is, for a given class F of admissible predictors φ : X → ˆ Y and an additive loss function (cid:96) :
Y → ˆ Y that measures discrepancies between true values and their estimated ﬁts, one aims at ﬁnding the mapping φ (cid:63) ∈ F that minimizes the expected risk
[Equation]
Because the joint distribution P X,Y is unknown, in practice the risk equation 1 (also called population risk ) cannot be computed directly; and, in the standard approach, one usually resorts to choosing the predictor with minimal risk on a training dataset consisting of n labeled samples { ( x i , y i ) } ni =1 that are drawn independently from the unknown joint distribution P X,Y .
Also, it is important to restrict the set F of admissible predictors to a low-complexity class to prevent overﬁtting.
This leads to the abstract inference problem shown in Figure 1.
In this paper, we study a generalization of this problem in which the prediction is to be performed in a distributed manner.
The model is shown in Figure 2.
Here, the prediction of the target variable Y ∈ Y is to be performed on the basis of samples of statistically correlated random variables ( X 1 , . . .
, X K ) that are observed each at a distinct predictor.
We investigate this problem in the case in which the loss function (cid:96) ( · ) is the logarithmic-loss ﬁdelity measure, given by
[Equation]
where ˆ y ( · ) designates a probability distribution on Y and ˆ y ( y ) is the value of this distribution evaluated for the outcome y ∈ Y .
The choice of a ‘good” loss function is often controversial in statistical learning theory, and although a complete and rigorous justiﬁcation of the usage of logarithmic loss as a ﬁdelity measure in learning theory is still awaited, partial explanations appeared in Jiao et al.
(2015) and, especially in Painsky and
Wornell (2018) where it is shown that, for binary classiﬁcation problems, by minimizing the logarithmic-loss one actually minimizes an upper bound to any choice of loss function that is smooth, proper (i.e., unbiased and
Fisher consistent) and convex.
Also, we constrain the complexity of the predictors by using mutual information as a regularizer term.
This is inline with recent works Xu and Raginsky (2017); Russo and Zou (2015) that show that the generalization error can be upper-bounded using the mutual information between the input dataset and the output of the predictor – see also Bousquet and Elisseeff (2002); Shalev-Shwartz et al.
(2010) where the stability of an algorithm is controlled by constraining the mutual information between its
P X | Y Y ∈ Y φ ψ b
Y ∈ Y X ∈ X
U = φ ( X )
P X 1 ,...,X K
| Y Y ∈ Y ··· φφ k ψ b
Y ∈ Y
X K
X
U
U K
1. A N EXAMPLE : MULTI - VIEW LEARNING
In many data analytics problems, data is collected from various sources of information or feature extractors; and is intrinsically heterogeneous .
For example, an image can be identiﬁed by its color or texture features; and a document may contain text and images.
Conventional machine learning approaches concatenate all available data into one big row vector (or matrix) on which a suitable algorithm is then applied.
Treating different observations as a single source might cause overﬁtting and is not physically meaningful because each group of data may have different statistical properties.
Alternatively, one may partition the data into groups according to samples homogeneity, and each group of data be regarded as a separate view .
This paradigm, termed multi-view learning Xu et al.
(2013), has received growing interest; and various algorithms exist, sometimes under references such as co-training Blum and Mitchell (1998); Dhillon et al.
; Kumar and Daum´e (2011);
G¨onen and Alpaydın (2011), multiple kernel learning G¨onen and Alpaydın (2011) and subspace learning Jia et al.
By using distinct encoder mappings to represent distinct groups of data, and jointly optimizing over all mappings to remove redundancy, multiview learning offers a degree of ﬂexibility that is not only desirable in practice but is likely to result in better learning capability.
Actually, as shown in Vapnik (2013), local learning algorithms produce less errors than global ones.
Viewing the problem as that of function approximation, the intuition is that it is usually non-easy to ﬁnd a unique function that holds good predictability properties in the entire data space.
1. I NFORMAL SUMMARY OF RESULTS
In this paper, ﬁrst we characterize the optimal tradeoff between relevance and complexity for the distributed learning model of Figure 2 for both discrete memoryless (DM) and memoryless vector Gaussian models.
While the result for the discrete data model (Theorem 1) is not difﬁcult to establish using connections with Courtade and Weissman (2014, Appendix B) which we explicit here, the result for the multivariate Gaussian data model (Theorem 2), which provides a sharp analytic characterization of optimal tradeoffs, is new and non-trivial (the proof of the converse part is not straightforward and was missing before this work in both theoretic learning and information theoretic communities including in the scalar case).
Second, we develop a variational bound on the optimal tradeoff that can be seen as a generalization of the ELBO and the β -VAE criteria Higgins et al.
(2016) to the distributed setting.
Furthermore, for both DM and Gaussian models, we also provide a variational inference type algorithm which is parametrized by neural networks and allows to compute the developed variational bound when the data distribution is not known.
Speciﬁcally, the main contributions of this paper are:
• In Section 3.2, we ﬁnd an explicit analytic characterization of optimal tradeoffs between relevance and complexity for the memoryless vector Gaussian model.
The result generalizes the Gaussian Information Bottleneck method of Globerson and Tishby (2004); Chechik et al.
(Feb. 2005) to the distributed learning scenario.
• In Section 3.3, we study the problem of maximizing relevance under a constraint on the sum complexity for which we establish a variational bound which generalizes the ELBO and the β -VAE criteria to the distributed setting.
• Section 3.4 is algorithmic-oriented.
We develop a variational inference type algorithm which enables to compute the bound.
This algorithm is obtained by parametrizing the encoders, the decoder, and the prior distributions via DNNs and using Monte-Carlo sampling.
Also, it makes usage of Kingma et
al.
’s re-parametrization trick Kingma and Welling (2013) and can be seen as a generalization of the variational information bottleneck algorithm in Alemi et al.
(2017) to the distributed setting.
• Section 4 contains some experimental results on real datasets which show the efﬁciency of the approaches and algorithms that we develop in this paper.
Most relevant to this paper is the single-encoder Information Bottleneck (IB) method of Tishby et al.
(1999) which readily and elegantly captures the above mentioned viewpoint of seeking the right balance between data ﬁt and generalization by using the mutual information both as a cost function and as a regularizer term.
Thus, the results of this paper can be seen as a generalization of those of Tishby et al.
(1999) for the DM model and Globerson and Tishby (2004)
; Chechik et al.
(Feb. 2005) for the Gaussian model to the distributed learning setting.
Remark:
Due to space constraints, the proofs of the results of this paper are deferred to the appendices section, which also contains additional experimental results.
1. N OTATION
Throughout, upper case letters denote random variables, e.g., X; lower case letters denote realizations of random variables, e.g., x ; and calligraphic letters denote sets, e.g., X .
The cardinality of a set is denoted by |X| .
For a random variable X with probability mass function (pmf) P X , we use P X ( x ) = p ( x ) , x ∈ X for short.
Boldface upper case letters denote vectors or matrices, e.g., X , where context should make the distinction clear.
For random variables ( X 1 , X 2 , . . . ) and a set of integers K ⊆ N , X K denotes the set of random variables with indices in the set K , i.e., X K =
{ X k : k ∈ K} .
If K = ∅ , X K = ∅ .
For k ∈ K we let
X K /k =
( X 1 , . . .
, X k − 1 , X k +1 , . . .
, X K ) , and assume that X
= X K +1 = ∅ .
Also, for zero-mean random vectors X and Y , the quantities Σ x , Σ x , y and Σ x | y denote respectively the covariance matrix of the vector
X , the covariance matric of vector ( X , Y ) and the conditional covariance matrix of X , conditionally on Y .
Finally, for two probability measures P X and Q X on the random variable X ∈ X , the relative entropy or
Kullback-Leibler divergence is denoted as D KL ( P X (cid:107) Q X ) .
F ORMAL P ROBLEM F ORMULATION
Let K ≥ 2 and ( X 1 , . . .
, X K , Y ) be a tuple of random variables with a given joint probability mass function (pmf) P X 1 ,...,X K ,Y ( x 1 , . . .
, x K , y ) for ( x 1 , . . .
, x K ) ∈
X 1 × . . .
× X K and y ∈ Y , where X k designates the alphabet of X k and Y that of Y .
Throughout, we assume that the following Markov chain holds for all k ∈ K :
= { 1 , . . .
, K } ,
[Equation]
The variable Y is a target variable; and we seek to characterize how accurate it can be predicted from a measurable random vector ( X 1 , . . .
, X K ) when the components of this vector are processed separately, each by a distinct encoder.
More speciﬁcally, let { ( X 1 ,i , . . .
, X K,i , Y i ) }
ni =1 be a collection of n independent copies of ( X 1 , . . .
, X K , Y ) .
Encoder k ∈ K only observes the sequence X nk ; and generates a description J k =
φ k
( X nk ) according to some mapping
[Equation]
[Equation]
) where M ( n ) k is an arbitrary set of descriptions.
The range of allowable description sets will be speciﬁed below.
A decoder ψ ( · ) collects all descriptions J K
=
( J 1 , . . .
, J K ) and returns an estimate ˆ Y n of Y n as
[Equation]
The relevance of the estimation ˆ Y n is measured in terms of the relevance , deﬁned here as the information that the descriptions φ 1 ( X n 1 ) , . .
. , φ K ( X nK ) collectively preserve about Y n , as measured by Shannon mutual information
[Equation]
where ˆ Y n = ψ ( φ 1 ( X n 1 ) , . .
φ K ( X nK )) and the subscript P X K ,Y indicates that the mutual information is computed under the joint distribution P X K ,Y .
There are various ways to control the complexity of the encoding functions { φ k } Kk =1 .
In this paper, we do so by restricting their ranges.
This is known as minimum description length complexity measure Hinton and van Camp (1993).
Speciﬁcally, the mapping φ k ( · ) at Encoder k ∈ K needs to satisfy
[Equation]
for all
[Equation]
Deﬁnition 1 A tuple (∆ , R 1 , . . .
, R K ) is said to be achievable if there exists an integer n , a family of encoding mappings { φ k } Kk =1 and a decoder mapping ψ such that
[Equation]
The relevance-complexity region IR DIB is given by the closure of all achievable tuples (∆ , R 1 , . . .
, R K ) .
In some cases, for given R K =
( R 1 , . . .
, R K ) , for the ease of the exposition we will be content with the relevance-complexity function ∆( R K , P X K ,Y ) deﬁned as
[Equation]
where the maximization is subjected to equation 7.
M AIN R ESULTS 3. D ISCRETE MEMORYLESS DATA MODEL
The following theorem (the proof of which can be found in the appendices section) provides a computable characterization of the relevance-complexity region IR DIB .
The result can be seen as a generalization of Tishby et al.
Tishby et al.
(1999) single encoder IB to the distributed learning model with K encoders.
Theorem 1 The relevance-complexity region IR DIB of the distributed learning problem with P X K ,Y for which the Markov chain equation 3 holds is given by the union of all tuples (∆ , R 1 , . . .
, R K ) ∈
R K +1+ that satisfy for all S ⊆ K ,
[Equation]
for some set of pmfs P :
= { P U k | X k ,T , . .
, P U k | X k ,T , P T } with joint distribution of the form
[Equation]
Remark
In Theorem 1, the random variable T stands for a convexiﬁcation of the region, i.e., convex combination of achievable relevance-complexity tuples is itself achievable.
For given T = t , the result of
Theorem1 comprises the optimization over K conditional distributions { P U K | X k ,t } .
For k ∈ K , the conditional distribution P U K | X k ,t represents a stochastic encoding of the feature X k into a latent variable U k .
Intuitively, the latent variable U k should capture all relevant information about Y that is contained in X k and non redundant with those carried out by { U i } i (cid:54) =
k .
The requirement of non-redundancy is mandated by the need to operate at the minimum possible complexity at which a desired relevance level is achievable (recall that minimum complexity, as expressed by algorithm’s input-output mutual information, translates directly into a better generalization capability).
Collectively, however, the set of all latent variables ( U 1 , . . .
, U K ) should be expressive enough to reproduce the target variable Y to within the desired relevance level.
For given T = t and ( U 1 , . . .
, U K ) =
( u 1 , . . .
, u K ) the optimal stochastic decoder is given by
[Equation]
Remark 2 Like for the single-encoder IB problem of Tishby et al.
(1999) and an increasing number of works that followed, including Courtade and Weissman (2014, Section III-F), our approach here is asymptotic.
In addition to that it leads to an exact characterization, the result also readily provides a lower bound on the performance in the non-asymptotic (e.g., one shot) setting.
For the latter setting known approaches (e.g., the functional representation lemma of Li and El Gamal (2018)) would lead to only non-matching inner and outer bounds on the region of optimal tradeoff pairs, as this is the case even for the single encoder case Li et al.
3. M EMORYLESS VECTOR G AUSSIAN DATA MODEL
We now turn to a continuous-alphabet setting.
Here, ( X 1 , . . .
, X K , Y ) is a zero-mean Gaussian random vector such that
[Equation]
where H k ∈ C n k × n y models the linear model connecting the target variable Y ∈ C n y to the observation at encoder k , and N k ∈ C n k , k = 1 , . .
, K , is the noise vector at encoder k , assumed to be Gaussian with zero-mean and covariance matrix Σ k , and independent from all other noises and the target variable Y .
We denote by Σ y the covariance matrix of of the target vector Y ∈ C n y .
For this model, we ﬁnd an explicit analytic characterization of optimal tradeoffs between relevance and complexity.
The proof relies on deriving an outer bound on the region described by equation 11, and showing that it is achievable with Gaussian pmfs, with no time-sharing.
In doing so, we use techniques that rely on the de Bruijn identity and the properties of Fisher information and minimum mean square error (MMSE).
Theorem 2 The relevance-complexity region IR GDIB for the vector Gaussian model is given by the union of all tuples (∆ , R 1 , . . .
, R L ) that satisfy for all S ⊆ K
[Equation]
for
[Table]
Proof: The proof of the direct part follows by evaluating the region of Theorem 1, which can be extended to the case of continuous alphabets using standard discretization (quantization) arguments, with the choices T = ∅ and p ( u k | x k , t ) =
CN ( x k , Σ 1 / 2 k ( Ω k − I ) Σ 1 / 2 k ) .
The main contribution in the proof is that of the converse part.
This proof is technical and rather lengthy and, for this reason, is deferred to the appendices section.
In the special case in which K = 1 , the result of Theorem 2 recovers that by Globerson and Tishby (2004) (see also Chechik et al.
(Feb. 2005)) which establishes the optimal relevance-complexity tradeoff of the single-encoder
Gaussian IB problem.
3. A VARIATIONAL BOUND
In this section, we consider the problem of learning encoders- and decoder mappings that maximize the relevance level for a given (ﬁxed) complexity level, i.e., those that perform at the vicinity of the boundary of the region
IR DIB .
First, we derive a parametrization of the relevance-complexity region; and, then, we develop a variational bound which expresses the optimal encoders’ and decoder mappings as the solution to an optimization problem – (an algorithm for solving this problem in the case of unknown distributions is given in the next section).
Let R sum :=
(cid:80)
Kk =1 R k .
Also, let IR sumDIB denote the region of achievable (relevance, sum-complexity) pairs,
IR
[Equation]
[Equation]
Applying Fourier-Motzkin Elimination (FME)
El Gamal and Kim (2011) to successively project out the variables
R 1 , . . .
, R K , and using equation 3, the region
IR sumDIB can be characterized as given in Proposition 1.
Proposition 1 The relevance-complexity region under sum-complexity constraint RI sumDIB is given by the convexhull of all tuples (∆ , R sum )
∈ R
+ satisfying ∆ ≤ ∆( R sum , P X K ,Y ) where
[Equation]
and where the maximization is over the set of pmfs P := { P U 1 | X 1 , . .
, P U K | X K } such that the joint pmf factorizes as p Y {+( y ) (cid:81) Kk =
 p X k | Y ( x k | y [Equation]+}
(cid:81)
Kk
[Equation]
1 p U
[Equation]
| [Equation] k ( u k | x k ) . The next proposition provides a characterization of the pairs (∆ , R sum ) that lie on the boundary of RI sumDIB in terms of a nonnegative parameter s ≥ 0 .
Proposition 2 For every pair (∆ , R sum )
∈ R 2+ that lies on the boundary of the relevance-complexity region
RI sumDIB there exist s ≥ 0 such that (∆ , R sum ) =
(∆ s , R s ) , where
[Equation]
and P ∗ is the set of conditional pmfs P that maximize the cost function
[Equation]
Using Proposition 2 it is clear that the encoders { P U k | X k } k ∈K that achieve the relevance-complexity pair (∆ s , R s ) can be computed by maximizing the regularized cost equation 18 for the corresponding value of s ≥ 0 .
The corresponding optimal decoder P Y | U K for these encoders can be found as in equation 13.
Different relevance-complexity pairs (∆ s , R s ) on the boundary of IR sumDIB and encoders- and decoder mappings that achieve it can be found by solving equation 18 for different values of s ≥ 0 and then evaluating equationand equation 17 for the obtained solution.
The optimization of equation 18 generally requires to compute marginal distributions involving the descriptions
U 1 , . . .
, U K , an aspect which can be non-easy computationally costly.
To overcome this limitation, in the following we derive a tight variational bound on L s ( P ) which lower bounds the DIB cost function with respect to some arbitrary distributions.
Let us consider the arbitrary decoder Q Y | U 1 ,...,
U K
( y | u 1 , . .
, u K ) for y ∈ Y , u 1 ∈ U 1 , . . .
, u K ∈
U K , the K decoders Q Y |
U k
( y | u k ) for k ∈ K for y ∈ Y , u k ∈ U k , and latent variable priors Q U k ( u k ) , k ∈ K , u k ∈ U k .
For short, we denote
[Equation]
Let us deﬁne the variational DIB cost function L VB s ( P , Q ) as
[Equation]
P , Q
[Equation] [Equation]
E [
[Equation]
Q Y | U K
[Equation]
Y | U K )]
[Equation]
The following lemma states that L VB s ( P , Q ) is a lower bound to L s ( P ) for all distributions Q .
Lemma 1 For ﬁxed pmfs P , we have
[Equation]
all pmfs Q
[Equation]
In addition, there exists a unique Q that achieves the maximum max Q L VB s ( P , Q ) =
L s ( P ) , and is given by
[Equation] [Equation]
where P U k , P Y |
U k and P Y
| U 1 ,...,U K are computed from the pmfs P .
[Equation]
Using the above, the optimization in equation 16 can be written in terms of the variational DIB cost function as
[Equation]
We close this section by noting that the cost function equation 19 can be seen as a generalization of the evidence lower bound (ELBO) as given in Rezende et al.
; Kingma and Welling (2013) for the single-encoder learning to the distributed setting.
Also, in the speciﬁc case in which Y =
( X 1 , . . .
, X K ) the bound generalizes the ELBO used for VAEs to the case of an arbitrary number of encoders.
3. C ASE OF UNKNOWN DISTRIBUTIONS : VARIATIONAL DISTRIBUTED IB ALGORITHM
In practice only a set of training samples { ( X 1 ,i , . . .
, X K,i , Y i ) }
ni =1 are available.
In this section, we provide a method to optimize equation 23 in this case by parametrizing the encoding and decoding distributions that are to optimize using a family of distributions whose parameters are determined by Deep Neural networks
(DNNs).
This allows us to formulate equation 23 in terms of the DNN parameters and optimize it by using the reparametrization trick Kingma and Welling (2013), Monte Carlo sampling, as well as stochastic gradient descent (SGD) type algorithms.
Let F eNN ,k denote the parametric family of encoding probability distributions
P U k |
X k over U k for each element on X k .
Each member of this collection, P U k | X k ; γ e k , is described by a parameter vector γ e k ∈ Γ e k ⊆ R l e k ,
where Γ e k ⊆ R l e k denotes the set of allowable parameter vectors.
The parameter vector γ e k is the output of a DNN f θ k : X k → Γ e k , with network parameters θ
k ∈
Θ k ⊆ R d e k , e.g., the weights of the network at all layers.
The
DNN f θ k takes X k as input and outputs the parameter vector γ e k , determining one of the probability members
P U k |
X k ; γ e k .
We have
[Equation]
For example, the family of multivariate Gaussian distributions is parameterized by the mean µ θk and covariance matrix Σ θk , i.e., γ k :=
( µ θk , Σ θk ) .
Therefore, given an observation X k , γ k := ( µ θk , Σ θk ) is determined by the output of the DNN f θ k and the corresponding family member of F eNN ,k is given by P U k | X k ; γ k ( u k | x k ) =
N ( u k ; µ θk , Σ θk ) .
Similarly, for decoders Q Y | U k over Y , deﬁne the family of distributions parametrized by a vector in Γ d k ⊆ R l d k determined by the output of a DNN f φ k : U k
→ Γ d k , with parameters φ k ∈ Φ k ⊆ R d d k , as
F
[Equation] [Equation]
and for the distribution Q Y | U K over Y for each element in U 1 × · · · × U K , deﬁne the family of distributions parameterized by the output of the DNN f φ K : U 1 × · · · × U K
→ Γ d K , with φ K ∈ Φ K ⊆ R d d K , and Γ d K ⊆
R d d K
F dNN , K
[Equation]
Finally, for the distributions Q ϕ k ( u k ) we deﬁne the family of distributions with parameter ϕ k ∈ Ψ k ⊆ R l p k
[Equation]
In the following, for brevity we use P θ k ( u k | x k ) , Q ψ k ( y | u k ) , Q ψ K ( y | u K ) and Q ϕ k ( u k ) to denote the distributions parametrized by the DNNs f θ k , f ψ k , f ψ K and ϕ k , respectively.
By restricting the optimization of the variational DIB cost in equation 23 to the encoder, decoder and priors within the families of distributions F eNN ,k , F dNN ,k , F dNN , K , F pNN ,k we get
[Equation]
(27) where we use the notation θ :
=
[ θ 1 , . .
, θ K ] , φ :=
[ φ 1 , . .
. , φ K , φ K ] and ϕ :=
[ ϕ 1 , . .
ϕ K ] to denote the
DNN and prior parameters and, the cost in equation 27 is given by
[Equation]
Next, we train the DNNs to maximize a Monte Carlo approximation of equation 27 over θ , φ , ϕ using SGD.
We use the reparameterization trick Kingma and Welling (2013), to sample from P θ k
( U k | X k ) .
In particular, we consider F eNN ,k to consist of a parametric family of distributions that can be sampled by ﬁrst sampling a random variable Z k with distribution P Z k
( z k ) , z k ∈ Z k
and then transforming the samples using some function g θ k :
X k × Z k
→ U k parameterized by θ k , such that U k
= g θ k
( x k , Z k ) ∼ P θ k
( U k | x k ) .
The reparametrization trick reduces the original optimization to estimating θ k of the deterministic function g θ k and allows to compute estimates of the gradient using backpropagation Kingma and Welling (2013).
The variational DIB cost in equation 27 can be approximated, by sampling m independent samples { u k,i,j } mj =1 ∼
P θ k ( u k | x k,i ) for each training sample ( x 1 ,i , . . .
, x K,i , y i ) , i = 1 , . . . , n .
Sampling is performed by using u k,i,j = g φ k ( x k,i , z k,j ) with { z k,j } mj =1 i.i.d.
sampled from P Z k .
We then have
[Equation]
E XPERIMENTS : R ESILIENCE TO N OISE , R OTATION AND O CCLUSION
In this experiment, we test the robustness of our method against noise, rotation and random occlusion on the MNIST dataset.
Speciﬁcally, we combine two types of random occlusions: the ﬁrst encoder observes a digit from the MNIST that is occluded by a square which is rotated randomly (rotation angle uniformly distributed over [ − 45 o , 45 o ] ); and the second encoder observes a noisy version of the same digit corrupted by additive noise (noise level uniform between 0 and 3).
The noisy pixels are clipped between 0 and 1, with more than 60% of the pixels occluded.
These occlusions make the problem signiﬁcantly more involved than the standard MNIST (for which application of our algorithm leads to an relevance of about 99.9%).
We considered a CNN deterministic networks with dropout which achieves a 99.8% for test data on the clean MNIST data.
Then, we have trained the same CNN architecture for each of the noisy inputs to the encoders, resulting in a relevance of 92.1% from the input to encoder 1 (randomly rotated occlusion) and 79.68% from the input to encoder 2 (noisy clipped image).
[Figure]
CNN Layers
Encoder k conv.
ker.
[5,5,32]-ReLu maxpool [2,2,2] conv.
ker.
[5,5,64]-ReLu maxpool [2,2,2] dense [1024]-ReLu dropout 0.
dense [256]-relu
Latent space k dense [256]-ReLu
Decoderdense [256]-ReLu
Decoder k dense [256]-ReLu
[Figure]
[Figure]
[Table]
We applied our D-VIB algorithm of Section 3.4 to this model with the CNN architecture of Table 1, in which
Encoder k = 1 , 2 is parametrized by an n u k = 256 dimensional multivariate Gaussian distribution N ( µ e k , Σ e k ) determined by the output of a DNN f θ k consisting of the concatenation of convolution, dense and maxpool layers with ReLu activations and dropout.
The output of the last layer is followed by a dense layer without activation that generate µ e k and Σ e k .
The prior is chosen as Q ϕ k ( u ) =
N ( 0 , I ) .
Each decoder takes the samples from P θ k
( U k | X k ) and processes its inputs with a dense layer DNN ( f φ K and f φ k ) each with 256 neurons and
ReLu activation, which outputs a vector ˆ y i of size |Y| = 10 normalized with a softmax, corresponding to a distribution over the one-hot encoding of the digit labels { 0 , . .
, 9 } from the K observations,
[Equation]
[Equation]
, (31) where Softmax ( p ) for p ∈ R
d is a vector with i -th entry as [ Softmax ( p )]
i = exp( p i ) / (cid:80) dj =1 exp( p j ) .
Figure 4 shows the relevance-complexity tradeoffs obtained using our D-VIB algorithm of Section 3.4, with n = 50 .
000 and 15 distinct s -values randomly chosen in the range [10 − 10 , 1] .
For comparison, we also present the performance obtained using three methods among state-of the-art multiview learning approaches: (i) applying a deterministic CNN on the two views concatenated (deterministic CNN), (ii) applying the singleencoder variational IB method of Alemi et al.
on the two views concatenated (C-VIB), and (iii) learning one function for each view via a distinct CNNs and optimize all CNNs independently (independent CNNs).
The achieved relevance is reported in Table 2.
Similar results were obtained for other datasets.
The interested reader may refer to the appendices section.
APPENDICES
P ROOFS OF MAIN THEOREMS , PROPOSITIONS AND LEMMAS
. A DDITIONAL EXPERIMENTAL RESULTS P ROOFS OF M AIN T HEOREMS , P ROPOSITIONS AND L EMMAS 5. A UXILIARY L EMMAS
Lemma 2 Dembo et al.
; Ekrem and Ulukus (2014) Let ( X , Y ) be a pair of random vectors with pmf p ( x , y ) .
We have log | ( πe )
[Equation]
− 1 ( X | Y )
[Equation]
|
[Equation]
) ≤
[Equation]
| ( πe )mmse( X | Y ) | , where the conditional Fischer information matrix is deﬁned as
J ( X | Y ) :
= E[ ∇ log p ( X | Y ) ∇ log p ( X | Y ) † ] , and the minimum mean squared error (MMSE) matrix is mmse( X | Y ) :
=
E[( X − E
[ X | Y ])
( X − E[ X | Y ]) † ] .
Lemma 3 Ekrem and Ulukus (2014) Let ( V 1 , V 2 ) be a random vector with ﬁnite second moments and
N ∼CN ( 0 , Σ N ) independent of ( V 1 , V 2 ) .
Then
[Equation]
5. P ROOF OF T HEOREM
If K = 1 the distributed learning problem that we study boils down to the well known Information Bottleneck (IB) problem of Tishby et al.
The single-encoder IB problem is essentially a remote point-to-point source coding problem Dobrushin and Tsybakov (1962) in which distortion is measured under the logarithm loss ﬁdelity criterion Harremoes and Tishby (2007).
In accordance with this analogy, for K ≥ 2 consider the multiterminal source coding problem under logarithmic loss in which the sequence Y n models a remote source that is observed by K spatially distributed agents; the agents observe noisy versions of the remote source and communicate independently with a decoder or Chief Executive Ofﬁcer (CEO) over rate-constrained noise-free links.
For instance, agent k , k ∈ K , observes X nk and uses R k bits per sample to describe it to the decoder.
The decoder wants to reconstruct the remote source Y n to within a prescribed ﬁdelity level, where incurred distortion is measured using the logarithmic loss criterion, i.e.,
[Equation]
where J = ( φ 1 ( X n 1 ) , . .
. , φ K ( X nK )) .
Here, ( X n 1 , . . .
, X nK , Y n ) is assumed to be distributed i.i.d.
according to the n -product of the pmf P X 1 ,...,X K ,Y , i.e., the Markov chain equation 3 holds.
Deﬁnition
A rate-distortion code (of blocklength n ) for the CEO problem consists of K encoding functions
[Equation] [Equation]
and a decoding function
[Equation]
A distortion-rate tuple ( D, R 1 , . . .
, R K ) is achievable for the DM CEO source coding problem with side information if there exist a blocklength n , encoding functions { ˜ φ k } Kk =1 and a decoding function ˜ ψ
such that
[Equation]
The distortion-rate region DR CEO of the CEO model is deﬁned as the closure of all non-negative tuples ( D, R 1 , . . .
, R K ) that are achievable.
[Equation]
Key to the proof of Theorem 1 is the following proposition which states that IR DIB and DR CEO can be inferred from each other.
Proposition 3 (∆ , R
[Equation] [Equation]
R K ) ∈
[Equation]
and only
[Equation] [Equation]
Proof: Let, for k = 1 , . .
, K , J k =
φ k ( X n k ) and J =
( J 1 , . . .
, J K ) .
Then,
[Equation]
where equation 38 is due to the non-negativity of the Kullback-Leibler divergence and the equality holds if and only if for ˆ P ( y n | j )
=
P ( y n | j ) where P ( y n | j ) =
Pr { Y n = y n | J = j } for all j and y n ∈ Y n .
Let an achievable tuple (∆ , R 1 , . .
, R K ) ∈ IR DIB be given.
Then, there must exist functions { φ k } Kk = such that equation 8 and equation 9 hold.
Using equation 38 that by letting the decoding function
˜ ψ ( J K )
= { P Y n | J K ( y n | J K ) } , we have E[ (cid:96) log ( Y n , ˆ Y n ) | J K ]
= H ( Y n | J K ) , which implies ( H ( Y ) − ∆ , R 1 , . . .
, R K ) ∈
DR CEO .
The result of Theorem 1 follows easily by combining (Courtade and Weissman, 2014, Theorem 10), which provides a single-letter characterization of the rate distortion region DR (cid:63) CEO of the CEO problem, and Proposition 3.
5. P ROOF OF T HEOREM
The proof of the direct part of Theorem 2 follows by evaluating the region of Theorem 1 with the choice T = ∅ and p ( u k | x k , t ) =
CN ( x k , Σ 1 / 2 k ( Ω k − I ) Σ 1 / 2 k ) .
The proof of the converse part is as follows.
Fix t ∈ T , S ⊆ K and a family of distributions { p ( u k | x k , t ) }
Kk = such that the joint distribution factorizes as equation 12.
Also, let 0 (cid:22) Ω
k,t (cid:22) Σ − 1 k and mmse
[Equation]
Such Ω k,t always exists since
[Equation]
Then, we have
I
[Equation]
where the inequality is due to Lemma 2; and equation 41 is due to equation 39.
Also, we have
[Equation] [Equation]
where equation 42 follows by using Lemma 2; and equation 43 holds by using the following equality
[Equation]
the proof of which uses a connection between MMSE and Fisher information as shown next.
For the proof of equation 44, ﬁrst note that from the MMSE estimation of Gaussian random vectors El Gamal and Kim (2011), we have
[Equation]
where G k = Σ y | x S c H † k Σ − 1 k and Z S c ∼ CN ( 0 , Σ y | x S c ) , with Σ
[Equation] [Equation]
Note that Z S c is independent of Y S c due to the orthogonality principle of the MMSE and its Gaussian distribution.
Hence, it is also independent of U S c ,q .
We have mmse
[Equation]
where equation 47 follows since the cross terms are zero due to the Markov chain ( U k,t , X k ) − (cid:10) − Y − (cid:10) − ( U K /k,t , X K /k ) ; and equation 48 follows due to equation 39 and G k .
Finally,
[Equation]
where equation 49 is due to Lemma 3; equation 50 is due to equation 48; and equation 51 follows due to equation 46.
Now
[Equation]
let ¯ Ω k := (cid:80) t ∈T p ( t )
Ω k,t .
The rest of the converse proof follows by averaging over the time sharing random variable to get
[Equation]
where equation 52 follows from the concavity of the log-det function and Jensen’s inequality.
Similarly to equation
[Equation]
from equation 43 and Jensen’s Inequality we have
[Equation]
Finally, using equation 52 and equation 53 in equation ?? , noting that Ω k = (cid:80) t ∈T p ( t ) Ω k,t (cid:22) Σ − 1 k since 0 (cid:22) Ω k,t (cid:22) Σ − 1 k , and taking the union over Ω k satisfying 0 (cid:22) Ω k (cid:22) Σ − 1 k , completes the proof of the converse part; and, hence, that of Theorem 2.
5. P ROOF OF P ROPOSITION
For simplicity of exposition, the proof is given for the case K = 2 encoders.
The proof for K > 2 follows similarly.
By the deﬁnition of IR sumDIB , the accuracy complexity tuple (∆ , R sum ) ∈ R 2+ is achievable for some random variables Y, X 1 , X 2 , U 1 , U 2 with joint pmf satisfying equation 12, if it holds that
[Equation]
The application of the Fourier-Motzkin elimination to project out R 1 and R 2 reduces the system on inequalities equation 54-equation 58 to the following system of inequalities
[Equation]
It follows due to the Markov chain U 1 − (cid:10) − X 1 − (cid:10) − Y − (cid:10) − X 2 − (cid:10) − U 2 that we have I ( Y ; U 1 , U 2 )
≤
I ( Y ; U 1 )
+ I
( Y ; U 2 ) .
Therefore, inequality equation 61 is redundant as it is implied by equation 59 and equation 60.
This completes the proof of Proposition 1.
5. P ROOF OF P ROPOSITION
Suppose that P ∗ yields the maximum in equation 16.
Then,
[Equation]
where equation 63 is due to the deﬁnition of L s ( P ) in equation 18; equation 64 follows since we have (cid:80)
Kk
[Equation]
[Equation]
k ; U ∗
[Equation]
Y | U ∗ k
[Equation]
] = R s −
[Equation]
Y [Equation] U ∗K ) + KH ( Y ) from the deﬁnition of R s in equation 17; and equation 66 follows from the deﬁnition in equation ?? .
Conversely, if P ∗ is the solution to the maximization in the function ∆( R sum , P X K ,Y ) in equation ?? such that ∆( R sum , P X K ,Y ) =
∆ s , then ∆ s ≤ I ( Y ; U ∗K ) and ∆ s ≤ R sum − (cid:80)
Kk
=
I ( X k ; U ∗ k | Y ) and we have, for any s ≥ 0 , that
[Equation]
where in equation 67 we have (cid:80) Kk
=
I ( X k ; U k | Y )
= − KH ( Y )
+ (cid:80)
Kk
=
I ( X k ; U k )
+ H ( Y
| U k ) due to the
Markov chain
U k − X k − Y − ( X K\ k , U K\ k ) ; equation 68 follows since L ∗ s is the maximum over all possible distributions P (not necessarily P ∗ maximizing ∆( R sum , P X K ,Y ) ); and equation 69 is due to equation 16.
Finally, equation 70 is valid for any R sum ≥ 0 and s ≥ 0 .
Given s , and hence (∆ s , R s ) , choosing R = R s yields ∆( R s , P X K ,Y ) ≤ ∆ s .
Together with equation 66, this completes the proof of Proposition 2.
5. P ROOF OF L EMMA
The proof follows by deriving the following bounds.
For any conditional pmf Q Y | Z ( y | z ) , y ∈ Y and z ∈ Z , e.g., Z = U K or Z = U k , proceeding similarly to equation 38 and averaging over Z , we have
[Equation]
Similarly, we have
[Equation]
Thus, we get
[Equation]
where equation 75 holds by the non-negativity of relative entropy: and the equality is met if and only if Q ∗ is as given by equation 21 and equation 22.
O THER E XPERIMENTAL R ESULTS (R EGRESSION FOR U NKNOWN G AUSSIAN M ODEL ) 6. D-VIB ALGORITHM FOR V ECTOR G AUSSIAN M ODEL For
the vector Gaussian data model equation 14 the optimal distributions P and Q in equation 23 lie within the family of multivariate Gaussian distributions.
Motivated by this observation, we consider the following parameterization for k ∈ K :
[Equation]
where µ e k , Σ e k are the output of a DNN f θ k with input X k that encodes the observations in a n u k -dimensional
Gaussian distribution, µ d K , Σ d K are the outputs of a DNN f φ K with inputs U 1 , . .
, U K , sampled from
P θ k ( u k | x k ) , and µ d k , Σ e k are the output of a DNN f φ k with input U k , k = 1 , . . .
, K .
With the above choice of parametric encoders and decoders, and using a single sample m = 1 , the empirical
DIB cost in equation 29 is given for the sample ( x 1 ,i , . . .
, x K,i , y i ) by
[Equation]
where ( µ d12 ,i , Σ d12 ,i ) denote the output of the DNN f φ K for the i -th sample ( x 1 ,i , . .
, x K,i , y i ) , and similarly for the other mean and covariance terms; and where we have used that each term in the empirical DIB cost equation 29 can be computed noting that for d -dimensional Gaussian pmfs N ( y ; µ , Σ ) we have
[Equation]
and the KL divergence between two multivariate Gaussian pmfs P 1 ∼ N ( µ 1 , Σ 1 ) and P 2 ∼ N ( µ 2 , Σ 2 ) in R d , is
[Equation]
The multivariate Gaussian parametrization of the encoders, decoders and prior distribution as given by equation 76-equation 79 can be used for other data models that are not necessary Gaussian.
For example, it is particularly suitable for regression problems in which Y lies on a continuous space.
Also, it is very often used in conjunction with VAE generative problems Rezende et al.
; Kingma and Welling (2013).
6. R EGRESSION FOR V ECTOR G AUSSIAN D ATA M ODEL
Consider a distributed learning model with K = 2 encoders, each observing a noisy version of an n y -dimensional
Gaussian vector Y ∼ N ( y ; 0 , I ) , as X k = H k Y + N k , where H k ∈ R n k × n y and the noises are distributed as N k ∼ N ( 0 , I ) for k = 1 , 2 .
For this model, the optimal accuracy-complexity region can be computed using Theorem 2.
In what follows, we evaluate the performance of our D-VIB of the previous section for regression.
The algorithm is trained using a dataset of n i.i.d.
samples { ( X 1 ,i , X 2 ,i , Y i ) } ni =1 form the described vector Gaussian data model.
We train the DNNs for various values of the parameter s .
We use the multivariate Gaussian parameterization in equation 76-equation 79 for the DNNs architecture shown in Table 6.2.
Speciﬁcally, Encoder k , k = 1 , 2 , consists of three dense layers of 512 neurons each followed by rectiﬁed linear unit (ReLu) activations.
The output of encoder k is processed by a dense layer without nonlinear activation to generate µ e k and Σ e k of sizeand 512 × 512 , respectively.
Each decoder consists of two dense layers of 512 neurons with ReLu activations.
The output of decoder 1 , 2 and 12 is processed, each, by a fully connected layer without activation to generate µ d k and Σ d k and µ d12 and Σ d12 , of size 2 and 2 × 2 .
[Figure]
l
[Figure]
v a n c
[Figure] [Figure]
Figure 5 shows the optimal relevance-complexity region of tuples (∆ , R sum ) obtained from Theorem 2 for a vector Gaussian model with K = 2 encoders, target variable dimension n y = 1 , and observations dimension n 1 = n 2 = 3 .
A set of 40 .
000 samples split among training ( 30 . 000 samples) and test ( 10 . 000 samples).
The ﬁgure depicts all accuracy-complexity pairs obtained by application of our algorithm D-VIB to this setting.
The results are compared to the case of inference with known joint distribution (referred to as D-IB, see next section) as well as the case of centralized inference (C-IB).
For the D-VIB algorithm, the the DNN architecture for the coders is shown in Table 6.2.
Figure 6 shows the evolution of the associated mean squared error (MSE) in the estimation of the label Y using our D-VIB algorithm.
As it can bee seen from both ﬁgures the performance of our D-VIB algorithm (which does not require knowledge of the joint label-feature distribution) is very close to that predicted by the theory, i.e., our Theorem 2.
Figure 7 shows similar curves for n y = 2 , n 1 = n 2 = 3 dimensions, for various sizes of the training datset.
As expected large training sets allow a more accurate prediction.
Noteworthy, that the performance during the training phase might be better than that of the centralized learning scenario is an indicator can be caused by overﬁtting.
Related to this aspect, recall that although the D-VIB algorithm does not estimate the underlying distribution explicitly, intuitively it does for the computation of the cost function.
This is related to that universal compressors also learn the actual distribution of the data that is being compressed.
Recall that since the plug-in estimator of entropy is biased downward, estimations of the mutual information terms that are involved in the cost function are then biased upward, which is an alternate explanation to the observed overﬁtting during the training phase.
[Figure]
DNN Layers
Encoder k dense [512]-ReLu dense [512]-ReLu dense [512]-ReLu
Lat.
space k dense
[256]-ReLu
Decoderdense [256]-ReLu
Decoder k dense [256]-ReLu
D ISTRIBUTED B LAHUT -A RIMOTO T YPE A LGORITHMS 7. D ISCRETE - ALPHABET SETTING
In this section, we derive an iterative method to optimize the variational DIB cost function in equation 23 when the data model is discrete and the joint distribution P X K ,Y is either known, or a good estimation of it can be obtained from the training samples.
In these cases, the maximizing distributions P , Q of the variational DIB cost in equation 23 can be efﬁciently found by an alternating optimization procedure over P and Q similar to the expectation-maximization (EM) algorithm
Dempster et al.
and
the standard Blahut-Arimoto (BA) methodBlahut (1972).
An extension to the vector Gaussian data model, which involves random variable with continuous alphabets, is also provided.
The main idea of the algorithm is that at iteration t , the optimal distributions P ( t ) that maximize the variational D-IB bound L VB s ( P , Q ( t ) ) for ﬁxed Q ( t ) can be optimized in closed form and, next, the maximizing pmfs Q ( t ) for given P ( t ) can be also found analytically.
So, starting from an initialization P (0) and Q (0) the algorithms performs the following computations successively and in this order, until convergence,
[Equation]
We refer to such algorithm as “Blahut-Arimoto Distributed Information Bottleneck Algorithm (BA-DIB)”.
Algorithm 1 describes the steps taken by BA-DIB to successively maximize L VB s ( P , Q ) by solving a concave optimization problem over P and over Q at each iteration.
We have the following lemma whose proof follows essentially by using the log-sum inequality Cover and Thomas (1991) and the convexity of the mapping x
(cid:55)→ x log x .
Lemma
The function L VB s ( P , Q ) is concave in P and in Q .
For ﬁxed P ( t ) , the optimal Q ( t ) maximizing the variational D-IB bound in equation 19 follows from Lemmaas given by equation 21-equation 22.
For ﬁxed Q ( t ) , the optimal P ( t ) can be found using the following lemma.
Lemma 5 For ﬁxed Q , there exists a P that achieves the maximum max P L VB s ( P , Q ) , where P U k | X k is given by
[Equation]
for u k ∈ U k and x k ∈ X k , k ∈ K , and where we deﬁne
[Equation]
Proof: Due to its concavity, to maximize L VB s ( P , Q ) with respect to P for given Q , we add the Lagrange multipliers λ x k ≥ 0 for each constraint (cid:80)
u k ∈U k p ( u k | x k ) = 1 with x k ∈ X k .
For each s , λ x k ≥ 0 and p ( u k | x k ) can be explicitly found by solving the KKT conditions, e.g.,
[Equation]
This completes the proof.
Set t = 0 and set P (0) with p ( u k | x k ) = 1 |U k | for u k ∈ U k , x k ∈ X
k , k = 1 , . . .
, K .
: repeat
Q ( t +1) using equation 21 and equation 22.
equation
7.1. C ONVERGENCE
Algorithm 1 essentially falls into the class of the Successive Upper-Bound Minimization (SUM) algorithms
Razaviyayn et al.
(2013) in which L VB s ( P , Q ) acts as a globally tight lower bound on L s ( P ) .
Algorithm 1 provides a sequence P ( t ) for each iteration t , which converges to a stationary point of the optimization problem equation 23.
Proposition 4 Every limit point of the sequence P ( t ) generated by Algorithm 1 converges to a stationary point of equation 23.
Proof: Let Q ∗ ( P ) =
arg max Q L VB s ( P , Q ) .
Using Lemma 1, for every P (cid:48) (cid:54)
=
P , it holds that
[Equation]
Since L s ( P ) and L VB s ( P , Q ∗ ( P (cid:48) )) satisfy the assumptions of (Razaviyayn et al., 2013, Proposition 1), then
L VB s ( P , Q ∗ ( P (cid:48) ))
satisﬁes A1-A4 in Razaviyayn et al.
Convergence to a stationary point of equationfollows from (Razaviyayn et al., 2013, Theorem 1).
The self consistent equations equation 21, equation 22 and equation 83 satisﬁed by any stationary point of the D-IB problem extend those of the standard point-to-point IB problem Globerson and Tishby (2004) to the distributed IB problem with K ≥ 2 encoders.
In particular, note the additional divergence term in equation 83.
7. G AUSSIAN SETTING
Recall Algorithm 1.
For ﬁnite alphabet sources the updating rules of Q ( t +1) and P ( t +1) in Algorithm 1 are relatively easy, but they become unfeasible for continuous alphabet sources.
We leverage on the optimality of
Gaussian test channels, shown in Theorem 2, to restrict the optimization of P to Gaussian distributions, which are easily represented by a ﬁnite set of parameters, namely mean and covariance.
We show that if P ( t ) are
Gaussian distributions, then P ( t +1) are also Gaussian distributions, which can be computed with an efﬁcient update algorithm of its representing parameters.
In particular, if at time t the k -th distributions P ( t ) U k | X k is given by
[Equation]
where Z tk ∼ CN ( 0 , Σ z tk ) , we show that at t + 1 , for P ( t +1) updated as in equation 82, the encoder P ( t +1)
U k
| X k corresponds to U t
+1 k
=
A t +1 k X k + Z t +1 k , where Z t +1 k ∼ CN ( 0 , Σ z t +1 k ) and Σ z t +1 k , A t +1 k are updated as Σ
[Equation]
[Equation]
The detailed update procedure is given in Algorithm 2 (see the following section for the details of the derivations).
[Equation] [Equation]
[Equation] [Equation]
[Equation]
7.2. D ERIVATION OF A LGORITHM
We derive the update rules of Algorithm 2 and show that the Gaussian distribution is invariant to the update rules in Algorithm 1, in line with Theorem 2.
First, we recall that if ( X 1 , X 2 ) are jointly Gaussian, then
[Equation]
where µ x 2 | x 1 :=
K x 2 | x 1 x 1 , with K x 2 | x 1 := Σ x 2 , x 1 Σ − 1 x 1 .
Then, for Q ( t +1) computed as in equation 21 and equation 22 from P ( t ) , which is a set of Gaussian distributions, we have
[Equation]
Next, we look at the update P ( t +1) as in equation 82 from given Q ( t +1) .
First, we have that p ( u tk ) is the marginal of U tk , given by U tk ∼ CN ( 0 , Σ u tk ) where Σ u tk = A tk
Σ x k
A t,Hk + Σ z tk .
Then, to compute ψ s ( u tk , x k ) , ﬁrst, we note that
E
[Equation]
and that for two generic multivariate Gaussian distributions P 1 ∼ CN ( µ 1 , Σ 1 ) and P 2 ∼ CN ( µ 2 , Σ 2 ) in C N , the KL divergence is computed as in equation 80 below.
Applying equation 91 and equation 80 in equation 83 and noting that all involved distributions are
Gaussian, it follows that ψ s ( u tk , x k ) is a quadratic form.
Then, since p ( u tk ) is Gaussian, the product log( p ( u tk ) exp( − ψ s ( u tk , x k ))) is also a quadratic form, and identifying constant, ﬁrst and second order terms, we can write
[Equation]
where Z ( x k ) is a normalization term independent of u k ,
[Equation]
and
[Equation] [Equation]
This shows that p ( t +1) ( u k | x k ) is a multivariate Gaussian distribution and that U t +1 k |{
X k = x k } is also a multivariate Gaussian distributed as CN ( µ u t
+1 k
| x k , Σ z t +1 k ) .
Next, we simplify equation 93 and equation 94 to obtain the update rules equation 86 and equation 87.
From the matrix inversion lemma, similarly to Chechik et al.
(Feb. 2005), for ( X 1 , X 2 ) jointly Gaussian we have
[Equation]
Applying equation 95, in equation 93 we have
[Equation]
where equation 97 is due to the Markov chain U k − (cid:10) − Y − (cid:10) − U K\ k .
Then, also from the matrix inversion lemma, we have for jointly Gaussian
( X 1 , X 2 ) ,
[Equation]
Applying equation 98 to equation 94, for the ﬁrst term in equation 94, we have
[Equation]
where Σ y , u tk =
A tk Σ x k , y ; and equation 100 is due to the deﬁnition of Σ x k | y .
Similarly, for the second term in equation 94, we have
[Equation]
where we use Σ u tk , yu t K\ k
=
A tk Σ x k , yu t K\ k ; and equation 102 is due to the Markov chain U k − (cid:10) − Y − (cid:10) − U K\ k .
For the third term in equation 94,
[Equation]
Equation equation 87 follows by noting that µ u t +1 k
| x k =
A t +1 k x k , and that from equation 94 A t +1 k can be identiﬁed as in equation 87.
Finally, we note that due to equation 85, Σ u tk |
y and Σ u tk | u t K\ k are given as in equation 88 and equation 89, where Σ x k | y =
Σ k and Σ x k |