T ARGET -E MBEDDING A UTOENCODERS FOR S UPERVISED R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings.
This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional.
We motivate and formalize the general framework of targetembedding autoencoders (T EA ) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets—encoding the prior that variations in targets are driven by a compact set of underlying factors.
As our theoretical contribution, we provide a guarantee of generalization for linear T EA s by demonstrating uniform stability, interpreting the beneﬁt of the auxiliary reconstruction task as a form of regularization.
As our empirical contribution, we extend validation of this approach beyond existing static classiﬁcation applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures—thereby underscoring the further generality of this framework beyond feedforward instantiations.
I NTRODUCTION
Representation learning deals with uncovering useful underlying structures of data, and autoencoders (Hinton & Salakhutdinov, 2006) have been a staple in a variety of problems.
While much research focuses on its use in unsupervised or semi-supervised settings with such diverse objectives as sparsity (Ranzato et al., 2007), generation (Kingma & Welling, 2013), and disentanglement (Chen et al., 2018), autoencoders are also useful in purely supervised settings—in particular, adding an auxiliary feature-reconstruction task to supervised classiﬁcation problems has been shown to empirically improve generalization (Le et al., 2018); in the linear case, the theoretically quantiﬁable beneﬁt matches that of simplistic norm-based regularization (Bousquet & Elisseeff, 2002; Rosasco & Poggio, 2009).
In this paper, we consider the inverse problem setting where the target space Y is high-dimensional; for instance, consider the multi-label classiﬁcation tasks of image tagging, text annotation, and image segmentation. This is in contrast to the vast majority of works designed to tackle a high-dimensional feature space X (where commonly |X| (cid:29) |Y| , such as in standard classiﬁcation problems).
In this setting, the usual (and universal) strategy of learning to reconstruct features (Weston et al., 2012; Kingma et al., 2014; Le et al., 2018) may not be most useful: learning latent representations that encapsulate the variation within X does not directly address the more challenging problem of mapping back up to a higher-dimensional Y .
Instead, we argue for leveraging intermediate representations that are compact and more easily predictable from features, yet simultaneously guaranteed to be predictive of targets.
In the process, we provide a uniﬁed theoretical perspective on recent applications of autoencoders to label-embedding in static, high-dimensional classiﬁcation problems (Yu et al., 2014; Girdhar et al., 2016; Yeh et al., 2017).
Extending into the temporal setting, we further empirically demonstrate the generality of target-embedding for recurrent, multi-variate sequence forecasting.
Our contributions are three-fold.
First, we motivate and formalize the target-embedding autoencoder (T EA ) framework: a general approach applicable to any underlying architecture.
Second, we provide a theoretical learning guarantee in the linear case by demonstrating uniform stability; speciﬁcally, we obtain an O (1 /N ) bound on instability by analogizing the beneﬁt of the auxiliary reconstruction task to a form of regularization—without incurring additional bias from explicit shrinkage.
Finally, we extend empirical validation of this approach beyond the domain of static classiﬁcation: using the task of multivariate disease trajectory forecasting as case study, we experimentally validate the advantage that T EA s confer on both linear and nonlinear architectures using real-world datasets with [Table] [Equation] [Table]
[Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] (null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> Y <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> X <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> f
o r w a r d ( s h a r e d ) e m b e d predict
Y <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Figure]
both continuous and discrete targets.
To the best of our knowledge, we are the ﬁrst to formalize and quantify the theoretical beneﬁt of autoencoder-based target-representation learning in a purely supervised setting, and to extend its application to the domain of multivariate sequence forecasting.
T ARGET -E MBEDDING A UTOENCODERS
Let X and Y be ﬁnite-dimensional vector spaces, and consider the supervised learning problem of predicting targets y ∈ Y from features x ∈ X .
With a ﬁnite batch of N training instances
D = { ( x n , y n ) }
Nn =1 , the objective is to learn a mapping h : X → Y that generalizes well to new samples from the same distribution.
The vast majority of existing work consider the setting—most commonly, classiﬁcation—where |X| (cid:29) |Y| ; under this scenario, autoencoders are often used to ﬁrst transform the input into some lower-dimensional representation
z ∈ Z amenable to the downstream task.
Doing so involves adding an auxiliary reconstruction loss (cid:96) r to the primary prediction loss (cid:96) p .
Formally, solutions of this form—in supervised and semi-supervised settings alike—consist of a shared forward model φ : X → Z , a reconstruction function r : Z → X , and a prediction function d :
Z → Y during training (where notation d reﬂects the downstream nature of the prediction task).
Denote ˜ x
= r ( φ ( x )) and ˆ y =
d ( φ ( x )) ; then the complete loss function takes the following form, [Equation]
[Equation]
In contrast, we focus on settings where the target space Y is high-dimensional, and where possibly |Y| > |X| .
In this case, we argue that learning to reconstruct the input is not necessarily most beneﬁcial.
In a simple classiﬁcation problem, autoencoding inputs leverages the hypothesis that a reconstructive representation is also likely discriminative.
In our setting, however, the more immediate problem is the high-dimensional structure of Y ; in particular, there is little guarantee that intermediate representations trained to encapsulate x are easily mapped back up to higher-dimensional targets.
Our goal is to make use of intermediate representations that are both predictable from features as well as predictive of targets.
A target-embedding autoencoder (T EA )—versus what we shall term a feature-embedding autoencoder (F EA )—ﬂips the model architecture around by learning an embedding of target vectors instead, which a predictor then learns a mapping into.
This involves an encoder e : Y → Z , an upstream predictor u : X → Z , and a shared forward model θ : Z → Y .
Denote ˜
y
= θ ( e ( y )) and ˆ y =
θ ( u ( x )) ; the complete loss function is now of the following form,
[Equation]
Abstractly, the general idea of target space reduction is not new; in particular, it has been present in various solutions in the domain of multi-label classiﬁcation (see Section 4 and Appendix B for discussions of related work).
Here we focus on target-embedding autoencoders ; they leverage the assumption that variations in (high-dimensional) target space are driven by a compact and predictable set of factors.
By construction, learning to reconstruct directly in output space ensures that latent representations are predictive of targets; at the same time, jointly training with the prediction loss ensures that latent representations are predictable from features.
Instead of learning representations for mapping out of (downstream), here we learn representations for mapping into (upstream); the shared forward model handles the rest.
See Figure 1 for high-level diagrams of T EA s versus F EA s.
Training and Inference .
Figure 2 gives block diagrams of component functions and objectives in (a) F EA s and (b) T EA s during training (see Algorithm 1 in Appendix C for pseudocode).
Training occurs in three stages. First, the autoencoder is trained (to learn representations): the encoder and decoder are trained on the reconstruction loss. Second, the prediction arm is trained to regress the learned
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` p @ ⇥ < latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` p @
W u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
<latexit sha1_base64="(null)">(null)</latexit> @` r @
W e <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @ ⇥ <
latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ` r <latexit
[Table]
=
[Table]
(null
[Table]
(null)<
[Table]
latexit> <latexit [Table] [Table] =
[Table]
(null)" [Table] (null)<
[Table]
(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Figure]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Figure]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit
[Figure]
latexit
[Figure]
_base
[Figure]
"(null)"
[Figure]
(null)
[Figure]
/latexit
[Figure]
<latexit sha1_base [Figure] "(null)
[Figure]
(null)</latexit> <latexit
[Figure]
[Figure]
"(null)">(null)</latexit> <latexit
[Figure]
_base64="(null
[Figure]
(null
[Figure]
</
[Figure]
[Figure]
[Figure]
(null)
[Figure] [Figure]
<
[Figure] [Figure] [Figure]
[Figure] [Figure] embeddings (generated by the encoder): while the predictor is trained (on the latent loss), parameters of the autoencoder are frozen. Finally, all three components are jointly trained on both the prediction loss and reconstruction loss: parameters of the predictor, embedding, and shared forward model are all trained simultaneously.
Note that during training, the forward model receives two types of latents as input: encodings of true targets, as well as encodings predicted from features. At inference time, the target-embedding arm is dropped, leaving the learned hypothesis h = θ ◦ u for prediction. Figure 4 (Appendix C) provides step-by-step block diagrams of both training and inference in greater detail.
We emphasize that T EA s—as is the case with F EA s—specify a general framework independent of the implementation details of each component.
For instance, the solutions to applications in Yu et al.
(2014), Yeh et al.
(2017), and Girdhar et al.
(2016) can be abstractly regarded as linear and nonlinear instances of this framework, with domain-speciﬁc architectures (see Section 4 and Appendix B for more detailed discussions).
Linear T EA s—which we study in greater detail in Section 3—involve parameterizations ( Θ , W u , W e ) of ( θ, u, e ) consisting of single hidden layers with linear activation.
S TABILITY -B ASED L EARNING G UARANTEE
Two questions are outstanding.
The ﬁrst is theoretical.
We are motivated by the prior that variations in target space are driven by a lower-dimensional set of underlying factors.
In this context, can we say something more rigorous about the beneﬁt of T EA s?
In this section, we take the ﬁrst step in showing that jointly learning target representations improves generalization performance in the supervised setting.
Speciﬁcally, we demonstrate that linear T EA s are characterized by uniform stability, from which theoretical guarantees are known to follow.
The second question is empirical.
We noted above that certain applications of label-embedding to classiﬁcation can be interpreted through this framework.
Does the beneﬁt extend beyond its static, feedforward instantiations—into the temporal setting for multi-variate sequence forecasting, with both continuous and discrete targets?
In Section 5, we ﬁrst validate our theoretical ﬁndings with linear models and sensitivities, as well as extending our empirical analysis to the realm of recurrent, nonlinear models for both regression and classiﬁcation.
Consider a linear T EA , where the upstream predictor is parameterized by W u ∈ R |Z|×|X| , targetembedding by W e ∈ R |Z|×|Y| , and shared forward model by Θ ∈ R |Y|×|Z| , where |Z| < |Y| .
The complete loss function is given by L = 1 N (cid:80) Nn
=
[ (cid:96) p ( ΘW u x n , y n ) + (cid:96) r
( ΘW e y n , y n )] following
Equation (2).
Interpreting the jointly learned autoencoding component as an auxiliary task, we show that the T EA algorithm for learning the shared forward model Θ is uniformly stable with respect to the domain of the supervised prediction task.
To establish our notation, ﬁrst recall the following:
Deﬁnition 1 (Generalization Bound)
Given a learning algorithm D (cid:55)→ h D that returns hypothesis h D , let R ( h D ) =
(cid:82) (cid:96) ( h D ( x ) , y ) dµ ( x , y ) denote the risk, and ˆ R ( h D )
= 1 N (cid:80) Nn
=1 (cid:96) ( h D ( x n ) , y n ) denote the empirical risk, where (cid:96) is some loss function.
A generalization bound is a probabilistic bound on the defect that takes the following form: R ( h D ) − ˆ R ( h D ) ≤ (cid:15) with some conﬁdence 1 − δ .
Deﬁnition 2 (Uniform Stability) Let D i denote a modiﬁcation of batch D where the i -th training instance ( x i , y i ) is replaced by an independent and identically distributed example ( x (cid:48)
i , y (cid:48) i ) .
A learning algorithm is said to be γ -uniformly stable with respect to the loss function (cid:96) if ∀D ∈ ( X ×Y ) N , ∀ i ∈ { 1 , ..., n } , ∀ ( x , y ) , ( x (cid:48)
i , y (cid:48) i )
∈
X ×Y : | (cid:96) ( h D ( x ) , y ) − (cid:96) ( h D i ( x ) , y ) | ≤ γ .
Uniform stability holds if the minimum value of γ converges to zero as batch size N increases without limit.
Uniform stability can be used to derive algorithm-dependent generalization bounds.
In particular,
Bousquet & Elisseeff (2002) ﬁrst showed that the defect (cid:15) of a γ -uniformly stable algorithm is is less than O (( γ + 1 /N )
(cid:112) N log(1 /δ )) with probability ≥ 1 − δ .
Feldman & Vondrak (2018) recently demonstrated an improved bound of O ( (cid:112) ( γ + 1 /N )
log(1 /δ )) .
Here, we show uniform stability for linear T EA s, where γ is O (1 /N ) —by which a tight generalization bound follows immediately.
Before we begin, we introduce two additional tools: c -strong convexity and σ -admissibility.
Note that these conditions are standard and easily satisﬁed—for instance, by the quadratic loss function; for more context see for example Bousquet & Elisseeff (2002), Liu et al.
(2016), and Mohri et al.
Deﬁnition 3 ( c -Strong Convexity) Differentiable loss function (cid:96) is c -strongly convex if ∀ h, h (cid:48) ∈
H : (cid:104) h (cid:48)
( x ) − h ( x ) , ∇ (cid:96) ( h (cid:48) ( x ) , y ) − ∇ (cid:96) ( h ( x ) , y ) (cid:105) ≥ c (cid:107) h (cid:48)
( x ) − h ( x ) (cid:107) 22 for some c ∈ R + , where ∇ (cid:96) ( h ( x ) , y ) denotes the gradient with respect to h ( x ) , and (cid:104)· , ·(cid:105) denotes the dot product operation.
Deﬁnition 4 ( σ -Admissibility)
Loss function (cid:96) is σ -admissible with respect to the underlying hypothesis class H if ∀ h, h (cid:48)
∈ H : | (cid:96) ( h (cid:48) ( x ) , y ) − (cid:96) ( h ( x ) , y ) | ≤ σ (cid:107) h (cid:48)
( x ) − h ( x ) (cid:107) 2 for some σ ∈ R + .
To obtain uniform stability, we make two assumptions—both analogous to prior work arguing from the beneﬁt of learning shared models between tasks.
Liu et al.
(2016) deals with learning multiple tasks in general, and Le et al.
(2018) deals with reconstructing inputs in what we describe as F EA s.
Now in multi-task learning, the separate tasks are usually chosen due to some prior relationship between them.
In the case of Assumption 1 in Liu et al.
and Assumption 5 in Le et al.
(2018), this is assumed to come from similarities in feature structures across tasks; hence their assumptions of cross-representability are made in feature space.
(Note that this restricts primary and auxiliary features to be elements of the same space).
Our setting is contrary: the inputs to primary and auxiliary tasks come from different spaces, but are trained to produce similar labels through a compact, shared latent space; hence our assumption of cross-representability will be made in this latent space instead.
Assumption 1 (Representative Vectors)
There exists a representative subset of target vectors
B = { b 1 , ..., b M } ⊂ { y 1 , ..., y N } such that the latent representation of any individual ( x , y ) can be linearly reconstructed from that of the representative subset with small error, i.e. W p x = (cid:80) Mm =
α m W e b m + η and W e y
= (cid:80)
Mm =
β m W e b m + η for some coefﬁcients α m , β m ∈ R , where
Σ Mm =
α 2 m ≤ r 2 α , Σ Mm
=1 β 2 m ≤ r 2 β for some r α , r β ∈ R + , and η is a small error satisfying (cid:107) η (cid:107) 2 ≤ ε .
Remark
This assumption is comparatively mild, even for ε = 0 .
Note that in Liu et al.
the features for separate tasks come from different examples in general, and the similarity of their distributions within X is simply assumed.
Here, each pair of inputs to the prediction and reconstruction tasks comes from the same instance, and similarity within Z is explicitly enforced through the (joint) training objective.
In addition, observe that the assumption will hold with zero error as long as the number of independent latent vectors is at least |Z| .
Furthermore, unlike the scenarios in Liu et al.
(2016) and Le et al.
we do not require that the input domains of the two tasks be identical.
Therefore for ease of exposition, we assume going forward that ε =
(see Remark 6 in Appendix A).
Remark 2 A comparison with Assumption 4 in Le et al.
(2018) sheds additional light on why we expect T EA s to be beneﬁcial where |Y| > |Z| , in contrast with the (more typical) scenario |X| > |Z| (cid:29) |Y| .
Critically, the technique in Le et al.
banks on the fact that the prediction arm projects the latent into a lower-dimensional target space.
Conversely, Assumption 1 here relies on the fact that the encoding arm maps into the latent space from a higher-dimensional target space (rendering cross-representability therein reasonable).
The distinction is crucial: we certainly do not expect any beneﬁt from autoencoding trivially low-dimensional vectors!
Note also that here the representative vectors are taken from Y ; to take them from X instead would be unreasonable.
For any compressive autoencoder, we generally expect if some subset { b 1 , ..., b M } ⊂ { y 1 , ..., y N } spans
Y that { W e b 1 , ..., W e b M } then also span Z in order to be maximally reconstructive.
The same cannot be said of subsets
[Equation]
M } [Equation] [Equation] ..., x N } that span X —for instance, take |X| ≤ |Z| .
In addition to being representative in terms of latent values, the set of representative points also needs to be representative in terms of the reconstruction error.
First, let L (cid:48) denote the counterpart to L where the i -th sample ( x i , y i ) is replaced by some new instance ( x (cid:48)
i , y (cid:48) i ) ∈
X × Y ; that is,
L (cid:48)
= 1 N [ (cid:96) p ( ΘW u x (cid:48)
i , y (cid:48) i ) + (cid:96) r
( ΘW e y (cid:48) i , y (cid:48) i ) + (cid:80) Nn =1; n (cid:54) =
i ( (cid:96) p ( ΘW u x n , y n ) + (cid:96) r
( ΘW e y n , y n ))] .
Then, let Θ ∗ , Θ (cid:48)∗ denote the optimal parameters corresponding to the two losses L and L (cid:48) .
Assumption 2 (Representative Errors) Let L (cid:48) r contain the reconstruction errors of the dataset without the i -th sample: L (cid:48)
r = (1 /N )
Σ Nn =1; n (cid:54) =
i (cid:96) r ( ΘW e y n , y n ) , and let L Br denote the reconstruction error of the representative subset: L Br = (1 /M )
Σ Mm =1 (cid:96) r ( ΘW e b m , b m ) .
Then there exists some a > 0 such that for any small κ > 0 :
L Br ( Θ ∗ ) − L Br ( κ Θ (cid:48)∗ + (1 − κ ) Θ ∗ ) + L Br ( Θ (cid:48)∗ ) − L Br ( κ Θ ∗ + (1 − κ ) Θ (cid:48)∗ ) ≤ a [ L (cid:48)
r ( Θ ∗ ) − L (cid:48)
r ( κ Θ (cid:48)∗ + (1 − κ ) Θ ∗ )] + a [ L (cid:48) r ( Θ (cid:48)∗ ) − L (cid:48) r ( κ Θ ∗ + (1 − κ ) Θ (cid:48)∗ )] .
That is, the difference in reconstruction error L Br between the two points Θ ∗ , Θ (cid:48)∗ is upper bounded by some constant factor of the corresponding difference in reconstruction error L (cid:48)
r at the two points.
Importantly, note that this does not require that the values of the errors L (cid:48) r and L Br themselves be similar, only that their differences be similar.
This assumption is identical to Assumption 6 in Le et al.
(2018), and plays an identical role: we make use of L Br —which is only dependent on M —to allow the bound to decay with N ; this is in contrast with the generic multi-task analysis of Liu et al.
(2016), which—if applied directly to T EA s (as with F EA s)—would give a bound that does not decay with N .
Theorem 1 (Uniform Stability) Let (cid:96) p and (cid:96) r be σ p -admissible and σ r -admissible loss functions, and let (cid:96) r be c -strongly convex.
Then under Assumptions 1 and 2, the following inequality holds,
[Equation]
cN Proof .
Appendix A.
Corollary 1 (Generalization Bound) Consider the same conditions as in Theorem 1; that is, let (cid:96) p and (cid:96) r be σ p -admissible and σ r -admissible losses, and let (cid:96) r be c -strongly convex.
Then under
Assumptions 1 and 2, the defect (cid:15) is less than O ( (cid:112) (1 /N ) log(1 /δ )) with probability at least 1 − δ .
Proof .
Follows immediately from Theorem 1 (above) and either of the following (similar results hold for both): Theorem 1.2 (Feldman & Vondrak, 2018), and Theorem 12 (Bousquet & Elisseeff, 2002).
In supervised learning, it is often easy to make an argument—on an intuitive level—for the “regularizing” effect of additional loss terms.
In contrast, this analysis allows us to unambiguously identify and quantify the beneﬁt of the embedding component as a regularizer (see Remark 4 in Appendix A).
Remark
In the linear label space reduction framework of Yu et al.
(2014), uniform convergence is also shown to hold via norm-based regularization.
Speciﬁcally for uniform stability, a similar bound can also be achieved by adding a strongly convex term to the objective, such as Tikhonov—and (cid:96) 2 —regularization (Bousquet & Elisseeff, 2002; Rosasco & Poggio, 2009; Shalev-Shwartz et al., 2010).
Here, however, the joint reconstruction task leverages a different kind of bias—precisely, the assumption that there exist compact and predictable representations of targets.
Therefore the signiﬁcance of this analysis is that we achieve an equivalent result independent of explicit regularization.
R ELATED W ORK
Our work straddles three threads of research: (1) supervised representation learning with autoencoders, (2) label space reduction for multi-label classiﬁcation, and (3) stability-based learning guarantees.
Appendix B provides a much expanded treatment, and presents summary tables for additional context.
Supervised representation learning .
While a great deal of research is devoted to uncovering useful underlying structures of data through autoencoders—with various properties such as sparsity (Ranzato et al., 2007) and disentanglement (Chen et al., 2018), among many others (Tschannen et al., 2018)—the goal of better representations is often for the beneﬁt of downstream tasks.
Semi-supervised autoencoders jointly optimized on partially-labeled data can obtain compact representations that improve prediction (Weston et al., 2012;
Kingma et al., 2014; Ghifary et al., 2016).
Furthermore, auxiliary reconstruction is also useful in a purely supervised setting: rather than focusing on how speciﬁc architectures better structure unlabeled data, Le et al.
(2018) show the simple beneﬁt of feature-reconstruction on supervised classiﬁcation—a special case of what we describe as F EA s.
In contrast, we focus on target -representation learning in the supervised setting, and analyze its beneﬁt under the prior that high-dimensional targets are driven by a compact and predictable set of factors.
We take inspiration from the empirical study of Girdhar et al.
(2016), where latent representations of 3D objects are jointly trained to be predictable from 2D images.
Their setup can be viewed as a speciﬁc instance of T EA s with (nonlinear) convolutional components, with a minor variation in training: in the joint stage, predictors continue to regress the learned embeddings, and gradients only backpropagate from latent space (instead of target space).
Unlike the symmetry of our losses (which we require for our analysis above), their common decoder is only shared indirectly (and predictions made indirectly).
As it turns out, this does not appear to matter for performance (see Section 5). In
Mostajabi et al. (2018), a two-stage procedure is used for semantic segmentation—loosely comparable to the ﬁrst two stages in T EA s; in contrast to our emphasis on joint training, they study the beneﬁt of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding, Dalca et al.
(2018) build anatomical priors for biomedical segmentation in unsupervised settings.
Multi-label classiﬁcation . The general idea of target space dimension reduction has been explored for multi-label classiﬁcation problems (commonly, annotation based on bags of features).
These ﬁrst derive a reduced label space, then subsequently associate inputs to it; methods include compressed sensing (Hsu et al., 2009), principal components (Tai & Lin, 2010), maximum-margin coding (Zhang & Schneider, 2012), and landmarking (Balasubramanian & Lebanon, 2012).
Closer to our theme of joint learning , Chen & Lin (2012)
ﬁrst propose simultaneously minimizing encoding and prediction errors via an SVD formulation.
Using generic empirical risk minimization, Yu et al.
(2014) formulate the problem as a linear model with a low-rank constraint.
While this captures an intuition (similar to ours) of restricted latent factors, their error bounds require norm-based regularization (unlike ours).
Recently, Yeh et al.
(2017) generalize the label-embedding approach to autoencoders .
This ﬂexibly accommodates custom losses to exploit correlations, as well as deep learning for nonlinearities.
Our work is related to this line of research, although we operate at a higher level of abstraction, with a signiﬁcant difference in focus.
Their problem is multi-label classiﬁcation, and their starting point is binary relevance (i.e. label by label).
During reduction, they worry about speciﬁc losses that capture dependencies within and among spaces.
In contrast, we worry about autoencoding at all —that is, we focus on the effect of joint reconstruction on learning the prediction model.
Problems can be of any form: classiﬁcation or regression, and our starting point is direct prediction (i.e. no reconstruction).
Stability and learning guarantees .
Generalizability via hypothesis stability is ﬁrst studied in Rogers & Wagner (1978) and Devroye & Wagner (1979); unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis, 1971; Pollard, 1984; Koltchinskii, 2001), these account for how the algorithm depends on the data.
Bousquet & Elisseeff (2002)
ﬁrst formalize the notion of uniform stability sufﬁcient for learnability, and Feldman & Vondrak (2018) use ideas related to differential privacy (Bassily et al., 2016) for further improvement.
Separately, while there is a wealth of research on dimensionality reduction and autoencoders (Singh et al., 2009; Mohri et al., 2015;
Gottlieb et al., 2016; Epstein & Meir, 2019), they either operate in the semi-supervised setting, or focus on the beneﬁt of feature representations (not targets) and also do not consider joint learning.
The beneﬁt of jointly learning multiple tasks through a common operator (Caruana, 1997) is explored with VC-based (Baxter, 2000) and Rademacher complexity-based (Maurer, 2006; Maurer et al., 2016) analyses.
Recently, Liu et al.
(2016) show that the algorithm for learning the shared model in a multi-task setting is uniformly stable.
While our argument is based on theirs, we are not interested in a generic bound for all tasks; closer to Le et al.
, we focus on the primary prediction task, and leverage the auxiliary reconstruction task for stability.
Similarly, we arrive at an O (1 /N ) on instability without an explicit regularization term as in Bousquet & Elisseeff (2002).
Unlike them, however, the fundamental distinction of our setting is that Y is high-dimensional (but where the underlying factors are assumed compact); in this sense our focus is the mirror opposite of theirs.
E XPERIMENTS AND D ISCUSSION
So far, we have formalized a general target-autoencoding framework for supervised learning, and quantiﬁed the beneﬁt via uniform stability.
Our overall goal in this section is to explore this beneﬁt in a simple controlled setting, such that we can identify and isolate its utility on the prediction task, and investigate any sensitivities of interest.
By way of preface, we emphasize two observations from above: (1) In the static, multi-label classiﬁcation setting, the gain from label-embedding has been studied, including the autoencoder approach of Yeh et al.
(2017)—which can be viewed as an instantiation of T EA s with sophisticated reﬁnements.
The beneﬁt of target-autoencoding is also [Table] demonstrated using nonlinear, convolutional architectures in Girdhar et al.
(2016)—which is also an instantiation of T EA s, also noting signiﬁcant gains.
Therefore a natural question of interest is:
• Does the utility of target-embedding extend to (nonlinear) recurrent models with sequential data for general, high-dimensional targets (i [Table]
e. regression and/or classiﬁcation)?
Disease Trajectories .
In this section, we take the ﬁrst step in answering this question—as our empirical contribution, we extend validation of target-embedding autoencoders to the domain of multivariate sequence forecasting , exploring its utility on linear and nonlinear sequence-to-sequence architectures.
What makes a good testbed?
In particular, the progression of diseases (and their markers) is high-dimensional in presentation; at the same time, their evolution is often driven by latent biological dynamics (Szczesniak et al., 2017; Pascoal et al., 2017; Alaa & van der Schaar, 2019).
With the increasing importance of early diagnosis and timely intervention in healthcare, the ability to forecast individual disease trajectories ( Y ) in the presence of limited windows of information ( X ) has become increasingly desirable (Donohue et al., 2014; Pham et al., 2017; Bhagwat et al., 2018).
Datasets .
We use three datasets in our experiments.
The ﬁrst consists of a cohort of patients enrolled in the UK Cystic Fibrosis registry ( UKCF ), which records follow-up trajectories for over 10, patients.
We are interested in forecasting future trajectories for the 11 possible infections andpossible comorbidities (all binary variables) recorded at each follow-up, using past trajectories and basic demographics as input.
The second consists of patients in the Alzheimer’s Disease
Neuroimaging Initiative study ( ADNI ), which tracks disease progression for over 1,700 patients.
We are interested in forecasting the evolution of the 8 primary biomarkers and 16 cognitive tests (all continuous variables) measured at each visit, using past measurements and basic demographics as input.
The third consists of a cohort of patients in intensive care units from the Medical Information
Mart for Intensive Care ( MIMIC ), which records physiological data streams for over 22,000 patients.
Likewise, we are interested in forecasting future trajectories for the 361 most frequently measured variables such as vital signs and lab tests (both binary and continuous variables), again using past measurements and basic demographics as input.
See Appendix D for more information on datasets.
Experimental Setup .
In each instance, the prediction input is a precedent window of up to width w x , and the prediction target is the succedent window of width w y .
For UKCF ( w x , w y ) =
(3 , 4) at 1-year resolution, for ADNI (4 , 8) at 6-month resolution, and for MIMIC (5 , 5) at 4-hour (resampled) resolution.
All models are implemented in Tensorﬂow.
Linear models consist of a single hidden layer with no nonlinearity; for the nonlinear case, we implement an RNN model for each component using GRUs.
See Appendix D for additional detail on model implementation and conﬁguration.
For evaluation, we measure the mean squared error (M SE ) for continuous targets (averaged across variables), and the area under the precision-recall curve (P RC ) and area under the receiver operating characteristic curve (R OC ) for binary targets (averaged across variables).
We use cross-validation on the training set for hyperparameter tuning, selecting the setting that gives the lowest validation loss averaged across folds.
For each model and dataset, we report the average and standard error of each performance metric across 10 different experiment runs, each with a different random train-test split.
Note that forecasting high-dimensional disease trajectories is challenging, and input information is deliberately limited (as is often the case in medical practice); the desired targets are similar or higherdimensional than the inputs (see Table 1).
This obviously results in an inherently difﬁcult prediction problem—but which makes for a good candidate setting to test the utility of target-representation learning.
RNN autoencoders have previously been proposed for learning representations of inputs (i.e. F EA s instantiated with RNNs) to improve classiﬁcation (Dai & Le, 2015),
prediction (Lyu et al., 2018), generation (Srivastava et al., 2015), and clustering (Baytas et al., 2017); similarly, their mission is not in excessively optimizing speciﬁc architectural novelties to match state-of-the-art, but rather in exploring the beneﬁt of the autoencoding framework.
Here, we learn representations of targets . [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]  M AIN R ESULTS
Overall Beneﬁt .
First, we examine the overall utility of T EA s. To verify the linear case ﬁrst, Tablesummarizes the performance of T EA and alternate setups on UKCF.
The temporal axis is ﬂattened to simulate ordinary static multi-label classiﬁcation
[Table]
and the base case is direct prediction (Base)—that is, absent any auxiliary representation learning or regularization.
Next, we allow for (cid:96) 2 -regularization over direct prediction (R EG ), as well as over all other methods.
F EA s differ only by the added feature-reconstruction, and T EA s only by the target-reconstruction; as an additional sensitivity, we also implement a combined approach (F/T EA ).
More generally, we also wish to examine the beneﬁt of T EA for the nonlinear case: Table 3 summarizes analogous results where component functions are implemented with GRU networks; results are shown for all datasets.
Ceteris paribus, we observe that target-representation learning has a notable positive impact on performance.
Interestingly, learning representations of inputs does not yield signiﬁcant beneﬁt, and the hybrid approach (F/T EA ) is worse than T EA ; this suggests that forcing intermediate representation to encode both features and targets may be overly constraining.
(Note that for the linear model, the instances are restricted to those for which the full input window is available; as a consequence, the results for linear and nonlinear cases are not directly comparable).
Figures 4 (Appendix C) and 5 (Appendix D) give training diagrams for all comparators.
Additional experiment results (by model, timestep, metric) are in Appendix E.1-2.
Source of Gain .
There are two (related) interpretations of T EA s. First, we studied the regularization view in Section 3; this concerns the beneﬁt of joint training using both prediction and reconstruction losses.
Ceteris paribus, we expect performance to improve purely by dint of the jointly trained T EA objective.
Second, the reduction view says that T EA s decompose the (difﬁcult) prediction problem into two (smaller) tasks: the autoencoder learns a compact representation z of y , and the predictor learns to map x to z .
This makes the potential beneﬁt of staged training (Section 2 and Appendix C) intuitively clear, and suggests an alternative—that of simply training the autoencoder and predictor arms in two stages—à la Mostajabi et al.
As a general framework, T EA s is a combination of both ideas: all three components are jointly trained in a third stage—à la Girdhar et al.
We now account for the improvement in performance due to these two sources of beneﬁt; Table 4 does so for the linear case (on UKCF), and Table 5 for the more general nonlinear case (on all datasets).
The “No Joint” setting isolates the beneﬁt from staged training only. This is analogous to basic unsupervised pretraining (though using targets), and corresponds to omitting the ﬁnal joint training stage in Algorithm 1.
The “No Staged” setting isolates the beneﬁt from joint training only (without pretraining the autoencoder or predictor), and corresponds to omitting the ﬁrst two training stages in
Algorithm 1.
The “Neither” setting is equivalent to vanilla prediction (R EG ) without leveraging either of the advantages.
We observe that while both sources of beneﬁt are individually important, neither setting performs quite as well as when both are combined.
See Appendix E.1-2 for extended results. [Table] [Table] [Table]
Variations .
Having established the utility of target-embedding, we can ask whether variations on the same theme perform similarly.
In particular, the embeddings in the empirical studies of Girdhar et al.
and
Yeh et al.
(2017) are jointly learned via the reconstruction loss (cid:96) r and latent loss (cid:96) z —that is, the prediction arm continues to regress learned embeddings during the joint training stage (Figure 4(d), in Appendix D).
The principle is similar, although (as noted in Section 4) the primary task is therefore learned indirectly ; this is in contrast to the vanilla T EA setup, where the primary task is learned directly via the prediction loss (cid:96) p .
Tables 4 and 5 also compare the performance of vanilla T EA s with this indirect variant (T EA ( L )), as well as a hybrid variant (T EA ( LP )) for which both latent and prediction losses are trained jointly with the reconstruction loss (Figure 4(e).
Perhaps as expected, we observe that performance across all three variants are more or less identical, afﬁrming the general beneﬁt of target-representation learning.
Again, see Appendix E.1-2 for extended results.
5. S ENSITIVITIES
Regularization .
Of course, target-representation learning is not a replacement for other regularization strategies; it is an additional tool that can be used in parallel where appropriate.
Figure 3(a) shows the performance of T EA and R EG with various coefﬁcients ν on (cid:96) 2 -regularization.
By itself, introducing (cid:96) 2 -regularization does improve performance up to a certain point, beyond which the additional shrinkage bias incurred begins to be counterproductive; this is not surprising.
Interestingly, introducing target-representation learning appears to leverage an orthogonal bias: it consistently improves prediction performance regardless of level of shrinkage.
This is a practical result of the theoretical observation in Remark 3: while prior works obtain stability through explicit (cid:96) 2 -regularization, the beneﬁt from target-embedding relies on a different bias entirely, which allows us to combine them.
While increasing the strength of either form of regularization reduces variability in results (see also below), excessive bias of either alone degrades performance.
See Appendix E.3 for full results.
Strength of Prior .
Target-embedding attempts to leverage the assumption that there exist compact and predictable representations of targets.
Realistically (e.g. due to measurement noise), of course, this will not hold perfectly.
In our experiments, we set the ratio of prediction and reconstruction losses to be 1 : 1 for T EA (as well as F EA and F/T EA ); that is, the “strength-of-prior” coefﬁcient λ on (cid:96) r is 0 .
In order to isolate the effect of λ during joint training, we observe the performance of
T EA s with joint training only (i.e. removing the confounding effect of staged training).
For large values of λ , we expect the reconstruction task to dominate in priority, which is (under an imperfect prior) not beneﬁcial for the ultimate prediction task—in general, a hidden representation that is most reconstructive is not necessarily also what is most predictable).
For small values of λ , the setup begins to resemble direct prediction.
Figure 3(b) veriﬁes our intuition.
Note that in the extreme case of λ = 1 , predictions are no better than random (see R OC ∼ 0 . 5 ).
See Appendix E.3 for full results. [Figure]
Sample Complexity .
Figure 3(c) shows the performance of T EA and R EG under various levels of data scarcity.
The beneﬁt conferred by T EA s is signiﬁcant especially when the amount of training data N is limited .
Importantly, note that we are operating strictly within the context of supervised learning: unlike in semi-supervised settings, here we are not just restricting access to paired data; we are restricting access to data per se .
(Though beyond the scope of this paper, we expect that extending T EA s to semi-supervised learning with additional unpaired data would yield larger gains).
Here, without the luxury of learning from unpaired data, we highlight the comparative advantage purely from the addition of target-representation learning.
Again, see Appendix E.3 for full results.
5. D ISCUSSION
By way of conclusion, we emphasize the importance of our central assumption: that there exist compact and predictable representations of the (high-dimensional) targets.
This is critical: targetembedding is not useful where this is not true.
Now obviously, learning representations of targets is unnecessary if the output dimension is trivially small (e.g. if the target is a single classiﬁcation label), or if the problem itself is trivially easy (e.g. if direct prediction is already perfect).
Also obvious is the situation where representations cannot possibly be compact (e.g. if all output dimensions are independent of each other), in which case any model with a compressive (bottleneck) representation as an intermediate target may make little sense to begin with.
Perhaps less obvious is that we cannot assume that the goals of prediction and reconstruction are always aligned.
Just as in learning featureembeddings (for downstream classiﬁcation), what is most reconstructive may not necessarily encode what is most discriminative; so too in learning target-embeddings (for upstream prediction), what is most reconstructive may not necessarily encode what is most predictable.
In the case of disease trajectories, it is medical knowledge that permits this assumption with some conﬁdence.
Appendix
E.4 gives an extreme (synthetic) counterexample where this prior is outright false—i.e. prediction and reconstruction are directly at odds.
While certainly contrived, it serves as a caveat about assumptions.
Using the deliberately challenging setting of disease trajectory forecasting with limited information, we have illustrated the nontrivial utility of target-representation learning in a controlled setting with baseline models.
While we appreciate that component models in the wild may be more tailored, this setting better allows us to identify and isolate the potential utility of target-autoencoding per se .
In addition to verifying our intuitions for the linear case, we have extended empirical validation of target-autoencoding to (nonlinear) sequence-to-sequence recurrent architectures; along the way, we explored the sources of gain from joint and staged training, as well as various sensitivities of interest.
Where the prior holds, target-embedding autoencoders are potentially applicable to any high-dimensional prediction task beyond static classiﬁcation and imaging applications, and exploring its utility for other speciﬁc domain-architectures may be a practical direction for future research.
A P ROOF OF T HEOREM
Deﬁnition 5 (Bregman Distance) Let (cid:96) be some differentiable, strictly convex loss.
For any h, h (cid:48) ∈
H , the Bregman distance associated with (cid:96) is given by D (cid:96) ( h (cid:48) (cid:107) h ) =
(cid:96) ( h (cid:48) )
− (cid:96) ( h ) − (cid:104) h (cid:48) − h, ∇ (cid:96) ( h ) (cid:105) .
Additivity and non-negativity are easy to see; that is, D (cid:96) ( h (cid:48) (cid:107) h ) ≥ 0 for any h, h (cid:48) ∈ H , and if (cid:96) =
(cid:96)
+ (cid:96) 2 and both (cid:96) 1 and (cid:96) 2 are convex functions, then D (cid:96) ( h (cid:48)
(cid:107) h ) =
D (cid:96)
( h (cid:48)
(cid:107) h )
+ D (cid:96) 2 ( h (cid:48)
(cid:107) h ) .
Theorem 1 (Uniform Stability) Let (cid:96) p and (cid:96) r be σ p -admissible and σ r -admissible loss functions, and let (cid:96) r be c -strongly convex.
Then under Assumptions 1 and 2, the following inequality holds,
[Equation]
Proof .
The overall strategy consists of three steps in sequence.
First, we want to bound the delta in prediction loss under Θ ∗ and Θ (cid:48)∗ using the set of representative vectors B .
Second, we bound the resulting expression in terms of the Bregman divergence of the complete loss functions L and L (cid:48) under Θ ∗ and Θ (cid:48)∗ .
Third, we express the divergence back in terms of the original expression itself (consisting of representative vectors), which allows us to solve for a bound on that expression.
Finally, combining the results from the three steps completes the proof.
We begin with the left-hand term,
[Equation]
where the ﬁrst inequality follows from the fact that (cid:96) p is σ p -admissible, the equality follows from
Assumption 1 for some coefﬁcients α m ∈ R , and the second inequality follows from the CauchySchwarz inequality.
As our second step, the goal is to upper-bound the term under the square-root,
M (cid:88)
[Equation]
where the ﬁrst inequality follows from the fact that (cid:96) r is c -strongly convex
[Equation]
and the ﬁnal equality follows from the deﬁnition of Bregman divergence (the standalone loss terms cancel).
We want an expression in terms of the loss functions L and L (cid:48)
[Equation]
which will subsequently allow us to obtain a bound expressed back in terms of the set of representative vectors.
Focusing on the term in the brackets,
[Equation]
where the ﬁrst and last equalities follows from the deﬁnition of Bregman divergence, and the second equality from the deﬁnition of directional derivatives.
The ﬁrst inequality follows from Assumption 2; for the second inequality, note that L (cid:48) r consists of a strict subset of the set of strictly convex losses that L consists of
[Equation]
and similarly for L (cid:48) .
Therefore by additivity and non-negativity of the Bregman distance, we have that D L
[Equation]
≤
[Equation]
and
[Equation]
(cid:48)
[Equation]
Θ
[Equation]
Our third and ﬁnal step is to go back and bound this term again using the set of representative vectors,
[Equation]
where for ﬁrst equality note that by construction the gradients of the losses L and L (cid:48) are zero at the respective optimal models Θ ∗ and Θ (cid:48)∗ .
The ﬁrst inequality follows from the fact that (cid:96) p and (cid:96) r are σ p -admissible and σ r -admissible respectively
[Equation]
and the second inequality follows from Assumptionand the Cauchy-Schwarz inequality.
Now, combining Equations (4), (5)
[Equation]
and (6) allows us to write
[Equation]
3)—completes the proof. Remark 4 Formulating the autoencoding component as an auxiliary task allows us to unambiguously interpret its beneﬁt as a regularizer. Speciﬁcally, the complete loss can be summarized and rewritten as L ( Θ ) = L p ( Θ ) + R 1 ( Θ ) + R 2 ( Θ ) ; that is, the T EA objective is a combination of the primary prediction loss L p ( Θ )
= 1 N (cid:80) Nn =1 (cid:96) p ( ΘW u x n , y n ) plus additional regularization, where R 1 ( Θ ) = 1 N (cid:80) Mm =1 (cid:96) r ( ΘW e b m , b m ) =
MN L Br ( Θ ) and R 2 ( Θ ) = 1 N (cid:80) Nn =1 (cid:96) r
( ΘW e y n , y n ) − MN L Br ( Θ ) .
In particular, the proof of Theorem 1 relies on R 1 ( Θ ) to upper-bound instability (Appendix A).
This precisely identiﬁes the regularizer in question, while Theorem 1 quantiﬁes its generalization beneﬁt.
Remark 5 Technicality: Moving from uniform stability (Theorem 1) to generalization bounds (Corollary 1) requires that the loss function not take on arbitrarily large values (see e.g. Bousquet & Elisseeff (2002)).
In practice, the label space itself is often bounded (see e.g. Assumption 1 in
Le et al.
; then the problem effectively reduces to that with a bounded loss function.
For example, consider a regression setting using the quadratic loss function, where the target data lie within [ − U, U ] ; then the loss function is bounded to be within [0 , 4 U 2 ] .
See Castro & Nowak (2018).
Remark 6 Technicality: Earlier we assumed ε = 0 for ease of exposition; carrying around the extra
O ( ε ) term (from Equations (3) and (6)) is not particularly illuminating.
Generalizing to ε (cid:54) = 0 can be done in a similar manner to Le et al.
(2018), with the additional assumptions of bounded spaces and that ε decreases as 1 /N (which they note is reasonable, since the more samples in the data, the more likely the cross-representativity assumption will hold with low error).
Again, note that ε = 0 holds as long as the number of independent latent vectors is at least |Z| ).
Similarly, Liu et al.
(2016) consider ε = 0 , noting in any case that they can increase N to obtain a small (cid:107) η (cid:107) 2 ; see their analysis for detail.
B E XPANDED R ELATED W ORK
In this paper [Table] we motivate and analyze a general autoencoder-based target-representation learning technique in the supervised setting, quantifying the generalization beneﬁt via an argument from uniform stability, as well as verifying its practical utility.
As such, our work lies at the intersection of three threads of research: (1) supervised representation learning using autoencoders, (2) label space reduction for multi-label classiﬁcation, as well as (3) algorithmic stability-based learning guarantees.
B.1 Supervised Representation Learning using Autoencoders
[Table]
-based representation learning (Hinton & Salakhutdinov, 2006) has long played important roles in unsupervised and semi-supervised settings.
Various inductive biases have been proposed to promote representations that are sparse (Ranzato et al., 2007), discrete (van den Oord et al., 2017), factorized (Chen et al., 2018), or hierarchical (Zhao et al., 2017), among others.
For a more thorough overview of various methods, we refer the reader to Bengio et al.
(2013) and Tschannen et al.
The goal of better representations is often for the beneﬁt of downstream tasks.
Semi-supervised autoencoders—trained on partially-labeled data—can be jointly optimized to obtain compact representations that improve generalization on supervised tasks (Ranzato & Szummer, 2008; Weston et al., 2012).
This naturally extends to representations that are generative (Kingma et al., 2014), disentangled (Narayanaswamy et al., 2017), or hierarchical (Rasmus et al., 2015).
In addition, semi-supervised autoencoders enable transfer learning across different domains via jointly-trained reconstruction-classiﬁcation networks Ghifary et al.
(2016), private-shared partitioned representations (Bousmalis et al., 2016), or by augmenting models with label-encoding layers (Zhuang et al., 2015).
Although less studied, more closely related to our work is the use of autoencoders in a purely supervised setting: rather than focusing on how speciﬁc architectural novelties may better structure unlabeled data, Le et al.
(2018) instead study the generalization beneﬁt by the simple addition of reconstruction to the supervised classiﬁcation task—a special case of what we describe as F EA s.
Now, all aforementioned studies operate on the basis of autoencoding features (for an explicit or implicit downstream prediction task).
In this paper, we instead focus on autoencoder-based target -representation learning (using T EA s) in the supervised setting, and—importantly—analyze the theoretical and empirical beneﬁts of the approach.
Unlike in simple classiﬁcation (for which
F EA s make sense), we are motivated by problems with high-dimensional output spaces, but where we operate under the assumption of a more compact and predictable set of underlying factors.
We take inspiration from the empirical investigation of Girdhar et al.
(2016), where latent representations of 3D objects (targets) are jointly trained to be predictable from 2D images (features); [Table]
Oktay et al.
(2017) deploy a similar approach for medical image segmentation.
On the other hand, in both cases the supervised task is truncated: the predictors are trained to regress the unsupervised embeddings (instead of ground-truth targets) [Table] and gradients only backpropagate from the latent space (instead of the target space).
This means that their common decoder function is only shared indirectly (and predictions made indirectly), versus the symmetric and simultaneously optimized forward model proposed for T EA s—an important distinction that our analysis relies on to obtain uniform stability. In
Mostajabi et al. (2018), a two-stage procedure is used for semantic segmentation—loosely comparable to the ﬁrst two stages in T EA s; in contrast to our emphasis on joint training [Table]
they study the beneﬁt of a frozen embedding branch in parallel with direct prediction. More broadly related to target-embedding,
Dalca et al. (2018) build anatomical priors for biomedical segmentation in unsupervised settings.
Label space dimension reduction comprises techniques that focus speciﬁcally on multi-label classiﬁcation.
Early approaches to multi-label classiﬁcation employ simplistic transformations such as label power-sets (Boutell et al., 2004), binary relevance (Tsoumakas et al., 2009), and label rankings (Fürnkranz et al., 2008); these are computationally inefﬁcient, and do not capture interdependencies between labels.
In contrast, label-embedding methods ﬁrst derive a latent label space with reduced dimensionality, and subsequently associate inputs to that latent space instead.
Encodings have been obtained via random projections (Hsu et al., 2009), principal components-based projections (Tai &
Lin, 2010), canonical correlation analysis (Zhang & Schneider, 2011), as well as maximum-margin coding (Zhang & Schneider, 2012).
A parallel thread of research has focused on selecting representative and reconstructive subsets of labels through group-sparse learning (Balasubramanian & Lebanon, 2012) and randomized sampling (Bi & Kwok, 2013).
Various extensions of label-embedding techniques abound, such as using bloom ﬁlters (Cisse et al., 2013), nearest-neighbors (Bhatia et al., 2015), handling missing data (Wu et al., 2014), as well as using binary compression (Zhou et al., 2017).
Closer our theme of joint learning by utilizing both features and targets, Chen & Lin (2012)
ﬁrst proposed simultaneously minimizing the encoding error (from labels) and prediction error (from features) through an SVD formulation.
Towards more ﬂexible learning, Lin et al.
(2014) did away with explicitly speciﬁed encoding functions, proposing to learn code matrices directly—making no assumptions whatsoever.
Unifying several prior methods, Yu et al.
cast label-embedding within the generic empirical risk minimization framework—as learning a linear model with a low-rank constraint; this perspective captures the generic intuition of a restricted number of latent factors, and admits generalization bounds based on norm-based regularization.
Recently, Yeh et al.
(2017) generalized the label-embedding approach to autoencoders ; this formulation ﬂexibly allows the [Table] addition of speciﬁc losses to exploit correlations—a tactic also used in Wang et al.
(2018) with multi-dimensional scaling.
Furthermore, nonlinearities can be handled by deep learning in component functions, unlike earlier approaches limited to kernel methods (Lin et al., 2014; Li & Guo, 2015).
Our work is related to this general autoencoder approach to label-embedding, although there are signiﬁcant differences in focus.
In particular, we operate at a higher level of abstraction.
Labelembedding techniques worry about label reduction [Table] and about speciﬁc loss functions that aim to preserve dependencies within and among spaces; their problem is one of multi-label classiﬁcation [Table]
and their baseline is binary relevance. In contrast, we worry about autoencoding at all—that is, we focus on the regularizing effect of the reconstruction loss on learning the prediction model; our baseline is direct prediction [Table] and the output can be of any form (classiﬁcation or regression).
In light of the sizable performance improvement of the autoencoder-based model of Yeh et al.
(2017) over comparators using direct prediction, our work can be regarded as a more generalized analysis of the contribution of the autoencoding component.
Moreover, unlike the uniform convergence-based analysis in Yu et al.
(2014), our bound does not rely on explicit norm-based regularization—instead, we interpret the embedding task itself as an intrinsic form of regularization to derive our stability-based guarantee.
Finally, also worth mentioning is the ﬁeld of extreme multi-label classiﬁcation (Bhatia et al., 2015), for which probabilistic methods such as Rai et al. (2015) and Kapoor et al. (2012) present sophisticated approaches to extremely high-dimensional classiﬁcation problems—with advantages in performance and use cases.
In light of the medical relevance of our experimental setting, we point out the application of Yan et al. (2010) to medical coding. See Bhatia et al. for a more detailed overview. B.3 Algorithmic Stability and Learning Guarantees
Generalizability is central to machine learning, and its analysis via hypothesis stability is ﬁrst studied in Rogers & Wagner (1978) and Devroye & Wagner (1979).
Unlike arguments based on the complexity of the search space (Vapnik & Chervonenkis, 1971; Pollard, 1984; Koltchinskii, 2001), stability-based approaches account for how the model produced by the algorithm depends on the data.
Based on concentration inequalities (McDiarmid, 1989), improved bounds are developed in Lugosi &
Pawlak (1994) by estimating posterior error probabilities.
The landmark work of Bousquet & Elisseeff (2002)
ﬁrst formalizes the notion of uniform stability sufﬁcient for learnability, obtaining relatively strong bounds for several regularization algorithms, and Feldman & Vondrak (2018) recently use ideas related to differential privacy (Bassily et al., 2016) for further improved bounds without additional assumptions.
For further context, see Mukherjee et al.
and Shalev-Shwartz et al.
For semi-supervised representation learning, Rigollet (2007)
ﬁrst introduces the notion of cluster excess-risk and convergence, formalizing the clustering criterion for unlabeled features to be useful (Seeger, 2000).
Based on the clustering assumption, Singh et al.
(2009) develops a ﬁnite sample analysis to quantify the performance improvement from unlabeled features.
Focusing on autoencoders,
Epstein & Meir (2019) adapt recent margin, norm, and compression-based results for deep networks (Bartlett et al., 2017; Neyshabur et al., 2017; Arora et al., 2018), and relate generalization of feature reconstructions to the beneﬁt of additional unlabeled features for the primary classiﬁcation task.
In the context of supervised problems, Mohri et al. (2015) and Gottlieb et al.
(2016) analyze the generalization properties of dimensionality reduction techniques for features with respect to a downstream task; however, rather than joint training, the primary task is optimized subsequently over the learned representations.
Taking a joint, multi-task approach (Caruana, 1997), Baxter (2000) ﬁrst leverages the inductive bias of a common optimal hypothesis class to obtain a VC-based generalization bound.
Maurer (2006) and Maurer et al.
(2016) argue from Rademacher complexity to illustrate the beneﬁt of the common operator; however, they only consider the task-averaged beneﬁt, whereas we want to focus speciﬁcally on the primary task.
There has been some work on generalization for each task (Ben-David & Schuller, 2003), but limited to binary classiﬁcation—contrary to our setting.
Arguing from stability, our approach is related to Liu et al.
in showing that the algorithm for learning the shared model in a multi-task setting is uniformly stable.
Our analysis also resembles Le et al.
(2018) in the more speciﬁc setting where the bound for the primary prediction task is obtained with assistance from the auxiliary reconstruction loss; unlike Liu et al. (2016), we are not interested in a generic bound for all tasks.
Again, however, the fundamental (and motivating) difference of our work stems from the (inverted) problem setting and resulting framework.
In the vast majority of works, the primary task is one of classiﬁcation (or more generally |X| (cid:29) |Y| ), where feature - embeddings make sense to learn.
Instead, we attend to the setting in which
Y is high-dimensional (but where the underlying factors are assumed to be compact).
In this setting, we argue (theoretically and empirically) that target -embeddings make more sense to learn in an auxiliary reconstruction task.
C E XPANDED A LGORITHM D ETAIL
In the following, Algorithm 1 gives pseudocode for T EA training.
Figure 4 gives detailed block diagrams of component functions and objectives corresponding to each training stage (and variant).
Input: D = { ( x n , y n ) } Nn =1 , learning rate ψ , minibatch size N s Output: Parameters Θ , W u , W e of components θ, u, e 1: Initialize: Θ , W u , W e 2: while not converged do (cid:46) Stage 1: Learn Target-Embedding 3:
Sample { ( x n , y n ) } N s n =1 i.i.d.
∼ D 4: for n ∈ { 1 , ..., N s } do 5:
z n = e ( y n ; W e ) (cid:46) Encode 6: ˜ y n = θ ( z n ; Θ ) (cid:46) Decode 7: end for 8:
L r = 1 N s (cid:80) N s n =1 (cid:96) r
(˜ y n , y n ) 9:
W e =
W e − ψ ∇ W e L r 10: Θ = Θ − ψ ∇ Θ L r 11: end while 12: while not converged do (cid:46) Stage 2: Regress Embeddings 13:
Sample { ( x n , y n ) } N s n =1 i.i.d.
∼ D 14: for n ∈ { 1 , ..., N s } do 15: ˆ z n = u ( x n ; W u )
(cid:46) Predict 16:
z n = e ( y n ; W e ) (cid:46) Encode 17: end for 18:
L
[Equation] [Equation] [Equation] L z 20: end while 21: while not converged do (cid:46) Stage 3: Joint Training 22:
Sample { ( x n , y n ) } N s n =1 i.i.d.
∼ D 23: for n ∈ { 1 , ..., N s } do 24: ˆ z [Equation] = u
[Equation]
x n
[Equation] W u ) (cid:46) Predict 25: z n = e ( y n [Equation]
W e )
(cid:46) Encode 26:
ˆ y n = θ (ˆ z n ; Θ ) (cid:46) Decode 27: ˜ y n = θ ( z n ; Θ ) (cid:46) Decode 28: end for 29:
L
[Equation]
, x n ) 30:
[Equation]
[Equation]
[Equation]
[Equation]
34: end while 35: return Θ , W u , W e
[Table]
2 Y <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> 2 X <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> 2 Z <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_
[Equation]
="(null)">
[Equation]
null)</
[Equation]
>
[Equation]
[Equation] [Equation]
>
[Equation]
[Equation]
[Equation]
[Equation]
null)"
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation] [Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
_base
[Equation]
(null)"
[Equation]
(null)
[Equation]
/latexit>
[Equation]
latexit
[Equation]
_base
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
>
[Equation]
latexit
[Equation]
_base
[Equation]
[Equation]
[Equation] [Equation]
null)
[Equation]
(null)
[Equation]
/latexit
[Equation]
latexit
[Equation]
_base
[Equation]
"(null)">(null)
[Equation]
/latexit>
[Equation]
latexit sha1_base
[Equation]
"(null)"
[Equation]
(null) [Equation] /latexit> [Equation]
latexit
[Equation]
base
[Equation]
"(null)" [Equation] (null) [Equation]
/latexit
[Equation]
latexit
[Equation]
[Equation]
[Equation]
null)
[Equation]
[Equation]
latexit
[Equation]
latexit
[Equation] [Equation] "(null)" [Equation]
(null)
[Equation]
latexit
[Equation]
latexit
[Equation]
_base
[Equation] "(null)">(null) [Equation] /latexit>
[Equation]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @
W e <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @ ⇥ <
latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ` r <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
@` z @
W u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
<latexit
[Equation]
_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Equation]
latexit sha1_base64="(null)"
[Equation]
(null)</latexit>
[Equation]
latexit [Equation] [Equation] "(null
[Equation]
(null
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
/latexit
[Equation]
latexit
[Equation]
_base
[Equation]
"(null
[Equation]
[Equation]
latexit>
[Table]
[Equation]
> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ˜ y
[Equation]
z
[Equation]
) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> z
[Equation]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_
[Table]
="(null)">(null)</latexit>
[Table]
<latexit sha1_
[Table]
="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit
[Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> [Table] ` [Table]
W
[Table]
<latexit sha1_base64="(null)">(
[Table]
> 2 Z <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
ˆ y = ✓ (ˆ z ; ⇥ ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ˆ z = u ( x ; W u ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ˜ y = ✓ ( z ; ⇥ ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> z = e ( y ; W e ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)
[Table]
null
[Table]
=
[Table]
<
[Table]
> <
[Table]
=
[Table]
>
[Table]
<
[Table]
>
[Table]
=
[Table]
null)
<
[Table]
>
[Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
W
[Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)
[Table]
[Table] )">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> [Equation]
latexit
[Equation]
_base
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
/latexit> <latexit
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
>
[Equation]
[Equation]
/latexit
[Equation]
latexit
[Equation]
latexit
[Equation]
latexit
[Equation]
_base
[Equation]
/latexit
[Equation]
latexit
[Equation]
_base
[Equation]
[Equation]
[Equation]
(null
[Equation]
(null
[Equation]
/latexit
[Equation]
latexit
[Equation]
_base
[Equation]
(null)
[Equation]
[Equation]
[Equation]
[Equation]
"(null)
[Equation]
(null)
[Equation]
/latexit
[Equation]
latexit
[Equation]
[Equation]
[Equation]
[Equation]
(null)
[Equation]
/latexit> <latexit sha1_base64="(null)">(null)</latexit
[Equation]
latexit sha1_base
[Equation]
"(null [Equation] [Equation] (null
[Equation] /latexit [Equation] Training (Stage 2) @
` z @
W u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ` z <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
z = e ( y ; W e ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @ ⇥ <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ` r <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
W u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @`
[Table]
W e <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
<
[Equation]
null
[Equation]
latexit [Equation] latexit
[Equation]
null
[Equation]
null
[Equation]
latexit
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table]
[Equation] latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> 2 Y <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> D A DDITIONAL E XPERIMENT D ETAIL
The UK Cystic Fibrosis registry records follow-up trajectories for over 10,000 patients over the period from 2008 and 2015 with a total of over 60,000 hospital visits. Each patient is associated withvariables over time, which includes data on treatments and diagnoses for 23 possible comorbidities (e.g. ABPA, diabetes, hypertension, pancreatitis), 11 possible infections (e.g. aspergillus, burkholderia cepacia, klebsiella pneumoniae), as well as static demographic information (e.g. gender, genetics, smoking status).
Using both static and temporal information in a precedent window, we forecast the future trajectories for the diagnoses of infections and comorbidities (all binary variables) recorded at each follow-up.
The Alzheimer’s Disease Neuroimaging Initiative study tracks disease progression for over 1,700 patients over the period from 2004 to 2016 with a total of over 10,000 (bi-annual)
[Figure]
[Table]
">(null)</latexit> @` p @ W d <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> `
[Equation]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Equation]
latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
latexit
[Equation] [Equation]
latexit
[Equation]
_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` p @  <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @
W r <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` r @
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> @` p @
W d <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> (b) F EA @` p @
W u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
<latexit sha1_base64="(null)">(null)</latexit>
ˆ z = u ( x ; W u ) <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
[Table]
[Equation]
Vector
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
null
[Table]
<
[Table]
null
[Table]
null
[Table]
<
[Table]
sha
[Table]
=
[Table]
null
[Table]
null
[Table]
<
[Table]
sha1_
[Table]
="
[Table]
null
[Table]
null
[Table]
<
[Table]
latexit sha1_
[Table]
"(null
[Table]
">(null)</latexit
[Table]
<latexit sha1_
[Table]
="
[Table]
null
[Table]
null
[Table]
</latexit
[Table]
<latexit
[Table] [Table]
[Table]
null
[Table] [Table]
null
[Table]
</
[Table]
latexit sha1_
[Table]
="
[Table]
null
[Table]
">(null)</latexit
[Table]
<
[Table] [Table]
="
[Table]
null
[Table]
">(null
[Table]
<
[Table]
<
[Table]
sha1_
[Table]
="
[Table]
null
[Table]
">(null
[Table]
</
[Table] [Table]
"(null
[Table] [Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_
[Table]
="
[Table]
null)">
[Table]
null)</latexit> <latexit sha1_
[Table]
="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit
[Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> ` r <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> clinical visits. We focus on the 8 primary quantitative biomarkers (e.g. entorhinal cortex, fusiform gyrus, hippocampus), 16 cognitive tests (e.g. ADAS11, CDR sum of boxes, mini mental state exam), as well as static demographic information (e.g. apolipoprotein E4, education level, ethnicity); we omit the remaining variables, for which the rate of missingness is over 50%. Using a precedent window, we forecast the future the evolution of the primary quantitative biomarkers and cognitive test results (all continuous variables) measured at each visit. The Medical Information Mart for Intensive
Care records physiological data streams patients admitted to intensive care units after 2008.
We use over 22,000 patients with a total of over 500,000 measurements (resampled at 4 hour intervals).
We focus on the most frequently measured vital signs and lab tests (e.g. heart rate, oxygen saturation, respiratory rate) recorded over time (with categorical variables binarized, this gives a total ofvariables), as well as static demographic information (e.g. admission type, gender, location, marital status); we omit the remaining variables, for which the rate of missingness is over 50%.
Using a precedent window, we forecast the subsequent window of those variables.
For each dataset, sequences are randomized at the patient level in order to obtain splits for training (and validation) and testing.
We implement all models using Tensorﬂow.
For the linear model, each component (encoder, decoder, and predictor) consists of a single layer, no bias term, and linear activation; static (demographic) and temporal data are concatenated and ﬂattened for both features and targets.
For the nonlinear case, we implement each component as an RNN using GRUs with the number of hidden layers ζ ∈ { 1 , 2 } where the number of hidden units is equal to the temporal feature dimension, and tanh is used for activation; the dimension of the latent space is (therefore) equal to the hidden state dimension.
(For even larger hidden capacities, the increased number of parameters rapidly degrades performance).
Static (demographic) information is incorporated as a mapping into the initial state for recurrent cells.
Training is performed using the ADAM optimizer with a learning rate of ψ ∈ { 3 e − 5 , 3 e − 4 , 3 e − 3 , 3 e − 2 } .
Models are trained until convergence up to a maximum of 10,000 iterations with a minibatch size of N s ∈ { 32 , 64 , 128 } ; the empirical loss is computed on the validation set every 50 iterations of training, and convergence is determined on the basis of that error.
Checkpointing is implemented every 50 iterations, and the best model parameters are restored (upon convergence) for use on the testing set.
For all models except “Base”, we allow the opportunity to select among the (cid:96) 2 -regularization coefﬁcients ν ∈ { 0 , 3 e − 5 , 3 e − 4 , 3 e − 3 , 3 e − 2 } .
We set the strength-of-prior coefﬁcient λ = 0 .
5 for F EA , F/T EA , as well as all variants of T EA (however, we do provide sensitivities on λ for T EA in our experiments).
For hyperparameter tuning ( ζ, ψ, ν, N s ), we use cross-validation on the training set using 20 iterations of random search, selecting the setting that gives the lowest validation loss averaged across folds.
For fair comparison (so as to isolate the effect of supervised representation learning over and above direct prediction), we apply the same setting chosen for R EG for F EA , F/T EA , and all variants of T EA ; therefore the only difference is the presence or absence of each autoencoding component.
For each model and dataset, the experiment is repeated for a total of 10 times (each with a different random split of data into training and held-out testing sets); all results are reported as means and standard errors of each performance metric across runs.
E A DDITIONAL E XPERIMENT R ESULTS E.1 Results for Linear Models
[Table] [Table] τ
No
[Table]
No Staged
[Table]
EA
[Table]
EA
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table]
E.2 Results for Recurrent Models
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table]
The
[Table]
isolates the
[Table]
from staged training only (analogous to basic unsupervised pretraining
[Table]
though using targets)
[Table]
the
[Table] Staged [Table] setting isolates the beneﬁt from joint training only (without pretraining)
[Table]
[Table] [Table] [Table] The "No Joint" setting isolates the beneﬁt from staged training only (analogous to
[Table]
[Table]
[Table] value [Table]
[Table]
[Table]
The two-sample t -test for a difference in means is conducted on the results.
An asterisk next to the comparator result is used to indicate a statistically signiﬁcant difference in means ( p -value < 0.05) relative to the T EA result.
Results are grouped over the temporal axis; note that the variance between splits is an artifact of this grouping.
[Table] [Table]
E.3 Results for Sensitivities
[Table] [Table]
controls
[Table]
are grouped
[Table] [Table]
[Table] [Table]
The λ coefﬁcient controls the strength of prior—i.e. the tradeoff between the prediction and reconstruction loss
[Table]
are reported
[Table]
[Table]
comorbidities (
[Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
E.4 Contrived Negative Example
Here with give a contrived example where the goals of prediction and reconstruction are directly at odds with each other.
Speciﬁcally, we show a setup where the “ and ” part of “compact and predictable” representations is impossible—that is, a compact target-representation that is more reconstructive ends up being less predictable.
Consider the following (true) data generating process: Latent variables p , u each consist of 10 independent dimensions, where p i ∼ N (0 , 1) and u i ∼ N (0 , 1) .
Feature vectors x are also of length 10, and are linear in p .
Target vectors y
=
[ y (cid:62) P , y (cid:62) U ] (cid:62) are of length 50, where y P is of length 10 and linear in p and y U is of length 40 and linear in u .
See Figure 7.
Note that the input features are inadequate for predicting all targets; our choice of lettering denotes elements that are in principle predictable “P”, and elements that are in principle unpredictable “U”. [Equation] [Table]
<latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> p <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> u <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit> <latexit sha1_base64="(null)">(null)</latexit>
Suppose this data generating process is unknown to us.
First, consider what happens with direct prediction: A linear model would learn to predict y P well, while predictions of y U would be no better than random.
So far, so good.
Now given the feature and target dimensions, consider the (not unreasonable) choice of T EA s with a latent dimension of 10.
This is an obvious problem:
During reconstruction, we naturally get more bang for our buck by encoding more of the (highly compressible)
y U instead of y P ; yet y U is entirely useless to encode, as it is not predictable from inputs anyway.
Reconstructing well is therefore directly at odds with predicting well.
This is certainly an extremely contrived scenario; nevertheless, without sufﬁcient domain knowledge, it serves as a caveat that—as with feature-embedding paradigms—target-embedding is only as good as its assumptions.
Synthetic scenario where prediction is directly at odds with reconstruction.
The prior (that we can leverage compact and predictable representations of targets) is hugely incorrect in this case; as a result, not only is target-autoencoding not beneﬁcial—it is positively harmful.
For T EA s, we observe that as the strength-of-prior coefﬁcient λ increases, the overall prediction error actually increases (while the reconstruction error decreases)
[Table] [Table] [Table] [Table]
E.5 Additional Sensitivities
One can also ask the (purely empirical) question of how much each model degrades on out-ofdistribution data—without additional training to ﬁne-tune the model to the new data.
In this context, we actually have no reason to expect T EA s to degrade any more or less than comparators.
For thoroughness, we show an additional experiment as an example for sensitivity analysis (using UKCF) as follows:
Each model is trained (only) on male patients and tested (only) on female patients, and vice versa.
The average results on held-out samples from in-distribution data and out-of-distribution data then allows us to compute the net degradation (
i [Table]
e. negative difference), which is reported below.
While T EA s individually perform better overall on both in-distribution and out-of-distribution samples, none of the differences in the amounts of degradation between models are statistically signiﬁcant:
[Table]
Finally
[Table]
given the staged training in Algorithm
[Table]
it should be clear that the order cannot be changed.
Stage 2 requires the encoder to already be trained to provide the requisite embeddings, so it must be preceded by Stage 1.
Therefore the only relevant possibilities are: (1) Stages 1-2 by themselves, without Stage 3; this is simply the “No Joint” setting.
(2) Stage 3 by itself, without Stages 1-2; this is simply the “No Staged” setting.
None of the stages altogether; this is simply the “Neither” setting.
(4) Stages 1,
[Table]
and 3 in order; this is simply Algorithm 1 itself.
The only remaining possibility is to have Stage 3 precede Stages 1-2.
This makes little sense, since when the reconstruction loss is trained by itself it is likely to “undo” the result of joint training.
For thoroughness, we run an additional sensitivity experiment (using UKCF) to conﬁrm this.
The following corresponds to the left half of
Table 4, with the additional column on the right (and the other columns labeled to reﬂect the training stages).
Verifying our intuitions, the setting “3-1-2” behaves almost identically with the setting “1-2”:
[Table]
[Table]