Original: Efﬁcient learning of nonlinear prediction models with time-series privileged information
Revised: Efﬁcient learning of nonlinear prediction models with time-series privileged information

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In domains where sample sizes are limited, efﬁcient learning algorithms are critical.
Revised: In domains where sample sizes are limited, efﬁcient learning algorithms are critical.

Original: Learning using privileged information (LuPI) offers increased sample efﬁciency byallowing prediction models access to types of information at training time whichis unavailable when the models are used.
Revised: Learning using privileged information (LuPI) offers increased sample efﬁciency byallowing prediction models access to types of information at training time whichis unavailable when the models are used.

Original: In recent work, it was shown that forprediction in linear-Gaussian dynamical systems, a LuPI learner with access tointermediate time series data is never worse and often better in expectation thanany unbiased classical learner.
Revised: In recent work, it was shown that forprediction in linear-Gaussian dynamical systems, a LuPI learner with access tointermediate time series data is never worse and often better in expectation thanany unbiased classical learner.

Original: We provide new insights into this analysis andgeneralize it to nonlinear prediction tasks in latent dynamical systems, extendingtheoretical guarantees to the case where the map connecting latent variables andobservations is known up to a linear transform.
Revised: We provide new insights into this analysis andgeneralize it to nonlinear prediction tasks in latent dynamical systems, extendingtheoretical guarantees to the case where the map connecting latent variables andobservations is known up to a linear transform.

Original: In addition, we propose algorithmsbased on random features and representation learning for the case when this map isunknown.
Revised: In addition, we propose algorithmsbased on random features and representation learning for the case when this map isunknown.

Original: A suite of empirical results conﬁrm theoretical ﬁndings and show thepotential of using privileged time-series information in nonlinear prediction.
Revised: A suite of empirical results conﬁrm theoretical ﬁndings and show thepotential of using privileged time-series information in nonlinear prediction.

Original:  Introduction
Revised:  Introduction

Original: In data-poor domains, making the best use of all available information is central to efﬁcient andeffective machine learning.
Revised: In data-poor domains, making the best use of all available information is central to efﬁcient andeffective machine learning.

Original: Despite this, supervised learning is often applied in such a way thatinformative data is ignored.
Revised: Despite this, supervised learning is often applied in such a way thatinformative data is ignored.

Original: A good example of this is learning to predict the condition of a patientat a set follow-up time based on information of the ﬁrst medical examination.
Revised: A good example of this is learning to predict the condition of a patientat a set follow-up time based on information of the ﬁrst medical examination.

Original: Classical supervisedlearning makes use only of the initial data to predict the disease status at the follow-up, althoughin many cases data about medications, laboratory tests or vital signs are routinely collected aboutpatients also at intermediate time points.
Revised: Classical supervisedlearning makes use only of the initial data to predict the disease status at the follow-up, althoughin many cases data about medications, laboratory tests or vital signs are routinely collected aboutpatients also at intermediate time points.

Original: This information is privileged (Vapnik and Vashist, 2009),as it is unavailable at the time of prediction but can be used for training a model.
Revised: This information is privileged (Vapnik and Vashist, 2009),as it is unavailable at the time of prediction but can be used for training a model.

Original: Learning using privileged information (LuPI) (Vapnik and Vashist, 2009), generalized distillation (Lopez-Paz et al., 2016) and multi-view learning (Rosenberg and Bartlett, 2007) have beenproposed to increase the sample efﬁciency by leveraging privileged information in learning.
Revised: Learning using privileged information (LuPI) (Vapnik and Vashist, 2009), generalized distillation (Lopez-Paz et al., 2016) and multi-view learning (Rosenberg and Bartlett, 2007) have beenproposed to increase the sample efﬁciency by leveraging privileged information in learning.

Original: Theoreti-
Revised: Theoreti-

Original: cal results guarantee improved learning rates (Pechyony and Vapnik, 2010; Vapnik et al., 2015) or tighter generalization bounds (Wang, 2019) for large sample sizes under appropriate assumptions.
Revised: cal results guarantee improved learning rates (Pechyony and Vapnik, 2010; Vapnik et al., 2015) or tighter generalization bounds (Wang, 2019) for large sample sizes under appropriate assumptions.

Original: However, privileged information is not always beneﬁcial; it must be related to the task at hand (Jon- schkowski et al., 2015).
Revised: However, privileged information is not always beneﬁcial; it must be related to the task at hand (Jon- schkowski et al., 2015).

Original: Previous works do not identify such settings.
Revised: Previous works do not identify such settings.

Original: Moreover, existing analyses do not state when learning with privileged information is preferable to classical learning for problems with small sample sizes—which is where efﬁciency is needed the most.
Revised: Moreover, existing analyses do not state when learning with privileged information is preferable to classical learning for problems with small sample sizes—which is where efﬁciency is needed the most.

Original: Karlsson et al.
Revised: Karlsson et al.

Original: (2021) studied LuPI in the context of predicting an outcome observed at the end of a time series based on variables collected at the ﬁrst time step.
Revised: (2022) studied LuPI in the context of predicting an outcome observed at the end of a time series based on variables collected at the ﬁrst time step.

Original: They showed that making use of data from intermediate time steps in particular settings always leads to lower or equal prediction risk—for any sample size—compared to the best unbiased model which does not make use of this privileged information.
Revised: They showed that making use of data from intermediate time steps in particular settings always leads to lower or equal prediction risk—for any sample size—compared to the best unbiased model which does not make use of this privileged information.

Original: However, their method called learning using privileged time series (LuPTS) was limited
Revised: However, their method called learning using privileged time series (LuPTS) was limited

Original: # ! $ # $ …
Revised: # ! $ # $ …

Original: Observed
Revised: Observed

Original: Privileged
Revised: Privileged

Original: Unobserved (a) Linear dynamical system, fully observed ! ! " !
Revised: Unobserved (a) Linear dynamical system, fully observed ! ! " !

Original: Ψ $ ! ! "
Revised: Ψ $ ! ! "

Original: " " $ " !
Revised: " " $ " !

Original: # " # ! $ " $ % & … Ψ Ψ Ψ
Revised: # " # ! $ " $ % & … Ψ Ψ Ψ

Original: Φ (b) Linear latent dynamics, nonlinear observations indicates the observation function of the latent system, with  its left inverse.
Revised: Φ (b) Linear latent dynamics, nonlinear observations indicates the observation function of the latent system, with  its left inverse.

Original: to settings where the outcome function, and estimators of it, are linear functions of baseline features.
Revised: to settings where the outcome function, and estimators of it, are linear functions of baseline features.

Original: Moreover, their analysis did not study how variance reduction behaves as a function of increasedinput dimension.
Revised: Moreover, their analysis did not study how variance reduction behaves as a function of increasedinput dimension.

Original: Hayashi et al.
Revised: Hayashi et al.

Original: (2019) also learned from privileged intermediate time points but theirstudy was limited to empirical results for classiﬁcation using generalized distillation.
Revised: (2019) also learned from privileged intermediate time points but theirstudy was limited to empirical results for classiﬁcation using generalized distillation.

Original: Contributions.
Revised: Contributions.

Original: We extend the LuPTS framework to nonlinear models and prediction tasks in latentdynamical systems (Section 3).
Revised: We extend the LuPTS framework to nonlinear models and prediction tasks in latentdynamical systems (Section 3).

Original: In this setting, we prove that learning with privileged informationleads to lower risk when the nonlinear map connecting latent variables and observations is known upto a linear transform (Section 3.1).
Revised: In this setting, we prove that learning with privileged informationleads to lower risk when the nonlinear map connecting latent variables and observations is known upto a linear transform (Section 3.1).

Original: In doing so, we also ﬁnd that when the representation dimensiongrows larger than the number of samples, the beneﬁt of privileged information vanishes.
Revised: In doing so, we also ﬁnd that when the representation dimensiongrows larger than the number of samples, the beneﬁt of privileged information vanishes.

Original: We show thatprivileged learners used with random feature maps can learn optimal models consistently, even whenthe relationship between latent and observed variables is unknown, but may suffer from bias in smallsamples (Section 3.2).
Revised: We show thatprivileged learners used with random feature maps can learn optimal models consistently, even whenthe relationship between latent and observed variables is unknown, but may suffer from bias in smallsamples (Section 3.2).

Original: As a remedy, we propose several representation learning algorithms aimedat trading off bias and variance (Section 3.3).
Revised: As a remedy, we propose several representation learning algorithms aimedat trading off bias and variance (Section 3.3).

Original: In experiments, we ﬁnd that privileged time-serieslearners with either random features or representation learning reduce variance and improve latentstate recovery in small-sample settings (Section 4) on both synthetic and real-world regression tasks.
Revised: In experiments, we ﬁnd that privileged time-serieslearners with either random features or representation learning reduce variance and improve latentstate recovery in small-sample settings (Section 4) on both synthetic and real-world regression tasks.

Original:  Prediction and privileged information in nonlinear time series
Revised:  Prediction and privileged information in nonlinear time series

Original: We aim to predict outcomes Y in Y ✓ R q based on covariates X 1 in X ✓ R k .
Revised: We aim to predict outcomes Y in Y ✓ R q based on covariates X 1 in X ✓ R k .

Original: X 1 are observed 53 at a baseline time point t = 1 , starting a time series on the form X 1 , X 2 , . . .
Revised: X 1 are observed 53 at a baseline time point t = 1 , starting a time series on the form X 1 , X 2 , . . .

Original: X T , Y .
Revised: X T , Y .

Original: Outcomes areassumed to be compositions of a representation  , a linear map ✓ and Gaussian noise ✏ ,
Revised: Outcomes areassumed to be compositions of a representation  , a linear map ✓ and Gaussian noise ✏ ,

Original: Y = h ( X 1 ) + ✏ , where h ( x 1 ) :
Revised: Y = h ( X 1 ) + ✏ , where h ( x 1 ) :

Original: = ✓ >  ( x 1 ) , and ✏ ⇠ N (0 , ˜  2 Y ) .
Revised: = ✓ >  ( x 1 ) , and ✏ ⇠ N (0 , ˜  2 Y ) .

Original: In addition to observations of X 1 and Y , privileged information (PI) is available in the form ofsamples of random variables, X 2 , ..., X T , from intermediate time points between X 1 and Y , alltaking values in X .
Revised: In addition to observations of X 1 and Y , privileged information (PI) is available in the form ofsamples of random variables, X 2 , ..., X T , from intermediate time points between X 1 and Y , alltaking values in X .

Original: Two data generating processes (DGPs) with this structure are illustrated in
Revised: Two data generating processes (DGPs) with this structure are illustrated in

Original: Figure 1.
Revised: Figure 1.

Original: Unlike baseline variables X 1 , privileged information is observed only at training time, notat test time .
Revised: Unlike baseline variables X 1 , privileged information is observed only at training time, notat test time .

Original: Therefore, it can only beneﬁt learning, and not inference.
Revised: Therefore, it can only beneﬁt learning, and not inference.

Original: Data sets of m 2 N + training samples, D := { ( x i, 1 , x i, 2 , ..., x i,T , y i ) } mi =1 , are drawn independently 61 and identically distributed from a ﬁxed, unknown distribution p over all random variables in oursystem.
Revised: Data sets of m 2 N + training samples, D := { ( x i, 1 , x i, 2 , ..., x i,T , y i ) } mi =1 , are drawn independently 61 and identically distributed from a ﬁxed, unknown distribution p over all random variables in oursystem.

Original: We let X t =
Revised: We let X t =

Original: [ x 1 ,t , ..., x m,t ] > denote the data matrix of features observed at time t = 1 , ..., Tand Y =
Revised: [ x 1 ,t , ..., x m,t ] > denote the data matrix of features observed at time t = 1 , ..., Tand Y =

Original: [ y 1 , ..., y m ] > the vector of all outcomes observed in D .
Revised: [ y 1 , ..., y m ] > the vector of all outcomes observed in D .

Original: A learning algorithm A : D !
Revised: A learning algorithm A : D !

Original: Hmaps data sets D 2 D to hypotheses ˆ h 2 H .
Revised: Hmaps data sets D 2 D to hypotheses ˆ h 2 H .

Original: A good algorithm A has small expected risk R p
Revised: An efﬁcient algorithm A has small expected risk R p

Original: ( A )65 with respecgt to a lossL :
Revised: ( A ) with respect to a loss

Original:  Y ⇥ Y !
Revised: L : Y ⇥ Y !

Original: R over training sets  drawn from p ,
Revised: R over training sets of ﬁxed size m drawn from p ,

Original: R p
Revised: R p

Original: ( A ) :=
Revised: ( A ) :=

Original: E D [ R p (ˆ h )] where ˆ h
Revised: E D [ R p (ˆ h )] where ˆ h

Original: =
Revised: =

Original: A ( D ) and R p (ˆ h ) :=
Revised: A ( D ) and R p (ˆ h ) :=

Original: E p [ L (ˆ h ( X 1 ) , Y )] .
Revised: E p [ L (ˆ h ( X 1 ) , Y )] .

Original: We study the regression setting with L the squared error, L ( y, y 0 ) =
Revised: We study the regression setting with L the squared error, L ( y, y 0 ) =

Original: k y  y 

Revised: k y  y

Original: k 22 .
Revised: k 22 .

Original: In analysis, we focuson univariate outcomes, q = 1 , but all results extend to multivariate outcomes, q > 1 .
Revised: In analysis, we focuson univariate outcomes, q = 1 , but all results extend to multivariate outcomes, q > 1 .

Original: Our main goal is to comparethe properties of privileged learners A P , which make use of privilegedinformation, to classical learners A C which learn without PI.
Revised: Our main goal is to compare privileged learners A P , which make use of privileged information, toclassical learners A C which learn without PI.

Original: We seek to identify conditions andalgorithms for which privileged learning  leads to lower risk in expectation, R ( A P )  R ( A C ) . In thenext section, we ﬁnd such a setting in the form of prediction in latent dynamical systems.
Revised: We seek to identify conditions and algorithms forwhich privileged learning is more efﬁcient, i.e., it leads to lower risk in expectation, R ( A P )  R ( A C )for the same number of training samples. We describe such a setting next.

Original: 2.1 Privileged information in latent dynamical systems
Revised: 2.1 Privileged information in latent dynamical systems

Original: We study tasks where data are produced by a latent dynamical process.
Revised: We study tasks where data are produced by a latent dynamical process.

Original: Observations X t are generatedfrom unobserved latent states Z t through a nonlinear transformation , see Figure 1b.
Revised: Observations X t are generatedfrom unobserved latent states Z t through a nonlinear transformation , see Figure 1b.

Original: This means
Revised: This means

Original: Input: Data D = ( { X t } , Y ) ; Repr.
Revised: Input: Data D = ( { X t } , Y ) ; Repr.

Original: ˆ  or kernel  if using a ﬁxed representation ˆ  then
Revised: ˆ  or kernel  if using a ﬁxed representation ˆ  then

Original: [Equation]
Revised: [Equation]

Original: for t
Revised: for t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: else if using kernel  then ˆ
Revised: else if using kernel  then ˆ

Original: [Equation]
Revised: [Equation]

Original: for t = 1 ,
Revised: for t = 1 ,

Original: [Equation]
Revised: [Equation]

Original: return ˆ h P
Revised: return ˆ h P

Original: Features, 0.
Revised: Features, 0.

Original: R ²
Revised: R ²

Original: OLS
Revised: OLS

Original: LuPTS that the target of learning, h
Revised: LuPTS that the target of learning, h

Original: ( X 1 ) =
Revised: ( X 1 ) =

Original: E [ Y | X 1 ] , is nonlinear in X 1 .
Revised: E [ Y | X 1 ] , is nonlinear in X 1 .

Original: Latent dynamical systems likethese have proven successful at modelling a variety of phenomena as for example ﬂuid dynamics inphysics (Lee and Carlberg, 2019) and human brain activity in neuroscience (Kao et al., 2015).
Revised: Latent dynamical systems likethese have proven successful at modelling a variety of phenomena as for example ﬂuid dynamics inphysics (Lee and Carlberg, 2019) and human brain activity in neuroscience (Kao et al., 2015).

Original: Assumption 1 (Latent linear-Gaussian system) .
Revised: Assumption 1 (Latent linear-Gaussian system) .

Original: Let Z 1 , ..., Z T be latent states related as a linearGaussian dynamical system in a space Z ✓ R d with Z 1 of arbitrary distribution.
Revised: Let Z 1 , ..., Z T be latent states related as a linearGaussian dynamical system in a space Z ✓ R d with Z 1 of arbitrary distribution.

Original: Further, let theobservation function be an injective map : Z !
Revised: Further, let theobservation function be an injective map : Z !

Original: X with left-inverse (representation)  : X !
Revised: X with left-inverse (representation)  : X !

Original: Z .
Revised: Z .

Original: Then, with A t 2 R d ⇥ d ,  2 R d assume thatvariables Z 1 , ..., Z T , X 1 , ..., X T , Y are generated as
Revised: With A 1 , ..., A T  1 2 R d ⇥ d ,  2 R d assume that Z 1 , ..., Z T , X 1 , ..., X T , Y are generated as

Original: Z t =
Revised: Z t =

Original: A > t  1 Z t  1 + ✏ t for t  2 , and 8 t : X t =
Revised: A > t  1 Z t  1 + ✏ t for t  2 , and 8 t : X t =

Original: ( Z t ) and
Revised: ( Z t ) and

Original: Y =  > Z T + ✏ Y .
Revised: Y =  > Z T + ✏ Y .

Original: where ✏ t ⇠ N (0 ,  2 t I ) and ✏ Y ⇠ N (0 ,  2 Y ) .
Revised: where ✏ t ⇠ N (0 ,  2 t I ) and ✏ Y ⇠ N (0 ,  2 Y ) .

Original: It is easy to verify that Assumption 1 is consistent with (1), but stronger: there are more systems thatmap
Revised: It is easy to verify that Assumption 1 is consistent with (1), but stronger: there are more systems thatmap

Original: X 1 to Y as ✓ >  ( X 1 ) than systems where additionally X t
Revised: X 1 to Y as ✓ >  ( X 1 ) than systems where additionally X t

Original: = ( Z t ) .
Revised: = ( Z t ) .

Original: Nevertheless, it is muchmore general than the results of Karlsson et al.
Revised: Nevertheless, it is muchmore general than the results of Karlsson et al.

Original: (2021), limited to the linear setting, i.e.  ( x 1 ) = x 1 .
Revised: (2022), limited to the linear setting, i.e.  ( x 1 ) = x 1 .

Original: Next, we present a generalized version of the LuPTS algorithm of Karlsson et al.
Revised: Next, we present a generalized version of the LuPTS algorithm of Karlsson et al.

Original: (2021) that isprovably preferable to classical learning when data is generated according to Assumption 1 and is known up to a linear transform.
Revised: (2022) that isprovably preferable to classical learning when data is generated according to Assumption 1 and is known up to a linear transform.

Original: Further, we discuss how this algorithm can be made universallyconsistent for unknown representations  when combined with random features.
Revised: Further, we discuss how this algorithm can be made universallyconsistent for unknown representations  when combined with random features.

Original:  Efﬁcient learning with time-series privileged information
Revised:  Efﬁcient learning with time-series privileged information

Original: We analyze and compare learners from the privileged and classical paradigms which produce estimateson the form ˆ h ( x 1 )
Revised: We analyze and compare learners from the privileged and classical paradigms which produce estimatesof the form ˆ h ( x 1 )

Original: =
Revised: =

Original: ˆ ✓ > ˆ  ( x 1 ) , as motivated by (1).
Revised: ˆ ✓ > ˆ  ( x 1 ) , as motivated by (1).

Original: We let each use representations ˆ  : X !
Revised: We let each use representations ˆ  : X !

Original: ˆ Z ✓
Revised: ˆ Z ✓

Original: R ˆ d from a family F shared by both paradigms , so that the hypothesis class H 

Revised: R ˆ d from a family F shared by both paradigms , so that the hypothesis class H

Original: ˆ h is shared as well.
Revised: ˆ h is shared as well.

Original: 3.1 Learning with true representations known up to a linear transform
Revised: 3.1 Learning with true representations known up to a linear transform

Original: In this section, we assume that privileged and classical learners have access to a common, ﬁxedrepresentation function
Revised: In this section, we assume that privileged and classical learners have access to a common, ﬁxedrepresentation function

Original: ˆ  which is related to the true representation function through a lineartransform, meaning there exists a matrix B such that ˆ  ( · )
Revised: ˆ  which is related to the true representation function through a lineartransform, meaning there exists a matrix B such that ˆ  ( · )

Original: =
Revised: =

Original: B  ( · ) .
Revised: B  ( · ) .

Original: Classical learning.
Revised: Classical learning.

Original: As a classical learner to serve as a strong comparison point for our privilegedlearners, we use the ordinary least-squares (OLS) linear estimator applied to the latent variables atthe ﬁrst time point inferred by ˆ  .
Revised: As a classical learner to serve as a strong comparison point for our privilegedlearners, we use the ordinary least-squares (OLS) linear estimator applied to the latent variables atthe ﬁrst time point inferred by ˆ  .

Original: With ˆ Z 1 =
Revised: With ˆ Z 1 =

Original: [ ˆ  ( x 1 , 1 ) , ..., ˆ  ( x m, 1 )] > 2 R m ⇥ ˆ d ,[Equation] where
Revised: [ˆ  ( x 1 , 1 ) , ..., ˆ  ( x m, 1 )] > 2 R m ⇥ ˆ d ,

Original: [Equation]
Revised: [Equation]

Original: When m > ˆ d and  is known up to linear transform, ˆ h C is the minimum-risk unbiased estimator of hwhich does not use privileged information.
Revised: When m > ˆ d and  is known up to linear transform, ˆ h C is the minimum-risk unbiased estimator of hwhich does not use privileged information.

Original: To accommodate the underdetermined case where m < ˆ d ,the matrix inverse( · )  1 is replaced by the Moore-Penrose pseudo inverse ( · ) † (Penrose, 1956).
Revised: To accommodate the underdetermined case where m < ˆ d ,the matrix inverse is replaced by the Moore-Penrose pseudo inverse ( · ) † (Penrose, 1956).

Original: Generalized LuPTS.
Revised: Generalized LuPTS.

Original: For privileged learning, we ﬁrst compute ˆ Z t =
Revised: For privileged learning, we ﬁrst compute ˆ Z t =

Original: [ˆ  ( x 1 ,t ) , ..., ˆ  ( x m,t )] > fort = 1 , ..., T .
Revised: [ˆ  ( x 1 ,t ) , ..., ˆ  ( x m,t )] > fort = 1 , ..., T .

Original: Then, we independently ﬁt parameter estimates ˆ A 1 , ..., ˆ A T , ˆ  of the dynamical systemshown in Figure 1b by minimizing the squared error in single-step predictions.
Revised: Then, we independently ﬁt parameter estimates ˆ A 1 , ..., ˆ A T , ˆ  of the dynamical systemshown in Figure 1b by minimizing the squared error in single-step predictions.

Original: This equates to aseries of OLS estimates in ˆ Z .
Revised: This equates to aseries of OLS estimates in ˆ Z .

Original: At test time, baseline variables x 1 are embedded with ˆ  and the latentdynamical system is simulated for T time steps to form a prediction ˆ h P ( x 1 ) =
Revised: At test time, baseline variables x 1 are embedded with ˆ  and the latentdynamical system is simulated for T time steps to form a prediction ˆ h P ( x 1 ) =

Original: ( ˆ A 1 · · · ˆ A T  ) >
Revised: ( ˆ A 1 · · · ˆ A T  ) >

Original: ˆ  ( x 1 ) .
Revised: ˆ  ( x 1 ) .

Original: Putting this together, we arrive at Algorithm 1 which we call generalized LuPTS .
Revised: Putting this together, we arrive at Algorithm 1 which we call generalized LuPTS .

Original: We may applya simple matrix identity and replace terms ˆ Z t ˆ Z > t in Algorithm 1 by the Gram matrices ˆ K t =
Revised: We may applya simple matrix identity and replace terms ˆ Z t ˆ Z > t in Algorithm 1 by the Gram matrices ˆ K t =

Original:  ( X t , X t ) of a reproducing kernel  with corresponding (implicit) feature map
Revised:  ( X t , X t ) of a reproducing kernel  with corresponding (implicit) feature map

Original: ˆ  ,  ( x, x 0 )
Revised: ˆ  ,  ( x, x 0 )

Original: =
Revised: =

Original: h ˆ  ( x ) , ˆ  ( x 0 ) i (Smola and Schölkopf, 1998).
Revised: h ˆ  ( x ) , ˆ  ( x 0 ) i (Smola and Schölkopf, 1998).

Original: This variant allows for learning with unknown  .
Revised: This variant allows for learning with unknown  .

Original: We now state a result which says that under Assumption 1, for an appropriate ﬁxed representation ˆ or kernel  , generalized LuPTS is never worse in expectation than the classical learner in (3).
Revised: We now state a result which says that under Assumption 1, for an appropriate ﬁxed representation ˆ or kernel  , generalized LuPTS is never worse in expectation than the classical learner in (3).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let D be a data set  drawn froma distribution p , consistent with Assumption 1.
Revised: Let D be a data set of size m drawn from p , consistent with Assumption 1.

Original: Assumethat the left inverse  : X !
Revised: Assume thatthe left inverse  : X !

Original: Z of the observation function is known up to a linear transform,explicitly or through a kernel  ( x, x 0 ) =
Revised: Z of the observation function is known up to a linear transform, explicitlyor through a kernel  ( x, x 0 ) =

Original: h ˆ  ( x ) , ˆ  ( x 0 ) i , i.e., there exists a matrix B with linearlyindependent columns such that ˆ  ( x ) =
Revised: h ˆ  ( x ) , ˆ  ( x 0 ) i , i.e., there exists a matrix B with linearly independentcolumns such that ˆ  ( x ) =

Original: B  ( x ) with ˆ  : X !
Revised: B  ( x ) with ˆ  : X !

Original: ˆ Z for all x 2 X .
Revised: ˆ Z for all x 2 X .

Original: Then, it holds forthe privileged learner A P ( D ) =
Revised: Then, it holds for the privilegedlearner A P ( D ) =

Original: ˆ h P (Algorithm 1) and the classical learner A C ( D )
Revised: ˆ h P (Algorithm 1) and the classical learner A C ( D )

Original: =
Revised: =

Original: ˆ h C (3) ,
Revised: ˆ h C (3) ,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: First, we show that the predictions made by generalized LuPTS are invariant to a lineartransform B applied to ˆ  ( · ) during training, see Appendix A.
Revised: First, we show that the predictions made by generalized LuPTS are invariant to a lineartransform B applied to ˆ  ( · ) during training, see Appendix A.

Original: Then we consider two cases: (i)
Revised: Then we consider two cases: (i)

Original: Whenˆ d
Revised: Whenˆ d

Original:  m we may re-purpose the proof of Theorem 1 in Karlsson et al.
Revised:  m we may re-purpose the proof of Theorem 1 in Karlsson et al.

Original: (2021), (ii) When m < ˆ d
Revised: (2022), (ii) When m < ˆ d

Original: Proposition 1 below directly implies E ˆ h P ,X 1 [Var D (ˆ h C ( X 1 ) | ˆ h P )]
Revised: Proposition 1 below directly implies E ˆ h P ,X 1 [Var D (ˆ h C ( X 1 ) | ˆ h P )]

Original: = 0 and R ( A P ) =
Revised: = 0 and R ( A P ) =

Original: R ( A C ) .
Revised: R ( A C ) .

Original: Theorem 1 implies preference for the privileged learner over the classical one since Var( · )  0 andthus R ( A P )  R ( A C )  .
Revised: Theorem 1 implies that the privileged learner is at least as sample efﬁcient as the classical one since Var( · )  0 and thus R ( A P )  R ( A C ) for the same number of training samples m .

Original: The result is a direct generalization of the main result in Karlsson et al.
Revised: The result is adirect generalization of the main result in Karlsson et al.

Original: We also observe that generalized LuPTS and the classical learner coincide whenever m < ˆ d .
Revised: We also observe that generalized LuPTS and the classical learner coincide under certain conditions when m  ˆ d .

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ˆ  : X !
Revised: Let ˆ  : X !

Original: ˆ Z ✓ R ˆ d be any map with corresponding kernel  .
Revised: ˆ Z ✓ R ˆ d be any map with corresponding kernel  .

Original: Let ˆ K t be the Gram matrix of  applied to X t and let ˆ h C , ˆ h P be classical (3) and privileged (Algorithm 1) estimates.
Revised: Let ˆ K t be the Gram matrix of  applied to X t and let ˆ h C , ˆ h P be classical (3) and privileged (Algorithm 1) estimates.

Original: Then, ˆ K t is invertible for all t = )
Revised: Then, ˆ K t is invertible for all t = )

Original: ˆ h P
Revised: ˆ h P

Original: =
Revised: =

Original: ˆ h C .
Revised: ˆ h C .

Original: ˆ K t is noninvertible whenever m > ˆ d , assuming linearly independent features.
Revised: ˆ K t is noninvertible whenever m > ˆ d , assuming linearly independent features.

Original: Proof sketch.
Revised: Proof sketch.

Original: When the Gram matrices K t are invertible, the pseudo-inverse coincides with theinverse, and factors K t K † t in the LuPTS estimators cancel, making LuPTS and CL equal.
Revised: When the Gram matrices K t are invertible, the pseudo-inverse coincides with theinverse, and factors K t K † t in the LuPTS estimators cancel, making LuPTS and CL equal.

Original: Remarks.
Revised: Remarks.

Original: Theorem 1 extends the applicability of LuPTS to a) nonlinear prediction through a ﬁxedfeature map or b) kernel estimation and c) to the underdetermined case of m < ˆ d .
Revised: Theorem 1 extends the applicability of LuPTS to a) nonlinear prediction through a ﬁxedfeature map or b) kernel estimation and c) to the underdetermined case of m < ˆ d .

Original: While previouswork is restricted to observed linear systems our result considers the case of latent linear dynamicswhich only need to be identiﬁed up to a linear transform (see Figure 1).
Revised: While previouswork is restricted to observed linear systems our result considers the case of latent linear dynamicswhich only need to be identiﬁed up to a linear transform (see Figure 1).

Original: Proposition 1 does not claimthat no preferable privileged learner exists; it is a statement only about generalized LuPTS.
Revised: Proposition 1 does not claimthat no preferable privileged learner exists; it is a statement only about generalized LuPTS.

Original: We mayrelate the result to the double descent characteristic previously observed for other linear estimators forﬁxed m and varying ˆ d (Loog et al., 2020).
Revised: We mayrelate the result to the double descent characteristic previously observed for other linear estimators forﬁxed m and varying ˆ d (Loog et al., 2020).

Original: After a phase transition around ˆ d = m LuPTS’s variancereduces for a second time when it becomes equivalent to the classical learner (see Figure 2).
Revised: After a phase transition around ˆ d = m LuPTS’s variancereduces for a second time when it becomes equivalent to the classical learner (see Figure 2).

Original: 3.2 Random feature maps for unknown representations
Revised: 3.2 Random feature maps for unknown representations

Original: When the true  is entirely unknown, as is often the case in practice, using a poor representationˆ  may yield biased results for both classical and privileged learners.
Revised: When the true  is entirely unknown, as is often the case in practice, using a poor representationˆ  may yield biased results for both classical and privileged learners.

Original: A common solution in nonlinear prediction is to use a universal kernel, such as the Gaussian-RBF kernel.
Revised: A common solution in nonlinear prediction is to use a universal kernel, such as the Gaussian-RBF kernel.

Original: These have densereproducing-kernel Hilbert spaces which allow approximation of any continuous function.
Revised: These have densereproducing-kernel Hilbert spaces which allow approximation of any continuous function.

Original: However,universal kernels also have positive-deﬁnite and thus invertible Gram matrices (Hofmann et al., 2008),which according to Proposition 1 eliminates any gain in sample efﬁciency of generalized LuPTS.
Revised: However,universal kernels also have positive-deﬁnite and thus invertible Gram matrices (Hofmann et al., 2008),which according to Proposition 1 eliminates any gain in sample efﬁciency of generalized LuPTS.

Original: Instead, we combine our algorithm with an approximation of universal kernels—random feature maps.
Revised: Instead, we combine our algorithm with an approximation of universal kernels—random feature maps.

Original: These methods project inputs x onto ˆ d features by a random linear map W 2 R k ⇥ ˆ d , and a nonlinear element-wise activation function.
Revised: These methods project inputs x onto ˆ d features by a random linear map W 2 R k ⇥ ˆ d , and a nonlinear element-wise activation function.

Original: By choosing ˆ d < m , we can beneﬁt from the function approximation properties of universal kernels (see discussion below) and the variance reduction of generalized LuPTS.
Revised: By choosing ˆ d < m , we can beneﬁt from the function approximation properties of universal kernels (see discussion below) and the variance reduction of generalized LuPTS.

Original: Popular random features include random Fourier features (RFF)
Revised: Popular random features include random Fourier features (RFF)

Original: (Rahimi and
Revised: (Rahimi and

Original: Recht, 2007) and random ReLU features (RRF) (Sun et al., 2018),
Revised: Recht, 2007) and random ReLU features (RRF) (Sun et al., 2018),

Original: [Equation]
Revised: [Equation]

Original: where f + ( z ) = max(0 , z ) is the rectiﬁer (ReLU) function and  > 0 is a bandwidth hyperparameter.
Revised: where f + ( z ) = max(0 , z ) is the rectiﬁer (ReLU) function and  > 0 is a bandwidth hyperparameter.

Original: For large enough numbers of random features and training samples, any continuous function h canbe approximated up to arbitrary precision by a linear map !
Revised: For large enough numbers of random features and training samples, any continuous function h canbe approximated up to arbitrary precision by a linear map !

Original: applied to the random features, e.g.,ˆ h ( x ) = !
Revised: applied to the random features, e.g.,ˆ h ( x ) = !

Original: >
Revised: >

Original: ˆ   RRF ( x ) , see Sun et al.
Revised: ˆ   RRF ( x ) , see Sun et al.

Original: ; Rudi and Rosasco (2017).
Revised: ; Rudi and Rosasco (2017).

Original: Applying the same argumentto the step-wise estimators of LuPTS we can justify using random features in Algorithm 1 by thefollowing observation: Generalized LuPTS with a random feature map can be made a universallyconsistent estimator of h ( x 1 ) =
Revised: Applying the same argumentto the step-wise estimators of LuPTS we can justify using random features in Algorithm 1 by thefollowing observation: Generalized LuPTS with a random feature map can be made a universallyconsistent estimator of h ( x 1 ) =

Original: E [ Y | x 1 ] under appropriate assumptions.
Revised: E [ Y | x 1 ] under appropriate assumptions.

Original: We refer to Appendix Bfor a precise statement.
Revised: We refer to Appendix Bfor a precise statement.

Original: As universal consistency describes the asymptotic behaviour in the limit ofinﬁnite samples and random features, this offers only limited insight into the beneﬁts of privilegedinformation in small sample settings, where performance will be a bias-variance trade-off.
Revised: As universal consistency describes the asymptotic behaviour in the limit ofinﬁnite samples and random features, this offers only limited insight into the beneﬁts of privilegedinformation in small sample settings, where performance will be a bias-variance trade-off.

Original: Variance reduction & bias ampliﬁcation.
Revised: Variance reduction & bias ampliﬁcation.

Original: Generalized LuPTS is only guaranteed lower variancecompared to the classical estimator under Assumption 1, although our empirical results (Section 4)suggest this applies more widely.
Revised: Generalized LuPTS is only guaranteed lower variancecompared to the classical estimator under Assumption 1, although our empirical results (Section 4)suggest this applies more widely.

Original: When ˆ  is a bad approximation of  , generalized LuPTS mayamplify bias, increasing with the number of privileged time points, compared to classical learning.
Revised: When ˆ  is a bad approximation of  , generalized LuPTS mayamplify bias, increasing with the number of privileged time points, compared to classical learning.

Original: We show this theoretically in Appendix C and also empirically in Appendix F.
Revised: We show this theoretically in Appendix C and also empirically in Appendix F.

Original: Whether generalized
Revised: Whether generalized

Original: LuPTS is still preferable to classical learning in terms of prediction risk appears to depend on theamount of bias that gets ampliﬁed.
Revised: LuPTS is still preferable to classical learning in terms of prediction risk appears to depend on theamount of bias that gets ampliﬁed.

Original: Our experiments imply the variance reduction mostly dominateswhen using random features, whereas this is not always the case for linear LuPTS.
Revised: Our experiments imply the variance reduction mostly dominateswhen using random features, whereas this is not always the case for linear LuPTS.

Original: The phenomenonof bias ampliﬁcation is familiar from e.g., model-based and model-free reinforcement learning (Koberet al., 2013).
Revised: The phenomenonof bias ampliﬁcation is familiar from e.g., model-based and model-free reinforcement learning (Koberet al., 2013).

Original: As the bias with random features may still be high for small sample settings, we nextpresent privileged representation learning algorithms to trade off bias and variance more efﬁciently.
Revised: As the bias with random features may still be high for small sample settings, we nextpresent privileged representation learning algorithms to trade off bias and variance more efﬁciently.

Original: 3.3 Privileged time-series representation learning
Revised: 3.3 Privileged time-series representation learning

Original: Up until now the representation ˆ  was considered ﬁxed, either because  was known up to a lineartransform (explicitly or implicitly) or because of the use of random feature methods.
Revised: Up until now the representation ˆ  was considered ﬁxed, either because  was known up to a lineartransform (explicitly or implicitly) or because of the use of random feature methods.

Original: Generalized
Revised: Generalized

Original: LuPTS (Algorithm 1) produces minimizers { ˆ A t } , ˆ  of the following objective for ﬁxed ˆ  ,
Revised: LuPTS (Algorithm 1) produces minimizers { ˆ A t } , ˆ  of the following objective for ﬁxed ˆ  ,

Original: [Equation]
Revised: [Equation]

Original: Objective (5) and the systems described by Assumption 1 lend themselves to methods which also learnthe representation  in addition to the latent dynamics { A t ,  } .
Revised: Objective (5) and the systems described by Assumption 1 lend themselves to methods which also learnthe representation  in addition to the latent dynamics { A t ,  } .

Original: Next, we present three algorithmswhich combine the ideas of generalized LuPTS with the expressiveness of deep representationlearning.
Revised: Next, we present three algorithmswhich combine the ideas of generalized LuPTS with the expressiveness of deep representationlearning.

Original: All learners use equivalent encoders to represent ˆ  ( · ) and linear layers to model therelations between the latent variables { Z t } and the outcome Y .
Revised: All learners use equivalent encoders to represent ˆ  ( · ) and linear layers to model therelations between the latent variables { Z t } and the outcome Y .

Original: The classical learner predicts theoutcome linearly from ˆ Z 1 .
Revised: The classical learner predicts theoutcome linearly from ˆ Z 1 .

Original: All architectures under consideration are visualized jointly in Figure 3a.
Revised: All architectures under consideration are visualized jointly in Figure 3a.

Original: SRL.
Revised: SRL.

Original: The ﬁrst privileged representation learner directly optimizes objective (5), just like generalized
Revised: The ﬁrst privileged representation learner directly optimizes objective (5), just like generalized

Original: LuPTS, but now also ﬁtting the representation ˆ  , parameterized by a neural network.
Revised: LuPTS, but now also ﬁtting the representation ˆ  , parameterized by a neural network.

Original: We refer to thismodel as stepwise representation learner (SRL).
Revised: We refer to thismodel as stepwise representation learner (SRL).

Original: As we will see in experiments, a drawback of thisapproach is that representations may favor predicting transitions ˆ z i,t !
Revised: As we will see in experiments, a drawback of thisapproach is that representations may favor predicting transitions ˆ z i,t !

Original: ˆ
Revised: ˆ

Original: z i,t +1 with small error, whilelosing information relevant for the target outcome in the process.
Revised: z i,t +1 with small error, whilelosing information relevant for the target outcome in the process.

Original: At test time, for a new input x 1 ,
Revised: At test time, for a new input x 1 ,

Original: SRL composes the stepwise dynamics to output ˆ hP ( x 1 ) = ˆ  > ˆ A > T . . . ˆ A >
Revised: SRL composes the stepwise dynamics to output ˆ h

Original: ˆ  ( x 
Revised: ( x ˆ ˆ

Original: ( "# ( $% ! )
Revised: ( "# ( $% ! )

Original: # or !$ ' !
Revised: # or !$ ' !

Original: $ % !
Revised: $ % !

Original: $ & &Φ !
Revised: $ & &Φ !

Original: ( "# ( $%
Revised: ( "# ( $%

Original: Privileged Rep. Learners ● SRL ● GRL ●● CRL
Revised: Privileged Rep. Learners ● SRL ● GRL ●● CRL

Original: Classic Rep. Learner &Φ t= t= t= t= t= x 1 x 2 x 3 x 4 x
Revised: Classic Rep. Learner &Φ t= t= t= t= t= x 1 x 2 x 3 x 4 x

Original: [Figure]
Revised: [Figure]

Original: CRL and GRL.
Revised: CRL and GRL.

Original: To make sure that the learned representation ˆ  retains information about Y , weadd linear outcome supervision to the representation
Revised: To make sure that the learned representation ˆ  retains information about Y , weadd linear outcome supervision to the representation

Original: ˆ  ( X t ) at each time step t .
Revised: ˆ  ( X t ) at each time step t .

Original: Recall that, by
Revised: Recall that, by

Original: Assumption 1, the expected outcome is linear in the latent state at any time step.
Revised: Assumption 1, the expected outcome is linear in the latent state at any time step.

Original: We introduce ahyperparameter  2 [0 , 1] to trade off the two types of losses and arrive at the combined representationlearner (CRL).
Revised: We introduce ahyperparameter  2 [0 , 1] to trade off the two types of losses and arrive at the combined representationlearner (CRL).

Original: With ↵ =
Revised: With ↵ =

Original: (ˆ  , { ˆ A t } , { ˆ ✓ t } ) the entire parameter vector, CRL minimizes the objective
Revised: (ˆ  , { ˆ A t } , { ˆ ✓ t } ) the entire parameter vector, CRL minimizes the objective

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ( ↵

Original: [Equation] [Equation] [Equation] We make test-time predictions using ˆ h P ( x 1 ) =
Revised: We make test-time predictions using ˆ h P ( x 1 ) =

Original: ˆ ✓ > 1 ˆ  ( x 1 ) .
Revised: ˆ ✓ > 1 ˆ  ( x 1 ) .

Original: In experiments, we highlight the case = 1 where only outcome supervision is used as greedy representation learner (GRL).
Revised: In experiments, we highlight the case = 1 where only outcome supervision is used as greedy representation learner (GRL).

Original: For a precisedeﬁnition of the GRL objective, see Appendix D. GRL is related to multi-view learning, in whichprediction of the same quantity is made from multiple “views” (cf. time points)
Revised: For a precisedeﬁnition of the GRL objective, see Appendix D. GRL is related to multi-view learning, in whichprediction of the same quantity is made from multiple “views” (cf. time points)

Original: (Zhao et al., 2017).
Revised: (Zhao et al., 2017).

Original:  Experiments
Revised:  Experiments

Original: We compare classical learning to variants of generalized LuPTS (Algorithm 1) and the privilegedrepresentation learners of Section 3.3 on two synthetic and two real-world time-series data sets.
Revised: We compare classical learning to variants of generalized LuPTS (Algorithm 1) and the privilegedrepresentation learners of Section 3.3 on two synthetic and two real-world time-series data sets.

Original: We(i) verify our theoretical ﬁndings by analyzing the sample efﬁciency and bias-variance characteristicsof the given algorithms; (ii) demonstrate that generalized LuPTS with random features succeeds insettings where linear LuPTS suffers from large bias; (iii) point out that privileged representationlearners offer even greater sample efﬁciency in practice and (iv) study how well these algorithmsrecover the true latent variables { Z t } and how this relates to predictive accuracy.
Revised: We(i) verify our theoretical ﬁndings by analyzing the sample efﬁciency and bias-variance characteristicsof the given algorithms; (ii) demonstrate that generalized LuPTS with random features succeeds insettings where linear LuPTS suffers from large bias; (iii) point out that privileged representationlearners offer even greater sample efﬁciency in practice and (iv) study how well these algorithmsrecover the true latent variables { Z t } and how this relates to predictive accuracy.

Original: Experimental setup.
Revised: Experimental setup.

Original: We report the mean coefﬁcient of determination ( R 2 ), proportional to thesquared-error risk R , for varying sample sizes, sequence lengths and prediction horizons.
Revised: We report the mean coefﬁcient of determination ( R 2 ), proportional to thesquared-error risk R , for varying sample sizes, sequence lengths and prediction horizons.

Original: Experimentsare repeated and averaged over different random seeds.
Revised: Experimentsare repeated and averaged over different random seeds.

Original: In each repetition, a given model performshyperparameter tuning on the training data using random search and ﬁve-fold cross-validation beforebeing re-trained on all training data.
Revised: In each repetition, a given model performshyperparameter tuning on the training data using random search and ﬁve-fold cross-validation beforebeing re-trained on all training data.

Original: The test set size is 1000 samples for synthetic data and 20% ofall data available for real-world data sets.
Revised: The test set size is 1000 samples for synthetic data and 20% ofall data available for real-world data sets.

Original: We consider six privileged learners of two groups.
Revised: We consider six privileged learners of two groups.

Original: The ﬁrstgroup comprises generalized LuPTS with the linear kernel (LuPTS) and the two random feature mapsshown in Section 3.2: Random Fourier features (Fourier RF) and random ReLU features (ReLU RF).
Revised: The ﬁrstgroup comprises generalized LuPTS with the linear kernel (LuPTS) and the two random feature mapsshown in Section 3.2: Random Fourier features (Fourier RF) and random ReLU features (ReLU RF).

Original: The classical learners for this group are OLS estimators used with the same kernel or feature map.
Revised: The classical learners for this group are OLS estimators used with the same kernel or feature map.

Original: The second group consists of the representation learners SRL, CRL and GRL.
Revised: The second group consists of the representation learners SRL, CRL and GRL.

Original: For tabular data, theirencoder is a multi-layer perceptron with three hidden layers of 25 neurons each.
Revised: For tabular data, theirencoder is a multi-layer perceptron with three hidden layers of 25 neurons each.

Original: For the image datathey use LeNet-5 (LeCun et al., 1989).
Revised: For the image datathey use LeNet-5 (LeCun et al., 1989).

Original: The classical learner (Classic Rep.) uses the same encoderwith a linear output layer.
Revised: The classical learner (Classic Rep.) uses the same encoderwith a linear output layer.

Original: The results presented were found to be robust to small changes in trainingparameters such as learning rate.
Revised: The results presented were found to be robust to small changes in trainingparameters such as learning rate.

Original: For details on the training process we refer to Appendix E. Allexperiments required less than 3000 GPU-h to complete using NVIDIA Tesla T4 GPUs.
Revised: For details on the training process we refer to Appendix E. Allexperiments required less than 3000 GPU-h to complete using NVIDIA Tesla T4 GPUs.

Original: Data sets
Revised: Data sets.

Original: We brieﬂy describe evaluation data sets and refer to Appendix E for further details.
Revised: We brieﬂy describe evaluation data sets and refer to Appendix E for further details.

Original: First, we createtwo synthetic data sets in which latent states and outcomes are generated from linear-Gaussiansystems as in Assumption 1.
Revised: First, we create two synthetic data sets in which latent states and outcomes are generated from linearGaussian systems as in Assumption 1.

Original: To produce the observations { X t } we use a deterministic nonlinear
Revised: To produce the observations { X t } we use a deterministic [Table] [Figure] Sign

Original: [Figure]
Revised: [Figure]

Revised: T

Original: [Figure] 
Revised: [Figure] [Figure] [Table] [Figure]

Original: function : Z !
Revised: : Z !

Original: X .
Revised: X .

Original: In the ﬁrst synthetic data set, which we call Square-Sign , the nonlineartransformation : R d !
Revised: In the ﬁrst synthetic data set, which we call Square-Sign , thenonlinear transformation : R d !

Original: R 2 d maps each latent feature Z ( t,k ) to a two dimensional vector such that
Revised: R 2 d maps each latent feature Z ( t,k ) to a two dimensional vectorsuch that

Original: [Equation]
Revised: [Equation]

Original: The second synthetic data set uses the same latent linear system with Z t 2 R 2 and produces squareimages (28 ⇥ 28 pixels) as observations.
Revised: The second synthetic data set uses the same latent linear system with Z t 2 R 2 and produces squareimages (28 ⇥ 28 pixels) as observations.

Original: As the images are reminiscent of clocks we refer to thisdata set as Clocks-LGS .
Revised: As the images are reminiscent of clocks we refer to thisdata set as Clocks-LGS .

Original: Example sequences of these observations are presented in Figure 3b.
Revised: Example sequences of these observations are presented in Figure 3b.

Original: Theangle, size and ﬁll of the two clock hands encode the value of the corresponding latent variable.
Revised: Theangle, size and ﬁll of the two clock hands encode the value of the corresponding latent variable.

Original: Theoutcome Y is a linear function of Z T with q = 1 .
Revised: Theoutcome Y is a linear function of Z T with q = 1 .

Original: For Square-Sign , q = 3 .
Revised: For Square-Sign , q = 3 .

Original: The Metro Interstate trafﬁc volume data set ( Trafﬁc ) (Hogue, 2012) contains hourly records of thetrafﬁc volume on the interstate 94 between Minneapolis and St. Paul, MN.
Revised: The Metro Interstate trafﬁc volume data set ( Trafﬁc ) (Hogue, 2012) contains hourly records of thetrafﬁc volume on the interstate 94 between Minneapolis and St. Paul, MN.

Original: In addition, the datacontains weather features and a holiday indication.
Revised: In addition, the datacontains weather features and a holiday indication.

Original: We predict the trafﬁc volume for a ﬁxed timehorizon given the present observations.
Revised: We predict the trafﬁc volume for a ﬁxed timehorizon given the present observations.

Original: Privileged information is observed every four hours.
Revised: Privileged information is observed every four hours.

Original: We also predict the progression of Alzheimer’s disease (AD) as measured by the outcome of the
Revised: We also predict the progression of Alzheimer’s disease (AD) as measured by the outcome of the

Original: Mini Mental State Examination (MMSE) (Galea and Woodward, 2005).
Revised: Mini Mental State Examination (MMSE) (Galea and Woodward, 2005).

Original: The anonymized data wereobtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004).
Revised: The anonymized data wereobtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004).

Original: Theoutcome of interest is the MMSE score 48 months after the ﬁrst examination.
Revised: Theoutcome of interest is the MMSE score 48 months after the ﬁrst examination.

Original: Privileged informationare the measurements at 12, 24 and 36 months.
Revised: Privileged informationare the measurements at 12, 24 and 36 months.

Original: In addition we tested our algorithms on the PM 2.5data set (Liang et al., 2016), where we predict the air quality in ﬁve Chinese cities (see Appendix F).
Revised: In addition we tested our algorithms on the PM 2.5data set (Liang et al., 2016), where we predict the air quality in ﬁve Chinese cities (see Appendix F).

Original: 4.1 Sample efﬁciency, bias and variance
Revised: 4.1 Sample efﬁciency, bias and variance

Original: The main goal of our work is to improve learning efﬁciency by incorporating privileged time-seriesinformation.
Revised: The main goal of our work is to improve learning efﬁciency by incorporating privileged time-seriesinformation.

Original: Across all prediction tasks and sample sizes, nonlinear variants of generalized LuPTSoutperform their classical counterpart in terms of sample efﬁciency as can be seen in Figure 4.
Revised: Across all prediction tasks and sample sizes, nonlinear variants of generalized LuPTSoutperform their classical counterpart in terms of sample efﬁciency as can be seen in Figure 4.

Original: On the
Revised: On the

Original: Trafﬁc prediction task, LuPTS ReLU RF outperforms linear LuPTS as the former appears to exhibit
Revised: Trafﬁc prediction task, LuPTS ReLU RF outperforms linear LuPTS as the former appears to exhibit

Original: [Figure]
Revised: [Figure]

Original: less bias.
Revised: less bias.

Original: On the synthetic data of Square-Sign , linear models reach their best accuracy quickly,while they are limited by their lack of expressiveness.
Revised: On the synthetic data of Square-Sign , linear models reach their best accuracy quickly,while they are limited by their lack of expressiveness.

Original: Generalized LuPTS ampliﬁes this bias, makinglinear LuPTS worse than OLS.
Revised: Generalized LuPTS ampliﬁes this bias, makinglinear LuPTS worse than OLS.

Original: Random feature methods attain higher accuracy here but are generallyless sample efﬁcient.
Revised: Random feature methods attain higher accuracy here but are generallyless sample efﬁcient.

Original: Generalized LuPTS combined with random features manages to decrease thisgap signiﬁcantly.
Revised: Generalized LuPTS combined with random features manages to decrease thisgap signiﬁcantly.

Original: On ADNI , we don’t see a beneﬁt of using nonlinear models in general.
Revised: On ADNI , we don’t see a beneﬁt of using nonlinear models in general.

Original: We makesimilar observations on the PM 2.5 air quality data set, see Appendix F.
Revised: We makesimilar observations on the PM 2.5 air quality data set, see Appendix F.

Original: The representation learners proposed in Section 3.3 are evaluated on the same data sets and on theimage prediction task Clocks-LGS .
Revised: The representation learners proposed in Section 3.3 are evaluated on the same data sets and on theimage prediction task Clocks-LGS .

Original: Example results are displayed in Figure 5.
Revised: Example results are displayed in Figure 5.

Original: These demonstratethat directly transferring the LuPTS objective to neural networks in the form of SRL results in subparperformance.
Revised: These demonstratethat directly transferring the LuPTS objective to neural networks in the form of SRL results in subparperformance.

Original: SRL does not seem to have a strong enough incentive to learn representations whichaccurately predict the outcome.
Revised: SRL does not seem to have a strong enough incentive to learn representations whichaccurately predict the outcome.

Original: Generalized distillation for privileged time series as suggested by
Revised: Generalized distillation for privileged time series as suggested by

Original: Hayashi et al.
Revised: Hayashi et al.

Original: (2019), does not improve upon classical learning in our tasks.
Revised: (2019), does not improve upon classical learning in our tasks.

Original: On all experimentsdisplayed in Figure 5, CRL and GRL outperform the classical learner.
Revised: On all experimentsdisplayed in Figure 5, CRL and GRL outperform the classical learner.

Original: The predictive accuracy ofthese models is similar on most tasks, as may be explained by the fact that CRL may reduce to GRLwhen choosing  = 1 in objective 6.
Revised: The predictive accuracy ofthese models is similar on most tasks, as may be explained by the fact that CRL may reduce to GRLwhen choosing  = 1 in objective 6.

Original: Noticeably, the general observation of GRL and CRL beingmore sample efﬁcient than the classic model, neither appears to depend on the neural architectureused for the encoder, nor does the modality of the data play an important role, as the image predictiontask Clocks-LGS (Figure 5c) demonstrates.
Revised: Noticeably, the general observation of GRL and CRL beingmore sample efﬁcient than the classic model, neither appears to depend on the neural architectureused for the encoder, nor does the modality of the data play an important role, as the image predictiontask Clocks-LGS (Figure 5c) demonstrates.

Original: For additional empirical results, we refer to Appendix F.
Revised: For additional empirical results, we refer to Appendix F.

Original: To analyze the bias and variance characteristics of our algorithms, we can estimate the expectedsquared prediction bias , E X 

Revised: To analyze the bias and variance characteristics of our algorithms, we can estimate the expectedsquared prediction bias , E X

Original: [( E D [ A ( D )]
Revised: [( E D [ A ( D )]

Original: ( X 1 )  E
Revised: ( X 1 )  E

Original: [ Y | X 1 ]) 2 ] , by computing E Y
Revised: [ Y | X 1 ]) 2 ] , by computing E Y

Original: [ Y | X 1 ] in synthetic
Revised: [ Y | X 1 ] in synthetic

Original: DGPs in addition to the variance of the different estimators.
Revised: DGPs in addition to the variance of the different estimators.

Original: Figure 6 depicts bias and variance for allmodels on the Square-Sign data.
Revised: Figure 6 depicts bias and variance for allmodels on the Square-Sign data.

Original: On the left panel, all variants of generalized LuPTS exhibit lowervariance than classical learning, despite being biased.
Revised: On the left panel, all variants of generalized LuPTS exhibit lowervariance than classical learning, despite being biased.

Original: This holds generally: Across all experiments,we never encountered an example where the use of privileged information has not resulted in lowervariance compared to classical learning.
Revised: This holds generally: Across all experiments,we never encountered an example where the use of privileged information has not resulted in lowervariance compared to classical learning.

Original: On the contrary, the privileged learners suffer higher biasthan the comparable classical learners because of the bias compounding over the individual predictionsteps as shown in Appendix C. For the random feature variants however, the bias decreases withthe number of samples.
Revised: On the contrary, the privileged learners suffer higher biasthan the comparable classical learners because of the bias compounding over the individual predictionsteps as shown in Appendix C. For the random feature variants however, the bias decreases withthe number of samples.

Original: The representation learners display similar characteristics on the right panelof Figure 6.
Revised: The representation learners display similar characteristics on the right panelof Figure 6.

Original: Learning the transitions between latent variables Z t appears to be associated with lowvariance and high bias as demonstrated by the results of SRL.
Revised: Learning the transitions between latent variables Z t appears to be associated with lowvariance and high bias as demonstrated by the results of SRL.

Original: GRL however, which does not modelthese transitions, exhibits the lowest bias and the largest variance of all privileged learners.
Revised: GRL however, which does not modelthese transitions, exhibits the lowest bias and the largest variance of all privileged learners.

Original: As CRLis able to trade off between these two objectives, the estimates for its variance and bias lie in betweenthe corresponding values of the other two privileged learners.
Revised: As CRLis able to trade off between these two objectives, the estimates for its variance and bias lie in betweenthe corresponding values of the other two privileged learners.

Original: 4.2 Latent variable recovery
Revised: 4.2 Latent variable recovery

Original: Under Assumption 1, it is sufﬁcient to identify the representation function  up to a linear transform
Revised: Under Assumption 1, it is sufﬁcient to identify the representation function  up to a linear transform

Original: B to have provable gains from privileged information over a classical learner.
Revised: B to have provable gains from privileged information over a classical learner.

Original: In synthetic data, wecan assess to what extent a representation ˆ  with this property has been found.
Revised: In synthetic data, wecan assess to what extent a representation ˆ  with this property has been found.

Original: As a proxy for theexistence of such a transform, we can compute a measure of correlation between ˆ  ( X ) and  ( X ) ,such as the Canonical Correlation Analysis (CCA) (Hotelling
Revised: As a proxy for theexistence of such a transform, we can compute a measure of correlation between ˆ  ( X ) and  ( X ) ,such as the Canonical Correlation Analysis (CCA) (Hotelling, 1936). Raghu et al.

Revised: (2017) introduceda modiﬁed version called Singular Vector Canonical Correlation Analysis (SVCCA) to computecorrelations when dealing with noisy dimensions in neural network representations.

Original: [Figure]
Revised: [Figure]

Original: The mean SVCCA coefﬁcients ¯ ⇢ and predictive accuracy of the representation learners are visualizedin Figure 7a.
Revised: The mean SVCCA coefﬁcients ¯ ⇢ and predictive accuracy of the representation learners are visualizedin Figure 7a.

Original: One notices that GRL and CRL produce higher correlation coefﬁcients than theclassical learner while also predicting the outcome on the Clocks-LGS task more accurately.
Revised: One notices that GRL and CRL produce higher correlation coefﬁcients than theclassical learner while also predicting the outcome on the Clocks-LGS task more accurately.

Original: Furthercomparing the representations learned by privileged and classical models, Figure 7b shows a visualexample of the improved latent recovery of CRL on the same task.
Revised: Furthercomparing the representations learned by privileged and classical models, Figure 7b shows a visualexample of the improved latent recovery of CRL on the same task.

Original: Concluding, the use of privilegedtime-series information does not only increase the sample efﬁciency of existing algorithms but canalso aid the recovery of latent variables in latent dynamical systems.
Revised: Concluding, the use of privilegedtime-series information does not only increase the sample efﬁciency of existing algorithms but canalso aid the recovery of latent variables in latent dynamical systems.

Original:  Related work
Revised:  Related work

Original: Existing analyses for learning with privileged information guarantee improvements in sample efﬁciency under strong assumptions but are insufﬁcient to establish a clear preference for LuPI learnersin small-sample settings, as argued in the Introduction.
Revised: Existing analyses for learning with privileged information guarantee improvements in sample efﬁciency under strong assumptions but are insufﬁcient to establish a clear preference for LuPI learnersin small-sample settings, as argued in the Introduction.

Original: We are not aware of results tight enoughto establish a lower bound on the risk of a classical learner larger than an upper bound for a PIlearner.
Revised: We are not aware of results tight enoughto establish a lower bound on the risk of a classical learner larger than an upper bound for a PIlearner.

Original: Additionally, our problem, see especially (5)–(6), is related to multi-task (representation)learning (Maurer et al., 2016).
Revised: Additionally, our problem, see especially (5)–(6), is related to multi-task (representation)learning (Maurer et al., 2016).

Original: However, our goal is different in that only a single task is of interestafter learning.
Revised: However, our goal is different in that only a single task is of interestafter learning.

Original: In estimation of causal effects, learning from surrogate outcomes (Prentice, 1989) hasbeen proposed as a way to increase sample efﬁciency.
Revised: In estimation of causal effects, learning from surrogate outcomes (Prentice, 1989) hasbeen proposed as a way to increase sample efﬁciency.

Original: Surrogates are variables related to the outcomewhich may be available even when the outcome is not (Athey et al., 2019).
Revised: Surrogates are variables related to the outcomewhich may be available even when the outcome is not (Athey et al., 2019).

Original: We can view these asprivileged information.
Revised: We can view these asprivileged information.

Original: While the problem shares structure with ours, the goal is to compensatefor missing outcomes and analytical results give no guarantees for improved efﬁciency when bothsurrogates outcomes are always observed (Chen et al., 2008;
Revised: While the problem shares structure with ours, the goal is to compensatefor missing outcomes and analytical results give no guarantees for improved efﬁciency when bothsurrogates outcomes are always observed (Chen et al., 2008;

Original: Kallus and Mao, 2020).
Revised: Kallus and Mao, 2020).

Original: Theorem 1 relies on the representation function  being identiﬁed up to a linear transformation.
Revised: Theorem 1 relies on the representation function  being identiﬁed up to a linear transformation.

Original: Nonlinear independent component analysis (ICA) aims to solve precisely this problem (Hyvärinenand Pajunen, 1999).
Revised: Nonlinear independent component analysis (ICA) aims to solve precisely this problem (Hyvärinenand Pajunen, 1999).

Original: It has been shown that non-stationary time series provide the conditions foridentiﬁability through contrastive learning (Hyvarinen and Morioka, 2016).
Revised: It has been shown that non-stationary time series provide the conditions foridentiﬁability through contrastive learning (Hyvarinen and Morioka, 2016).

Original: These results make for apotentially interesting connection between learning using privileged time series and nonlinear ICA.
Revised: These results make for apotentially interesting connection between learning using privileged time series and nonlinear ICA.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented learning algorithms for predicting nonlinear outcomes by utilizing time seriesprivileged information during training.
Revised: We have presented learning algorithms for predicting nonlinear outcomes by utilizing time seriesprivileged information during training.

Original: We prove that our estimator is preferable to classical learningwhen data is generated from a latent-variable dynamical system with partially known components.
Revised: We prove that our estimator is preferable to classical learningwhen data is generated from a latent-variable dynamical system with partially known components.

Original: When the system is entirely unknown, we show that using random ReLU features results in aconsistent privileged estimator.
Revised: When the system is entirely unknown, we show that using random ReLU features results in aconsistent privileged estimator.

Original: We also propose representation learning methods of related form anddemonstrate the beneﬁts of all learners over classical learning empirically.
Revised: We also propose representation learning methods of related form anddemonstrate the beneﬁts of all learners over classical learning empirically.

Original: Remaining challengesinclude providing risk guarantees for learning with biased representations, with regularized estimators,and for more general data generating processes with weaker structural assumptions.
Revised: Remaining challengesinclude providing risk guarantees for learning with biased representations, with regularized estimators,and for more general data generating processes with weaker structural assumptions.

Original: We are hopefulthat the utility demonstrated in this work will inspire future research to overcome these limitations.Checklist
Revised: We are hopefulthat the utility demonstrated in this work will inspire future research to overcome these limitations.

Original: For all authors...(

Original: a) Do the main

Original: claims

Original: made

Original: in the abstract and introduction accurately reﬂect

Original: the

Original: paper’s

Original: contributions

Original: and

Original: scope?

Original: Yes

Original: ] The introduction refers to the section where eachcontribution is made

Original: b) Did you

Original: describe

Original: the

Original: limitations

Original: of

Original: your

Original: work?

Original: Yes

Original: ] See

Original: e

Original: g., the discussion in

Original: Section 6 and

Original: on the applicability of Theorem 1 in Section 3.

Original: and

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We have

Original: not identiﬁed any potential negative societal impact directly related to our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assump-

Original: tion 1 and Theorem 1.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Yes, see the Appendix.3.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Code was

Original: submitted as supplementary information.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] See Section 4 and Appendix E

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] Each result

Original: ﬁgure

Original: includes error regions.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix E.2(b)

Original: Did you mention the license of the assets?

Original: [Yes] See Appendix E.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: Only original code.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [Yes] ADNI data is anonymized.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]