Original: Efficient learning of nonlinear prediction models with time-series privileged information
Revised: Efficient learning of nonlinear prediction models with time-series privileged information

Original: Abstract
Revised: Abstract

Original: In domains where sample sizes are limited, efficient learning algorithms are critical.
Revised: In domains where sample sizes are limited, efficient learning algorithms are critical.

Original: Learning using privileged information (LuPI) offers increased sample efficiency by allowing prediction models access to types of information at training time which is unavailable when the models are used.
Revised: Learning using privileged information (LuPI) offers increased sample efficiency by allowing prediction models access to auxiliary information at training time which is unavailable when the models are used.

Original: In recent work, it was shown that for prediction in linear-Gaussian dynamical systems, a LuPI learner with access to intermediate time series data is never worse and often better in expectation than any unbiased classical learner.
Revised: In recent work, it was shown that for prediction in linear-Gaussian dynamical systems, a LuPI learner with access to intermediate time series data is never worse and often better in expectation than any unbiased classical learner.

Original: We provide new insights into this analysis and generalize it to nonlinear prediction tasks in latent dynamical systems, extending theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform.
Revised: We provide new insights into this analysis and generalize it to nonlinear prediction tasks in latent dynamical systems, extending theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform.

Original: In addition, we propose algorithms based on random features and representation learning for the case when this map is unknown.
Revised: In addition, we propose algorithms based on random features and representation learning for the case when this map is unknown.

Original: A suite of empirical results confirm theoretical findings and show the potential of using privileged time-series information in nonlinear prediction.
Revised: A suite of empirical results confirm theoretical findings and show the potential of using privileged time-series information in nonlinear prediction.

Original:  Introduction
Revised:  Introduction

Original: In data-poor domains making the best use of all available information is central to efficient and effective machine learning.
Revised: In data-poor domains making the best use of all available information is central to efficient and effective machine learning.

Original: Despite this, supervised learning is often applied in such a way that informative data is ignored.
Revised: Despite this, supervised learning is often applied in such a way that informative data is ignored.

Original: A good example of this is learning to predict the condition of a patient at a set follow-up time based on information of the first medical examination.
Revised: A good example of this is learning to predict the condition of a patient at a set follow-up time based on information of the first medical examination.

Original: Classical supervised learning makes use only of the initial data to predict the disease status at the follow-up, although in many cases data about medications, laboratory tests or vital signs are routinely collected about patients also at intermediate time points.
Revised: Classical supervised learning makes use only of the initial data to predict the disease status at the follow-up, although in many cases data about medications, laboratory tests or vital signs are routinely collected about patients also at intermediate time points.

Original: This information is privileged (Vapnik and Vashist, 2009), as it is unavailable at the time of prediction but can be used for training a model.
Revised: This information is privileged (Vapnik and Vashist, 2009), as it is unavailable at the time of prediction but can be used for training a model.

Original: Learning using privileged information (LuPI) (Vapnik and Vashist, 2009), generalized distillation (Lopez-Paz et al., 2016) and multi-view learning (Rosenberg and Bartlett, 2007) have been proposed to increase the sample efficiency by leveraging privileged information in learning.
Revised: Learning using privileged information (LuPI) (Vapnik and Vashist, 2009), generalized distillation (Lopez-Paz et al., 2016) and multi-view learning (Rosenberg and Bartlett, 2007) have been proposed to increase the sample efficiency by leveraging privileged information in learning.

Original: Theoretical results guarantee improved learning rates (Pechyony and Vapnik, 2010; Vapnik et al., 2015) or tighter generalization bounds (Wang, 2019) for large sample sizes under appropriate assumptions.
Revised: Theoretical results guarantee improved learning rates (Pechyony and Vapnik, 2010; Vapnik et al., 2015) or tighter generalization bounds (Wang, 2019) for large sample sizes under appropriate assumptions.

Original: However, privileged information is not always beneficial; it must be related to the task at hand (Jonschkowski et al., 2015).
Revised: However, privileged information is not always beneficial; it must be related to the task at hand (Jonschkowski et al., 2015).

Original: Previous works do not identify such settings.
Revised: Previous works do not identify such settings.

Original: Moreover, existing analyses do not state when learning with privileged information is preferable to classical learning for problems with small sample sizes—which is where efficiency is needed the most.
Revised: Moreover, existing analyses do not state when learning with privileged information is preferable to classical learning for problems with small sample sizes—which is where efficiency is needed the most.

Original: Karlsson et al.
Revised: Karlsson et al.

Original: (2022) studied LuPI in the context of predicting an outcome observed at the end of a time series based on variables collected at the first time step.
Revised: (2022) studied LuPI in the context of predicting an outcome observed at the end of a time series based on variables collected at the first time step.

Original: They showed that making use of data from intermediate time steps in particular settings always leads to lower or equal prediction risk—for any sample size—compared to the best unbiased model which does not make use of this privileged information.
Revised: They showed that making use of data from intermediate time steps in particular settings always leads to lower or equal prediction risk—for any sample size—compared to the best unbiased model which does not make use of this privileged information.

Original: However, their method called learning using privileged time series (LuPTS) was limited to settings where the outcome function, and estimators of it, are linear functions of baseline features.
Revised: However, their method called learning using privileged time series (LuPTS) was limited to settings where the outcome function, and estimators of it, are linear functions of baseline features.

Original: # ! $ # $ …
Revised: # ! $ # $ …

Original: Observed
Revised: Observed

Original: Privileged
Revised: Privileged

Original: Unobserved (
Revised: Unobserved (

Original: [Figure]
Revised: [Figure]

Original: Ψ $ ! ! "
Revised: Ψ $ ! ! "

Original: " " $ " !
Revised: " " $ " !

Original: # " # ! $ " $ % & … Ψ Ψ Ψ
Revised: # " # ! $ " $ % & … Ψ Ψ Ψ

Original: Φ (b) Linear latent dynamics, nonlinear observations
Revised: Φ (b) Linear latent dynamics, nonlinear observations

Original: Moreover, their analysis did not study how variance reduction behaves as a function of increased input dimension.
Revised: Moreover, their analysis did not study how variance reduction behaves as a function of increased input dimension.

Original: Hayashi et al.
Revised: Hayashi et al.

Original: (2019) also learned from privileged intermediate time points but their study was limited to empirical results for classification using generalized distillation.
Revised: (2019) also learned from privileged intermediate time points but their study was limited to empirical results for classification using generalized distillation.

Original: There is an abundance of real-world prediction tasks with fixed follow-up times which can be framed as having access to privileged time-series information.
Revised: There is an abundance of real-world prediction tasks with fixed follow-up times which can be framed as having access to privileged time-series information.

Original: Examples include predicting 90-day patient mortality (Karhade et al., 2019) or patient readmission in healthcare (Mortazavi et al., 2016), the churn of users of an online service over a fixed period (Huang et al., 2012) or yearly crop yields from satellite imagery of farms (You et al., 2017).
Revised: Examples include predicting 90-day patient mortality (Karhade et al., 2019) or patient readmission in healthcare (Mortazavi et al., 2016), the churn of users of an online service over a fixed period (Huang et al., 2012) or yearly crop yields from satellite imagery of farms (You et al., 2017).

Original: In these cases, privileged time-series information comprises daily patient vitals, intermediate user interactions and daily satellite imagery respectively.
Revised: In these cases, privileged time-series information comprises daily patient vitals, intermediate user interactions and daily satellite imagery respectively.

Original: We are motivated by finding sample-efficient learning algorithms that utilize privileged time-series information to improve upon classical learning alternatives in such settings.
Revised: We are motivated by finding sample-efficient learning algorithms that utilize privileged time-series information to improve upon classical learning alternatives in such settings.

Original: Contributions.
Revised: Contributions.

Original: We extend the LuPTS framework to nonlinear models and prediction tasks in latent dynamical systems (Section 3).
Revised: We extend the LuPTS framework to nonlinear models and prediction tasks in latent dynamical systems (Section 3).

Original: In this setting, we prove that learning with privileged information leads to lower risk when the nonlinear map connecting latent variables and observations is known up to a linear transform (Section 3.1).
Revised: In this setting, we prove that learning with privileged information leads to lower risk when the nonlinear map connecting latent variables and observations is known up to a linear transform (Section 3.1).

Original: In doing so, we also find that when the representation dimension grows larger than the number of samples, the benefit of privileged information vanishes.
Revised: In doing so, we also find that when the representation dimension grows larger than the number of samples, the benefit of privileged information vanishes.

Original: We show that a privileged learner using random feature maps can learn optimal models consistently, even when the relationship between latent and observed variables is unknown, and give a practical algorithm based on this idea (Section 3.2).
Revised: We show that a privileged learner using random feature maps can learn optimal models consistently, even when the relationship between latent and observed variables is unknown, and give a practical algorithm based on this idea (Section 3.2).

Original: However, random feature methods may suffer from bias in small samples.
Revised: However, random feature methods may suffer from bias in small samples.

Original: As a remedy, we propose several representation learning algorithms aimed at trading off bias and variance (Section 3.3).
Revised: As a remedy, we propose several representation learning algorithms aimed at trading off bias and variance (Section 3.3).

Original: In experiments, we find that privileged time-series learners with either random features or representation learning reduce variance and improve latent state recovery in small-sample settings (Section 4) on both synthetic and real-world regression tasks.
Revised: In experiments, we find that privileged time-series learners with either random features or representation learning reduce variance and improve latent state recovery in small-sample settings (Section 4) on both synthetic and real-world regression tasks.

Original:  Prediction and privileged information in nonlinear time series
Revised:  Prediction and privileged information in nonlinear time series

Original: We aim to predict outcomes Y in Y
Revised: We aim to predict outcomes Y in Y

Original: ⊆
Revised: ⊆

Original: R q based on covariates
Revised: R q based on covariates

Original: X 1 in X ⊆ R k .
Revised: X 1 in X ⊆ R k .

Original: X 1 are observed at a baseline time point t = 1 , starting a  time series on the form X 1 , X 2 , . . .
Revised: X 1 are observed at a baseline time point t = 1 , starting a discrete time series on the form X 1 , X 2 , . . .

Original: X T , Y .
Revised: X T , Y .

Original: Outcomes are assumed to be compositions of a representation Φ , a linear map θ and Gaussian noise ϵ ,
Revised: Outcomes are assumed to be compositions of a representation Φ , a linear map θ and Gaussian noise ϵ ,

Original: Y = h ( X 1 ) + ϵ, where h ( x 1 ) :
Revised: Y = h ( X 1 ) + ϵ, where h ( x 1 ) :

Original: = θ ⊤ Φ( x 1 ) , and ϵ ∼ N (0 , ˜ σ 2 Y ) .
Revised: = θ ⊤ Φ( x 1 ) , and ϵ ∼ N (0 , ˜ σ 2 Y ) .

Original: In addition to observations of X 1 and Y , privileged information (PI) is available in the form of samples of random variables, X 2 , ..., X T , from intermediate time points between X 1 and Y , all taking values in X .
Revised: In addition to observations of X 1 and Y , privileged information (PI) is available in the form of samples of random variables, X 2 , ..., X T , from intermediate time points between X 1 and Y , all taking values in X .

Original: Two data generating processes (DGPs) with this structure are illustrated in Figure 1.
Revised: Two data generating processes (DGPs) with this structure are illustrated in Figure 1.

Original: Unlike baseline variables X 1 , privileged information is observed only at training time, not at test time .
Revised: Unlike baseline variables X 1 , privileged information is observed only at training time, not at test time .

Original: Therefore, it can only benefit learning, and not inference.
Revised: Therefore, it can only benefit learning, and not inference.

Original: Data sets of m ∈ N + training samples,
Revised: Data sets of m ∈ N + training samples,

Original: D := { ( x i, 1 , x i, 2 , ..., x i,T , y i ) } mi =1 , are drawn independently and identically distributed from a fixed, unknown distribution p over all random variables in our system.
Revised: D := { ( x i, 1 , x i, 2 , ..., x i,T , y i ) } mi =1 , are drawn independently and identically distributed from a fixed, unknown distribution p over all random variables in our system.

Original: We let X t =
Revised: We let X t =

Original: [ x 1 ,t , ..., x m,t ] ⊤ denote the data matrix of features observed at time t = 1 , ..., T and Y =
Revised: [ x 1 ,t , ..., x m,t ] ⊤ denote the data matrix of features observed at time t = 1 , ..., T and Y =

Original: [ y 1 , ..., y m ] ⊤ the vector of all outcomes observed in D .
Revised: [ y 1 , ..., y m ] ⊤ the vector of all outcomes observed in D .

Original: A learning algorithm A : D → H maps data sets D ∈ D to hypotheses ˆ h ∈ H .
Revised: A learning algorithm A : D → H maps data sets D ∈ D to hypotheses ˆ h ∈ H .

Original: An efficient algorithm A has small expected risk R p ( A ) with respect to a loss
Revised: An efficient algorithm A has small expected risk R p ( A ) with respect to a loss

Original: L : Y × Y → R over training sets of fixed size m drawn from p ,
Revised: L : Y × Y → R over training sets of fixed size m drawn from p ,

Original: R p
Revised: R p

Revised: [Equation]

Original: A ) :=
Revised: A ) :=

Original: E D [ R p (ˆ h )] where ˆ h
Revised: E D [ R p (ˆ h )] where ˆ h

Original: =
Revised: =

Original: A ( D ) and R p (ˆ h ) :=
Revised: A ( D ) and R p (ˆ h ) :=

Original: E p [ L (ˆ h ( X 1 ) , Y )] 
Revised: E p [ L (ˆ h ( X 1 ) , Y )] .

Original: [Equation]
Revised: [Equation]

Original: We study the regression setting with L the squared error, L ( y, y ′ )
Revised: We study the regression setting with L the squared error, L ( y, y ′ )

Original: =
Revised: =

Original: ∥
Revised: ∥

Original: y − y ′
Revised: y − y ′

Original: ∥ 22 .
Revised: ∥ 22 .

Original: In analysis, we focus on univariate outcomes, q = 1 , but all results extend to multivariate outcomes, q > 1 .
Revised: In analysis, we focus on univariate outcomes, q = 1 , but all results extend to multivariate outcomes, q > 1 .

Original: Input: Data D = ( { X t } , Y ) ; Repr.
Revised: Input: Data D = ( { X t } , Y ) ; Repr.

Original: ˆΦ or kernel κ if using a fixed representation ˆΦ then
Revised: ˆΦ or kernel κ if using a fixed representation ˆΦ then

Original: [Equation]
Revised: [Equation]

Original: ˆΦ
Revised: ˆΦ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: else if using kernel κ then ˆ
Revised: else if using kernel κ then ˆ

Original: [Equation]
Revised: [Equation]

Original: for t =
Revised: for t =

Original: [Equation]
Revised: [Equation]

Original: return ˆ h P
Revised: return ˆ h P

Original: Features, 0.
Revised: Features, 0.

Original: R ²
Revised: R ²

Original: OLS
Revised: OLS

Original: LuPTS
Revised: LuPTS

Original: Our main goal is to compare privileged learners
Revised: Our main goal is to compare privileged learners

Original: A P , which make use of privileged information, to classical learners A C which learn without PI.
Revised: A P , which make use of privileged information, to classical learners A C which learn without PI.

Original: We seek to identify conditions and algorithms for which privileged learning is more efficient, i.e., it leads to lower risk in expectation, R ( A P ) ≤ R ( A C ) for the same number of training samples.
Revised: We seek to identify conditions and algorithms for which privileged learning is more efficient, i.e., it leads to lower risk in expectation, R ( A P ) ≤ R ( A C ) for the same number of training samples.

Original: We describe such a setting next.
Revised: We describe such a setting next.

Original: 2.1 Privileged information in latent dynamical systems
Revised: 2.1 Privileged information in latent dynamical systems

Original: We study tasks where data are produced by a latent dynamical process.
Revised: We study tasks where data are produced by a latent dynamical process.

Original: Observations X t are generated from unobserved latent states Z t through a nonlinear transformation Ψ , see Figure 1b.
Revised: Observations X t are generated from unobserved latent states Z t through a nonlinear transformation Ψ , see Figure 1b.

Original: This means that the target of learning, h
Revised: This means that the target of learning, h

Original: ( X 1 ) =
Revised: ( X 1 ) =

Original: E [ Y | X 1 ] , is nonlinear in X 1 .
Revised: E [ Y | X 1 ] , is nonlinear in X 1 .

Original: Latent dynamical systems like these have proven successful at modelling a variety of phenomena as for example fluid dynamics in physics (Lee and Carlberg, 2019) and human brain activity in neuroscience (Kao et al., 2015).
Revised: Latent dynamical systems like these have proven successful at modelling a variety of phenomena as for example fluid dynamics in physics (Lee and Carlberg, 2019) and human brain activity in neuroscience (Kao et al., 2015).

Original: Assumption 1 (Latent linear-Gaussian system) .
Revised: Assumption 1 (Latent linear-Gaussian system) .

Original: Let Z 1 , ..., Z T be latent states related as a linearGaussian dynamical system in a space Z ⊆ R d with Z 1 of arbitrary distribution.
Revised: Let Z 1 , ..., Z T be latent states related as a linearGaussian dynamical system in a space Z ⊆ R d with Z 1 of arbitrary distribution.

Original: Further, let the observation function be an injective map Ψ :
Revised: Further, let the observation function be an injective map Ψ :

Original: Z → X with left-inverse (representation) Φ : X → Z .
Revised: Z → X with left-inverse (representation) Φ : X → Z .

Original: With A 1 , ..., A T − 1 ∈ R d × d , β ∈ R d assume that Z 1 , ..., Z T , X 1 , ..., X T , Y are generated as
Revised: With A 1 , ..., A T − 1 ∈ R d × d , β ∈ R d assume that Z 1 , ..., Z T , X 1 , ..., X T , Y are generated as

Original: Z t =
Revised: Z t =

Original: A ⊤ t − 1 Z t − 1 + ϵ t for t ≥ 2 , and ∀ t : X t =
Revised: A ⊤ t − 1 Z t − 1 + ϵ t for t ≥ 2 , and ∀ t : X t =

Original: Ψ( Z t ) and
Revised: Ψ( Z t ) and

Original: Y = β ⊤ Z T + ϵ Y .
Revised: Y = β ⊤ Z T + ϵ Y .

Original: where ϵ t ∼ N (0 , σ 2 t I ) and ϵ Y ∼ N (0 , σ 2 Y ) .
Revised: where ϵ t ∼ N (0 , σ 2 t I ) and ϵ Y ∼ N (0 , σ 2 Y ) .

Original: It is easy to verify that Assumption 1 is consistent with (1), but stronger: there are more systems that map X 1 to Y as θ ⊤ Φ( X 1 ) than systems where additionally X t
Revised: It is easy to verify that Assumption 1 is consistent with (1), but stronger: there are more systems that map X 1 to Y as θ ⊤ Φ( X 1 ) than systems where additionally X t

Original: = Ψ( Z t ) .
Revised: = Ψ( Z t ) .

Original: Nevertheless, it is much more general than the results of Karlsson et al.
Revised: Nevertheless, it is much more general than the results of Karlsson et al.

Original: (2022), limited to the linear setting, i.e. Φ( x 1 ) = x 1 .
Revised: (2022), limited to the linear setting, i.e. Φ( x 1 ) = x 1 .

Original: Next, we present a generalized version of the LuPTS algorithm of Karlsson et al.
Revised: Next, we present a generalized version of the LuPTS algorithm of Karlsson et al.

Original: (2022) that is provably preferable to classical learning when data is generated according to Assumption 1 and Φ is known up to a linear transform.
Revised: (2022) that is provably preferable to classical learning when data is generated according to Assumption 1 and Φ is known up to a linear transform.

Original: Further, we discuss how a privileged learner can be made universally consistent for unknown representations Φ when combined with random features.
Revised: Further, we discuss how a privileged learner can be made universally consistent for unknown representations Φ when combined with random features.

Original:  Efficient learning with time-series privileged information
Revised:  Efficient learning with time-series privileged information

Original: We analyze and compare learners from the privileged and classical paradigms which produce estimates of the form ˆ h ( x 1 )
Revised: We analyze and compare learners from the privileged and classical paradigms which produce estimates of the form ˆ h ( x 1 )

Original: =
Revised: =

Original: ˆ θ ⊤ ˆΦ( x 1 ) , as motivated by (1).
Revised: ˆ θ ⊤ ˆΦ( x 1 ) , as motivated by (1).

Original: We let each use representations ˆΦ : X → ˆ Z ⊆
Revised: We let each use representations ˆΦ : X → ˆ Z ⊆

Original: R ˆ d from a family F shared by both paradigms , so that the hypothesis class H ∋ ˆ h is shared as well.
Revised: R ˆ d from a family F shared by both paradigms , so that the hypothesis class H ∋ ˆ h is shared as well.

Original: 3.1 Learning with true representations known up to a linear transform
Revised: 3.1 Learning with true representations known up to a linear transform

Original: In this section, we assume that privileged and classical learners have access to a common, fixed representation function ˆΦ which is related to the true representation function through a linear transform, meaning there exists a matrix B such that ˆΦ( · ) =
Revised: In this section, we assume that privileged and classical learners have access to a common, fixed representation function ˆΦ which is related to the true representation function through a linear transform, meaning there exists a matrix B such that ˆΦ( · ) =

Original: B Φ( · ) .
Revised: B Φ( · ) .

Original: Classical learning.
Revised: Classical learning.

Original: As a classical learner to serve as a strong comparison point for our privileged learners, we use the ordinary least-squares (OLS) linear estimator applied to the latent variables at the first time point inferred by ˆΦ .
Revised: As a classical learner to serve as a strong comparison point for our privileged learners, we use the ordinary least-squares (OLS) linear estimator applied to the latent variables at the first time point inferred by ˆΦ .

Original: With ˆ Z 1 = [ˆΦ( x 1 , 1 ) , ..., ˆΦ( x m, 1 )] ⊤ ∈ R m × ˆ d ,
Revised: With ˆ Z 1 = [ˆΦ( x 1 , 1 ) , ..., ˆΦ( x m, 1 )] ⊤ ∈ R m × ˆ d ,

Original: [Equation]
Revised: [Equation]

Original: ) where ˆ
Revised: ) where ˆ

Original: [Equation]
Revised: [Equation]

Original: When m > ˆ d and Φ is known up to linear transform, ˆ h C is the minimum-risk unbiased estimator of h which does not use privileged information.
Revised: When m ≥ ˆ d and Φ is known up to linear transform, ˆ h C is the minimum-risk unbiased estimator of h which does not use privileged information.

Original: To accommodate the underdetermined case where m < ˆ d , the matrix inverse is replaced by the Moore-Penrose pseudo inverse ( · ) † (Penrose, 1956).
Revised: To accommodate the underdetermined case where m < ˆ d , the matrix inverse is replaced by the Moore-Penrose pseudo inverse ( · ) † (Penrose, 1956).

Original: Generalized LuPTS.
Revised: Generalized LuPTS.

Original: For privileged learning, we first compute ˆ Z t = [ˆΦ( x 1 ,t ) , ..., ˆΦ( x m,t )]
Revised: For privileged learning, we first compute ˆ Z t = [ˆΦ( x 1 ,t ) , ..., ˆΦ( x m,t )]

Original: ⊤ for t = 1 , ..., T .
Revised: ⊤ for t = 1 , ..., T .

Original: Then, we independently fit parameter estimates ˆ A 1 , ..., ˆ A T , ˆ β of the dynamical system shown in Figure 1b by minimizing the squared error in single-step predictions.
Revised: Then, we independently fit parameter estimates ˆ A 1 , ..., ˆ A T , ˆ β of the dynamical system shown in Figure 1b by minimizing the squared error in single-step predictions.

Original: This equates to a series of OLS estimates in ˆ Z .
Revised: This equates to a series of OLS estimates in ˆ Z .

Original: At test time, baseline variables x 1 are embedded with ˆΦ and the latent dynamical system is simulated for T time steps to form a prediction ˆ h P ( x 1 ) =
Revised: At test time, baseline variables x 1 are embedded with ˆΦ and the latent dynamical system is simulated for T time steps to form a prediction ˆ h P ( x 1 ) =

Original: ( ˆ A 1 · · · ˆ A T β ) ⊤ ˆΦ( x 1 ) .
Revised: ( ˆ A 1 · · · ˆ A T β ) ⊤ ˆΦ( x 1 ) .

Original: Putting this together, we arrive at Algorithm 1 which we call generalized LuPTS .
Revised: Putting this together, we arrive at Algorithm 1 which we call generalized LuPTS .

Original: We may apply a simple matrix identity and replace terms ˆ Z t ˆ Z ⊤ t in Algorithm 1 by the Gram matrices ˆ K t
Revised: We may apply a simple matrix identity and replace terms ˆ Z t ˆ Z ⊤ t in Algorithm 1 by the Gram matrices ˆ K t

Original: = κ ( X t , X t ) of a reproducing kernel κ with corresponding (implicit) feature map ˆΦ , κ ( x, x ′ )
Revised: = κ ( X t , X t ) of a reproducing kernel κ with corresponding (implicit) feature map ˆΦ , κ ( x, x ′ )

Original: = ⟨ ˆ Φ( x ) , ˆ Φ( x ′ ) ⟩ (Smola and Schölkopf, 1998).
Revised: = ⟨ ˆ Φ( x ) , ˆ Φ( x ′ ) ⟩ (Smola and Schölkopf, 1998).

Original: This variant allows for learning with unknown Φ .
Revised: This variant allows for learning with unknown Φ .

Original: We now state a result which says that under Assumption 1, for an appropriate fixed representation ˆΦ or kernel κ , generalized LuPTS is never worse in expectation than the classical learner in (3).
Revised: We now state a result which says that under Assumption 1, for an appropriate fixed representation ˆΦ or kernel κ , generalized LuPTS is never worse in expectation than the classical learner in (3).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let D be a data set of size m drawn from p , consistent with Assumption 1.
Revised: Let D be a data set of size m drawn from p , consistent with Assumption 1.

Original: Assume that the left inverse Φ : X → Z of the observation function Ψ is known up to a linear transform, explicitly or through a kernel κ ( x, x ′ )
Revised: Assume that the left inverse Φ : X → Z of the observation function Ψ is known up to a linear transform, explicitly or through a kernel κ ( x, x ′ )

Original: = ⟨ ˆΦ( x ) , ˆΦ( x ′ ) ⟩ , i.e., there exists a matrix B with linearly independent columns such that ˆΦ( x ) =
Revised: = ⟨ ˆΦ( x ) , ˆΦ( x ′ ) ⟩ , i.e., there exists a matrix B with linearly independent columns such that ˆΦ( x ) =

Original: B Φ( x ) with ˆΦ : X → ˆ Z for all x ∈ X .
Revised: B Φ( x ) with ˆΦ : X → ˆ Z for all x ∈ X .

Original: Then, it holds for the privileged learner A P ( D ) =
Revised: Then, it holds for the privileged learner A P ( D ) =

Original: ˆ h P (Algorithm 1) and the classical learner A C ( D )
Revised: ˆ h P (Algorithm 1) and the classical learner A C ( D )

Original: =
Revised: =

Original: ˆ h C
Revised: ˆ h C

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: First, we show that the predictions made by generalized LuPTS are invariant to a linear transform B applied to ˆΦ( · ) during training, see Appendix A.
Revised: First, we show that the predictions made by generalized LuPTS are invariant to a linear transform B applied to ˆΦ( · ) during training, see Appendix A.

Original: Then we consider two cases: (i) When ˆ d ≤ m we may re-purpose the proof of Theorem 1 in Karlsson et al.
Revised: Then we consider two cases: (i) When ˆ d ≤ m we may re-purpose the proof of Theorem 1 in Karlsson et al.

Original: (2022), (ii) When m < ˆ d
Revised: (2022), (ii) When m < ˆ d

Original: Proposition 1 below directly implies E ˆ h P ,X 1 [Var D (ˆ h C ( X 1 ) | ˆ h P )]
Revised: Proposition 1 below directly implies E ˆ h P ,X 1 [Var D (ˆ h C ( X 1 ) | ˆ h P )]

Original: = 0 and R ( A P ) =
Revised: = 0 and R ( A P ) =

Original: R ( A C ) .
Revised: R ( A C ) .

Original: Theorem 1 implies that the privileged learner is at least as sample efficient as the classical one since
Revised: Theorem 1 implies that the privileged learner is at least as sample efficient as the classical one since

Original: Var( · ) ≥ 0 and thus R ( A P ) ≤ R ( A C ) for the same number of training samples m .
Revised: Var( · ) ≥ 0 and thus R ( A P ) ≤ R ( A C ) for the same number of training samples m .

Original: The result is a direct generalization of the main result in Karlsson et al.
Revised: The result is a direct generalization of the main result in Karlsson et al.

Original: We also observe that generalized
Revised: We also observe that generalized

Original: LuPTS and the classical learner coincide under certain conditions when m ≤ ˆ d .
Revised: LuPTS and the classical learner coincide under certain conditions when m ≤ ˆ d .

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ˆ Φ :
Revised: Let ˆ Φ :

Original: X → ˆ Z ⊆
Revised: X → ˆ Z ⊆

Original: R ˆ d be any map with corresponding kernel κ .
Revised: R ˆ d be any map with corresponding kernel κ .

Original: Let ˆ K t be the Gram matrix of κ applied to X t and let ˆ h C , ˆ h P be classical (3) and privileged (Algorithm 1) estimates.
Revised: Let ˆ K t be the Gram matrix of κ applied to X t and let ˆ h C , ˆ h P be classical (3) and privileged (Algorithm 1) estimates.

Original: Then, ˆ K t is invertible for all t =
Revised: Then, ˆ K t is invertible for all t =

Original: ⇒ ˆ h P =
Revised: ⇒ ˆ h P =

Original: ˆ h C .
Revised: ˆ h C .

Original: ˆ K t is noninvertible whenever m > ˆ d , assuming linearly independent features.
Revised: ˆ K t is noninvertible whenever m > ˆ d , assuming linearly independent features.

Original: Proof sketch.
Revised: Proof sketch.

Original: When the Gram matrices K t are invertible, the pseudo-inverse coincides with the inverse, and factors K t K † t in the LuPTS estimators cancel, making LuPTS and CL equal.
Revised: When the Gram matrices K t are invertible, the pseudo-inverse coincides with the inverse, and factors K t K † t in the LuPTS estimators cancel, making LuPTS and CL equal.

Original: Remarks.
Revised: Remarks.

Original: Theorem 1 extends the applicability of LuPTS to a) nonlinear prediction through a fixed feature map or b) kernel estimation and c) to the underdetermined case of m < ˆ d .
Revised: Theorem 1 extends the applicability of LuPTS to a) nonlinear prediction through a fixed feature map or b) kernel estimation and c) to the underdetermined case of m < ˆ d .

Original: While previous work is restricted to observed linear systems our result considers the case of latent linear dynamics which only need to be identified up to a linear transform (see Figure 1).
Revised: While previous work is restricted to observed linear systems our result considers the case of latent linear dynamics which only need to be identified up to a linear transform (see Figure 1).

Original: Proposition 1 does not claim that no preferable privileged learner exists; it is a statement only about generalized LuPTS.
Revised: Proposition 1 does not claim that no preferable privileged learner exists; it is a statement only about generalized LuPTS.

Original: We may relate the result to the double descent characteristic previously observed for other linear estimators for fixed m and varying ˆ d
Revised: We may relate the result to the double descent characteristic previously observed for other linear estimators for fixed m and varying ˆ d

Original: (Loog et al., 2020).
Revised: (Loog et al., 2020).

Original: After a phase transition around ˆ d = m LuPTS’s variance reduces for a second time when it becomes equivalent to the classical learner (see Figure 2).
Revised: After a phase transition around ˆ d = m LuPTS’s variance reduces for a second time when it becomes equivalent to the classical learner (see Figure 2).

Original: 3.2 Random feature maps for unknown representations
Revised: 3.2 Random feature maps for unknown representations

Original: When the true Φ is entirely unknown, as is often the case in practice, using a poor representation ˆΦ may yield biased results for both classical and privileged learners.
Revised: When the true Φ is entirely unknown, as is often the case in practice, using a poor representation ˆΦ may yield biased results for both classical and privileged learners.

Original: A common solution in nonlinear prediction is to use a universal kernel, such as the Gaussian-RBF kernel.
Revised: A common solution in nonlinear prediction is to use a universal kernel, such as the Gaussian-RBF kernel.

Original: These have dense reproducing-kernel Hilbert spaces which allow approximation of any continuous function.
Revised: These have dense reproducing-kernel Hilbert spaces which allow approximation of any continuous function.

Original: However, universal kernels also have positive-definite and thus invertible Gram matrices (Hofmann et al., 2008), which according to Proposition 1 eliminates any gain in sample efficiency of generalized LuPTS.
Revised: However, universal kernels also have positive-definite and thus invertible Gram matrices (Hofmann et al., 2008), which according to Proposition 1 eliminates any gain in sample efficiency of generalized LuPTS.

Original: Instead, we combine our algorithm with an approximation of universal kernels—random feature maps.
Revised: Instead, we combine our algorithm with an approximation of universal kernels—random feature maps.

Original: These methods project inputs x onto ˆ d features by a random linear map W ∈ R k × ˆ d , and a nonlinear element-wise activation function.
Revised: These methods project inputs x onto ˆ d features by a random linear map W ∈ R k × ˆ d , and a nonlinear element-wise activation function.

Original: By choosing ˆ d < m , we can benefit from the function approximation properties of universal kernels (see discussion below) and the variance reduction of generalized LuPTS.
Revised: By choosing ˆ d < m , we can benefit from the function approximation properties of universal kernels (see discussion below) and the variance reduction of generalized LuPTS.

Original: Popular random features include random Fourier features (RFF)
Revised: Popular random features include random Fourier features (RFF)

Original: (Rahimi and
Revised: (Rahimi and

Original: Recht, 2007) and random ReLU features (RRF) (Sun et al., 2018),
Revised: Recht, 2007) and random ReLU features (RRF) (Sun et al., 2018),

Original: [Equation]
Revised: [Equation]

Original: where f + ( z ) = max(0 , z ) is the rectifier (ReLU) function and γ > 0 is a bandwidth hyperparameter.
Revised: where f + ( z ) = max(0 , z ) is the rectifier (ReLU) function and γ > 0 is a bandwidth hyperparameter.

Original: For large enough numbers of random features and training samples, any continuous function h can be approximated up to arbitrary precision by a linear map ω applied to the random features, e.g., ˆ h ( x ) =
Revised: For large enough numbers of random features and training samples, any continuous function h can be approximated up to arbitrary precision by a linear map ω applied to the random features, e.g., ˆ h ( x ) =

Original: ω ⊤ ˆΦ γ RRF ( x ) , see Sun et al.
Revised: ω ⊤ ˆΦ γ RRF ( x ) , see Sun et al.

Original: ; Rudi and Rosasco (2017).
Revised: ; Rudi and Rosasco (2017).

Original: Applying the same argument to the step-wise estimators of LuPTS we can justify using random features in Algorithm 1 by the following observation: Under appropriate assumptions, we can construct a privileged learner using random feature maps which is a universally consistent estimator of h ( x 1 ) =
Revised: Applying the same argument to the step-wise estimators of LuPTS we can justify using random features in Algorithm 1 by the following observation: Under appropriate assumptions, we can construct a privileged learner using random feature maps which is a universally consistent estimator of h ( x 1 ) =

Original: E [ Y | x 1 ] .
Revised: E [ Y | x 1 ] .

Original: The precise construction deviates somewhat from Algorithm 1, but follows the same structure.
Revised: The precise construction deviates somewhat from Algorithm 1, but follows the same structure.

Original: We refer to
Revised: We refer to

Original: Appendix B for a precise statement.
Revised: Appendix B for a precise statement.

Original: As universal consistency describes the asymptotic behaviour in the limit of infinite samples and random features, this offers only limited insight into the benefits of privileged information in small sample settings, where performance will be a bias-variance trade-off.
Revised: As universal consistency describes the asymptotic behaviour in the limit of infinite samples and random features, this offers only limited insight into the benefits of privileged information in small sample settings, where performance will be a bias-variance trade-off.

Original: Variance reduction & bias amplification.
Revised: Variance reduction & bias amplification.

Original: Generalized LuPTS is only guaranteed lower variance compared to the classical estimator under Assumption 1, although our empirical results (Section 4) suggest this applies more widely.
Revised: Generalized LuPTS is only guaranteed lower variance compared to the classical estimator under Assumption 1, although our empirical results (Section 4) suggest this applies more widely.

Original: When ˆΦ is a bad approximation of Φ , generalized LuPTS may amplify bias, increasing with the number of privileged time points, compared to classical learning.
Revised: When ˆΦ is a bad approximation of Φ , generalized LuPTS may amplify bias, increasing with the number of privileged time points, compared to classical learning.

Original: We show this theoretically in Appendix C and also empirically in Appendix F.
Revised: We show this theoretically in Appendix C and also empirically in Appendix F.

Original: Whether generalized
Revised: Whether generalized

Original: LuPTS is still preferable to classical learning in terms of prediction risk appears to depend on the amount of bias that gets amplified.
Revised: LuPTS is still preferable to classical learning in terms of prediction risk appears to depend on the amount of bias that gets amplified.

Original: Our experiments imply the variance reduction mostly dominates when using random features, whereas this is not always the case for linear LuPTS.
Revised: Our experiments imply the variance reduction mostly dominates when using random features, whereas this is not always the case for linear LuPTS.

Original: The phenomenon of bias amplification is familiar from e.g., model-based and model-free reinforcement learning (Kober et al., 2013).
Revised: The phenomenon of bias amplification is familiar from e.g., model-based and model-free reinforcement learning (Kober et al., 2013).

Original: As the bias with random features may still be high for small sample settings, we next present privileged representation learning algorithms to trade off bias and variance more efficiently.
Revised: As the bias with random features may still be high for small sample settings, we next present privileged representation learning algorithms to trade off bias and variance more efficiently.

Original: 3.3 Privileged time-series representation learning
Revised: 3.3 Privileged time-series representation learning

Original: Up until now the representation ˆΦ was considered fixed, either because Φ was known up to a linear transform (explicitly or implicitly) or because of the use of random feature methods.
Revised: Up until now the representation ˆΦ was considered fixed, either because Φ was known up to a linear transform (explicitly or implicitly) or because of the use of random feature methods.

Original: Generalized
Revised: Generalized

Original: LuPTS (Algorithm 1) produces minimizers { ˆ A t } , ˆ β of the following objective for fixed ˆΦ ,
Revised: LuPTS (Algorithm 1) produces minimizers { ˆ A t } , ˆ β of the following objective for fixed ˆΦ ,

Original: [Equation]
Revised: [Equation]

Original: Objective (5) and the systems described by Assumption 1 lend themselves to methods which also learn the representation Φ in addition to the latent dynamics { A t , β } .
Revised: Objective (5) and the systems described by Assumption 1 lend themselves to methods which also learn the representation Φ in addition to the latent dynamics { A t , β } .

Original: Next, we present three algorithms which combine the ideas of generalized LuPTS with the expressiveness of deep representation learning.
Revised: Next, we present three algorithms which combine the ideas of generalized LuPTS with the expressiveness of deep representation learning.

Original: All learners use equivalent encoders to represent ˆΦ( · ) and linear layers to model the relations between the latent variables { Z t } and the outcome Y .
Revised: All learners use equivalent encoders to represent ˆΦ( · ) and linear layers to model the relations between the latent variables { Z t } and the outcome Y .

Original: The classical learner predicts the outcome linearly from ˆ Z 1 .
Revised: The classical learner predicts the outcome linearly from ˆ Z 1 .

Original: All architectures under consideration are visualized jointly in Figure 3a.
Revised: All architectures under consideration are visualized jointly in Figure 3a.

Original: ( "# ( $% ! )
Revised: ( "# ( $% ! )

Original: # or !$ ' !
Revised: # or !$ ' !

Original: $ % !
Revised: $ % !

Original: $ & &Φ !
Revised: $ & &Φ !

Original: ( "# ( $%
Revised: ( "# ( $%

Original: Privileged Rep. Learners ● SRL ● GRL ●● CRL
Revised: Privileged Rep. Learners ● SRL ● GRL ●● CRL

Original: Classic Rep. Learner &Φ t= t= t= t= t=
Revised: Classic Rep. Learner &Φ t= t= t= t= t=

Original: [Figure]
Revised: [Figure]

Original: SRL.
Revised: SRL.

Original: The first privileged representation learner directly optimizes objective (5), just like generalized
Revised: The first privileged representation learner directly optimizes objective (5), just like generalized

Original: LuPTS, but now also fitting the representation ˆΦ , parameterized by a neural network.
Revised: LuPTS, but now also fitting the representation ˆΦ , parameterized by a neural network.

Original: We refer to this model as stepwise representation learner (SRL).
Revised: We refer to this model as stepwise representation learner (SRL).

Original: As we will see in experiments, a drawback of this approach is that representations may favor predicting transitions ˆ z i,t → ˆ z i,t +1 with small error, while losing information relevant for the target outcome in the process.
Revised: As we will see in experiments, a drawback of this approach is that representations may favor predicting transitions ˆ z i,t → ˆ z i,t +1 with small error, while losing information relevant for the target outcome in the process.

Original: At test time, for a new input x 1 ,
Revised: At test time, for a new input x 1 ,

Original: SRL composes the stepwise dynamics to output ˆ h P ( x 1 ) =
Revised: SRL composes the stepwise dynamics to output ˆ h P ( x 1 ) =

Original: ˆ β ⊤ ˆ
Revised: ˆ β ⊤ ˆ

Original: A ⊤ T . . .
Revised: A ⊤ T . . .

Original: ˆ A ⊤ 1 ˆ Φ( x 1 ) .
Revised: ˆ A ⊤ 1 ˆ Φ( x 1 ) .

Original: CRL and GRL.
Revised: CRL and GRL.

Original: To make sure that the learned representation ˆΦ retains information about Y , we add linear outcome supervision to the representation ˆΦ( X t ) at each time step t .
Revised: To make sure that the learned representation ˆΦ retains information about Y , we add linear outcome supervision to the representation ˆΦ( X t ) at each time step t .

Original: Recall that, by
Revised: Recall that, by

Original: Assumption 1, the expected outcome is linear in the latent state at any time step.
Revised: Assumption 1, the expected outcome is linear in the latent state at any time step.

Original: We introduce a hyperparameter λ ∈ [0 , 1] to trade off the two types of losses and arrive at the combined representation learner (CRL).
Revised: We introduce a hyperparameter λ ∈ [0 , 1] to trade off the two types of losses and arrive at the combined representation learner (CRL).

Original: With α = (ˆΦ , { ˆ A t } , { ˆ θ t } ) the entire parameter vector, CRL minimizes the objective
Revised: With α = (ˆΦ , { ˆ A t } , { ˆ θ t } ) the entire parameter vector, CRL minimizes the objective

Original: L CRL (
Revised: L CRL (

Original: [Equation]
Revised: [Equation]

Original: We make test-time predictions using ˆ h P ( x 1 )
Revised: We make test-time predictions using ˆ h P ( x 1 )

Original: =
Revised: =

Original: ˆ θ ⊤ 1 ˆΦ( x 1 ) .
Revised: ˆ θ ⊤ 1 ˆΦ( x 1 ) .

Original: In experiments, we highlight the case λ = 1 where only outcome supervision is used as greedy representation learner (GRL).
Revised: In experiments, we highlight the case λ = 1 where only outcome supervision is used as greedy representation learner (GRL).

Original: For a precise definition of the GRL objective, see Appendix D. GRL is related to multi-view learning, in which prediction of the same quantity is made from multiple “views” (cf. time points)
Revised: For a precise definition of the GRL objective, see Appendix D. GRL is related to multi-view learning, in which prediction of the same quantity is made from multiple “views” (cf. time points)

Original: (Zhao et al., 2017).
Revised: (Zhao et al., 2017).

Original:  Experiments
Revised:  Experiments

Original: We compare classical learning to variants of generalized LuPTS (Algorithm 1) and the privileged representation learners of Section 3.3 on two synthetic and two real-world time-series data sets.
Revised: We compare classical learning to variants of generalized LuPTS (Algorithm 1) and the privileged representation learners of Section 3.3 on two synthetic and two real-world time-series data sets.

Original: We (i) verify our theoretical findings by analyzing the sample efficiency and bias-variance characteristics of the given algorithms; (ii) demonstrate that generalized LuPTS with random features succeeds in settings where linear LuPTS suffers from large bias; (iii) point out that privileged representation learners offer even greater sample efficiency in practice and (iv) study how well these algorithms recover the true latent variables { Z t } and how this relates to predictive accuracy.
Revised: We (i) verify our theoretical findings by analyzing the sample efficiency and bias-variance characteristics of the given algorithms; (ii) demonstrate that generalized LuPTS with random features succeeds in settings where linear LuPTS suffers from large bias; (iii) point out that privileged representation learners offer even greater sample efficiency in practice and (iv) study how well these algorithms recover the true latent variables { Z t } and how this relates to predictive accuracy.

Original: Experimental setup.
Revised: Experimental setup.

Original: We report the mean coefficient of determination ( R 2 ), proportional to the squared-error risk R , for varying sample sizes, sequence lengths and prediction horizons.
Revised: We report the mean coefficient of determination ( R 2 ), proportional to the squared-error risk R , for varying sample sizes, sequence lengths and prediction horizons.

Original: Experiments are repeated and averaged over different random seeds.
Revised: Experiments are repeated and averaged over different random seeds.

Original: In each repetition, a given model performs hyperparameter tuning on the training data using random search and five-fold cross-validation before being re-trained on all training data.
Revised: In each repetition, a given model performs hyperparameter tuning on the training data using random search and five-fold cross-validation before being re-trained on all training data.

Original: The test set size is 1000 samples for synthetic data and 20% of all data available for real-world data sets.
Revised: The test set size is 1000 samples for synthetic data and 20% of all data available for real-world data sets.

Original: We consider six privileged learners of two groups.
Revised: We consider six privileged learners of two groups.

Original: The first group comprises generalized LuPTS with the linear kernel (LuPTS) and the two random feature maps shown in Section 3.2: Random Fourier features (Fourier RF) and random ReLU features (ReLU RF).
Revised: The first group comprises generalized LuPTS with the linear kernel (LuPTS) and the two random feature maps shown in Section 3.2: Random Fourier features (Fourier RF) and random ReLU features (ReLU RF).

Original: The classical learners for this group are OLS estimators used with the same kernel or feature map.
Revised: The classical learners for this group are OLS estimators used with the same kernel or feature map.

Original: The second group consists of the representation learners SRL, CRL and GRL.
Revised: The second group consists of the representation learners SRL, CRL and GRL.

Original: For tabular data, their encoder is a multi-layer perceptron with three hidden layers of 25 neurons each.
Revised: For tabular data, their encoder is a multi-layer perceptron with three hidden layers of 25 neurons each.

Original: For the image data they use LeNet-5 (LeCun et al., 1989).
Revised: For the image data they use LeNet-5 (LeCun et al., 1989).

Original: The classical learner (Classic Rep.) uses the same encoder with a linear output layer.
Revised: The classical learner (Classic Rep.) uses the same encoder with a linear output layer.

Original: The results presented were found to be robust to small changes in training
Revised: The results presented were found to be robust to small changes in training

Original: [Figure]
Revised: [Figure]

Original: parameters such as learning rate.
Revised: parameters such as learning rate.

Original: For details on the training process we refer to Appendix E. All experiments required less than 3000 GPU-h to complete using NVIDIA Tesla T4 GPUs.
Revised: For details on the training process we refer to Appendix E. All experiments required less than 3000 GPU-h to complete using NVIDIA Tesla T4 GPUs.

Original: Data sets.
Revised: Data sets.

Original: We briefly describe evaluation data sets and refer to Appendix E for further details.
Revised: We briefly describe evaluation data sets and refer to Appendix E for further details.

Original: First, we create two synthetic data sets in which latent states and outcomes are generated from linearGaussian systems as in Assumption 1.
Revised: First, we create two synthetic data sets in which latent states and outcomes are generated from linearGaussian systems as in Assumption 1.

Original: To produce the observations { X t } we use a deterministic nonlinear function Ψ : Z −→ X .
Revised: To produce the observations { X t } we use a deterministic nonlinear function Ψ : Z −→ X .

Original: In the first synthetic data set, which we call Square-Sign , the nonlinear transformation Ψ : R d −→ R 2 d maps each latent feature Z ( t,k ) to a two dimensional vector such that
Revised: In the first synthetic data set, which we call Square-Sign , the nonlinear transformation Ψ : R d −→ R 2 d maps each latent feature Z ( t,k ) to a two dimensional vector such that

Original: [Equation]
Revised: [Equation]

Original: The second synthetic data set uses the same latent linear system with Z t ∈ R 2 and produces square images (28 × 28 pixels) as observations.
Revised: The second synthetic data set uses the same latent linear system with Z t ∈ R 2 and produces square images (28 × 28 pixels) as observations.

Original: As the images are reminiscent of clocks we refer to this data set as Clocks-LGS .
Revised: As the images are reminiscent of clocks we refer to this data set as Clocks-LGS .

Original: Example sequences of these observations are presented in Figure 3b.
Revised: Example sequences of these observations are presented in Figure 3b.

Original: The angle, size and fill of the two clock hands encode the value of the corresponding latent variable.
Revised: The angle, size and fill of the two clock hands encode the value of the corresponding latent variable.

Original: The outcome Y is a linear function of Z T with q = 1 .
Revised: The outcome Y is a linear function of Z T with q = 1 .

Original: For Square-Sign , q = 3 .
Revised: For Square-Sign , q = 3 .

Original: The Metro Interstate traffic volume data set ( Traffic ) (Hogue, 2012) contains hourly records of the traffic volume on the interstate 94 between Minneapolis and St. Paul, MN.
Revised: The Metro Interstate traffic volume data set ( Traffic ) (Hogue, 2012) contains hourly records of the traffic volume on the interstate 94 between Minneapolis and St. Paul, MN.

Original: In addition, the data contains weather features and a holiday indication.
Revised: In addition, the data contains weather features and a holiday indication.

Original: We predict the traffic volume for a fixed time horizon given the present observations.
Revised: We predict the traffic volume for a fixed time horizon given the present observations.

Original: Privileged information is observed every four hours.
Revised: Privileged information is observed every four hours.

Original: We also predict the progression of Alzheimer’s disease (AD) as measured by the outcome of the
Revised: We also predict the progression of Alzheimer’s disease (AD) as measured by the outcome of the

Original: Mini Mental State Examination (MMSE) (Galea and Woodward, 2005).
Revised: Mini Mental State Examination (MMSE) (Galea and Woodward, 2005).

Original: The anonymized data were obtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004).
Revised: The anonymized data were obtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004).

Original: The outcome of interest is the MMSE score 48 months after the first examination.
Revised: The outcome of interest is the MMSE score 48 months after the first examination.

Original: Privileged information are the measurements at 12, 24 and 36 months.
Revised: Privileged information are the measurements at 12, 24 and 36 months.

Original: In addition we tested our algorithms on the PM 2.
Revised: In addition we tested our algorithms on the PM 2.

Original: data set (Liang et al., 2016), where we predict the air quality in five Chinese cities (see Appendix F).
Revised: data set (Liang et al., 2016), where we predict the air quality in five Chinese cities (see Appendix F).

Original: 4.1 Sample efficiency, bias and variance
Revised: 4.1 Sample efficiency, bias and variance

Original: The main goal of our work is to improve learning efficiency by incorporating privileged time-series information.
Revised: The main goal of our work is to improve learning efficiency by incorporating privileged time-series information.

Original: Across  all prediction tasks and sample sizes, nonlinear variants of generalized LuPTS outperform their classical counterpart in terms of sample efficiency as can be seen in Figure 4.
Revised: Across almost all prediction tasks and sample sizes, nonlinear variants of generalized LuPTS outperform their classical counterpart in terms of sample efficiency as can be seen in Figure 4.

Original: On the

Original:  Traffic prediction task, LuPTS ReLU RF outperforms linear LuPTS as the former appears to exhibit less bias.
Revised: On the Traffic prediction task, LuPTS ReLU RF outperforms linear LuPTS as the former appears to exhibit less bias.

Original: On the synthetic data of Square-Sign , linear models reach their best accuracy quickly, while they are limited by their lack of expressiveness.
Revised: On the synthetic data of Square-Sign , linear models reach their best accuracy quickly, while they are limited by their lack of expressiveness.

Original: Generalized LuPTS amplifies this bias, making linear LuPTS worse than OLS.
Revised: Generalized LuPTS amplifies this bias, making linear LuPTS worse than OLS.

Original: Random feature methods attain higher accuracy here but are generally less sample efficient.
Revised: Random feature methods attain higher accuracy here but are generally less sample efficient.

Original: Generalized LuPTS combined with random features manages to decrease this gap significantly.
Revised: Generalized LuPTS combined with random features manages to decrease this gap significantly.

Original: On ADNI , we don’t see a benefit of using nonlinear models in general.
Revised: On ADNI , we don’t see a benefit of using nonlinear models in general.

Original: We make similar observations on the PM 2.5 air quality data set, see Appendix F.
Revised: We make similar observations on the PM 2.5 air quality data set, see Appendix F.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: The representation learners proposed in Section 3.3 are evaluated on the same data sets and on the image prediction task Clocks-LGS .
Revised: The representation learners proposed in Section 3.3 are evaluated on the same data sets and on the image prediction task Clocks-LGS .

Original: Example results are displayed in Figure 5.
Revised: Example results are displayed in Figure 5.

Original: These demonstrate that directly transferring the LuPTS objective to neural networks in the form of SRL results in subpar performance.
Revised: These demonstrate that directly transferring the LuPTS objective to neural networks in the form of SRL results in subpar performance.

Original: SRL does not seem to have a strong enough incentive to learn representations which accurately predict the outcome.
Revised: SRL does not seem to have a strong enough incentive to learn representations which accurately predict the outcome.

Original: Generalized distillation for privileged time series as suggested by
Revised: Generalized distillation for privileged time series as suggested by

Original: Hayashi et al.
Revised: Hayashi et al.

Original: (2019), does not improve upon classical learning in our tasks.
Revised: (2019), does not improve upon classical learning in our tasks.

Original: On all experiments displayed in Figure 5, CRL and GRL outperform the classical learner.
Revised: On all experiments displayed in Figure 5, CRL and GRL outperform the classical learner.

Original: The predictive accuracy of these models is similar on most tasks, as may be explained by the fact that CRL may reduce to GRL when choosing λ = 1 in objective 6.
Revised: The predictive accuracy of these models is similar on most tasks, as may be explained by the fact that CRL may reduce to GRL when choosing λ = 1 in objective 6.

Original: Noticeably, the general observation of GRL and CRL being more sample efficient than the classic model, neither appears to depend on the neural architecture used for the encoder, nor does the modality of the data play an important role, as the image prediction task Clocks-LGS (Figure 5c) demonstrates.
Revised: Noticeably, the general observation of GRL and CRL being more sample efficient than the classic model, neither appears to depend on the neural architecture used for the encoder, nor does the modality of the data play an important role, as the image prediction task Clocks-LGS (Figure 5c) demonstrates.

Original: For additional empirical results, we refer to Appendix F.
Revised: For additional empirical results, we refer to Appendix F.

Original: To analyze the bias and variance characteristics of our algorithms, we can estimate the expected squared prediction bias , E X 

Revised: To analyze the bias and variance characteristics of our algorithms, we can estimate the expected squared prediction bias , E X

Original: [( E D [ A ( D )]( X 1 ) − E
Revised: [( E D [ A ( D )]( X 1 ) − E

Original: [ Y | X 1 ]) 2 ] , by computing E Y
Revised: [ Y | X 1 ]) 2 ] , by computing E Y

Original: [ Y | X 1 ] in synthetic
Revised: [ Y | X 1 ] in synthetic

Original: DGPs in addition to the variance of the different estimators.
Revised: DGPs in addition to the variance of the different estimators.

Original: Figure 6 depicts bias and variance for all models on the Square-Sign data.
Revised: Figure 6 depicts bias and variance for all models on the Square-Sign data.

Original: On the left panel, all variants of generalized LuPTS exhibit lower variance than classical learning, despite being biased.
Revised: On the left panel, all variants of generalized LuPTS exhibit lower variance than classical learning, despite being biased.

Original: This holds generally: Across all experiments, we never encountered an example where the use of privileged information has not resulted in lower variance compared to classical learning.
Revised: This holds generally: Across all experiments, we never encountered an example where the use of privileged information has not resulted in lower variance compared to classical learning.

Original: On the contrary, the privileged learners suffer higher bias than the comparable classical learners because of the bias compounding over the individual prediction steps as shown in Appendix C. For the random feature variants however, the bias decreases with the number of samples.
Revised: On the contrary, the privileged learners suffer higher bias than the comparable classical learners because of the bias compounding over the individual prediction steps as shown in Appendix C. For the random feature variants however, the bias decreases with the number of samples.

Original: The representation learners display similar characteristics on the right panel of Figure 6.
Revised: The representation learners display similar characteristics on the right panel of Figure 6.

Original: Learning the transitions between latent variables Z t appears to be associated with low variance and high bias as demonstrated by the results of SRL.
Revised: Learning the transitions between latent variables Z t appears to be associated with low variance and high bias as demonstrated by the results of SRL.

Original: GRL however, which does not model these transitions, exhibits the lowest bias and the largest variance of all privileged learners.
Revised: GRL however, which does not model these transitions, exhibits the lowest bias and the largest variance of all privileged learners.

Original: As CRL is able to trade off between these two objectives, the estimates for its variance and bias lie in between the corresponding values of the other two privileged learners.
Revised: As CRL is able to trade off between these two objectives, the estimates for its variance and bias lie in between the corresponding values of the other two privileged learners.

Original: [Figure]
Revised: [Figure]

Original: 4.2 Latent variable recovery
Revised: 4.2 Latent variable recovery

Original: Under Assumption 1, it is sufficient to identify the representation function Φ up to a linear transform
Revised: Under Assumption 1, it is sufficient to identify the representation function Φ up to a linear transform

Original: B to have provable gains from privileged information over a classical learner.
Revised: B to have provable gains from privileged information over a classical learner.

Original: In synthetic data, we can assess to what extent a representation ˆΦ with this property has been found.
Revised: In synthetic data, we can assess to what extent a representation ˆΦ with this property has been found.

Original: As a proxy for the existence of such a transform, we can compute a measure of correlation between ˆΦ( X ) and Φ( X ) , such as the Canonical Correlation Analysis (CCA) (Hotelling, 1936).
Revised: As a proxy for the existence of such a transform, we can compute a measure of correlation between ˆΦ( X ) and Φ( X ) , such as the Canonical Correlation Analysis (CCA) (Hotelling, 1936).

Original: Raghu et al.
Revised: Raghu et al.

Original: (2017) introduced a modified version called Singular Vector Canonical Correlation Analysis (SVCCA) to compute correlations when dealing with noisy dimensions in neural network representations.
Revised: (2017) introduced a modified version called Singular Vector Canonical Correlation Analysis (SVCCA) to compute correlations when dealing with noisy dimensions in neural network representations.

Original: The mean SVCCA coefficients ¯ ρ and predictive accuracy of the representation learners are visualized in Figure 7a.
Revised: The mean SVCCA coefficients ¯ ρ and predictive accuracy of the representation learners are visualized in Figure 7a.

Original: One notices that GRL and CRL produce higher correlation coefficients than the classical learner while also predicting the outcome on the Clocks-LGS task more accurately.
Revised: One notices that GRL and CRL produce higher correlation coefficients than the classical learner while also predicting the outcome on the Clocks-LGS task more accurately.

Original: Further comparing the representations learned by privileged and classical models, Figure 7b shows a visual example of the improved latent recovery of CRL on the same task.
Revised: Further comparing the representations learned by privileged and classical models, Figure 7b shows a visual example of the improved latent recovery of CRL on the same task.

Original: Concluding, the use of privileged time-series information does not only increase the sample efficiency of existing algorithms but can also aid the recovery of latent variables in latent dynamical systems.
Revised: Concluding, the use of privileged time-series information does not only increase the sample efficiency of existing algorithms but can also aid the recovery of latent variables in latent dynamical systems.

Original:  Related work
Revised:  Related work

Original: Existing analyses for learning with privileged information guarantee improvements in  sample efficiency under strong assumptions  but are insufficient to establish a clear preference for LuPI learners in small-sample settings, as argued in the Introduction.
Revised: Existing analyses for learning with privileged information guarantee improvements in asymptotic sample efficiency under strong assumptions (Vapnik and Vashist, 2009) but are insufficient to establish a clear preference for LuPI learners for a fixed sample size.

Revised: For example, Pechyony and Vapnik (2010) showed that for a specialized problem construction, empirical risk minimization (ERM) using privileged information can achieve fast learning rates, O (1 /n ) , while classical (non-privileged) ERM can only achieve slow rates, O (1 / √ n ) .

Original: We are not aware of results tight enough to establish a lower bound on the risk of a classical learner larger than an upper bound for a PI learner.
Revised: We are not aware of any generalization theory tight enough to establish a lower bound on the risk of a classical learner larger than an upper bound for a PI learner.

Original: Additionally, our problem, see especially (5)–(6), is  related to multi-task (representation) learning (Maurer et al., 2016).
Revised: Our problem is also related to multi-task (representation) learning (Maurer et al., 2016), see especially (5)–(6).

Original: However, our goal is different in that only a single task is of interest after learning.
Revised: However, our goal is different in that only a single task is of interest after learning.

Original: In estimation of causal effects, learning from surrogate outcomes (Prentice, 1989) has been proposed as a way to increase sample efficiency.
Revised: In estimation of causal effects, learning from surrogate outcomes (Prentice, 1989) has been proposed as a way to increase sample efficiency.

Original: Surrogates are variables related to the outcome which may be available even when the outcome is not (Athey et al., 2019).
Revised: Surrogates are variables related to the outcome which may be available even when the outcome is not (Athey et al., 2019).

Original: We can view these as privileged information.
Revised: We can view these as privileged information.

Original: While the problem shares structure with ours, the goal is to compensate for missing outcomes and analytical results give no guarantees for improved efficiency when both surrogates outcomes are always observed (Chen et al., 2008;
Revised: While the problem shares structure with ours, the goal is to compensate for missing outcomes and analytical results give no guarantees for improved efficiency when both surrogates outcomes are always observed (Chen et al., 2008;

Original: Kallus and Mao, 2020).
Revised: Kallus and Mao, 2020).

Original: Theorem 1 relies on the representation function Φ being identified up to a linear transformation. Nonlinear independent component analysis (ICA) aims to solve precisely this problem (Hyvärinen and Pajunen, 1999). It has been shown that the nonstationarity of time series provides the conditions for identifiability through time-contrastive learning (Hyvarinen and Morioka, 2016).
Revised: Guo and Perkovi´c (2022) showed that, in the context of a linear-Gaussian system on a directed acyclic graph, a recursive least-squares estimator is the asymptotically most efficient estimator of causal effects using only the sample covariance.

Original: These results make for a potentially interesting connection between learning using privileged time series and theidentifiability of nonlinear ICA. We hope to explore this connection in future work.
Revised: In the linear case on a path graph, their estimator coincides with ours. However, no analysis is provided for the nonlinear case or for fixed sample sizes.

Original:  Discussion
Revised:  Discussion

Original: We have presented learning algorithms for predicting nonlinear outcomes by utilizing time-series privileged information during training.
Revised: We have presented learning algorithms for predicting nonlinear outcomes by utilizing time-series privileged information during training.

Original: We prove that our estimator is preferable to classical learning when data is generated from a latent-variable dynamical system with partially known components.
Revised: We prove that our estimator is preferable to classical learning when data is generated from a latent-variable dynamical system with partially known components.

Original: The proof holds for the case when the latent dynamical system is recovered by the representation function up to a linear transform, assuming that a left inverse of the true observation generating function exists.
Revised: The proof holds for the case when the latent dynamical system is recovered by the representation function up to a linear transform, assuming that a left inverse of the true observation generating function exists.

Original: However, this assumption does not appear to be necessary, as our empirical results demonstrate that privileged learning is preferable to classical learning even when these conditions cannot be guaranteed.
Revised: However, this assumption does not appear to be necessary, as our empirical results demonstrate that privileged learning is preferable to classical learning even when these conditions cannot be guaranteed.

Original: Consequently, a more general theoretical result where less is known about the latent system seems attainable.
Revised: Consequently, a more general theoretical result where less is known about the latent system seems attainable.

Original: For example, one might consider a case in which only a few independent components of the latent variables are recovered by the representation used by the learning algorithm, while other components are treated as noise.
Revised: For example, one might consider a case in which only a few independent components of the latent variables are recovered by the representation used by the learning algorithm, while other components are treated as noise.

Original: When the latent dynamical system is entirely unknown, we create practical estimators using random feature embeddings which outperform the corresponding classical learner across experiments.
Revised: When the latent dynamical system is entirely unknown, we create practical estimators using random feature embeddings which outperform the corresponding classical learner across experiments.

Original: We show that a universally consistent learner can be constructed based on this idea, with slightly different form.
Revised: We show that a universally consistent learner can be constructed based on this idea, with slightly different form.

Original: As a further alternative, we propose representation learning methods of related form using neural networks and demonstrate the empirical benefits also of this estimator over classical learning.
Revised: As a further alternative, we propose representation learning methods of related form using neural networks and demonstrate the empirical benefits also of this estimator over classical learning.

Original: In experiments, we analyze how the gap in risk between privileged and classical learning changes for different prediction horizons, as displayed in Figure 16 in Appendix F.
Revised: In experiments, we analyze how the gap in risk between privileged and classical learning changes for different prediction horizons, as displayed in Figure 16 in Appendix F.

Original: The results suggest that the risk advantage of privileged learners grows with the sequence length of the prediction task, despite the fact that the task becomes more difficult at the same time.
Revised: The results suggest that the risk advantage of privileged learners grows with the sequence length of the prediction task, despite the fact that the task becomes more difficult at the same time.

Original: Our work focuses on the setting where data is observed as a time series.
Revised: Our work focuses on the setting where data is observed as a time series.

Original: This setting is chosen for its simple causal structure given by (latent) linear-Gaussian systems and because it can be motivated from many different applications.
Revised: This setting is chosen for its simple causal structure given by (latent) linear-Gaussian systems and because it can be motivated from many different applications.

Original: However, the ideas presented are not specifically tied to time and also apply in the case when all variables are observed simultaneously as long as the causal structure remains sequential.
Revised: However, the ideas presented are not specifically tied to time and also apply in the case when all variables are observed simultaneously as long as the causal structure remains sequential.

Original: Moreover, we believe that the theory presented here is not limited to sequential settings and generalizes to other causal structures, in particular directed acyclic graphs that connect the baseline covariates to the outcome.
Revised: Moreover, we believe that the theory presented here is not limited to sequential settings and generalizes to other causal structures, in particular directed acyclic graphs that connect the baseline covariates to the outcome.

Original: In either case, one might only have access to the baseline variable at test time.
Revised: In either case, one might only have access to the baseline variable at test time.

Original: For example, the timely collection of data for all covariates at test time might be very expensive or even impossible.
Revised: For example, the timely collection of data for all covariates at test time might be very expensive or even impossible.

Original: As pointed out before, Theorem 1 requires the recovery of the latent variables up to a linear transformation.
Revised: As pointed out before, Theorem 1 requires the recovery of the latent variables up to a linear transformation.

Original: Nonlinear independent component analysis (ICA) (Hyvarinen and Morioka, 2016) aims to solve precisely this problem and has been applied to time series via time-contrastive learning.
Revised: Nonlinear independent component analysis (ICA) (Hyvarinen and Morioka, 2016) aims to solve precisely this problem and has been applied to time series via time-contrastive learning.

Original: This makes for an interesting connection between learning using privileged information and nonlinear ICA, as the experiments of Section 4.2 suggest that privileged time-series information aids the recovery of latent variables.
Revised: This makes for an interesting connection between learning using privileged information and nonlinear ICA, as the experiments of Section 4.2 suggest that privileged time-series information aids the recovery of latent variables.

Original: Other remaining challenges include providing risk guarantees for learning with biased representations (including deep neural networks), with regularized estimators, and for more general data generating processes with weaker structural assumptions.
Revised: Other remaining challenges include providing risk guarantees for learning with biased representations (including deep neural networks), with regularized estimators, and for more general data generating processes with weaker structural assumptions.

Original: We are hopeful that the utility demonstrated in this work will inspire future research to overcome these limitations.
Revised: We are hopeful that the utility demonstrated in this work will inspire future research to overcome these limitations.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank Anton Matsson and Rickard Karlsson for insightful feedback and the
Revised: We would like to thank Anton Matsson and Rickard Karlsson for insightful feedback and the

Original: Alzheimer’s Neuroimaging Initiative (ADNI) for collecting and providing the data on Alzheimer’s disease used in this project.
Revised: Alzheimer’s Neuroimaging Initiative (ADNI) for collecting and providing the data on Alzheimer’s disease used in this project.

Original: The present work was funded in part by the Wallenberg AI, Autonomous
Revised: The present work was funded in part by the Wallenberg AI, Autonomous

Original: Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Revised: Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] The introduction refers to the section where each contribution is made.
Revised: [Yes] The introduction refers to the section where each contribution is made.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See e
Revised: [Yes] See e

Original: g., the discussion in Section 6 and on the applicability of Theorem 1 in Section 3.1 and 3.2.
Revised: g., the discussion in Section 6 and on the applicability of Theorem 1 in Section 3.1 and 3.2.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: We have not identified any potential negative societal impact directly related to our work.
Revised: We have not identified any potential negative societal impact directly related to our work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption 1 and Theorem 1.
Revised: [Yes] See Assumption 1 and Theorem 1.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Yes, see the Appendix. 3.
Revised: [Yes] Yes, see the Appendix. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] A URL to a code repository is included in the paper.
Revised: [Yes] A URL to a code repository is included in the paper.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 4 and Appendix E (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Section 4 and Appendix E (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Each result figure includes error regions.
Revised: [Yes] Each result figure includes error regions.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section4.
Revised: [Yes] See Section4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix E. (b) Did you mention the license of the assets?
Revised: [Yes] See Appendix E. (b) Did you mention the license of the assets?

Original: [Yes] See Appendix E. (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Appendix E. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] Only original code.
Revised: [Yes] Only original code.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] ADNI data is anonymized.
Revised: [Yes] ADNI data is anonymized.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Appendix A Proof of Theorem
Revised: Appendix A Proof of Theorem

Original: Our proof requires an additional technical assumption: that the matrix of true latent states
Revised: Our proof requires an additional technical assumption: that the matrix of true latent states

Original: Z t =
Revised: Z t =

Original: [Φ( x 1 ,t ) , ..., Φ( x m,t )] ⊤ , for all t , for a random data set D has independent columns with probability 1.
Revised: [Φ( x 1 ,t ) , ..., Φ( x m,t )] ⊤ , for all t , for a random data set D has independent columns with probability 1.

Original: This implies that rank ( Z t ) =
Revised: This implies that rank ( Z t ) =

Original: d and m ≥ d .
Revised: d and m ≥ d .

Original: We consider this a minor restriction since it would only be violated if either a) two or more components of Z t were perfectly correlated—in this case, a smaller system with same distributions over observations could always be constructed—or b) if we observe fewer samples than necesary to determine the system ( m < d ).
Revised: We consider this a minor restriction since it would only be violated if either a) two or more components of Z t were perfectly correlated—in this case, a smaller system with same distributions over observations could always be constructed—or b) if we observe fewer samples than necesary to determine the system ( m < d ).

Original: Note that this does not require that the dimension ˆ d of the estimated representation ˆΦ is smaller than m .
Revised: Note that this does not require that the dimension ˆ d of the estimated representation ˆΦ is smaller than m .

Original: We begin by proving that both classical and LuPTS estimators are invariant to a particular form of linear transformation of the representation ˆΦ .
Revised: We begin by proving that both classical and LuPTS estimators are invariant to a particular form of linear transformation of the representation ˆΦ .

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume we have a latent linear Gaussian system as defined in Assumption 1 such that for a data set D of m samples, the matrix of true latent states Z t =
Revised: Assume we have a latent linear Gaussian system as defined in Assumption 1 such that for a data set D of m samples, the matrix of true latent states Z t =

Original: [Φ( X 1 ,t ); ... ; Φ( X m,t )] has linearly independent columns with probability 1.
Revised: [Φ( X 1 ,t ); ... ; Φ( X m,t )] has linearly independent columns with probability 1.

Original: Let A Φ P be the LuPTS algorithm using the system’s true map Φ( · ) with Φ(Ψ( z ))
Revised: Let A Φ P be the LuPTS algorithm using the system’s true map Φ( · ) with Φ(Ψ( z ))

Original: =
Revised: =

Original: z ∀ z ∈ Z .
Revised: z ∀ z ∈ Z .

Original: Let A ˆΦ P be the same algorithm using a different map ˆ Φ( · ) .
Revised: Let A ˆΦ P be the same algorithm using a different map ˆ Φ( · ) .

Original: We assume that ∃ B : ˆΦ( x ) =
Revised: We assume that ∃ B : ˆΦ( x ) =

Original: B Φ( x ) ∀ x ∈ X .
Revised: B Φ( x ) ∀ x ∈ X .

Original: Analogously, we denote the classical learners
Revised: Analogously, we denote the classical learners

Original: A ˆΦ C and
Revised: A ˆΦ C and

Original: A Φ C .
Revised: A Φ C .

Original: If B ∈ R ˆ d × d has linearly independent columns we have
Revised: If B ∈ R ˆ d × d has linearly independent columns we have

Original: [Equation]
Revised: [Equation]

Original: and ˆ
Revised: and ˆ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let Z t ∈ R m × d be made up of the rows Z t ( i, :) =
Revised: Let Z t ∈ R m × d be made up of the rows Z t ( i, :) =

Original: Φ( X t ( i, :) ) when X t ∈ R m × k is the design matrix belonging to data set D .
Revised: Φ( X t ( i, :) ) when X t ∈ R m × k is the design matrix belonging to data set D .

Original: In the same fashion we define ˆ Z t ∈ R m × ˆ d using the map ˆΦ instead.
Revised: In the same fashion we define ˆ Z t ∈ R m × ˆ d using the map ˆΦ instead.

Original: By assumption, Z t and B ∈ R ˆ d × d have independent columns such that B † B
Revised: By assumption, Z t and B ∈ R ˆ d × d have independent columns such that B † B

Original: = I .
Revised: = I .

Original: These assumptions are used for matrix identities involving the Moore-Penrose inverse below.
Revised: These assumptions are used for matrix identities involving the Moore-Penrose inverse below.

Original: We compute the prediction on a new test point x for the classical learner:
Revised: We compute the prediction on a new test point x for the classical learner:

Original: [Equation]
Revised: [Equation]

Original: The arguments for the privileged learners are analogous:
Revised: The arguments for the privileged learners are analogous:

Original: [Equation]
Revised: [Equation]

Original: ⊤
Revised: ⊤

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We consider the generalized LuPTS estimator ˆ h ˆΦ P ( · ) treating different cases for the number of samples m and latent state dimension d in turn.
Revised: We consider the generalized LuPTS estimator ˆ h ˆΦ P ( · ) treating different cases for the number of samples m and latent state dimension d in turn.

Original: By the added technical assumption, that the true latent state Z t =
Revised: By the added technical assumption, that the true latent state Z t =

Original: [Φ( x 1 ,t ) , ..., Φ( x m,t )] ⊤ has rank d for all t with probability 1, and the assumption that ˆΦ is linearly close to Φ , by a matrix B ∈ R ˆ d × d of rank d such that ˆ Z t =
Revised: [Φ( x 1 ,t ) , ..., Φ( x m,t )] ⊤ has rank d for all t with probability 1, and the assumption that ˆΦ is linearly close to Φ , by a matrix B ∈ R ˆ d × d of rank d such that ˆ Z t =

Original: Z t B ⊤ , we get that rank (ˆ Z t )
Revised: Z t B ⊤ , we get that rank (ˆ Z t )

Original: =
Revised: =

Original: d for all t .
Revised: d for all t .

Original: This also implies d ≤ ˆ d .
Revised: This also implies d ≤ ˆ d .

Original: (i) m
Revised: (i) m

Original: =
Revised: =

Original: d : In this case, the Gram matrix K t
Revised: d : In this case, the Gram matrix K t

Original: = ˆ Z t ˆ Z ⊤ t has full rank and thus is invertible for all t .
Revised: = ˆ Z t ˆ Z ⊤ t has full rank and thus is invertible for all t .

Original: By
Revised: By

Original: Proposition 1, LuPTS coincides with the classical learner.
Revised: Proposition 1, LuPTS coincides with the classical learner.

Original: Hence, E ˆ h P ,X 

Revised: Hence, E ˆ h P ,X

Original: [Var D (ˆ h C ( X 1 ) | ˆ h P )]
Revised: [Var D (ˆ h C ( X 1 ) | ˆ h P )]

Original: =and R ( A P ) =
Revised: =and R ( A P ) =

Original: R ( A C ) .
Revised: R ( A C ) .

Original: (ii) m <
Revised: (ii) m <

Original: d : In this case, there does not exists a linearly close
Revised: d : In this case, there does not exists a linearly close

Original: as
Revised: as

Original: defined above, representation ˆΦ to Φ since the rank of ˆ Z t must be smaller than d .
Revised: defined above, representation ˆΦ to Φ since the rank of ˆ Z t must be smaller than d .

Original: This contradicts that rank
Revised: This contradicts that rank

Original: (ˆ Z t ) =
Revised: (ˆ Z t ) =

Original: d .
Revised: d .

Original: Independently, if the conditions of Proposition 1 hold, the same equivalence holds as in the case m = d .
Revised: Independently, if the conditions of Proposition 1 hold, the same equivalence holds as in the case m = d .

Original: (iii) m
Revised: (iii) m

Original: > d : In this case, the kernel Gram matrix K t
Revised: > d : In this case, the kernel Gram matrix K t

Original: = ˆ Z t ˆ Z ⊤ t has rank
Revised: = ˆ Z t ˆ Z ⊤ t has rank

Original: d
Revised: d

Original: <
Revised: <

Original: m
Revised: m

Original: and is
Revised: and is

Original: never invertible.
Revised: never invertible.

Original: Three sub-cases remain: a) When ˆ d = d , the matrix B is invertible and square, the covariance matrix ˆΣ t
Revised: Three sub-cases remain: a) When ˆ d = d , the matrix B is invertible and square, the covariance matrix ˆΣ t

Original: = ˆ Z ⊤ t
Revised: = ˆ Z ⊤ t

Original: ˆ Z t is invertible for all t and our estimator coincides with linear LuPTS (Karlsson et al., 2022) in the space implied by ˆΦ .
Revised: ˆ Z t is invertible for all t and our estimator coincides with linear LuPTS (Karlsson et al., 2022) in the space implied by ˆΦ .

Original: To see this, note that Lemma 1 implies that ˆ h ˆΦ P ( · ) makes the same predictions as a different generalized LuPTS estimator
Revised: To see this, note that Lemma 1 implies that ˆ h ˆΦ P ( · ) makes the same predictions as a different generalized LuPTS estimator

Original: ˆ h Φ P ( · ) using the true map Φ when the two representation functions are related through B , as defined in the Theorem statement.
Revised: ˆ h Φ P ( · ) using the true map Φ when the two representation functions are related through B , as defined in the Theorem statement.

Original: Consequently, we may analyze the latter estimator instead of the first.
Revised: Consequently, we may analyze the latter estimator instead of the first.

Original: It uses the parameter
Revised: It uses the parameter

Original: [Equation]
Revised: [Equation]

Original: We know by assumption that the covariance matrices Σ t =
Revised: We know by assumption that the covariance matrices Σ t =

Original: ( Z ⊤ t Z t ) ∈
Revised: ( Z ⊤ t Z t ) ∈

Original: R d × d have full rank for all t .
Revised: R d × d have full rank for all t .

Original: This implies that the Moore-Penrose pseudoinverse ( · ) † may be replaced by the regular matrix inverse ( · ) − 1 in the expression above, yielding
Revised: This implies that the Moore-Penrose pseudoinverse ( · ) † may be replaced by the regular matrix inverse ( · ) − 1 in the expression above, yielding

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to the LuPTS estimator of Karlsson et al.
Revised: which is equivalent to the LuPTS estimator of Karlsson et al.

Original: (2022) used on a linear-Gaussian system in the space Z rather than in X .
Revised: (2022) used on a linear-Gaussian system in the space Z rather than in X .

Original: In this case, Theorem 1 from Karlsson et al.
Revised: In this case, Theorem 1 from Karlsson et al.

Original: (2022) yields the desired result.
Revised: (2022) yields the desired result.

Original: b) If ˆ d > d , ˆΣ is not invertible but, due to Lemma 1, we can instead study a representation which is an appropriate linear transform B ∈ R ˆ d × d away from ˆ Z , and apply the Karlsson et al.
Revised: b) If ˆ d > d , ˆΣ is not invertible but, due to Lemma 1, we can instead study a representation which is an appropriate linear transform B ∈ R ˆ d × d away from ˆ Z , and apply the Karlsson et al.

Original: (2022) result as described for the case ˆ d = d .
Revised: (2022) result as described for the case ˆ d = d .

Original: Note that in this case B is non-square but has linearly independent columns as required.
Revised: Note that in this case B is non-square but has linearly independent columns as required.

Original: c)
Revised: c)

Original: If ˆ d < d , the assumed matrix B cannot exist with the stated conditions (the assumptions of Theorem 1 are not satisfied).
Revised: If ˆ d < d , the assumed matrix B cannot exist with the stated conditions (the assumptions of Theorem 1 are not satisfied).

Original: B Universality of random features
Revised: B Universality of random features

Original: A learning algorithm
Revised: A learning algorithm

Original: A is said to be universally consistent if, for any continuous function h , the output of A converges in probability to h .
Revised: A is said to be universally consistent if, for any continuous function h , the output of A converges in probability to h .

Original: That is, for a random dataset D m of m i.i.d.
Revised: That is, for a random dataset D m of m i.i.d.

Original: samples drawn from a distribution p , and any ϵ > 0 ,
Revised: samples drawn from a distribution p , and any ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Sun et al.
Revised: Sun et al.

Original: (2019) prove that (norm-bounded) linear regression applied to random ReLU features (RRF) is universally consistent.
Revised: (2019) prove that (norm-bounded) linear regression applied to random ReLU features (RRF) is universally consistent.

Original: Specifically, for any ϵ, δ > 0 , there is a finite number of random features ˆ d and samples m , such that the estimator
Revised: Specifically, for any ϵ, δ > 0 , there is a finite number of random features ˆ d and samples m , such that the estimator

Original: [Equation]
Revised: [Equation]

Original: achieves an error of at most ϵ with probability ≥ 1 − δ for univariate continuous functions of x .
Revised: achieves an error of at most ϵ with probability ≥ 1 − δ for univariate continuous functions of x .

Original: Universal consistency (as ˆ d → ∞ , m > ˆ d ) ) follows as a result.
Revised: Universal consistency (as ˆ d → ∞ , m > ˆ d ) ) follows as a result.

Original: We can apply the same idea to a version of generalized LuPTS by considering each parameter estimate of the latent dynamical system given ˆΦ , with norms restricted by R .
Revised: We can apply the same idea to a version of generalized LuPTS by considering each parameter estimate of the latent dynamical system given ˆΦ , with norms restricted by R .

Original: We drop the subscript RRF from ˆΦ moving forward, and continue to use random ReLU features.
Revised: We drop the subscript RRF from ˆΦ moving forward, and continue to use random ReLU features.

Original: For the final prediction step of Y , we let
Revised: For the final prediction step of Y , we let

Original: [Equation]
Revised: [Equation]

Original: Progressing recursively backward from t =
Revised: Progressing recursively backward from t =

Original: T , we let
Revised: T , we let

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˆΦ γ, ˆ d t are random features specific to t .
Revised: where ˆΦ γ, ˆ d t are random features specific to t .

Original: Since the target of each prediction at t + 1 is fixed with respect to the features used as input at t , the result from Sun et al.
Revised: Since the target of each prediction at t + 1 is fixed with respect to the features used as input at t , the result from Sun et al.

Original: (2019) can be used to give a learning guarantee for each step.
Revised: (2019) can be used to give a learning guarantee for each step.

Original: The construction differs from the standard generalized LuPTS formulation as ˆΦ is not shared between time steps, and so ˆ A t will be non-square in general.
Revised: The construction differs from the standard generalized LuPTS formulation as ˆΦ is not shared between time steps, and so ˆ A t will be non-square in general.

Original: We believe that this is merely a limitation of the proof technique and that the results hold for shared random features and square transitions.
Revised: We believe that this is merely a limitation of the proof technique and that the results hold for shared random features and square transitions.

Original: Let H ReLU denote the set of linear functions applied to ˆ d random ReLU features, with uniform random projection coefficients
Revised: Let H ReLU denote the set of linear functions applied to ˆ d random ReLU features, with uniform random projection coefficients

Original: ω j ∼ U ( S d ) , j = 1 , ..., ˆ d .
Revised: ω j ∼ U ( S d ) , j = 1 , ..., ˆ d .

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 (Follows from Proposition 5 in Sun et al.
Revised: Corollary 1 (Follows from Proposition 5 in Sun et al.

Original: Let Assumption 1 hold with noiseless transitions and outcomes, ϵ t = 0 for t = 2 , ..., T , and ϵ Y = 0 .
Revised: Let Assumption 1 hold with noiseless transitions and outcomes, ϵ t = 0 for t = 2 , ..., T , and ϵ Y = 0 .

Original: Define g ∗ t ( x t ) :
Revised: Define g ∗ t ( x t ) :

Original: = Ψ( A ⊤ t Φ( x t )) =
Revised: = Ψ( A ⊤ t Φ( x t )) =

Original: x t +1 and assume that for any fixed RRF representation ˆΦ γ, ˆ d t +1 , each component j = 1 , ..., ˆ d t + of the transition target satisfies ˆΦ γ, ˆ d t +1 ( g ∗ t ( · ))
Revised: x t +1 and assume that for any fixed RRF representation ˆΦ γ, ˆ d t +1 , each component j = 1 , ..., ˆ d t + of the transition target satisfies ˆΦ γ, ˆ d t +1 ( g ∗ t ( · ))

Original: ( j ) ∈ H ReLU , ˆ d t .
Revised: ( j ) ∈ H ReLU , ˆ d t .

Original: Let ˆ A t be the minimizer of the single-step transitions as defined in (8) .
Revised: Let ˆ A t be the minimizer of the single-step transitions as defined in (8) .

Original: Then, for any δ > 0 , ϵ > 0 there is a number of random features
Revised: Then, for any δ > 0 , ϵ > 0 there is a number of random features

Original: ˆ d = ˆ d ( ϵ, δ ) and samples m = m ( ϵ, δ ) , such that with probability ≥ 1 − δ ,
Revised: ˆ d = ˆ d ( ϵ, δ ) and samples m = m ( ϵ, δ ) , such that with probability ≥ 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: The result follows from Proposition 5 in Sun et al.
Revised: The result follows from Proposition 5 in Sun et al.

Original: (2019) applied to the transition functions in our problem.
Revised: (2019) applied to the transition functions in our problem.

Original: Putting this together for all time-steps, we get the following result.
Revised: Putting this together for all time-steps, we get the following result.

Original: Proposition 2 (Universal consistency of RRF privileged learner) .
Revised: Proposition 2 (Universal consistency of RRF privileged learner) .

Original: Let Assumption 1 hold with ϵ t = 0 , ϵ Y = 0 .
Revised: Let Assumption 1 hold with ϵ t = 0 , ϵ Y = 0 .

Original: By Corollary1 and Sun et al.
Revised: By Corollary1 and Sun et al.

Original: (2019), we have for any δ > 0 , ϵ > 0 , γ > 0 and a sequence ( ˆ d 1 , ..., ˆ d T ) of sufficiently large numbers of random features and samples m , that with probability at least 1 − δ/T , for the privileged estimator defined in (8) , (7) ,
Revised: (2019), we have for any δ > 0 , ϵ > 0 , γ > 0 and a sequence ( ˆ d 1 , ..., ˆ d T ) of sufficiently large numbers of random features and samples m , that with probability at least 1 − δ/T , for the privileged estimator defined in (8) , (7) ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then, further assume that the largest eigenvalue λ max ( ˆ A t ) ≤ C for any t and ∥ ˆ β ∥ ≤ C .
Revised: Then, further assume that the largest eigenvalue λ max ( ˆ A t ) ≤ C for any t and ∥ ˆ β ∥ ≤ C .

Original: Then, with probability at least 1 − δ ,
Revised: Then, with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let ∥ · ∥ = ∥ ·
Revised: Let ∥ · ∥ = ∥ ·

Original: ∥
Revised: ∥

Original: L 2 ( p ) .
Revised: L 2 ( p ) .

Original: Then, letting ˆΦ t = ˆΦ γ, ˆ d t , and applying a union bound to each of the T ( ϵ, δ ) -assumptions, and a series of Cauchy-Schwarz inequalities,
Revised: Then, letting ˆΦ t = ˆΦ γ, ˆ d t , and applying a union bound to each of the T ( ϵ, δ ) -assumptions, and a series of Cauchy-Schwarz inequalities,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 2 shows that a privileged learner with random ReLU features can be turned into a universally consistent estimator of any (noiseless) continuous function of X 1 by letting each time step have its own random feature representation of appropriate size and adding a norm constraint to each linear transformation.
Revised: Proposition 2 shows that a privileged learner with random ReLU features can be turned into a universally consistent estimator of any (noiseless) continuous function of X 1 by letting each time step have its own random feature representation of appropriate size and adding a norm constraint to each linear transformation.

Original: The construction in (8), (7) deviates from Algorithm 1 primarily in that the random feature representations used at each time step are different
Revised: The construction in (8), (7) deviates from Algorithm 1 primarily in that the random feature representations used at each time step are different

Original: [Equation]
Revised: [Equation]

Original: but the overall structure is maintained: At training, predictions of Y are made from an embedding ˆ Z T = ˆΦ T ( X T ) of X T , predictions of ˆ Z T are made from an embedding ˆ Z T − 1 = ˆΦ T − 1 ( X T − 1 ) , and so on.
Revised: but the overall structure is maintained: At training, predictions of Y are made from an embedding ˆ Z T = ˆΦ T ( X T ) of X T , predictions of ˆ Z T are made from an embedding ˆ Z T − 1 = ˆΦ T − 1 ( X T − 1 ) , and so on.

Original: At test time, the transition matrices ˆ A 1 , ..., ˆ A T , ˆ β are multiplied and applied to ˆ Z 1 = ˆΦ 1 ( X 1 ) .
Revised: At test time, the transition matrices ˆ A 1 , ..., ˆ A T , ˆ β are multiplied and applied to ˆ Z 1 = ˆΦ 1 ( X 1 ) .

Original: We conjecture that a similar argument can be applied to the construction in Algorithm 1 by letting the dimension of each time step approach ∞ .
Revised: We conjecture that a similar argument can be applied to the construction in Algorithm 1 by letting the dimension of each time step approach ∞ .

Original: C Compounding bias
Revised: C Compounding bias

Original: We can describe the compounding bias of the LuPTS estimator due to a biased representation ˆΦ , in comparison with the standard OLS estimator, by propagating the error in ˆΦ through the estimates.
Revised: We can describe the compounding bias of the LuPTS estimator due to a biased representation ˆΦ , in comparison with the standard OLS estimator, by propagating the error in ˆΦ through the estimates.

Original: Assume that
Revised: Assume that

Original: [Equation]
Revised: [Equation]

Original: Then, let Z t =
Revised: Then, let Z t =

Original: Φ( X t ) and for an estimate ˆΦ , assumed for simplicity to have the same dimension, ˆ d = d ,
Revised: Φ( X t ) and for an estimate ˆΦ , assumed for simplicity to have the same dimension, ˆ d = d ,

Original: [Equation]
Revised: [Equation]

Original: where R t is the residual w.r.t.
Revised: where R t is the residual w.r.t.

Original: Φ .
Revised: Φ .

Original: Let bold-face variables indicate multi-sample equivalents of all variables.
Revised: Let bold-face variables indicate multi-sample equivalents of all variables.

Original: Further, define Σ t =
Revised: Further, define Σ t =

Original: Z ⊤ t Z t and ˆΣ t = ˆ Z ⊤ t ˆ Z t .
Revised: Z ⊤ t Z t and ˆΣ t = ˆ Z ⊤ t ˆ Z t .

Original: Fitting ˆ θ to ˆΦ using the classical learner (OLS) yields an estimate
Revised: Fitting ˆ θ to ˆΦ using the classical learner (OLS) yields an estimate

Original: [Equation]
Revised: [Equation]

Original: Now, define Ω t =
Revised: Now, define Ω t =

Original: R ⊤ t ˆ Z t + Z ⊤ t R t
Revised: R ⊤ t ˆ Z t + Z ⊤ t R t

Original: and we have
Revised: and we have

Original: [Equation]
Revised: [Equation]

Original: where ∆ 1 = − Σ − 1 1 Ω 1 (Σ 1 + Ω 1 ) − 1 , ˆ θ ∗ c is the OLS estimate of θ for the true Φ and the second line follows from the Woodbury matrix identity.
Revised: where ∆ 1 = − Σ − 1 1 Ω 1 (Σ 1 + Ω 1 ) − 1 , ˆ θ ∗ c is the OLS estimate of θ for the true Φ and the second line follows from the Woodbury matrix identity.

Original: The norm of ∆ 1 is related to the condition number of Σ 1 .
Revised: The norm of ∆ 1 is related to the condition number of Σ 1 .

Original: The expectation of the first term is θ , and the expectation of the remaining terms is the bias.
Revised: The expectation of the first term is θ , and the expectation of the remaining terms is the bias.

Original: Now, we can do the same thing for the privileged estimator.
Revised: Now, we can do the same thing for the privileged estimator.

Original: Let’s start with T = 2 .
Revised: Let’s start with T = 2 .

Original: [Equation]
Revised: [Equation]

Original: Thus, the difference in bias between the two estimators is
Revised: Thus, the difference in bias between the two estimators is

Original: [Equation]
Revised: [Equation]

Original: More generally
Revised: More generally

Original: [Equation]
Revised: [Equation]

Original: we can express this difference recursively as below.
Revised: we can express this difference recursively as below.

Original: Proposition 3.
Revised: Proposition 3.

Original: Let ˆ θ p :
Revised: Let ˆ θ p :

Original: =
Revised: =

Original: ˆ A 1 · · · ˆ A T ˆ β be a privileged estimator using a linearly biased representation ˆΦ
Revised: ˆ A 1 · · · ˆ A T ˆ β be a privileged estimator using a linearly biased representation ˆΦ

Original: [Equation]
Revised: [Equation]

Original: and let ˆ A ∗ t be the same estimator using an unbiased representation Φ ∗ .
Revised: and let ˆ A ∗ t be the same estimator using an unbiased representation Φ ∗ .

Original: Then, the bias of ˆ θ p is
Revised: Then, the bias of ˆ θ p is

Original: [Equation]
Revised: [Equation]

Original: where E t is the compounded error in transition dynamics
Revised: where E t is the compounded error in transition dynamics

Original: [Equation]
Revised: [Equation]

Original: computed recursively as follows
Revised: computed recursively as follows

Original: [Equation]
Revised: [Equation]

Original: A ∗ t ) , with E 0 = 0 .
Revised: A ∗ t ) , with E 0 = 0 .

Original: In the worst case, the bias of ˆ θ p grows exponentially with T .
Revised: In the worst case, the bias of ˆ θ p grows exponentially with T .

Original: D Privileged time series representation learners
Revised: D Privileged time series representation learners

Original: We expand on the description of the greedy representation learner (GRL) described as a special case of CRL in Section 3.3.
Revised: We expand on the description of the greedy representation learner (GRL) described as a special case of CRL in Section 3.3.

Original: To avoid the information loss of SRL, we consider its conceptual opposite, using privileged time series information only to predict the outcome.
Revised: To avoid the information loss of SRL, we consider its conceptual opposite, using privileged time series information only to predict the outcome.

Original: To do this, a linear output layer ˆ θ ⊤ t ˆ Z t is used to predict Y at every time step t .
Revised: To do this, a linear output layer ˆ θ ⊤ t ˆ Z t is used to predict Y at every time step t .

Original: Recall that, by Assumption 1, the expected outcome is linear in the latent state at any time step.
Revised: Recall that, by Assumption 1, the expected outcome is linear in the latent state at any time step.

Original: The method is related to multi-view learning, in which prediction of the same quantity is made from multiple “views” (Zhao et al., 2017).
Revised: The method is related to multi-view learning, in which prediction of the same quantity is made from multiple “views” (Zhao et al., 2017).

Original: We dub the model greedy privileged representation learner (GRL), which minimizes the objective
Revised: We dub the model greedy privileged representation learner (GRL), which minimizes the objective

Original: [Equation]
Revised: [Equation]

Original: During inference this algorithm returns ˆ h P ( · ) =
Revised: During inference this algorithm returns ˆ h P ( · ) =

Original: ˆ θ ⊤ 1 ˆΦ( · ) .
Revised: ˆ θ ⊤ 1 ˆΦ( · ) .

Original: Compared to the objective of CRL in 6, we introduce an additional hyperparameter λ ∈ (0 , 1) to place more weight on the loss term that is relevant at inference time.
Revised: Compared to the objective of CRL in 6, we introduce an additional hyperparameter λ ∈ (0 , 1) to place more weight on the loss term that is relevant at inference time.

Original: As a consequence, we choose w t = λ for t = 1 and w t = 1 − λ otherwise.
Revised: As a consequence, we choose w t = λ for t = 1 and w t = 1 − λ otherwise.

Original: We expect GRL to have less bias than SRL, but higher variance since less structure is imposed on the representation ˆΦ .
Revised: We expect GRL to have less bias than SRL, but higher variance since less structure is imposed on the representation ˆΦ .

Original: E Experiment setup & data processing E.1 Detailed experiment setup
Revised: E Experiment setup & data processing E.1 Detailed experiment setup

Original: In the following we give a detailed description of the experimental setup used to obtain the results presented in Section 4 as well as the additional results that are part of this section.
Revised: In the following we give a detailed description of the experimental setup used to obtain the results presented in Section 4 as well as the additional results that are part of this section.

Original: For a given data set, we select a combination of training set sizes and sequence length.
Revised: For a given data set, we select a combination of training set sizes and sequence length.

Original: For each unique combination of these parameters the models of interest are trained repeatedly with different random sampling.
Revised: For each unique combination of these parameters the models of interest are trained repeatedly with different random sampling.

Original: For each repetition the data is split into a train and a test set randomly before hyperparameter tuning and model training are performed.
Revised: For each repetition the data is split into a train and a test set randomly before hyperparameter tuning and model training are performed.

Original: At last each model’s predictions on the test set are scored by computing the coefficient of determination R 2 .
Revised: At last each model’s predictions on the test set are scored by computing the coefficient of determination R 2 .

Original: On synthetic data the test set contains 1000 samples.
Revised: On synthetic data the test set contains 1000 samples.

Original: In the case of real-world data, where samples are limited, we test on 20% of all available data.
Revised: In the case of real-world data, where samples are limited, we test on 20% of all available data.

Original: [Table]
Revised: [Table]

Original: The preprocessing used for real-world data and the generation procedure of synthetic data is unique to each data set.
Revised: The preprocessing used for real-world data and the generation procedure of synthetic data is unique to each data set.

Original: We refer to the data set specific subsections for detailed descriptions of how each data set is processed.
Revised: We refer to the data set specific subsections for detailed descriptions of how each data set is processed.

Original: During the experiments each model uses standardized data for training and inference.
Revised: During the experiments each model uses standardized data for training and inference.

Original: To perform the data rescaling we use the StandardScaler implementation that is part of scikit-learn (Pedregosa et al., 2011).
Revised: To perform the data rescaling we use the StandardScaler implementation that is part of scikit-learn (Pedregosa et al., 2011).

Original: Hyperparameter tuning.
Revised: Hyperparameter tuning.

Original: The tuning of hyperparameters is carried out for each repetition and is implemented using random search and five-fold cross-validation.
Revised: The tuning of hyperparameters is carried out for each repetition and is implemented using random search and five-fold cross-validation.

Original: Each hyperparameter is sampled from a fixed interval of possible values.
Revised: Each hyperparameter is sampled from a fixed interval of possible values.

Original: An overview of the ranges of different hyperparameters determined through random-search is provided in Table 1.
Revised: An overview of the ranges of different hyperparameters determined through random-search is provided in Table 1.

Original: For the experiments with variants of generalized LuPTS we sample ten sets of hyperparameters before retraining on all training data using the best set of parameters.
Revised: For the experiments with variants of generalized LuPTS we sample ten sets of hyperparameters before retraining on all training data using the best set of parameters.

Original: For the representation learners we merely sample five values for λ .
Revised: For the representation learners we merely sample five values for λ .

Original: Neural network training.
Revised: Neural network training.

Original: The training of neural networks involves many choices and hyperparameters.
Revised: The training of neural networks involves many choices and hyperparameters.

Original: We choose PyTorch’s implementation (Paszke et al., 2019) of the Adam optimizer (Kingma and Ba, 2017) to train the representation learning models.
Revised: We choose PyTorch’s implementation (Paszke et al., 2019) of the Adam optimizer (Kingma and Ba, 2017) to train the representation learning models.

Original: If not specified otherwise the results shown in this project are obtained using a learning rate of 0 . 0001 , a batch size of 30 , leaky ReLU activations and a maximum of 1500 training epochs.
Revised: If not specified otherwise the results shown in this project are obtained using a learning rate of 0 . 0001 , a batch size of 30 , leaky ReLU activations and a maximum of 1500 training epochs.

Original: In the case of neural network models, the sample sizes reported as part of the experiments denote the combined size of the training and validation set, where the validation set contains 20% of those samples.
Revised: In the case of neural network models, the sample sizes reported as part of the experiments denote the combined size of the training and validation set, where the validation set contains 20% of those samples.

Original: We use early stopping during the training process by keeping track of the validation loss.
Revised: We use early stopping during the training process by keeping track of the validation loss.

Original: If a model does not improve the validation loss over a waiting period of 200 epochs we stop training early and set the network parameters to the values that obtained the lowest validation loss up until that point.
Revised: If a model does not improve the validation loss over a waiting period of 200 epochs we stop training early and set the network parameters to the values that obtained the lowest validation loss up until that point.

Original: In order to make sure that results are not dependent on the specific choice of the parameters just described, we performed additional experiments with different parameter choices.
Revised: In order to make sure that results are not dependent on the specific choice of the parameters just described, we performed additional experiments with different parameter choices.

Original: The results were found to be robust to small changes in these parameters.
Revised: The results were found to be robust to small changes in these parameters.

Original: Generalized distillation.
Revised: Generalized distillation.

Original: In order to compare our algorithms to the alternative of using generalized distillation for privileged time series as presented by Hayashi et al.
Revised: In order to compare our algorithms to the alternative of using generalized distillation for privileged time series as presented by Hayashi et al.

Original: (2019), we implemented a model that (i) produces hypotheses of the same class as our other algorithms and (ii) that adopts the learning paradigm of a student model incorporating soft targets produced by a teacher model into its loss function.
Revised: (2019), we implemented a model that (i) produces hypotheses of the same class as our other algorithms and (ii) that adopts the learning paradigm of a student model incorporating soft targets produced by a teacher model into its loss function.

Original: For tabular data our teacher model is a multi-layer-perceptron (MLP) with T ∗ k input neurons such that all { X t } are concatenated and then used as input.
Revised: For tabular data our teacher model is a multi-layer-perceptron (MLP) with T ∗ k input neurons such that all { X t } are concatenated and then used as input.

Original: The teacher MLP makes use of five hidden layers, each consisting of 100 neurons.
Revised: The teacher MLP makes use of five hidden layers, each consisting of 100 neurons.

Original: In the case of the image data generated by
Revised: In the case of the image data generated by

Original: Clocks-LGS , the teacher uses an implementation of LeNet-5 on all variables X t (while sharing the encoder parameters) before concatenating the 25-dimensional output of this encoder from different time steps.
Revised: Clocks-LGS , the teacher uses an implementation of LeNet-5 on all variables X t (while sharing the encoder parameters) before concatenating the 25-dimensional output of this encoder from different time steps.

Original: This combined representation is then processed by an MLP with a single hidden layer with 25 neurons.
Revised: This combined representation is then processed by an MLP with a single hidden layer with 25 neurons.

Original: The loss function used for the student model producing the estimate ˆ h C is architecturally identical to the classic representation learner used in each of the experiments.
Revised: The loss function used for the student model producing the estimate ˆ h C is architecturally identical to the classic representation learner used in each of the experiments.

Original: When training the student model the mean squared error on the data and the error corresponding to the soft targets of the teacher model are combined via a hyperparameter λ :
Revised: When training the student model the mean squared error on the data and the error corresponding to the soft targets of the teacher model are combined via a hyperparameter λ :

Original: [Equation]
Revised: [Equation]

Original: The hyperparameter is determined via hyperparameter tuning in all repetitions as described for other hyperparameters in this section.
Revised: The hyperparameter is determined via hyperparameter tuning in all repetitions as described for other hyperparameters in this section.

Original: All other training procedures follow the same logic as described above.
Revised: All other training procedures follow the same logic as described above.

Original: Resources.
Revised: Resources.

Original: For the training of the representation learning algorithms we use a cluster of graphics processing units (GPUs) in order to reach the number of experiment repetitions required for our work.
Revised: For the training of the representation learning algorithms we use a cluster of graphics processing units (GPUs) in order to reach the number of experiment repetitions required for our work.

Original: A single experiment like shown in Figure 5c takes several hours on 100 NVIDIA Tesla T4 GPUs.
Revised: A single experiment like shown in Figure 5c takes several hours on 100 NVIDIA Tesla T4 GPUs.

Original: While the random feature methods do not require GPU training, they still require hyperparameter tuning which is why we compute results such as presented in Figure 4 on many CPU cores in parallel.
Revised: While the random feature methods do not require GPU training, they still require hyperparameter tuning which is why we compute results such as presented in Figure 4 on many CPU cores in parallel.

Original: While the experiments on neural networks cannot reasonably be reproduced on a single desktop machine, this is still possible within a few days for the random feature methods.
Revised: While the experiments on neural networks cannot reasonably be reproduced on a single desktop machine, this is still possible within a few days for the random feature methods.

Original: Latent variable recovery and SVCCA.
Revised: Latent variable recovery and SVCCA.

Original: In order to assess to what extent a representation has been found that is linearly related to the true latent variables we use SVCCA as described by Raghu et al.
Revised: In order to assess to what extent a representation has been found that is linearly related to the true latent variables we use SVCCA as described by Raghu et al.

Original: (2017), meaning we first use PCA retaining at least 99% of variation before then applying CCA.
Revised: (2017), meaning we first use PCA retaining at least 99% of variation before then applying CCA.

Original: For the visualization shown in Figure 7b we construct a grid of points (150 times 150) around the origin and assign each point a unique color.
Revised: For the visualization shown in Figure 7b we construct a grid of points (150 × 150) around the origin and assign each point a unique color.

Original: Then we compute an image using the observation generating function Ψ : Z −→ X of the Clocks-LGS data set for each point.
Revised: Then we compute an image using the observation generating function Ψ : Z −→ X of the Clocks-LGS data set for each point.

Original: The observations are passed to the encoder ˆ Φ of the two representation learners producing estimates of the latent variables.
Revised: The observations are passed to the encoder ˆ Φ of the two representation learners producing estimates of the latent variables.

Original: We then map the estimates to the ground truth linearly using SVCCA before plotting the result.
Revised: We then map the estimates to the ground truth linearly using SVCCA before plotting the result.

Original: E.2 Alzheimer progression
Revised: E.2 Alzheimer progression

Original: To test our algorithms on the task of predicting the progression of Alzheimer’s disease (AD) we use an anonymized data set obtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004) under the LONI Research License.
Revised: To test our algorithms on the task of predicting the progression of Alzheimer’s disease (AD) we use an anonymized data set obtained through the Alzheimer’s Disease Neuroimaging Initiative ( ADNI ) (ADNI, 2004) under the LONI Research License.

Original: The initiative is large multi-site research study on the brains of over 2000 AD patients which collects many features such as genetic, imaging and biospecimen biomarkers.
Revised: The initiative is large multi-site research study on the brains of over 2000 AD patients which collects many features such as genetic, imaging and biospecimen biomarkers.

Original: The data consists of measurements taken every 3 months with some observations missing.
Revised: The data consists of measurements taken every 3 months with some observations missing.

Original: The outcome of interest in our experiments is the Mini Mental State Experiment (MMSE) score 48 months after the first measurement(Galea and Woodward, 2005).
Revised: The outcome of interest in our experiments is the Mini Mental State Experiment (MMSE) score 48 months after the first measurement(Galea and Woodward, 2005).

Original: Privileged information are the measurements taken between those time points, at 12, 24 and 36 months into the program.
Revised: Privileged information are the measurements taken between those time points, at 12, 24 and 36 months into the program.

Original: Data processing.
Revised: Data processing.

Original: The processing procedure used in this project is borrowed directly from the work of Karlsson et al.
Revised: The processing procedure used in this project is borrowed directly from the work of Karlsson et al.

Original: There is a large amount of missing information in the ADNI data set.
Revised: There is a large amount of missing information in the ADNI data set.

Original: The missingness varies with the time of when measurements were taken.
Revised: The missingness varies with the time of when measurements were taken.

Original: Further some subjects were not present at some of the follow-up examinations.
Revised: Further some subjects were not present at some of the follow-up examinations.

Original: To deal with the missingness patients without an observation for the final follow up (the outcome Y ) are excluded from our experiments.
Revised: To deal with the missingness patients without an observation for the final follow up (the outcome Y ) are excluded from our experiments.

Original: Further, we also require that patients are present at all intermediate time steps (12, 24 and 36 months after the first measurement) which we use as privileged information.
Revised: Further, we also require that patients are present at all intermediate time steps (12, 24 and 36 months after the first measurement) which we use as privileged information.

Original: We one-hot encode categorical features and exclude features for which more than 70% of the observations are missing.
Revised: We one-hot encode categorical features and exclude features for which more than 70% of the observations are missing.

Original: To deal with the remaining missing values, mean imputation is used.
Revised: To deal with the remaining missing values, mean imputation is used.

Original: Due to the filtering that we apply as a result of the missing data we only obtain 502 suitable sequences that we can use for our experiments.
Revised: Due to the filtering that we apply as a result of the missing data we only obtain 502 suitable sequences that we can use for our experiments.

Original: E.3 Traffic data
Revised: E.3 Traffic data

Original: The Traffic data set (Hogue, 2012) obtained through the UCI machine learning repository (Dua and
Revised: The Traffic data set (Hogue, 2012) obtained through the UCI machine learning repository (Dua and

Original: Graff, 2017) contains hourly measurements of the traffic volume as well as weather features and holiday information.
Revised: Graff, 2017) contains hourly measurements of the traffic volume as well as weather features and holiday information.

Original: The raw data contains 48.204 records.
Revised: The raw data contains 48.204 records.

Original: An overview of all available features is given in Table 2.
Revised: An overview of all available features is given in Table 2.

Original: Data processing.
Revised: Data processing.

Original: We noticed extreme outliers in the data set as well as implausible numerical values for the temperature and rain features.
Revised: We noticed extreme outliers in the data set as well as implausible numerical values for the temperature and rain features.

Original: Further, records for some of the hours of the timeframe (2012 - 2018) covered by this data set are missing.
Revised: Further, records for some of the hours of the timeframe (2012 - 2018) covered by this data set are missing.

Original: To deal with the extreme outliers we calculate the mean and standard deviation of each feature and remove records which contain values that are further than six standard deviations from the mean of a particular feature.
Revised: To deal with the extreme outliers we calculate the mean and standard deviation of each feature and remove records which contain values that are further than six standard deviations from the mean of a particular feature.

Original: We also remove a feature entirely if there is no variation left after this filtering.
Revised: We also remove a feature entirely if there is no variation left after this filtering.

Original: This is the case for the snowfall feature as snow is very rare in Minneapolis.
Revised: This is the case for the snowfall feature as snow is very rare in Minneapolis.

Original: From the date and time of each record we calculate the weekday which we add
Revised: From the date and time of each record we calculate the weekday which we add

Original: [Table]
Revised: [Table]

Original: categories like clear, clouds, or rain
Revised: categories like clear, clouds, or rain

Original: [Table]
Revised: [Table]

Original: as a one-hot encoded feature and also represent the hour of the day h ∈ { 0 , 1 , . . .
Revised: as a one-hot encoded feature and also represent the hour of the day h ∈ { 0 , 1 , . . .

Original: 23 } as two separate periodic features given by
Revised: 23 } as two separate periodic features given by

Original: [Equation]
Revised: [Equation]

Original: This ensures that a timestamp just before midnight produces similar features compared to just after midnight.
Revised: This ensures that a timestamp just before midnight produces similar features compared to just after midnight.

Original: We one-hot encode the holiday information, making no difference between different types of holidays, and make this feature persist over a full calendar day.
Revised: We one-hot encode the holiday information, making no difference between different types of holidays, and make this feature persist over a full calendar day.

Original: In the original data set the holiday information is only specified for the first hour of the day.
Revised: In the original data set the holiday information is only specified for the first hour of the day.

Original: The column Weather_Main contains some weather conditions that are very rare, such as smoke and squall.
Revised: The column Weather_Main contains some weather conditions that are very rare, such as smoke and squall.

Original: As a consequence we group the different conditions before encoding them as binary variables.
Revised: As a consequence we group the different conditions before encoding them as binary variables.

Original: In particular we make drizzle, rain and squall one single feature while also grouping together fog, haze, mist and smoke as they all affect visibility.
Revised: In particular we make drizzle, rain and squall one single feature while also grouping together fog, haze, mist and smoke as they all affect visibility.

Original: time series selection.
Revised: Time series selection.

Original: After this preprocessing that leaves only numerical values and one-hot encoded categorical values we group the data together as time series used for the experiments.
Revised: After this preprocessing, that leaves only numerical values and one-hot encoded categorical values, we group the data together as time series used for the experiments.

Original: In order to do so we specify a desired sequence length T + 1 and a sequence step size in hours.
Revised: In order to do so we specify a desired sequence length T + 1 and a sequence step size in hours.

Original: With this information we iterate through the data set assembling time series with i) no values missing ii) the correct length and step size and iii) at least a seven hour gap between each pair of sequences.
Revised: With this information we iterate through the data set assembling time series with i) no values missing ii) the correct length and step size and iii) at least a seven hour gap between each pair of sequences.

Original: The third condition is introduced to make sure one does not end up with very similar cases (for short sequences in particular) in training and test set.
Revised: The third condition is introduced to make sure one does not end up with very similar cases (for short sequences in particular) in training and test set.

Original: E.4 Square-Sign
Revised: E.4 Square-Sign

Original: The Square-Sign data set serves as a test environment for learning from privileged time series information where one can assure the conditions necessary for Assumption 1 to hold.
Revised: The Square-Sign data set serves as a test environment for learning from privileged time series information where one can assure the conditions necessary for Assumption 1 to hold.

Original: In particular this means creating a linear-Gaussian system which remains unobserved and combining it with an observation generating function ψ : Z −→ X .
Revised: In particular this means creating a linear-Gaussian system which remains unobserved and combining it with an observation generating function ψ : Z −→ X .

Original: Latent linear-Gaussian system.
Revised: Latent linear-Gaussian system.

Original: The first component that makes up the generation process for
Revised: The first component that makes up the generation process for

Original: Square-Sign (and Clocks-LGS ) is the linear-Gaussian system which is latent, just as depicted on the right side of Figure 1b with Z t ∈ R d .
Revised: Square-Sign (and Clocks-LGS ) is the linear-Gaussian system which is latent, just as depicted on the right side of Figure 1b with Z t ∈ R d .

Original: The first step in the data creation process is sampling each of the d components of Z 0 from N (0 , 5) .
Revised: The first step in the data creation process is sampling each of the d components of Z 0 from N (0 , 5) .

Original: Then the subsequent latent variables Z t +1 are computed as
Revised: Then the subsequent latent variables Z t +1 are computed as

Original: [Equation]
Revised: [Equation]

Original: For the outcome we use the same form but with different dimensionality:
Revised: For the outcome we use the same form but with different dimensionality:

Original: [Equation]
Revised: [Equation]

Original: Off-diagonal elements of the transition matrices
Revised: Off-diagonal elements of the transition matrices

Original: A t ∈ R d × d are sampled from a Normal distribution
Revised: A t ∈ R d × d are sampled from a Normal distribution

Original: N (0 , 0 . 2) while the diagonal elements are set to one.
Revised: N (0 , 0 . 2) while the diagonal elements are set to one.

Original: In a second step we compute the spectral radius of the randomly created matrices A t via eigenvalue decomposition, obtaining the components U Λ U ⊤ .
Revised: In a second step we compute the spectral radius of the randomly created matrices A t via eigenvalue decomposition, obtaining the components U Λ U ⊤ .

Original: We then set the spectral radius to a predefined value λ max = 1 . 3 and reassemble the matrix as
Revised: We then set the spectral radius to a predefined value λ max = 1 . 3 and reassemble the matrix as

Original: [Equation]
Revised: [Equation]

Original: The coefficients of β are drawn from the same normal distribution as the ones of A t but undergo no further changes.
Revised: The coefficients of β are drawn from the same normal distribution as the ones of A t but undergo no further changes.

Original: Observation generating function.
Revised: Observation generating function.

Original: As the dimensionality d of the latent space Z is not fixed we use an observation generating function that is not restricted to a specific value of d .
Revised: As the dimensionality d of the latent space Z is not fixed we use an observation generating function that is not restricted to a specific value of d .

Original: For each element in Z t ∈
Revised: For each element in Z t ∈

Original: Z
Revised: Z

Original: = R
Revised: = R

Original: d we create two elements in X t ∈ X
Revised: d we create two elements in X t ∈ X

Original: = R 2 d by denoting its sign separately from its square.
Revised: = R 2 d by denoting its sign separately from its square.

Original: This gives the following nonlinear observation generating function:
Revised: This gives the following nonlinear observation generating function:

Original: [Equation]
Revised: [Equation]

Original: E.5 Clocks-LGS
Revised: E.5 Clocks-LGS

Original: This data set serves the purpose of testing our algorithms on a different modality with high dimensional data.
Revised: This data set serves the purpose of testing our algorithms on a different modality with high dimensional data.

Original: In particular the idea was to use image data as this is a domain where neural networks have been very successful.
Revised: In particular the idea was to use image data as this is a domain where neural networks have been very successful.

Original: For this reason we combine a latent dynamical system with an image generation process which we explain in detail in this section.
Revised: For this reason we combine a latent dynamical system with an image generation process which we explain in detail in this section.

Original: Latent linear-Gaussian system.
Revised: Latent linear-Gaussian system.

Original: We use exactly the same setup as we do for the Square-Sign latent dynamical system as described in Section E.4.
Revised: We use exactly the same setup as we do for the Square-Sign latent dynamical system as described in Section E.4.

Original: The only difference here is the dimensionality of the latent variables, transition matrices and the outcome.
Revised: The only difference here is the dimensionality of the latent variables, transition matrices and the outcome.

Original: For Clocks-LGS we generally have d =and q ∈ { 1 , 2 } .
Revised: For Clocks-LGS we generally have d =and q ∈ { 1 , 2 } .

Original: Image generation.
Revised: Image generation.

Original: The second part of Clocks-LGS is creating images from two dimensional latent vectors Z t =
Revised: The second part of Clocks-LGS is creating images from two dimensional latent vectors Z t =

Original: [ Z (1) t , Z (2) t ] ⊤ .
Revised: [ Z (1) t , Z (2) t ] ⊤ .

Original: The goal was to keep it the process simple while using small black and white images of 28 × 28 pixels.
Revised: The goal was to keep it the process simple while using small black and white images of 28 × 28 pixels.

Original: In addition we wanted each image to have no ambiguity with respect to the latent state it represents.
Revised: In addition we wanted each image to have no ambiguity with respect to the latent state it represents.

Original: We represent the first component by a clock hand mounted at the image center.
Revised: We represent the first component by a clock hand mounted at the image center.

Original: One can think of Z (1) t as the angle in radian, meaning the hand points straight up for
Revised: One can think of Z (1) t as the angle in radian, meaning the hand points straight up for

Original: Z (1) t = 0 or Z (1) t = 2 π and straight down for Z (1) t = π .
Revised: Z (1) t = 0 or Z (1) t = 2 π and straight down for Z (1) t = π .

Original: To visualize a full rotation we increase the size of the cirle around the image center in discrete steps for each mutliple of 2 π .
Revised: To visualize a full rotation we increase the size of the cirle around the image center in discrete steps for each mutliple of 2 π .

Original: For negative values the circle is empty (black) while it is filled (white) for positive values.
Revised: For negative values the circle is empty (black) while it is filled (white) for positive values.

Original: For the second component we make use of the same logic but instead of a clock hand, we only use a circle that orbits the image center.
Revised: For the second component we make use of the same logic but instead of a clock hand, we only use a circle that orbits the image center.

Original: The two hands cannot obscure each other as the orbiting cirle uses a larger radius.
Revised: The two hands cannot obscure each other as the orbiting cirle uses a larger radius.

Original: Figureshows three examples of pairs of corresponding latent vectors and generated images.
Revised: Figureshows three examples of pairs of corresponding latent vectors and generated images.

Original: E.6 PM 2 . 5 air quality
Revised: E.6 PM 2 . 5 air quality

Original: Due to health concerns the air quality in Chinese cities has become an important topic.
Revised: Due to health concerns the air quality in Chinese cities has become an important topic.

Original: The PM 2.
Revised: The PM 2.

Original: data set contains hourly meteorologic information and the concentration of small particles (PM 2 . 5 ) for the cities Beijing, Shanghai, Guangzhou, Chengdu and Shenyang (Liang et al., 2016).
Revised: data set contains hourly meteorologic information and the concentration of small particles (PM 2 . 5 ) for the cities Beijing, Shanghai, Guangzhou, Chengdu and Shenyang (Liang et al., 2016).

Original: The individual features available for all cities are listed in Table 3.
Revised: The individual features available for all cities are listed in Table 3.

Original: In addition to the features listed, the data includes the date and time of each record.
Revised: In addition to the features listed, the data includes the date and time of each record.

Original: Just like in the preprocessing of Traffic we compute a periodic time feature using expression 10 to represent the time of day of each record.
Revised: Just like in the preprocessing of Traffic we compute a periodic time feature using expression 10 to represent the time of day of each record.

Original: For each numerical feature we calculate the mean and standard deviation and remove rows with values that are more extreme than six standard deviations from the mean.
Revised: For each numerical feature we calculate the mean and standard deviation and remove rows with values that are more extreme than six standard deviations from the mean.

Original: We also remove rows with missing categorical features, which are then represented as one-hot vectors.
Revised: We also remove rows with missing categorical features, which are then represented as one-hot vectors.

Original: Apart from differences in the preprocessing we consider the same prediction task as Karlsson et al.
Revised: Apart from differences in the preprocessing we consider the same prediction task as Karlsson et al.

Original: (2022) which is predicting the future particle concentration for a fixed time horizon given current observations.
Revised: (2022) which is predicting the future particle concentration for a fixed time horizon given current observations.

Original: [Table]
Revised: [Table]

Original: F Additional experiment results
Revised: F Additional experiment results

Original: In the course of this section we present a larger scope of our experimental results.
Revised: In the course of this section we present a larger scope of our experimental results.

Original: We demonstrate the predictive accuracy of the algorithms introduced in Sections 3.2 and 3.3 in terms of the mean coefficient of determination R 2 over different settings on five data sets.
Revised: We demonstrate the predictive accuracy of the algorithms introduced in Sections 3.2 and 3.3 in terms of the mean coefficient of determination R 2 over different settings on five data sets.

Original: The variation of the results over repetitions is represented by the shaded areas in the visualizations, which denotes one standard deviation above and below the mean value.
Revised: The variation of the results over repetitions is represented by the shaded areas in the visualizations, which denotes one standard deviation above and below the mean value.

Original: We also show empirically that the bias of generalized
Revised: We also show empirically that the bias of generalized

Original: LuPTS increases with the number of privileged time steps when a poor representation function is used.
Revised: LuPTS increases with the number of privileged time steps when a poor representation function is used.

Original: This can be seen in Figure 16a where we test privileged and classical learners over 500 different systems of the Square-Sign type over different sequence lengths.
Revised: This can be seen in Figure 16a where we test privileged and classical learners over 500 different systems of the Square-Sign type over different sequence lengths.

Original: In addition we further illustrate how privileged information can improve the recovery of latent variables of the data generating process by providing more visualizations in the style of Figure 7a on the Clocks-LGS and Square-Sign data set.
Revised: In addition we further illustrate how privileged information can improve the recovery of latent variables of the data generating process by providing more visualizations in the style of Figure 7a on the Clocks-LGS and Square-Sign data set.

Original: These can be found in Figures 13 and 16.
Revised: These can be found in Figures 13 and 16.

Original: In the following we first provide the experiment details for visualizing the phase transition of generalized LuPTS as seen in Figure 2.
Revised: In the following we first provide the experiment details for visualizing the phase transition of generalized LuPTS as seen in Figure 2.

Original: In the subsections thereafter the material is organized by data set.
Revised: In the subsections thereafter the material is organized by data set.

Original: F.1 Two regimes of generalized LuPTS
Revised: F.1 Two regimes of generalized LuPTS

Original: As seen in Figure 2 and demonstrated by Proposition 1, generalized LuPTS becomes equivalent to the corresponding classical learner when the number of features ˆ d is larger than the number of samples m .
Revised: As seen in Figure 2 and demonstrated by Proposition 1, generalized LuPTS becomes equivalent to the corresponding classical learner when the number of features ˆ d is larger than the number of samples m .

Original: In the following we provide the experiment details that led to Figure 2.
Revised: In the following we provide the experiment details that led to Figure 2.

Original: In order to evaluate the dependency on the number of features we used linear LuPTS and OLS on a synthetically generated linear-Gaussian system as displayed in Figure 1a.
Revised: In order to evaluate the dependency on the number of features we used linear LuPTS and OLS on a synthetically generated linear-Gaussian system as displayed in Figure 1a.

Original: We use the same setup as for the latent dynamics in Square-Sign but without a nonlinear observation generating function.
Revised: We use the same setup as for the latent dynamics in Square-Sign but without a nonlinear observation generating function.

Original: For each number of features k = ˆ d = d we sample 50 such systems with different dynamics, each producing a training set with m = 100 and test set of 1000 samples.
Revised: For each number of features k = ˆ d = d we sample 50 such systems with different dynamics, each producing a training set with m = 100 and test set of 1000 samples.

Original: The systems are all configured with T = 3 and q = 10 .
Revised: The systems are all configured with T = 3 and q = 10 .

Original: We train and score both estimators on all of the data generating systems before computing the mean coefficient of determination over all systems with the same number of features.
Revised: We train and score both estimators on all of the data generating systems before computing the mean coefficient of determination over all systems with the same number of features.

Original: F.2 Alzheimer progression F.3 Traffic dataF.4 Clock_LGS F.5 Square-Sign
Revised: [Figure] [Figure] [Figure] F.3 Traffic data

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: F.6 PM

Original: [Figure]
Revised: [Figure]

Revised: 5 air quality

Original: [Figure]
Revised: [Figure]

Revised: ²

Original: [Figure] [Figure]
Revised: [Figure] [Figure]