Original: T OWARDS P ROVABLE C ONTROL FOR U NKNOWN L INEAR D YNAMICAL S YSTEMS
Revised: T OWARDS P ROVABLE C ONTROL FOR U NKNOWN L INEAR D YNAMICAL S YSTEMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the control of symmetric linear dynamical systems with unknown dynamics and a hidden state.
Revised: We study the control of symmetric linear dynamical systems with unknown dynamics and a hidden state.

Original: Using a recent spectral ﬁltering technique for concisely representing such systems in a linear basis, we formulate optimal control in this setting as a convex program.
Revised: Using a recent spectral ﬁltering technique for concisely representing such systems in a linear basis, we formulate optimal control in this setting as a convex program.

Original: This approach eliminates the need to solve the nonconvex problem of explicit identiﬁcation of the system and its latent state, and allows for provable optimality guarantees for the control signal.
Revised: This approach eliminates the need to solve the nonconvex problem of explicit identiﬁcation of the system and its latent state, and allows for provable optimality guarantees for the control signal.

Original: We give the ﬁrst efﬁcient algorithm for ﬁnding the optimal control signal with an arbitrary time horizon 𝑇 , with sample complexity (number of training rollouts) polynomial only in log 𝑇 and other relevant parameters.
Revised: We give the ﬁrst efﬁcient algorithm for ﬁnding the optimal control signal with an arbitrary time horizon 𝑇 , with sample complexity (number of training rollouts) polynomial only in log 𝑇 and other relevant parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent empirical successes of reinforcement learning involve using deep nets to represent the underlying MDP and policy.
Revised: Recent empirical successes of reinforcement learning involve using deep nets to represent the underlying MDP and policy.

Original: However, we lack any supporting theory, and are far from developing algorithms with provable guarantees for such settings.
Revised: However, we lack any supporting theory, and are far from developing algorithms with provable guarantees for such settings.

Original: We can make progress by addressing simpler setups, such as those provided by control theory.
Revised: We can make progress by addressing simpler setups, such as those provided by control theory.

Original: Control theory concerns the control of dynamical systems, a non-trivial task even if the system is fully speciﬁed and provable guarantees are not required.
Revised: Control theory concerns the control of dynamical systems, a non-trivial task even if the system is fully speciﬁed and provable guarantees are not required.

Original: This is true even in the simplest setting of a linear dynamical system (LDS) with quadratic costs, since the resulting optimization problems are high-dimensional and sensitive to noise.
Revised: This is true even in the simplest setting of a linear dynamical system (LDS) with quadratic costs, since the resulting optimization problems are high-dimensional and sensitive to noise.

Original: The task of controlling an unknown linear system is signiﬁcantly more complex, often giving rise to non-convex and high-dimensional optimization problems.
Revised: The task of controlling an unknown linear system is signiﬁcantly more complex, often giving rise to non-convex and high-dimensional optimization problems.

Original: The standard practice in the literature is to ﬁrst solve the non-convex problem of system identiﬁcation —that is, recover a model that accurately describes the system—and then apply standard robust control methods.
Revised: The standard practice in the literature is to ﬁrst solve the non-convex problem of system identiﬁcation —that is, recover a model that accurately describes the system—and then apply standard robust control methods.

Original: The non-convex problem of system identiﬁcation is the main reason that we have essentially no provable algorithms for controlling even the simplest linear dynamical systems with unknown latent states.
Revised: The non-convex problem of system identiﬁcation is the main reason that we have essentially no provable algorithms for controlling even the simplest linear dynamical systems with unknown latent states.

Original: In this paper, we take the ﬁrst step towards a provably efﬁcient control algorithm for linear dynamical systems.
Revised: In this paper, we take the ﬁrst step towards a provably efﬁcient control algorithm for linear dynamical systems.

Original: Despite the highly non-convex and high-dimensional formulation of the problem, we can efﬁciently ﬁnd the optimal control signal in polynomial time with optimal sample complexity.
Revised: Despite the highly non-convex and high-dimensional formulation of the problem, we can efﬁciently ﬁnd the optimal control signal in polynomial time with optimal sample complexity.

Original: Our method is based on wave-ﬁltering , a recent spectral representation technique for symmetric LDSs (Hazan et al., 2017).
Revised: Our method is based on wave-ﬁltering , a recent spectral representation technique for symmetric LDSs (Hazan et al., 2017).

Original: 1. O UR RESULTS
Revised: 1. O UR RESULTS

Original: A dynamical system converts input signals { 𝑥 1 , . .
Revised: A dynamical system converts input signals { 𝑥 1 , . .

Original: . , 𝑥 𝑇 } ∈ R 𝑛 into output signals { 𝑦 1 , . .
Revised: . , 𝑥 𝑇 } ∈ R 𝑛 into output signals { 𝑦 1 , . .

Original: . , 𝑦 𝑇 } ∈ R 𝑚 , incurring a sequence of costs 𝑐 1 , . .
Revised: . , 𝑦 𝑇 } ∈ R 𝑚 , incurring a sequence of costs 𝑐 1 , . .

Original: 𝑐 𝑇 ∈ R .
Revised: 𝑐 𝑇 ∈ R .

Original: We are interested in controlling unknown dynamical systems with hidden states (which can be thought of as being partially “observed”via the output signals).
Revised: We are interested in controlling unknown dynamical systems with hidden states (which can be thought of as being partially “observed”via the output signals).

Original: A vast body of work focuses on linear dynamical systems with quadratic costs, in which the { 𝑦 𝑡 } and { 𝑐 𝑡 } are governed by the following dynamics:
Revised: A vast body of work focuses on linear dynamical systems with quadratic costs, in which the { 𝑦 𝑡 } and { 𝑐 𝑡 } are governed by the following dynamics:

Original: [Equation]
Revised: [Equation]

Original: where ℎ 1 , . .
Revised: where ℎ 1 , . .

Original: , ℎ 𝑇 ∈ R 𝑑 is a sequence of hidden states starting with a ﬁxed ℎ 1 .
Revised: , ℎ 𝑇 ∈ R 𝑑 is a sequence of hidden states starting with a ﬁxed ℎ 1 .

Original: Matrices 𝐴, 𝐵, 𝐶, 𝐷, 𝑄, 𝑅 of appropriate dimension describe the system and cost objective; the { 𝜂 𝑡 } are
Revised: Matrices 𝐴, 𝐵, 𝐶, 𝐷, 𝑄, 𝑅 of appropriate dimension describe the system and cost objective; the { 𝜂 𝑡 } are

Original: Gaussian noise vectors.
Revised: Gaussian noise vectors.

Original: All of these matrices, as well as the parameters of the Gaussian, can be unknown.
Revised: All of these matrices, as well as the parameters of the Gaussian, can be unknown.

Original: The most fundamental control problem involves controlling the system for some time horizon 𝑇 :
Revised: The most fundamental control problem involves controlling the system for some time horizon 𝑇 :

Original: ﬁnd a signal 𝑥 1 , . .
Revised: ﬁnd a signal 𝑥 1 , . .

Original: , 𝑥 𝑇 that minimizes the sum of these quadratic output costs (cid:80) 𝑡 𝑐 𝑡 .
Revised: , 𝑥 𝑇 that minimizes the sum of these quadratic output costs (cid:80) 𝑡 𝑐 𝑡 .

Original: Clearly, any algorithm for doing so must ﬁrst learn the system parameters in some form, and this is often the source of computational intractability (meaning algorithms that take time exponential in the number of system parameters).
Revised: Clearly, any algorithm for doing so must ﬁrst learn the system parameters in some form, and this is often the source of computational intractability (meaning algorithms that take time exponential in the number of system parameters).

Original: Previously known algorithms are of two types.
Revised: Previously known algorithms are of two types.

Original: The ﬁrst type tries to solve the non-convex problem, with algorithms that lack provable guarantees and may take exponential time in the worst case: e.g., expectation-maximization (EM) or gradient-based methods (back-propagation through time, like the training of RNNs) which identify both the hidden states and system parameters.
Revised: The ﬁrst type tries to solve the non-convex problem, with algorithms that lack provable guarantees and may take exponential time in the worst case: e.g., expectation-maximization (EM) or gradient-based methods (back-propagation through time, like the training of RNNs) which identify both the hidden states and system parameters.

Original: Algorithms of the second type rely upon regression, often used in time-series analysis.
Revised: Algorithms of the second type rely upon regression, often used in time-series analysis.

Original: Since 𝑇 -step dynamics of the system involve the ﬁrst 𝑇 powers of 𝐴 , the algorithm represents these powers as new variables and learns the 𝑇 -step dynamics via regression (e.g., the so-called VARX( 𝑝, 𝑠 ) model) assuming that the system is well-conditioned; see Appendix A. This has moderate computational complexity but high sample complexity since the number of parameters in the regression scales with 𝑇 , the number of time steps.
Revised: Since 𝑇 -step dynamics of the system involve the ﬁrst 𝑇 powers of 𝐴 , the algorithm represents these powers as new variables and learns the 𝑇 -step dynamics via regression (e.g., the so-called VARX( 𝑝, 𝑠 ) model) assuming that the system is well-conditioned; see Appendix A. This has moderate computational complexity but high sample complexity since the number of parameters in the regression scales with 𝑇 , the number of time steps.

Original: Our new method obtains the best of both results: few parameters to train resulting in low sample complexity, as well as polynomial computational complexity.
Revised: Our new method obtains the best of both results: few parameters to train resulting in low sample complexity, as well as polynomial computational complexity.

Original: Table 1 below compares the different methods, where we denote by | D | = max( ‖ 𝐵 ‖ 𝐹 ,
Revised: Table 1 below compares the different methods, where we denote by | D | = max( ‖ 𝐵 ‖ 𝐹 ,

Original: ‖ 𝐶 ‖ 𝐹 , ‖ 𝐷
Revised: ‖ 𝐶 ‖ 𝐹 , ‖ 𝐷

Original: ‖ 𝐹 , ‖ 𝑄 ‖ 𝐹 , 𝑚, 𝑛 ) the size of the system, and by 𝑇 the time horizon for planning.
Revised: ‖ 𝐹 , ‖ 𝑄 ‖ 𝐹 , 𝑚, 𝑛 ) the size of the system, and by 𝑇 the time horizon for planning.

Original: [Table]
Revised: [Table]

Original: System
Revised: System

Original: [Table]
Revised: [Table]

Original: of control algorithms.
Revised: of control algorithms.

Original: In Section 2 we state the precise algorithm.
Revised: In Section 2 we state the precise algorithm.

Original: The informal result is as follows.
Revised: The informal result is as follows.

Original: Theorem 1.1 (Controlling an unknown LDS; informal) .
Revised: Theorem 1.1 (Controlling an unknown LDS; informal) .

Original: Let D be a linear dynamical system with a symmetric transition matrix 𝐴 and with size | D | .
Revised: Let D be a linear dynamical system with a symmetric transition matrix 𝐴 and with size | D | .

Original: Then, for every 𝜀 > 0 , Algorithm 1 produces a sequence of controls (ˆ 𝑥 1 , . . .
Revised: Then, for every 𝜀 > 0 , Algorithm 1 produces a sequence of controls (ˆ 𝑥 1 , . . .

Original: ˆ 𝑥 𝑇 )
Revised: ˆ 𝑥 𝑇 )

Original: ∈ B 𝑇 

Revised: ∈ B 𝑇

Original: with ‖ ˆ 𝑥 1: 𝑡 ‖ 2 ≤ 𝐿 , such that
Revised: with ‖ ˆ 𝑥 1: 𝑡 ‖ 2 ≤ 𝐿 , such that

Original: [Equation]
Revised: [Equation]

Original: Assuming i.i.d.
Revised: Assuming i.i.d.

Original: Gaussian noise
Revised: Gaussian noise

Original: 𝜂 𝑡 ∼ 𝒩 (0 , Σ) , the algorithm samples ˜ 𝑂 (poly ( | D | , 𝐿, Tr Σ , 1 /𝜀 )) trajectories from D , and runs in time polynomial in the same parameters.
Revised: 𝜂 𝑡 ∼ 𝒩 (0 , Σ) , the algorithm samples ˜ 𝑂 (poly ( | D | , 𝐿, Tr Σ , 1 /𝜀 )) trajectories from D , and runs in time polynomial in the same parameters.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: The ﬁeld of optimal control for dynamical systems is extremely broad and brings together literature from machine learning, statistical time-series analysis, dynamical system tracking and Kalman ﬁltering, system identiﬁcation, and optimal control.
Revised: The ﬁeld of optimal control for dynamical systems is extremely broad and brings together literature from machine learning, statistical time-series analysis, dynamical system tracking and Kalman ﬁltering, system identiﬁcation, and optimal control.

Original: For an extensive survey of the ﬁeld, see e.g.
Revised: For an extensive survey of the ﬁeld, see e.g.

Original: (Todorov, 2006; Bertsekas, 2000).
Revised: (Todorov, 2006; Bertsekas, 2000).

Original: Tracking a known system.
Revised: Tracking a known system.

Original: A less ambitions goal than control is tracking of a dynamical system, or prediction of the output give a known input.
Revised: A less ambitions goal than control is tracking of a dynamical system, or prediction of the output give a known input.

Original: For the special case of LDS, the well-known
Revised: For the special case of LDS, the well-known

Original: Kalman ﬁlter (Kalman, 1960) is an optimal recursive least-squares solution for maximum likelihood estimation (MLE) under Gaussian perturbations to a linear dynamical system.
Revised: Kalman ﬁlter (Kalman, 1960) is an optimal recursive least-squares solution for maximum likelihood estimation (MLE) under Gaussian perturbations to a linear dynamical system.

Original: System identiﬁcation.
Revised: System identiﬁcation.

Original: When the underlying dynamical system is unknown, there are essentially no provably efﬁcient methods for recovering it.
Revised: When the underlying dynamical system is unknown, there are essentially no provably efﬁcient methods for recovering it.

Original: For various techniques used in practice, see the classic survey (Ljung, 1998).
Revised: For various techniques used in practice, see the classic survey (Ljung, 1998).

Original: Roweis & Ghahramani (1999) suggest using the EM algorithm to learn the parameters of an LDS, nowadays widespread, but it is well-known that optimality is not guaranteed.
Revised: Roweis & Ghahramani (1999) suggest using the EM algorithm to learn the parameters of an LDS, nowadays widespread, but it is well-known that optimality is not guaranteed.

Original: The recent result of Hardt et al.
Revised: The recent result of Hardt et al.

Original: (2016) gives a polynomial time algorithm for system recovery, although it applies only to the single-input-single-output case and makes various statistical assumptions on the inputs.
Revised: (2016) gives a polynomial time algorithm for system recovery, although it applies only to the single-input-single-output case and makes various statistical assumptions on the inputs.

Original: Model-free tracking.
Revised: Model-free tracking.

Original: Our methods depend crucially on a new algorithm for LDS sequence prediction, at the heart of which is a new convex relaxation for the tracking formulation (Hazan et al., 2017).
Revised: Our methods depend crucially on a new algorithm for LDS sequence prediction, at the heart of which is a new convex relaxation for the tracking formulation (Hazan et al., 2017).

Original: In particular, this method circumvent the obstacle of explicit system identiﬁcation.
Revised: In particular, this method circumvent the obstacle of explicit system identiﬁcation.

Original: We detail our usage of this result in Deﬁnition 2.3.
Revised: We detail our usage of this result in Deﬁnition 2.3.

Original: We note an intriguing connection to the recently widespread use of deep neural networks to represent an unknown MDP in reinforcement learning: the main algorithm queries the unknown dynamical system with exploration signals, and uses its responses to build a compact representation (denoted by ˆ 𝑀 in Algorithm 1) which estimates the behavior of the system.
Revised: We note an intriguing connection to the recently widespread use of deep neural networks to represent an unknown MDP in reinforcement learning: the main algorithm queries the unknown dynamical system with exploration signals, and uses its responses to build a compact representation (denoted by ˆ 𝑀 in Algorithm 1) which estimates the behavior of the system.

Original: Time-series analysis.
Revised: Time-series analysis.

Original: One of the most common approaches to modeling dynamical systems is the autoregressive-moving average (ARMA) model and its derivatives in the time-series analysis literature (Hamilton, 1994; Box et al., 1994; Brockwell & Davis, 2009).
Revised: One of the most common approaches to modeling dynamical systems is the autoregressive-moving average (ARMA) model and its derivatives in the time-series analysis literature (Hamilton, 1994; Box et al., 1994; Brockwell & Davis, 2009).

Original: At the heart of this method is the autoregressive form of a time series, namely,
Revised: At the heart of this method is the autoregressive form of a time series, namely,

Original: [Equation]
Revised: [Equation]

Original: Using online learning techniques, it is possible to completely identify an autoregressive model, even in the presence of adversarial noise (Anava et al., 2013).
Revised: Using online learning techniques, it is possible to completely identify an autoregressive model, even in the presence of adversarial noise (Anava et al., 2013).

Original: This technique lies at the heart of a folklore regression method for optimal control, given in the second row of table 1.
Revised: This technique lies at the heart of a folklore regression method for optimal control, given in the second row of table 1.

Original: Optimal control.
Revised: Optimal control.

Original: The most relevant and fundamental primitive from control theory, as applied to the control of linear dynamical systems, is the linear-quadratic-Gaussian (LQG) problem.
Revised: The most relevant and fundamental primitive from control theory, as applied to the control of linear dynamical systems, is the linear-quadratic-Gaussian (LQG) problem.

Original: In this setting, the system dynamics are assumed to be known, and the task is to ﬁnd a sequence of inputs which minimize a given quadratic cost.
Revised: In this setting, the system dynamics are assumed to be known, and the task is to ﬁnd a sequence of inputs which minimize a given quadratic cost.

Original: A common solution, the LQG controller, is to combine
Revised: A common solution, the LQG controller, is to combine

Original: Kalman ﬁltering with a linear-quadratic regulator , a controller selected by solving the Bellman equation for the problem.
Revised: Kalman ﬁltering with a linear-quadratic regulator , a controller selected by solving the Bellman equation for the problem.

Original: Such an approach admits theoretical guarantees under varied assumptions on the system; see, for example, Dean et al.
Revised: Such an approach admits theoretical guarantees under varied assumptions on the system; see, for example, Dean et al.

Original: Our setting also involves a linear dynamical system with quadratic costs, and thus can be seen as a special case of the LQG setup, in which the process noise is zero, and the transition matrix is assumed to be symmetrics.
Revised: Our setting also involves a linear dynamical system with quadratic costs, and thus can be seen as a special case of the LQG setup, in which the process noise is zero, and the transition matrix is assumed to be symmetrics.

Original: However, our results are not analogous: our task also includes learning the system’s dynamics.
Revised: However, our results are not analogous: our task also includes learning the system’s dynamics.

Original: As such, our main algorithm for control takes a very different approach than that of the standard LQR: rather than solving a recursive system of equations, we provide a formulation of control as a one-shot convex program.
Revised: As such, our main algorithm for control takes a very different approach than that of the standard LQR: rather than solving a recursive system of equations, we provide a formulation of control as a one-shot convex program.

Original:  S TATEMENT OF MAIN THEOREM
Revised:  S TATEMENT OF MAIN THEOREM

Original: First, we state the formal deﬁnitions of the key objects of interest.
Revised: First, we state the formal deﬁnitions of the key objects of interest.

Original: Deﬁnition 2.1 (Dynamical system) .
Revised: Deﬁnition 2.1 (Dynamical system) .

Original: A dynamical system D is a mapping that takes a sequence of input vectors 𝑥 1 , . . .
Revised: A dynamical system D is a mapping that takes a sequence of input vectors 𝑥 1 , . . .

Original: , 𝑥 𝑇 ∈ B 2 = { 𝑥 ∈ R 𝑛 : ‖ 𝑥 ‖ 2 ≤ 1 } to a sequence of output vectors 𝑦 1 , . . .
Revised: , 𝑥 𝑇 ∈ B 2 = { 𝑥 ∈ R 𝑛 : ‖ 𝑥 ‖ 2 ≤ 1 } to a sequence of output vectors 𝑦 1 , . . .

Original: , 𝑦 𝑇 ∈ R 𝑚 and costs 𝑐 1 , . .
Revised: , 𝑦 𝑇 ∈ R 𝑚 and costs 𝑐 1 , . .

Original: . , 𝑐 𝑇 ∈ R .
Revised: . , 𝑐 𝑇 ∈ R .

Original: Denote 𝑥 𝑠 : 𝑡 =
Revised: Denote 𝑥 𝑠 : 𝑡 =

Original: [ 𝑥 𝑠 ; . . .
Revised: [ 𝑥 𝑠 ; . . .

Original: ; 𝑥 𝑡 ] as the concatenation of all input vectors from time 𝑠 to 𝑡 , and write
Revised: ; 𝑥 𝑡 ] as the concatenation of all input vectors from time 𝑠 to 𝑡 , and write

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 2.2 (Linear dynamical system) .
Revised: Deﬁnition 2.2 (Linear dynamical system) .

Original: A linear dynamical system (LDS) is a dynamical system whose outputs and costs are deﬁned by
Revised: A linear dynamical system (LDS) is a dynamical system whose outputs and costs are deﬁned by

Original: [Equation]
Revised: [Equation]

Original: ∼ 𝒩
Revised: ∼ 𝒩

Original: [Equation]
Revised: [Equation]

Original: where ℎ 1 , . .
Revised: where ℎ 1 , . .

Original: , ℎ 𝑇 ∈ R 𝑑 is a sequence of hidden states starting with ﬁxed ℎ 1 , and 𝐴, 𝐵, 𝐶, 𝐷, 𝑄, 𝑅 are matrices (or vectors) of appropriate dimension.
Revised: , ℎ 𝑇 ∈ R 𝑑 is a sequence of hidden states starting with ﬁxed ℎ 1 , and 𝐴, 𝐵, 𝐶, 𝐷, 𝑄, 𝑅 are matrices (or vectors) of appropriate dimension.

Original: We assume ‖ 𝐴
Revised: We assume ‖ 𝐴

Original: ‖ op ≤ 1 , i.e., all singular values of 𝐴 are at most one, and that 𝑄 (cid:60) 0 , 𝑅 (cid:60) 0 .
Revised: ‖ op ≤ 1 , i.e., all singular values of 𝐴 are at most one, and that 𝑄 (cid:60) 0 , 𝑅 (cid:60) 0 .

Original: Our algorithm and its guarantees depend on the construction of a family of orthonormal vectors in R 𝑇 , which are interpreted as convolution ﬁlters on the input time series.
Revised: Our algorithm and its guarantees depend on the construction of a family of orthonormal vectors in R 𝑇 , which are interpreted as convolution ﬁlters on the input time series.

Original: We deﬁne the wave-ﬁltering matrix below; for more details, see Section 3 of Hazan et al.
Revised: We deﬁne the wave-ﬁltering matrix below; for more details, see Section 3 of Hazan et al.

Original: Deﬁnition 2.3 (Wave-ﬁltering matrix) .
Revised: Deﬁnition 2.3 (Wave-ﬁltering matrix) .

Original: Fix any 𝑛, 𝑇 , and 1 ≤ 𝑘 ≤ 𝑇 .
Revised: Fix any 𝑛, 𝑇 , and 1 ≤ 𝑘 ≤ 𝑇 .

Original: Let 𝜑 𝑗 be the eigenvector corresponding to the 𝑗 -th largest eigenvalue of the Hankel matrix
Revised: Let 𝜑 𝑗 be the eigenvector corresponding to the 𝑗 -th largest eigenvalue of the Hankel matrix

Original: 𝑍 𝑇 ∈ R 𝑇 × 𝑇 , with entries 𝑍 𝑖𝑗 := 2 ( 𝑖 + 𝑗 ) 3 − ( 𝑖 + 𝑗 ) .
Revised: 𝑍 𝑇 ∈ R 𝑇 × 𝑇 , with entries 𝑍 𝑖𝑗 := 2 ( 𝑖 + 𝑗 ) 3 − ( 𝑖 + 𝑗 ) .

Original: The wave-ﬁltering matrix Φ ∈ R 𝑛𝑘 × 𝑛𝑇 is deﬁned by 𝑘 vertically stacked block matrices { Φ ( 𝑗 ) ∈ R 𝑛 × 𝑛𝑇 } , deﬁned by horizontally stacked multiples of the identity matrix:
Revised: The wave-ﬁltering matrix Φ ∈ R 𝑛𝑘 × 𝑛𝑇 is deﬁned by 𝑘 vertically stacked block matrices { Φ ( 𝑗 ) ∈ R 𝑛 × 𝑛𝑇 } , deﬁned by horizontally stacked multiples of the identity matrix:

Original: [Equation]
Revised: [Equation]

Original: Then, letting 𝑡 range from 1 to 𝑇 , Φ 𝑋 𝑡 : 𝑡 − 𝑇 then gives a dimension-wise convolution of the input time series by the ﬁlters { 𝜑 𝑗 } of length 𝑇 .
Revised: Then, letting 𝑡 range from 1 to 𝑇 , Φ 𝑋 𝑡 : 𝑡 − 𝑇 then gives a dimension-wise convolution of the input time series by the ﬁlters { 𝜑 𝑗 } of length 𝑇 .

Original: Theorem 3.3 uses a structural result from Hazan et al.
Revised: Theorem 3.3 uses a structural result from Hazan et al.

Original: (2017), which guarantees the existence of a concise representation of D in the basis of these ﬁlters.
Revised: (2017), which guarantees the existence of a concise representation of D in the basis of these ﬁlters.

Original: The main theorem we prove is the following.
Revised: The main theorem we prove is the following.

Original: Theorem 2.4 (Controlling an unknown LDS) .
Revised: Theorem 2.4 (Controlling an unknown LDS) .

Original: Let D be a LDS with a symmetric transition matrix 𝐴 and with ‖ 𝐵 ‖ 𝐹 , ‖ 𝐶
Revised: Let D be a LDS with a symmetric transition matrix 𝐴 and with ‖ 𝐵 ‖ 𝐹 , ‖ 𝐶

Original: ‖ 𝐹 , ‖ 𝐷
Revised: ‖ 𝐹 , ‖ 𝐷

Original: ‖ 𝐹 , ‖ 𝑄 ‖ op ≤ 𝜌 .
Revised: ‖ 𝐹 , ‖ 𝑄 ‖ op ≤ 𝜌 .

Original: For every 𝜀 > 0 , Algorithm 1, with a choice of 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) and 𝑆 = Ω (cid:0) 𝐿 2 𝜌 2 𝑛𝑘 Tr(Σ) log( 𝐿/𝛿 ) 𝜀 − 2 (cid:1) , produces a sequence of controls (ˆ 𝑥 1 , . . .
Revised: For every 𝜀 > 0 , Algorithm 1, with a choice of 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) and 𝑆 = Ω (cid:0) 𝐿 2 𝜌 2 𝑛𝑘 Tr(Σ) log( 𝐿/𝛿 ) 𝜀 − 2 (cid:1) , produces a sequence of controls (ˆ 𝑥 1 , . . .

Original: ˆ 𝑥 𝑇 )
Revised: ˆ 𝑥 𝑇 )

Original: ∈ B 𝑇 2 , such that with probability at least 1 − 𝛿 ,
Revised: ∈ B 𝑇 2 , such that with probability at least 1 − 𝛿 ,

Original: [Equation]
Revised: [Equation]

Original: assuming that
Revised: assuming that

Original: [Equation]
Revised: [Equation]

Original: where 𝜆 is such that 𝑄
Revised: where 𝜆 is such that 𝑄

Original: [Equation]
Revised: [Equation]

Original: 𝜆𝐼 .
Revised: 𝜆𝐼 .

Original: Further, the algorithm samples poly (cid:0) 1 𝜀 , log (cid:0) 1 𝛿 (cid:1) , log 𝑇, log 𝜌, 𝐿, 𝑚, 𝑛, Tr(Σ) (cid:1) trajectories from the dynamical system, and runs in time polynomial in the same parameters.
Revised: Further, the algorithm samples poly (cid:0) 1 𝜀 , log (cid:0) 1 𝛿 (cid:1) , log 𝑇, log 𝜌, 𝐿, 𝑚, 𝑛, Tr(Σ) (cid:1) trajectories from the dynamical system, and runs in time polynomial in the same parameters.

Original: We remark on condition (3).
Revised: We remark on condition (3).

Original: Tr( 𝑄 Σ) is inevitable loss due to background noise, so (3) is an assumption on the system’s controllability.
Revised: Tr( 𝑄 Σ) is inevitable loss due to background noise, so (3) is an assumption on the system’s controllability.

Original: 𝜆 is bounded away from 0 when we suffer loss in all directions of 𝑦 𝑡 .
Revised: 𝜆 is bounded away from 0 when we suffer loss in all directions of 𝑦 𝑡 .

Original: We set up some notation for the algorithm.
Revised: We set up some notation for the algorithm.

Original: Let Φ ∈ R 𝑛𝑘
Revised: Let Φ ∈ R 𝑛𝑘

Original: ×
Revised: ×

Original: 𝑇 be the wave-ﬁltering matrix from Deﬁnition 2.3.
Revised: 𝑇 be the wave-ﬁltering matrix from Deﬁnition 2.3.

Original: Let 𝑥 𝑖 = 0 for 𝑖 ≤ 0 , and let 𝑋 𝑡 = 𝑥 𝑡 : 𝑡 − 𝑇 +1 .
Revised: Let 𝑥 𝑖 = 0 for 𝑖 ≤ 0 , and let 𝑋 𝑡 = 𝑥 𝑡 : 𝑡 − 𝑇 +1 .

Original: Let 𝜌 = max( ‖ 𝐵 ‖ 𝐹 ,
Revised: Let 𝜌 = max( ‖ 𝐵 ‖ 𝐹 ,

Original: ‖ 𝐶
Revised: ‖ 𝐶

Original: ‖ 𝐹 , ‖ 𝐷
Revised: ‖ 𝐹 , ‖ 𝐷

Original: ‖ 𝐹 , ‖ 𝑄 ‖ 𝐹 ) be an upper bound for the matrices involved in generating the outputs and costs.
Revised: ‖ 𝐹 , ‖ 𝑄 ‖ 𝐹 ) be an upper bound for the matrices involved in generating the outputs and costs.

Original:  P ROOF OF MAIN THEOREM
Revised:  P ROOF OF MAIN THEOREM

Original: To prove Theorem 2.4, we invoke Lemma 3.1 and Lemma 3.2, proved in Subsection 3.1 and Subsection 3.2, respectively.
Revised: To prove Theorem 2.4, we invoke Lemma 3.1 and Lemma 3.2, proved in Subsection 3.1 and Subsection 3.2, respectively.

Original: Lemma 3.1 (Learning dynamics) .
Revised: Lemma 3.1 (Learning dynamics) .

Original: For every 𝜀 > 0 and symmetric LDS D , with probability ≥ 1 − 𝛿 , setting 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) and 𝑆 = Ω Å 𝐿 2 𝑛𝑘 Tr(Σ)ln
Revised: For every 𝜀 > 0 and symmetric LDS D , with probability ≥ 1 − 𝛿 , setting 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) and 𝑆 = Ω Å 𝐿 2 𝑛𝑘 Tr(Σ)ln

Original: ( 𝑇𝛿 ) 𝜀ã , step 7 in Algorithm 1 computes a matrix ˆ 𝑀 ∈ R 𝑚 × 𝑛𝑘 such that for every sequence of input signals ( 𝑥 1 , . .
Revised: ( 𝑇𝛿 ) 𝜀ã , step 7 in Algorithm 1 computes a matrix ˆ 𝑀 ∈ R 𝑚 × 𝑛𝑘 such that for every sequence of input signals ( 𝑥 1 , . .

Original: . , 𝑥 𝑇 )
Revised: . , 𝑥 𝑇 )

Original: ∈
Revised: ∈

Original: B 𝑇 2 satisfying ‖ 𝑥 1: 𝑇 ‖ 2 ≤ 𝐿 and 1 ≤ 𝑡 ≤ 𝑇 , we have that
Revised: B 𝑇 2 satisfying ‖ 𝑥 1: 𝑇 ‖ 2 ≤ 𝐿 and 1 ≤ 𝑡 ≤ 𝑇 , we have that

Original: [Equation]
Revised: [Equation]

Original: 5) where ˆ 𝑦 𝑡 = ˆ 𝑀 Φ 𝑋 𝑡 + 𝑧 𝑡 and 𝑦 𝑡 = D 𝑡 𝑦 ( 𝑥 1: 𝑡 ) .
Revised: 5) where ˆ 𝑦 𝑡 = ˆ 𝑀 Φ 𝑋 𝑡 + 𝑧 𝑡 and 𝑦 𝑡 = D 𝑡 𝑦 ( 𝑥 1: 𝑡 ) .

Original: Average
Revised: Average

Original: 𝑆
Revised: 𝑆

Original: =1 𝑦
Revised: =1 𝑦

Original: 𝑇
Revised: 𝑇

Original: 7 Let ˆ 𝑀 be the matrix whose 𝑗 -th column is ˆ 𝑀 𝑗 := 𝑦 ( 𝑗 ) 𝑇 − 𝑧 𝑇 .
Revised: 7 Let ˆ 𝑀 be the matrix whose 𝑗 -th column is ˆ 𝑀 𝑗 := 𝑦 ( 𝑗 ) 𝑇 − 𝑧 𝑇 .

Original: 8 Solve the following convex program to obtain controls (ˆ 𝑥 1 , . . .
Revised: 8 Solve the following convex program to obtain controls (ˆ 𝑥 1 , . . .

Original: ˆ 𝑥 𝑇 ) :
Revised: ˆ 𝑥 𝑇 ) :

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.2 (Robustness of control to uncertainty in dynamics) .
Revised: Lemma 3.2 (Robustness of control to uncertainty in dynamics) .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: where ˆ 𝑀 ∈ R 𝑚 × 𝑛𝑘 is such that for every sequence of input signals ( 𝑥 1 , . .
Revised: where ˆ 𝑀 ∈ R 𝑚 × 𝑛𝑘 is such that for every sequence of input signals ( 𝑥 1 , . .

Original: . , 𝑥 𝑇 )
Revised: . , 𝑥 𝑇 )

Original: ∈ B 𝑇 

Revised: ∈ B 𝑇

Original: with ‖ 𝑥 1: 𝑇 ‖ 2 ≤ 𝐿 and 𝑡 ≤ 𝑇 , Equation (5) holds with ˆ 𝑦 𝑡
Revised: with ‖ 𝑥 1: 𝑇 ‖ 2 ≤ 𝐿 and 𝑡 ≤ 𝑇 , Equation (5) holds with ˆ 𝑦 𝑡

Original: = ˆ D 𝑡 ( 𝑥 1: 𝑡 ) .
Revised: = ˆ D 𝑡 ( 𝑥 1: 𝑡 ) .

Original: Assume (3) .
Revised: Assume (3) .

Original: Then it is true that
Revised: Then it is true that

Original: E (cid:34)
Revised: E (cid:34)

Original: [Equation]
Revised: [Equation]

Original: cost
Revised: cost

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Moreover, the minimization problem posed above is convex (for 𝑄, 𝑅 (cid:60) 0 ), and can be solved to within 𝜀 opt acccuracy in poly(log 𝑇, 𝜌, 𝜎, 1 /𝜀 opt ) time using the ellipsoid method.
Revised: Moreover, the minimization problem posed above is convex (for 𝑄, 𝑅 (cid:60) 0 ), and can be solved to within 𝜀 opt acccuracy in poly(log 𝑇, 𝜌, 𝜎, 1 /𝜀 opt ) time using the ellipsoid method.

Original: Proof of Theorem 2.4.
Revised: Proof of Theorem 2.4.

Original: Use Lemma 3.1 with 𝜀 ← (cid:91) 𝑂 Ä 𝜀𝜌 ä .
Revised: Use Lemma 3.1 with 𝜀 ← (cid:91) 𝑂 Ä 𝜀𝜌 ä .

Original: Note that 𝑥 1: 𝑇 = 𝜑 𝑗 is a valid input to the
Revised: Note that 𝑥 1: 𝑇 = 𝜑 𝑗 is a valid input to the

Original: LDS because ‖ 𝜑 𝑗
Revised: LDS because ‖ 𝜑 𝑗

Original: ‖ 2 = 1 .
Revised: ‖ 2 = 1 .

Original: Now use Lemma 3.2 on the conclusion of Lemma 3.1.
Revised: Now use Lemma 3.2 on the conclusion of Lemma 3.1.

Original: 3. L EARNING THE DYNAMICS
Revised: 3. L EARNING THE DYNAMICS

Original: To prove Lemma 3.1, we will use the following structural result from Hazan et al.
Revised: To prove Lemma 3.1, we will use the following structural result from Hazan et al.

Original: (2017) restated to match the setting in consideration.
Revised: (2017) restated to match the setting in consideration.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Let D be a symmetric LDS with the stated conditions from Section 2, with ﬁxed ℎ 1 .
Revised: Let D be a symmetric LDS with the stated conditions from Section 2, with ﬁxed ℎ 1 .

Original: Let 𝜀 > 0 , and let 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) .
Revised: Let 𝜀 > 0 , and let 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) .

Original: Let 𝑧 1: 𝑡 be the output if the input signal is 0 .
Revised: Let 𝑧 1: 𝑡 be the output if the input signal is 0 .

Original: Let 𝑀 ′ be the matrix such that for all input-output pairs ( 𝑥 1: 𝑇 , 𝑦 1: 𝑇 ) , E ( 𝑦 𝑇 − 𝑧 𝑇 ) =
Revised: Let 𝑀 ′ be the matrix such that for all input-output pairs ( 𝑥 1: 𝑇 , 𝑦 1: 𝑇 ) , E ( 𝑦 𝑇 − 𝑧 𝑇 ) =

Original: 𝑀 ′
Revised: 𝑀 ′

Original: 𝑋 𝑇 .
Revised: 𝑋 𝑇 .

Original: Then for all 𝑥 1: 𝑇 ∈ B ⊤ 2 , the matrix 𝑀 =
Revised: Then for all 𝑥 1: 𝑇 ∈ B ⊤ 2 , the matrix 𝑀 =

Original: 𝑀 ′
Revised: 𝑀 ′

Original: Φ ⊤ satisﬁes
Revised: Φ ⊤ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This follows from Theorem 3b in Hazan et al.
Revised: This follows from Theorem 3b in Hazan et al.

Original: (2017) after noting four things.
Revised: (2017) after noting four things.

Original: E ( 𝑦 𝑡 − 𝑧 𝑡 ) are the outputs when the system is started at ℎ 1 = 0 with inputs 𝑥 1: 𝑡 and no noise.
Revised: E ( 𝑦 𝑡 − 𝑧 𝑡 ) are the outputs when the system is started at ℎ 1 = 0 with inputs 𝑥 1: 𝑡 and no noise.

Original: A linear relationship 𝑦 𝑡 − 𝑧 𝑡 = 𝑀 ′
Revised: A linear relationship 𝑦 𝑡 − 𝑧 𝑡 = 𝑀 ′

Original: 𝑋 𝑡 holds by unfolding
Revised: 𝑋 𝑡 holds by unfolding

Original: the recurrence relation.
Revised: the recurrence relation.

Original: Examining the construction of 𝑀 in the proof of Theorem 3b, the 𝑀 is exactly the projection of 𝑀 ′ onto the subspace deﬁned by Φ .
Revised: Examining the construction of 𝑀 in the proof of Theorem 3b, the 𝑀 is exactly the projection of 𝑀 ′ onto the subspace deﬁned by Φ .

Original: (Note we are using the fact that the rows of Φ are orthogonal.)
Revised: (Note we are using the fact that the rows of Φ are orthogonal.)

Original: Theorem 3b is stated in terms of the quantity 𝐿 𝑦 , which is bounded by 𝜌 2 by Lemma F. in Hazan et al.
Revised: Theorem 3b is stated in terms of the quantity 𝐿 𝑦 , which is bounded by 𝜌 2 by Lemma F. in Hazan et al.

Original: We can modify
Revised: We can modify

Original: the
Revised: the

Original: system
Revised: system

Original: so
Revised: so

Original: that
Revised: that

Original: 𝐷
Revised: 𝐷

Original: = 𝑂 by replacing (
Revised: = 𝑂 by replacing (

Original: 𝐴, 𝐵, 𝐶, 𝐷
Revised: 𝐴, 𝐵, 𝐶, 𝐷

Original: with (( 𝐴 𝑂 𝑂 𝑂 ) , ( 𝐵𝐼 ) , ( 𝐶 𝐷 ) , 𝑂 ) .
Revised: with (( 𝐴 𝑂 𝑂 𝑂 ) , ( 𝐵𝐼 ) , ( 𝐶 𝐷 ) , 𝑂 ) .

Original: This replaces 𝜌 by 𝑂 (
Revised: This replaces 𝜌 by 𝑂 (

Original: max(
Revised: max(

Original: 𝜌, √
Revised: 𝜌, √

Original: 𝑛 )
Revised: 𝑛 )

Original: Theorem 3b
Revised: Theorem 3b

Original: originally had dependence of
Revised: originally had dependence of

Original: 𝑦
Revised: 𝑦

Original: 𝑡
Revised: 𝑡

Original: on both Φ 𝑋 𝑡 and 𝑥 𝑡 , but if 𝐷 = 𝑂 then the
Revised: on both Φ 𝑋 𝑡 and 𝑥 𝑡 , but if 𝐷 = 𝑂 then the

Original: dependence is only on Φ 𝑋 𝑡 .
Revised: dependence is only on Φ 𝑋 𝑡 .

Original: Proof of Lemma 3.1.
Revised: Proof of Lemma 3.1.

Original: Take 𝑆 =
Revised: Take 𝑆 =

Original: Ω Ä 𝑛𝑘𝐿 2 𝜀 2 Tr(Σ) log (cid:0)
Revised: Ω Ä 𝑛𝑘𝐿 2 𝜀 2 Tr(Σ) log (cid:0)

Original: 𝑇𝛿 (cid:1) ä .
Revised: 𝑇𝛿 (cid:1) ä .

Original: Letting 𝑀 ′ be the matrix such that 𝑦 𝑡 − 𝑧 𝑡 = 𝑀 ′ 𝑋 𝑡 , and 𝑀 =
Revised: Letting 𝑀 ′ be the matrix such that 𝑦 𝑡 − 𝑧 𝑡 = 𝑀 ′ 𝑋 𝑡 , and 𝑀 =

Original: 𝑀 ′
Revised: 𝑀 ′

Original: Φ ⊤ as in Theorem 3.3, we have that
Revised: Φ ⊤ as in Theorem 3.3, we have that

Original: [Equation]
Revised: [Equation]

Original: Let 1 ≤ 𝑡 ≤ 𝑇 .
Revised: Let 1 ≤ 𝑡 ≤ 𝑇 .

Original: We bound the error under controls 𝑥 1: 𝑡 ∈ B 𝑡 2 , ‖ 𝑥 1: 𝑡 ‖ 2 ≤ 𝐿 using the triangle inequality.
Revised: We bound the error under controls 𝑥 1: 𝑡 ∈ B 𝑡 2 , ‖ 𝑥 1: 𝑡 ‖ 2 ≤ 𝐿 using the triangle inequality.

Original: Letting 𝑦 1: 𝑡 be the output under 𝑥 1: 𝑡 ,
Revised: Letting 𝑦 1: 𝑡 be the output under 𝑥 1: 𝑡 ,

Original: [Equation]
Revised: [Equation]

Original: By Theorem 3.3, for 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) , choosing constants appropriately, the ﬁrst term is ≤ 𝜀 4 .
Revised: By Theorem 3.3, for 𝑘 = Ω(log 2 𝑇 log( 𝑛𝜌/𝜀 )) , choosing constants appropriately, the ﬁrst term is ≤ 𝜀 4 .

Original: To bound the second term in (10), we show 𝑧 𝑡 concentrates around E 𝑧 𝑡 .
Revised: To bound the second term in (10), we show 𝑧 𝑡 concentrates around E 𝑧 𝑡 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: By concentration of sums of 𝜒 2 random variables (see Hsu et al.
Revised: By concentration of sums of 𝜒 2 random variables (see Hsu et al.

Original: (2012), for example),
Revised: (2012), for example),

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: as long as 𝑆 ≥ 5 𝜀 ′ 2 Tr(Σ) log Å𝛿 ã .
Revised: as long as 𝑆 ≥ 5 𝜀 ′ 2 Tr(Σ) log Å𝛿 ã .

Original: Take 𝛿 ′ = 𝛿 2 𝑇 and 𝜀 ′ =
Revised: Take 𝛿 ′ = 𝛿 2 𝑇 and 𝜀 ′ =

Original: 𝜀 4 𝐿 √ 𝑛𝑘 and note 𝑆 was chosen to satisfy the above.
Revised: 𝜀 4 𝐿 √ 𝑛𝑘 and note 𝑆 was chosen to satisfy the above.

Original: Use the union bound to get that
Revised: Use the union bound to get that

Original: [Equation]
Revised: [Equation]

Original: To bound the third term in (10), we ﬁrst show that ˆ 𝑀 concentrates around 𝑀 .
Revised: To bound the third term in (10), we ﬁrst show that ˆ 𝑀 concentrates around 𝑀 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: (14) so ˆ 𝑀
Revised: (14) so ˆ 𝑀

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By 𝜒 2 concentration,
Revised: By 𝜒 2 concentration,

Original: [Equation]
Revised: [Equation]

Original: We also have (cid:13)(cid:13)
Revised: We also have (cid:13)(cid:13)

Original: ( E 𝑧 𝑇 − 𝑧 𝑇 ) 1 ⊤ (cid:13)(cid:13)
Revised: ( E 𝑧 𝑇 − 𝑧 𝑇 ) 1 ⊤ (cid:13)(cid:13)

Original: 𝐹 ≤ √ 𝑛𝑘 ‖ E 𝑧 𝑇 − 𝑧 𝑇
Revised: 𝐹 ≤ √ 𝑛𝑘 ‖ E 𝑧 𝑇 − 𝑧 𝑇

Original: With ≥ 1 − 𝛿 probability, we avoid both the bad events in (13) and (18) so
Revised: With ≥ 1 − 𝛿 probability, we avoid both the bad events in (13) and (18) so

Original: ‖ 𝑧 𝑡 − E 𝑧 𝑡 ‖ 2 ≤
Revised: ‖ 𝑧 𝑡 − E 𝑧 𝑡 ‖ 2 ≤

Original: [Equation]
Revised: [Equation]

Original: √ 𝑛𝑘 for all 1 ≤ 𝑡 ≤ 𝑇 and
Revised: √ 𝑛𝑘 for all 1 ≤ 𝑡 ≤ 𝑇 and

Original: [Equation]
Revised: [Equation]

Original: and for all ‖ 𝑥 1: 𝑡 ‖ ≤ 𝐿 , the third term of (10) is bounded by (note ‖ Φ ‖ op = 1 because it has orthogonal rows)
Revised: and for all ‖ 𝑥 1: 𝑡 ‖ ≤ 𝐿 , the third term of (10) is bounded by (note ‖ Φ ‖ op = 1 because it has orthogonal rows)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus by (10), (cid:13)(cid:13)(cid:13) E 𝑦 𝑡 − 𝑧 𝑡 − ˆ 𝑀 Φ 𝑋 𝑡 (cid:13)(cid:13)(cid:13) 2 ≤ 𝜀 with probability ≥ 1 − 𝛿 .
Revised: Thus by (10), (cid:13)(cid:13)(cid:13) E 𝑦 𝑡 − 𝑧 𝑡 − ˆ 𝑀 Φ 𝑋 𝑡 (cid:13)(cid:13)(cid:13) 2 ≤ 𝜀 with probability ≥ 1 − 𝛿 .

Original: 3. R OBUSTNESS OF C ONTROL TO U NCERTAINTY IN D YNAMICS
Revised: 3. R OBUSTNESS OF C ONTROL TO U NCERTAINTY IN D YNAMICS

Original: To prove Lemma 3.2, we need the following helpful lemma.
Revised: To prove Lemma 3.2, we need the following helpful lemma.

Original: Lemma 3.4.
Revised: Lemma 3.4.

Original: For a symmetric LDS D with 𝜌 = max( ‖ 𝐵 ‖ 𝐹 , ‖ 𝐶 ‖ 𝐹 , ‖ 𝐷 ‖ 𝐹 , ‖ 𝑃 ‖ op ) and 𝐴 (cid:52) 𝐼 , 𝑄 (cid:60) 𝜆𝐼 , and an approximation ˆ D 𝑡 whose predictions ˆ 𝑦 𝑡 satisfy the conclusion of Lemma 3.1, we have that for every sequence of controls ( 𝑥 1 , . . .
Revised: For a symmetric LDS D with 𝜌 = max( ‖ 𝐵 ‖ 𝐹 , ‖ 𝐶 ‖ 𝐹 , ‖ 𝐷 ‖ 𝐹 , ‖ 𝑃 ‖ op ) and 𝐴 (cid:52) 𝐼 , 𝑄 (cid:60) 𝜆𝐼 , and an approximation ˆ D 𝑡 whose predictions ˆ 𝑦 𝑡 satisfy the conclusion of Lemma 3.1, we have that for every sequence of controls ( 𝑥 1 , . . .

Original: , 𝑥 𝑇 )
Revised: , 𝑥 𝑇 )

Original: ∈ B 𝑇 2 , and for every 1 ≤ 𝑡 ≤ 𝑇 ,
Revised: ∈ B 𝑇 2 , and for every 1 ≤ 𝑡 ≤ 𝑇 ,

Original: [Equation]
Revised: [Equation]

Original: 𝑇 (cid:88)
Revised: 𝑇 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Ã
Revised: Ã

Original: [Equation]
Revised: [Equation]

Original: + 𝑇𝜌𝜀
Revised: + 𝑇𝜌𝜀

Original: [Equation]
Revised: [Equation]

Original: (23) where cost 𝑡 ^ D ( 𝑥 1: 𝑡 ) =
Revised: (23) where cost 𝑡 ^ D ( 𝑥 1: 𝑡 ) =

Original: ˆ D 𝑡 ( 𝑥 1: 𝑡 ) ⊤ 𝑄 ˆ D 𝑡 ( 𝑥 1: 𝑡 ) + 𝑥 ⊤ 𝑡 𝑅𝑥 𝑡 .
Revised: ˆ D 𝑡 ( 𝑥 1: 𝑡 ) ⊤ 𝑄 ˆ D 𝑡 ( 𝑥 1: 𝑡 ) + 𝑥 ⊤ 𝑡 𝑅𝑥 𝑡 .

Original: Proof of Lemma 3.4.
Revised: Proof of Lemma 3.4.

Original: Let ˆ 𝑦 𝑡 = ˆ D 𝑦 ( 𝑥 1: 𝑡 ) and 𝑦 𝑡 = D
Revised: Let ˆ 𝑦 𝑡 = ˆ D 𝑦 ( 𝑥 1: 𝑡 ) and 𝑦 𝑡 = D

Original: 𝑦
Revised: 𝑦

Original: (ˆ 𝑥 1: 𝑡 ) =
Revised: (ˆ 𝑥 1: 𝑡 ) =

Original: E [ 𝑦 𝑡 ] + 𝜂 𝑡 .
Revised: E [ 𝑦 𝑡 ] + 𝜂 𝑡 .

Original: Using the assumption that E [ 𝜂 𝑡 𝜂 ⊤ 𝑡 ] = Σ ,
Revised: Using the assumption that E [ 𝜂 𝑡 𝜂 ⊤ 𝑡 ] = Σ ,

Original: [Equation]
Revised: [Equation]

Original: (24) =
Revised: (24) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: using Cauchy-Schwarz
Revised: using Cauchy-Schwarz

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 3.2.
Revised: Proof of Lemma 3.2.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Letting 𝑉 =
Revised: Letting 𝑉 =

Original: (cid:80) 𝑇𝑡 =

Revised: (cid:80) 𝑇𝑡 =

Original: [ E [ cost 𝑡 D ( 𝑥 * 1: 𝑡 )] − Tr( 𝑄 Σ)] , for inputs 𝑥 * 1: 𝑡 .
Revised: [ E [ cost 𝑡 D ( 𝑥 * 1: 𝑡 )] − Tr( 𝑄 Σ)] , for inputs 𝑥 * 1: 𝑡 .

Original: By assumption (3), 𝑉 = 𝑜 ( 𝑇𝜆 ) .
Revised: By assumption (3), 𝑉 = 𝑜 ( 𝑇𝜆 ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Now by Lemma 3.4,
Revised: Now by Lemma 3.4,

Original: [Equation]
Revised: [Equation]

Original:  Letting ˆ 𝑦 1: 𝑇 be the outputs under ˆ D under the control ˆ 𝑥 1: 𝑇 , note that similar to (32),[Equation]
Revised: [Equation] Letting ˆ 𝑦 1: 𝑇 be the outputs under ˆ D under the control ˆ 𝑥 1: 𝑇 , note that similar to (32),

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 3.4,
Revised: Using Lemma 3.4,

Original: [Equation]
Revised: [Equation]

Original: Now 
Revised: Now cost

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The assumptionon 𝑉  gives the lemma.
Revised: The assumption 𝑉 = 𝑜 ( 𝑇𝜆 ) gives the lemma.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented an algorithm for ﬁnding the optimal control inputs for an unknown symmetric linear dynamical system, which requires querying the system only a polylogarithmic number of times in the number of such inputs 𝑇 , while running in polynomial time.
Revised: We have presented an algorithm for ﬁnding the optimal control inputs for an unknown symmetric linear dynamical system, which requires querying the system only a polylogarithmic number of times in the number of such inputs 𝑇 , while running in polynomial time.

Original: Deviating signiﬁcantly from previous approaches, we circumvent the non-convex optimization problem of system identiﬁcation by a new learned representation of the system.
Revised: Deviating signiﬁcantly from previous approaches, we circumvent the non-convex optimization problem of system identiﬁcation by a new learned representation of the system.

Original: We see this as a ﬁrst step towards provable, efﬁcient methods for the traditionally non-convex realm of control and reinforcement learning.
Revised: We see this as a ﬁrst step towards provable, efﬁcient methods for the traditionally non-convex realm of control and reinforcement learning

Original: A A UTOREGRESSIVE M ODELS AS A R ELAXATION OF LDS
Revised: A A UTOREGRESSIVE M ODELS AS A R ELAXATION OF LDS

Original: In this section, we verify the statement made in Section 1 on the time and sample complexity of approximating a linear dynamical system with an autoregressive model.
Revised: In this section, we verify the statement made in Section 1 on the time and sample complexity of approximating a linear dynamical system with an autoregressive model.

Original: Although this is wellknown (see, for example, Section 6 of Hardt et al.
Revised: Although this is wellknown (see, for example, Section 6 of Hardt et al.

Original: , we present a self-contained presentation for convenience and to unify notation.
Revised: , we present a self-contained presentation for convenience and to unify notation.

Original: The vector autoregressive model with exogenous variables , or VARX( 𝑝, 𝑠 ) , is a touchstone in timeseries analysis.
Revised: The vector autoregressive model with exogenous variables , or VARX( 𝑝, 𝑠 ) , is a touchstone in timeseries analysis.

Original: Given a time series of inputs (sometimes known as biases) { 𝜉 𝑡 } , it generates the time series of responses { 𝑦 𝑡 } by the following recurrence:
Revised: Given a time series of inputs (sometimes known as biases) { 𝜉 𝑡 } , it generates the time series of responses { 𝑦 𝑡 } by the following recurrence:

Original: [Equation]
Revised: [Equation]

Original: Here, 𝑝 and 𝑠 are memory parameters, the { 𝐴 𝑖 } and { 𝐵 𝑖 } are matrices of appropriate dimension, and the { 𝜉 𝑡 } are noise vectors.
Revised: Here, 𝑝 and 𝑠 are memory parameters, the { 𝐴 𝑖 } and { 𝐵 𝑖 } are matrices of appropriate dimension, and the { 𝜉 𝑡 } are noise vectors.

Original: In the special case of 𝑝 = 0 , the problem can be solved efﬁciently with linear regression: in this case, 𝑦 𝑡 is a linear function of the concatenated inputs [ 𝑥 𝑡 ; 𝑥 𝑡 − 1 ; . .
Revised: In the special case of 𝑝 = 0 , the problem can be solved efﬁciently with linear regression: in this case, 𝑦 𝑡 is a linear function of the concatenated inputs [ 𝑥 𝑡 ; 𝑥 𝑡 − 1 ; . .

Original: , 𝑥 𝑡 − 𝑠 +1 ] .
Revised: , 𝑥 𝑡 − 𝑠 +1 ] .

Original: A VARX(0 , 𝑠 ) model is speciﬁed by 𝑀 =
Revised: A VARX(0 , 𝑠 ) model is speciﬁed by 𝑀 =

Original: [ 𝑀 (0) , . . .
Revised: [ 𝑀 (0) , . . .

Original: , 𝑀 ( 𝑠 − 1) ]
Revised: , 𝑀 ( 𝑠 − 1) ]

Original: ∈ R 𝑚 × 𝑛𝑠 and predicts 𝑦 𝑡 =
Revised: ∈ R 𝑚 × 𝑛𝑠 and predicts 𝑦 𝑡 =

Original: 𝑀𝑥 𝑡 : 𝑡 − 𝑠 +1 .
Revised: 𝑀𝑥 𝑡 : 𝑡 − 𝑠 +1 .

Original: We quantify the relationship between VARX(0 , 𝑠 ) and linear dynamical systems, with a statement analogous to Theorem 3.1:
Revised: We quantify the relationship between VARX(0 , 𝑠 ) and linear dynamical systems, with a statement analogous to Theorem 3.1:

Original: Theorem A.1.
Revised: Theorem A.1.

Original: Let D be an LDS with size ℒ , ﬁxed ℎ 1 , and noise 𝜂 𝑡 = 0 , producing outputs { 𝑦 1 , . .
Revised: Let D be an LDS with size ℒ , ﬁxed ℎ 1 , and noise 𝜂 𝑡 = 0 , producing outputs { 𝑦 1 , . .

Original: . , 𝑦 𝑇 } from inputs { 𝑥 1 , . .
Revised: . , 𝑦 𝑇 } from inputs { 𝑥 1 , . .

Original: . , 𝑥 𝑇 } .
Revised: . , 𝑥 𝑇 } .

Original: Suppose that the transition matrix of D has operator norm at most 𝛼 < 1 .
Revised: Suppose that the transition matrix of D has operator norm at most 𝛼 < 1 .

Original: Then, for each 𝜀 > 0 , there is a VARX(0 , 𝑠 ) model with 𝑠 = 𝑂 (
Revised: Then, for each 𝜀 > 0 , there is a VARX(0 , 𝑠 ) model with 𝑠 = 𝑂 (

Original: [Equation]
Revised: [Equation]

Original: − 𝛼 log( ℒ /𝜀 )) , speciﬁed by a matrix 𝑀 , such that
Revised: − 𝛼 log( ℒ /𝜀 )) , speciﬁed by a matrix 𝑀 , such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the modiﬁcation of D given in the proof of Theorem 3.3, we may assume without loss of generality that 𝐷 = 𝑂 .
Revised: By the modiﬁcation of D given in the proof of Theorem 3.3, we may assume without loss of generality that 𝐷 = 𝑂 .

Original: Also, as in the discussion of Theorem 3.1, it can be assumed that the initial hidden state ℎ 1 is zero.
Revised: Also, as in the discussion of Theorem 3.1, it can be assumed that the initial hidden state ℎ 1 is zero.

Original: Then, we construct the block of 𝑀 corresponding to lag 𝑖 as
Revised: Then, we construct the block of 𝑀 corresponding to lag 𝑖 as

Original: [Equation]
Revised: [Equation]

Original: This is well-deﬁned for all 1 ≤ 𝑖 ≤ 𝑡 .
Revised: This is well-deﬁned for all 1 ≤ 𝑖 ≤ 𝑡 .

Original: Note that when 𝑠 ≥ 𝑡 , the autoregressive model completely speciﬁes the system D , which is determined by its (inﬁnite-horizon) impulse response function.
Revised: Note that when 𝑠 ≥ 𝑡 , the autoregressive model completely speciﬁes the system D , which is determined by its (inﬁnite-horizon) impulse response function.

Original: Furthermore, by deﬁnition of 𝛼 , we have
Revised: Furthermore, by deﬁnition of 𝛼 , we have

Original: [Equation]
Revised: [Equation]

Original: Noting that
Revised: Noting that

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: implying the claim by the stated choice of 𝑠 .
Revised: implying the claim by the stated choice of 𝑠 .

Original: VARX(0 , 𝑠 ) only serves as a good approximation of an LDS whose hidden state decays on a time scale shorter than 𝑠 ; when the system is ill-conditioned ( 𝛼 is close to 1), this can get arbitrarily large, requiring the full time horizon 𝑠
Revised: VARX(0 , 𝑠 ) only serves as a good approximation of an LDS whose hidden state decays on a time scale shorter than 𝑠 ; when the system is ill-conditioned ( 𝛼 is close to 1), this can get arbitrarily large, requiring the full time horizon 𝑠

Original: = 𝑇 .
Revised: = 𝑇 .

Original: On the other hand, it is clear that both the time and sample complexity of learning a VARX(0 , 𝑠 ) model grows linearly in 𝑠 .
Revised: On the other hand, it is clear that both the time and sample complexity of learning a VARX(0 , 𝑠 ) model grows linearly in 𝑠 .

Original: This veriﬁes the claim in the introduction.
Revised: This veriﬁes the claim in the introduction.