Original: A DYNAMIC GAME APPROACH TO TRAINING ROBUST DEEP POLICIES .
Revised: A DYNAMIC GAME APPROACH TO TRAINING ROBUST DEEP POLICIES .

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies.
Revised: We present a method for evaluating the sensitivity of deep reinforcement learning (RL) policies.

Original: We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies.
Revised: We also formulate a zero-sum dynamic game for designing robust deep reinforcement learning policies.

Original: Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies.
Revised: Our approach mitigates the brittleness of policies when agents are trained in a simulated environment and are later exposed to the real world where it is hazardous to employ RL policies.

Original: This framework for training deep RL policies involve a zero-sum dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region.
Revised: This framework for training deep RL policies involve a zero-sum dynamic game against an adversarial agent, where the goal is to drive the system dynamics to a saddle region.

Original: Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm.
Revised: Using a variant of the guided policy search algorithm, our agent learns to adopt robust policies that require less samples for learning the dynamics and performs better than the GPS algorithm.

Original: Without loss of generality, we demonstrate that deep
Revised: Without loss of generality, we demonstrate that deep

Original: RL policies trained in this fashion will be maximally robust to a “worst” possible adversarial disturbances
Revised: RL policies trained in this fashion will be maximally robust to a “worst” possible adversarial disturbances

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (RL) for complex agent behavior in realistic environments usually combines function approximation techniques with learning-based control.
Revised: Deep reinforcement learning (RL) for complex agent behavior in realistic environments usually combines function approximation techniques with learning-based control.

Original: A good RL controller should guarantee fulﬁllment of performance speciﬁcations under external disturbances, or modeling errors.
Revised: A good RL controller should guarantee fulﬁllment of performance speciﬁcations under external disturbances, or modeling errors.

Original: Quite often in practice, however, this is not the case – with deep RL policies not often generalizing well to real-world scenarios.
Revised: Quite often in practice, however, this is not the case – with deep RL policies not often generalizing well to real-world scenarios.

Original: This can be attributed to the inherent differences between the training and testing environments.
Revised: This can be attributed to the inherent differences between the training and testing environments.

Original: Recently, there have been efforts at integrating function approximation techniques with learning-based control, in an end-to-end fashion, in order to have systems that optimize objectives while guaranteeing generalization to environmental uncertainties.
Revised: Recently, there have been efforts at integrating function approximation techniques with learning-based control, in an end-to-end fashion, in order to have systems that optimize objectives while guaranteeing generalization to environmental uncertainties.

Original: Examples include trajectory-based optimization for known dynamics ([17, 26]), or trajectory optimization for unknown dynamics such as guided policy search algorithms [1, 14, 16].
Revised: Examples include trajectory-based optimization for known dynamics ([17, 26]), or trajectory optimization for unknown dynamics such as guided policy search algorithms [1, 14, 16].

Original: While these methods produce performance efﬁciency for agent tasks in the real world, there are sensitivity questions of such policies that need to be addressed such as, how to guarantee maximally robust deep RL policies in the presence of external disturbances, or modeling errors.
Revised: While these methods produce performance efﬁciency for agent tasks in the real world, there are sensitivity questions of such policies that need to be addressed such as, how to guarantee maximally robust deep RL policies in the presence of external disturbances, or modeling errors.

Original: A typical approach employed in minimizing sample inefﬁciency is to engineer an agent’s policy in a simulated environment, and later transfer such policies to physical environments.
Revised: A typical approach employed in minimizing sample inefﬁciency is to engineer an agent’s policy in a simulated environment, and later transfer such policies to physical environments.

Original: However, questions of robustness persist in such scenarios as the agent often has to cope with modeling errors and new sensory inputs from a different environment.
Revised: However, questions of robustness persist in such scenarios as the agent often has to cope with modeling errors and new sensory inputs from a different environment.

Original: For continuous control tasks, learned policies may become brittle in the presence of external perturbations, or a slight change in the system dynamics may signiﬁcantly affect the performance of the learned controller [21] – defeating the purpose of having a robust policy that is learned through environmental interaction .
Revised: For continuous control tasks, learned policies may become brittle in the presence of external perturbations, or a slight change in the system dynamics may signiﬁcantly affect the performance of the learned controller [21] – defeating the purpose of having a robust policy that is learned through environmental interaction .

Original: The contribution of this paper is two-fold:
Revised: The contribution of this paper is two-fold:

Original: • ﬁrst, we provide a framework that demonstrates the brittleness of a state-of-the-art deep RL policy; speciﬁcally, given a trained RL policy, we pose an adversarial agent against the ﬁxed trained policy; the goal is to perturb the parameter space of the learned policy.
Revised: • ﬁrst, we provide a framework that demonstrates the brittleness of a state-of-the-art deep RL policy; speciﬁcally, given a trained RL policy, we pose an adversarial agent against the ﬁxed trained policy; the goal is to perturb the parameter space of the learned policy.

Original: We demonstrate that the most sophisticated deep policies fail in the presence of adversarial perturbations.
Revised: We demonstrate that the most sophisticated deep policies fail in the presence of adversarial perturbations.

Original: • second, we formulate an iterative dynamic zero-sum, two player game, where each agent executes an opposite reaction to its pair: a concave-convex problem follows explicitly
Revised: • second, we formulate an iterative dynamic zero-sum, two player game, where each agent executes an opposite reaction to its pair: a concave-convex problem follows explicitly

Original: and
Revised: and

Original: our goal is to achieve a saddle point equilibrium, where the state is everywhere deﬁned but possibly inﬁnite-valued).
Revised: our goal is to achieve a saddle point equilibrium, where the state is everywhere deﬁned but possibly inﬁnite-valued).

Original: Noting that lack of generalization of learned reward functions to the real-world can be thought of as external disturbance that perturb the system dynamics, we formulate the learning of robust control policies as a zero-sum two player Markov game – an iterative dynamic game (iDG) – that pits an adversarial agent against a protagonist agent.
Revised: Noting that lack of generalization of learned reward functions to the real-world can be thought of as external disturbance that perturb the system dynamics, we formulate the learning of robust control policies as a zero-sum two player Markov game – an iterative dynamic game (iDG) – that pits an adversarial agent against a protagonist agent.

Original: The controller aims to minimize a given cost while the second agent, an adversary aims to maximize the given cost in the presence of an additive disturbance.
Revised: The controller aims to minimize a given cost while the second agent, an adversary aims to maximize the given cost in the presence of an additive disturbance.

Original: We run the algorithm in ﬁnite episodic settings and show a dynamic game approach aimed at generating policies that are maximally robust.
Revised: We run the algorithm in ﬁnite episodic settings and show a dynamic game approach aimed at generating policies that are maximally robust.

Original: The content of this paper is thus organized: we review relevant literature to our contribution in
Revised: The content of this paper is thus organized: we review relevant literature to our contribution in

Original: Sec. 2; we then provide an H ∞ background in Sec. 3.
Revised: Sec. 2; we then provide an H ∞ background in Sec. 3.

Original: This H ∞ technical introduction will be used in formulating the design of perturbation signals in Sec. 4.
Revised: This H ∞ technical introduction will be used in formulating the design of perturbation signals in Sec. 4.

Original: Without loss of generality, we provide a formal treatment of the iDG algorithm within the guided policy search framework in
Revised: Without loss of generality, we provide a formal treatment of the iDG algorithm within the guided policy search framework in

Original: Sec. 5.
Revised: Sec. 5.

Original: Experimental evaluation on multiple robots is provided in Sec. 6 followed by conclusions in
Revised: Experimental evaluation on multiple robots is provided in Sec. 6 followed by conclusions in

Original: Sec. 7.
Revised: Sec. 7.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Robustness studies in classical control have witnessed the formalization of algorithms and computation necessary to carry out stable feedback control and dynamic game tasks (e.g. [2, 15, 19]).
Revised: Robustness studies in classical control have witnessed the formalization of algorithms and computation necessary to carry out stable feedback control and dynamic game tasks (e.g. [2, 15, 19]).

Original: There now exist closed-form and iterative-based algorithms to quantify the sensitivity of a control system and design robust feedback controllers.
Revised: There now exist closed-form and iterative-based algorithms to quantify the sensitivity of a control system and design robust feedback controllers.

Original: These methods are well-studied in classical H ∞ control theory.
Revised: These methods are well-studied in classical H ∞ control theory.

Original: While questions of robustness of policies have existed for long in connectionist RL settings[25], only recently have researchers started addressing the question of incorporating robustness guarantees into deep RL controllers.
Revised: While questions of robustness of policies have existed for long in connectionist RL settings[25], only recently have researchers started addressing the question of incorporating robustness guarantees into deep RL controllers.

Original: Heess et.
Revised: Heess et.

Original: al [9] posit that rich, robust performance will emerge if an agent is simulated in a sufﬁciently rich and diverse environment.
Revised: al [9] posit that rich, robust performance will emerge if an agent is simulated in a sufﬁciently rich and diverse environment.

Original: [9] proposed a learning framework for agents in locomotion tasks which involved choosing simple reward functions, but exposing the agent to various levels of difﬁcult environments as a way of achieving ostensibly sophisticated performance objectives.
Revised: [9] proposed a learning framework for agents in locomotion tasks which involved choosing simple reward functions, but exposing the agent to various levels of difﬁcult environments as a way of achieving ostensibly sophisticated performance objectives.

Original: Incorporating various levels of difﬁculty in obstacles, height and terrain smoothness to an agent’s environment for every episodic task, they achieved robust behaviors for difﬁcult locomotion tasks after many episodes ( ≈ 10 6 ) of training.
Revised: Incorporating various levels of difﬁculty in obstacles, height and terrain smoothness to an agent’s environment for every episodic task, they achieved robust behaviors for difﬁcult locomotion tasks after many episodes ( ≈ 10 6 ) of training.

Original: However, this strategy defeats one of the primary objectives of RL namely, to make an agent discover good policies with ﬁnite data based on little interaction with the environment.
Revised: However, this strategy defeats one of the primary objectives of RL namely, to make an agent discover good policies with ﬁnite data based on little interaction with the environment.

Original: An ideal robust RL controller must come from data-efﬁcient samples or imitations.
Revised: An ideal robust RL controller must come from data-efﬁcient samples or imitations.

Original: Furthermore, this approach takes a qualitative measure at building robust signals into the reward function via means such as locomotion hurdles with variations in height, slopes, and slalom walls.
Revised: Furthermore, this approach takes a qualitative measure at building robust signals into the reward function via means such as locomotion hurdles with variations in height, slopes, and slalom walls.

Original: We reckon that building such physical barriers for an agent is expensive in the real-world and learning such emergent locomotion behaviors takes a long training time.
Revised: We reckon that building such physical barriers for an agent is expensive in the real-world and learning such emergent locomotion behaviors takes a long training time.

Original: Pinto et.
Revised: Pinto et.

Original: al.
Revised: al.

Original: [20], posed the learning of robust RL rewards in a zero-sum, two-player markov decision process (MDP) deﬁned by the standard RL tuple {S , A 1 , A 2 , P , R , γ, s 0 } , where A 1 and A 2 denote the continuous action spaces of the two players.
Revised: [20], posed the learning of robust RL rewards in a zero-sum, two-player markov decision process (MDP) deﬁned by the standard RL tuple {S , A 1 , A 2 , P , R , γ, s 0 } , where A 1 and A 2 denote the continuous action spaces of the two players.

Original: Both players share a joint transition probability
Revised: Both players share a joint transition probability

Original: P and reward R .
Revised: P and reward R .

Original: Pinto’s approach assumed a knowledge of the underlying dynamics so that an adversarial policy, π adv θ ( u t | x t ) , can exploit the weakness in a protagonist agent’s policy, π prot θ ( u t | x t ) .
Revised: Pinto’s approach assumed a knowledge of the underlying dynamics so that an adversarial policy, π adv θ ( u t | x t ) , can exploit the weakness in a protagonist agent’s policy, π prot θ ( u t | x t ) .

Original: This relied on a minimax alternating optimization process: optimizing for one set of actions while holding the other ﬁxed, to the end of ensuring robustness of the learned reward function.
Revised: This relied on a minimax alternating optimization process: optimizing for one set of actions while holding the other ﬁxed, to the end of ensuring robustness of the learned reward function.

Original: While it introduced H ∞ control as a robustness measure for classical RL problems, it falls short of adapting
Revised: While it introduced H ∞ control as a robustness measure for classical RL problems, it falls short of adapting

Original: H ∞ for complex agent tasks and policy optimizations.
Revised: H ∞ for complex agent tasks and policy optimizations.

Original: Moreover, there are no theoretical analyses of saddle-/pareto-point or Nash equilibrium guarantees and the global optimum that assures maximal robustness at π prot θ ( u t | x t ) =
Revised: Moreover, there are no theoretical analyses of saddle-/pareto-point or Nash equilibrium guarantees and the global optimum that assures maximal robustness at π prot θ ( u t | x t ) =

Original: π adv θ ( u t | x t ) is left unaddressed.
Revised: π adv θ ( u t | x t ) is left unaddressed.

Original: Perhaps the closest formulation to this work is [10]’s neural ﬁctitious self-play for large games with imperfect state information whereby players select best responses to their opponents’ average strategies.
Revised: Perhaps the closest formulation to this work is [10]’s neural ﬁctitious self-play for large games with imperfect state information whereby players select best responses to their opponents’ average strategies.

Original: [10] showed that with deep reinforcement learning, self-play in imperfect-information environments approached a Nash equilibrium where other reinforcement learning methods diverged.
Revised: [10] showed that with deep reinforcement learning, self-play in imperfect-information environments approached a Nash equilibrium where other reinforcement learning methods diverged.

Original: From a methodical perspective, we formulate the robustness of RL controllers within an H ∞ framework (see [19]) for deep robot motor tasks.
Revised: From a methodical perspective, we formulate the robustness of RL controllers within an H ∞ framework (see [19]) for deep robot motor tasks.

Original: Similar to a matrix game with two agents, we let both agents play a zero-sum, two person game where each agent’s action strategy or security level never falls below that of the other.
Revised: Similar to a matrix game with two agents, we let both agents play a zero-sum, two person game where each agent’s action strategy or security level never falls below that of the other.

Original: The ordering according to which the players act so that each player acts optimally in a “min max” fashion does not affect the convergence to saddle point in our formulation.
Revised: The ordering according to which the players act so that each player acts optimally in a “min max” fashion does not affect the convergence to saddle point in our formulation.

Original: We consider the case where the security levels of both players coincide so that the strategy pair of both agents constitute a saddle-point pure strategy [3, p. 19].
Revised: We consider the case where the security levels of both players coincide so that the strategy pair of both agents constitute a saddle-point pure strategy [3, p. 19].

Original:  B ACKGROUND AND P RELIMINARIES
Revised:  B ACKGROUND AND P RELIMINARIES

Original: Reinforcement learning in robot control tasks consists of selecting control commands u from the space of a control policy π (often parameterized by θ ) that act on a high-dimensional state x .
Revised: Reinforcement learning in robot control tasks consists of selecting control commands u from the space of a control policy π (often parameterized by θ ) that act on a high-dimensional state x .

Original: The x typically composed of internal (e.g. joint angles and velocities) and external (e.g. object pose, positional information in the world) components.
Revised: The x typically composed of internal (e.g. joint angles and velocities) and external (e.g. object pose, positional information in the world) components.

Original: For a stochastic policy π ( u t | x t ) the commands inﬂuence the state of the robot based on the transition distribution π θ ( u t | x t , t ) .
Revised: For a stochastic policy π ( u t | x t ) the commands inﬂuence the state of the robot based on the transition distribution π θ ( u t | x t , t ) .

Original: The state and action pairs constitute a trajectory distribution τ
Revised: The state and action pairs constitute a trajectory distribution τ

Original: =
Revised: =

Original: ( x 1 , u 1 , x 2 , u 2 , · · · , x T , u T ) .
Revised: ( x 1 , u 1 , x 2 , u 2 , · · · , x T , u T ) .

Original: The performance of the robot on an episodic motor task is evaluated by an accumulated reward function R ( τ ) deﬁned as
Revised: The performance of the robot on an episodic motor task is evaluated by an accumulated reward function R ( τ ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: for an instantaneous reward function, r t , and a ﬁnal reward, r t f .
Revised: for an instantaneous reward function, r t , and a ﬁnal reward, r t f .

Original: Many tasks in robot learning domains can be formulated as above, whereby we choose a locally optimal policy π (cid:63) θ that optimizes the expectation of the accumulated reward
Revised: Many tasks in robot learning domains can be formulated as above, whereby we choose a locally optimal policy π (cid:63) θ that optimizes the expectation of the accumulated reward

Original: [Equation]
Revised: [Equation]

Original: where p π θ ( τ ) denotes distribution over trajectories τ and is deﬁned as
Revised: where p π θ ( τ ) denotes distribution over trajectories τ and is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: p ( x t +1 | x t , u t ) above represents the robot’s dynamics and its environment.
Revised: p ( x t +1 | x t , u t ) above represents the robot’s dynamics and its environment.

Original: Given the inadequacy of value function approximation methods in managing high-dimensional continuous state and action spaces as well as the difﬁculty of carrying out arbitrary exploration given hardware constraints [7], we resolve to use policy search (PS) methods as they operate in the parameter space of parameterized policies.
Revised: Given the inadequacy of value function approximation methods in managing high-dimensional continuous state and action spaces as well as the difﬁculty of carrying out arbitrary exploration given hardware constraints [7], we resolve to use policy search (PS) methods as they operate in the parameter space of parameterized policies.

Original: However, direct PS are often specialized algorithms that produce optimal policies for a particular task (often using policy gradient methods), and they come with the negative effects of not generalizing well to ﬂexible trajectory optimizations and large representations e.g. using neural network policies.
Revised: However, direct PS are often specialized algorithms that produce optimal policies for a particular task (often using policy gradient methods), and they come with the negative effects of not generalizing well to ﬂexible trajectory optimizations and large representations e.g. using neural network policies.

Original: Guided policy search (GPS) algorithms [1, 14, 16] are able to guide the search for parameterized policies from poor local minima using an alternating block coordinate ascent of the optimization problem, made up of a C-Step and an S-Step .
Revised: Guided policy search (GPS) algorithms [1, 14, 16] are able to guide the search for parameterized policies from poor local minima using an alternating block coordinate ascent of the optimization problem, made up of a C-Step and an S-Step .

Original: In the C-step , a well-posed cost function is minimized with respect to the trajectory samples, generating guiding distributions p
Revised: In the C-step , a well-posed cost function is minimized with respect to the trajectory samples, generating guiding distributions p

Original: i ( u t | x t ) ; and in the Sstep , the locally learned time-varying control laws, p i ( u t | x t ) , are parameterized by a nonlinear, neural network policy using supervised learning.
Revised: i ( u t | x t ) ; and in the Sstep , the locally learned time-varying control laws, p i ( u t | x t ) , are parameterized by a nonlinear, neural network policy using supervised learning.

Original: The S-step ﬁts policies of the form π θ ( u t | x t ) =
Revised: The S-step ﬁts policies of the form π θ ( u t | x t ) =

Original: N ( µ π ( x t ) , Σ π ( x t )) to the local controllers p i ( u t | x t ) , where µ π ( x t ) , and Σ π ( x t ) are functions that are estimated.
Revised: N ( µ π ( x t ) , Σ π ( x t )) to the local controllers p i ( u t | x t ) , where µ π ( x t ) , and Σ π ( x t ) are functions that are estimated.

Original: In order to ensure the learned policy for a dynamical system is robust to external uncertainties, modeling and transfer learning errors, we propose an iterative dynamic game consisting of an agent within an environment, and an adversarial agent, interacting with the original agent in the closed-loop environment E , over a ﬁnite horizon, T (it could also be extended to the inﬁnite horizon case).
Revised: In order to ensure the learned policy for a dynamical system is robust to external uncertainties, modeling and transfer learning errors, we propose an iterative dynamic game consisting of an agent within an environment, and an adversarial agent, interacting with the original agent in the closed-loop environment E , over a ﬁnite horizon, T (it could also be extended to the inﬁnite horizon case).

Original: The adversary could represent a spooﬁng agent in the world or modeling errors between the plant and dynamics.
Revised: The adversary could represent a spooﬁng agent in the world or modeling errors between the plant and dynamics.

Original: The states evolve according to the following stochastic dynamics p ( x t +

Revised: The states evolve according to the following stochastic dynamics p ( x t +

Original: | x t , u t , v t ) , ∀ t = 0 , ..., T where x t ∈ X t is a markovian system state,
Revised: | x t , u t , v t ) , ∀ t = 0 , ..., T where x t ∈ X t is a markovian system state,

Original: u t ∈
Revised: u t ∈

Original: U t is the action taken by the agent (henceforth called the protagonist), v t ∈ V t is the action taken by an adversarial agent.
Revised: U t is the action taken by the agent (henceforth called the protagonist), v t ∈ V t is the action taken by an adversarial agent.

Original: The subscripts denote time steps t ∈ [1 , T ] , allowing for a simpler structuring of the individual policies per time step [7].
Revised: The subscripts denote time steps t ∈ [1 , T ] , allowing for a simpler structuring of the individual policies per time step [7].

Original: The problems we consider are control tasks with complex dynamics, having continuous state and action spaces, and with trajectories deﬁned as ¯ τ = { x 1 , u 1 , v 1 , x 2 , u 2 , v 2 , · · · , x t f , u t f , v t f } .
Revised: The problems we consider are control tasks with complex dynamics, having continuous state and action spaces, and with trajectories deﬁned as ¯ τ = { x 1 , u 1 , v 1 , x 2 , u 2 , v 2 , · · · , x t f , u t f , v t f } .

Original: At time t , the system’s controller visits states with high rewards while the adversarial agent wants to visit states with low rewards.
Revised: At time t , the system’s controller visits states with high rewards while the adversarial agent wants to visit states with low rewards.

Original: The solution to this zero-sum game follows with an equilibrium at the origin – a saddle-point solution ensues.
Revised: The solution to this zero-sum game follows with an equilibrium at the origin – a saddle-point solution ensues.

Original: The policies that govern the behavior of the agents are deﬁned as π θ ( u t | x t ) and π θ ( v k | x k ) respectively, and the learned local linear time-varying Gaussian controllers are deﬁned as p ( u k | x k ) , p ( v k | x k ) .
Revised: The policies that govern the behavior of the agents are deﬁned as π θ ( u t | x t ) and π θ ( v k | x k ) respectively, and the learned local linear time-varying Gaussian controllers are deﬁned as p ( u k | x k ) , p ( v k | x k ) .

Original: In the next subsection, we show that learned motor policies, p ( u k | x k ) , are sensitive to minute additive disturbances; we later on propose how to build robustness to such trained neural network policies.
Revised: In the next subsection, we show that learned motor policies, p ( u k | x k ) , are sensitive to minute additive disturbances; we later on propose how to build robustness to such trained neural network policies.

Original: This is important in learning tasks where the robustness margins of a trained controller need to be known in advance before being introduced to a new execution environment.
Revised: This is important in learning tasks where the robustness margins of a trained controller need to be known in advance before being introduced to a new execution environment.

Original: Our goal is to select a suitable policy parameterization, so as to assure robustness and stability guarantees [4].
Revised: Our goal is to select a suitable policy parameterization, so as to assure robustness and stability guarantees [4].

Original: In this paper, we speciﬁcally use convex variant of the mirror descent version of
Revised: In this paper, we speciﬁcally use convex variant of the mirror descent version of

Original: GPS [16].
Revised: GPS [16].

Original:  C ASE FOR R OBUSTNESS IN PS A LGORITHMS
Revised:  C ASE FOR R OBUSTNESS IN PS A LGORITHMS

Original: In this section, we show why guided policy search algorithms are non-robust to even the simplest form of perturbations – additive disturbance.
Revised: In this section, we show why guided policy search algorithms are non-robust to even the simplest form of perturbations – additive disturbance.

Original: Before we proceed, we note that policy search algorithms are popular among the RL tools available because they have a “modest” level of robustness built into them e.g.
Revised: Before we proceed, we note that policy search algorithms are popular among the RL tools available because they have a “modest” level of robustness built into them e.g.

Original: • by requiring the learning controller to start the policy parameterization from multiple initial states, • adding a Kullback-Leibler (KL) constraint term to the reward function e.g. [1, 14], • solving a motor task in multiple ways where more than one solution exist [7], • or by introducing additive noise into the system as white noise e.g. differential dynamic programming (DDP) methods [11] or their iLQG variants[23].
Revised: • by requiring the learning controller to start the policy parameterization from multiple initial states, • adding a Kullback-Leibler (KL) constraint term to the reward function e.g. [1, 14], • solving a motor task in multiple ways where more than one solution exist [7], • or by introducing additive noise into the system as white noise e.g. differential dynamic programming (DDP) methods [11] or their iLQG variants[23].

Original: A fundamental drawback of these robustness mechanisms, however, is that the learned policy can only tolerate disturbance for slightly changing conditions; parametric uncertainty is not suitably modeled as white noise, and treating the error as an extra input might be relative to the size of the inputs (drawn from the environment) – necessitating the need for a formal treatment of robustness in PS algorithms.
Revised: A fundamental drawback of these robustness mechanisms, however, is that the learned policy can only tolerate disturbance for slightly changing conditions; parametric uncertainty is not suitably modeled as white noise, and treating the error as an extra input might be relative to the size of the inputs (drawn from the environment) – necessitating the need for a formal treatment of robustness in PS algorithms.

Original: A methodical way of solving the robustness problem in deep RL would be to consider techniques formalized in H ∞ control theory, where controller sensitivity and robustness are solved within the framework of a differential game.
Revised: A methodical way of solving the robustness problem in deep RL would be to consider techniques formalized in H ∞ control theory, where controller sensitivity and robustness are solved within the framework of a differential game.

Original: We conjecture that the lack of robustness of a RL trained policy arises from the difference between a plant model and the real system, or the difference in learning environments.
Revised: We conjecture that the lack of robustness of a RL trained policy arises from the difference between a plant model and the real system, or the difference in learning environments.

Original: If we can measure the sensitivity of a system, γ , then we can aim for policy robustness by ensuring that γ is sufﬁciently small to reject disturbance arising from the training environment or modeling errors if the gain of mapping from the error space to the disturbance is less than γ − 1 [27].
Revised: If we can measure the sensitivity of a system, γ , then we can aim for policy robustness by ensuring that γ is sufﬁciently small to reject disturbance arising from the training environment or modeling errors if the gain of mapping from the error space to the disturbance is less than γ − 1 [27].

Original: The ﬁgure to the right depicts the standard H ∞ control problem.
Revised: The ﬁgure to the right depicts the standard H ∞ control problem.

Original: Suppose G in the left inset is a plant for which we can ﬁnd an internally stabilizing controller, Σ K , that ensures stable transfer of input u to measurement y , the H ∞ control objective is to ﬁnd the “worst” possible disturbance, v , which produces an undesired output, z ; we want to minimize the effect of z .
Revised: Suppose G in the left inset is a plant for which we can ﬁnd an internally stabilizing controller, Σ K , that ensures stable transfer of input u to measurement y , the H ∞ control objective is to ﬁnd the “worst” possible disturbance, v , which produces an undesired output, z ; we want to minimize the effect of z .

Original: In the right inset in the ﬁgure, we treat unmodeled dynamics, transfer errors and other uncertainty as an additional feedback to which we would like to adapt with respect to the worst possible disturbance in a prescribed range.
Revised: In the right inset in the ﬁgure, we treat unmodeled dynamics, transfer errors and other uncertainty as an additional feedback to which we would like to adapt with respect to the worst possible disturbance in a prescribed range.

Original: Σ ∆ in the right inset represents these uncertainties; our goal is to ﬁnd the closed-loop optimal policy for which the plant, G , will satisfy performance requirements and maintain robustness for a large range of systems Σ ∆ .
Revised: Σ ∆ in the right inset represents these uncertainties; our goal is to ﬁnd the closed-loop optimal policy for which the plant, G , will satisfy performance requirements and maintain robustness for a large range of systems Σ ∆ .

Original: We focus on conditions under
Revised: We focus on conditions under

Original: Algorithm
Revised: Algorithm

Original: 1: for iteration k ∈ { 1 , . . .
Revised: 1: for iteration k ∈ { 1 , . . .

Original: , K } do 2:
Revised: , K } do 2:

Original: C-step : p i ← arg min p i
Revised: C-step : p i ← arg min p i

Original: E p
Revised: E p

Original: i ( τ ) (cid:104)(cid:80) Tt =1 r ( x t , u t ) (cid:105)
Revised: i ( τ ) (cid:104)(cid:80) Tt =1 r ( x t , u t ) (cid:105)

Original: such that D KL
Revised: such that D KL

Original: ( p i ( τ ) (cid:107) π θ ( τ )) ≤ (cid:15) 3:
Revised: ( p i ( τ ) (cid:107) π θ ( τ )) ≤ (cid:15) 3:

Original: S-step : π θ ← arg min θ (cid:80) i D KL ( p i ( τ ) (cid:107) π θ ( τ ))
Revised: S-step : π θ ← arg min θ (cid:80) i D KL ( p i ( τ ) (cid:107) π θ ( τ ))

Original: (from supervised learning) 4: end for which we can make the H ∞ norm of the system less than a given prior, γ .
Revised: (from supervised learning) 4: end for which we can make the H ∞ norm of the system less than a given prior, γ .

Original: Speciﬁcally, we want to design a controller Σ K that minimizes the H ∞ norm of the closed-loop transfer function T zv from disturbance v to output z deﬁned as (cid:107) T zv (cid:107) ∞
Revised: Speciﬁcally, we want to design a controller Σ K that minimizes the H ∞ norm of the closed-loop transfer function T zv from disturbance v to output z deﬁned as (cid:107) T zv (cid:107) ∞

Original: =
Revised: =

Original: sup v (cid:107) z (cid:107) 2 (cid:107) v (cid:107) 2 .
Revised: sup v (cid:107) z (cid:107) 2 (cid:107) v (cid:107) 2 .

Original: From the small-gain theorem, the system in the right ﬁgure above will be stable for any stable mapping ∆ :
Revised: From the small-gain theorem, the system in the right ﬁgure above will be stable for any stable mapping ∆ :

Original: z → v for (cid:107) ∆ (cid:107) ∞ < γ − 1 [19].
Revised: z → v for (cid:107) ∆ (cid:107) ∞ < γ − 1 [19].

Original: In a differential game setting, we can consider a minmax solution to the H ∞ problem for the plant G with dynamics given by ˙ x = f ( x, u, w ) so that we solve an H ∞ problem that satisﬁes the constraint (cid:107) T zv (cid:107) ∞ =
Revised: In a differential game setting, we can consider a minmax solution to the H ∞ problem for the plant G with dynamics given by ˙ x = f ( x, u, w ) so that we solve an H ∞ problem that satisﬁes the constraint (cid:107) T zv (cid:107) ∞ =

Original: sup v (cid:107) z (cid:107) 22 (cid:107) v (cid:107) 22 ≤ γ 2 , or ﬁnd a control input u that satisﬁes the constraint V = (cid:82) T t =0 (cid:0) z
Revised: sup v (cid:107) z (cid:107) 22 (cid:107) v (cid:107) 22 ≤ γ 2 , or ﬁnd a control input u that satisﬁes the constraint V = (cid:82) T t =0 (cid:0) z

Original: Tt z t − γ 2 v Tt v t (cid:1) dt ≤ 0 for all possible disturbances v for which x 0 = 0 .
Revised: Tt z t − γ 2 v Tt v t (cid:1) dt ≤ 0 for all possible disturbances v for which x 0 = 0 .

Original: We consider a differential game for which the best control u that minimizes V , and the worst disturbance v that maximizes V are derived from
Revised: We consider a differential game for which the best control u that minimizes V , and the worst disturbance v that maximizes V are derived from

Original: [Equation]
Revised: [Equation]

Original: The optimal value function is determined from the Hamilton-Jacobi-Isaacs (HJI) equation,
Revised: The optimal value function is determined from the Hamilton-Jacobi-Isaacs (HJI) equation,

Original: [Equation]
Revised: [Equation]

Original: from which the optimal u and v can be computed.
Revised: from which the optimal u and v can be computed.

Original: (cid:15) is adjusted based on the formulation in [16].
Revised: (cid:15) is adjusted based on the formulation in [16].

Original: The dynamics p i ( x k +1 | x k , u k , v k ) =
Revised: The dynamics p i ( x k +1 | x k , u k , v k ) =

Original: N ( f x k x k + f u k u k
Revised: N ( f x k x k + f u k u k

Original: + f v k v k , F k ) are ﬁtted to samples { x ik +1 , x ik , u ik , v ik } using a mixture of Gaussian models to the generated samples { x ik +1 , x ik , u ik , v ik } at iteration i for all time k .
Revised: + f v k v k , F k ) are ﬁtted to samples { x ik +1 , x ik , u ik , v ik } using a mixture of Gaussian models to the generated samples { x ik +1 , x ik , u ik , v ik } at iteration i for all time k .

Original: Speciﬁcally, we incorporate a normal-inverse-Wishart prior on the Gaussian model as described in [14, § A.3].
Revised: Speciﬁcally, we incorporate a normal-inverse-Wishart prior on the Gaussian model as described in [14, § A.3].

Original: 4. S ENSITIVITY OF A LEARNED RL P OLICY
Revised: 4. S ENSITIVITY OF A LEARNED RL P OLICY

Original: This section offers guidance on testing the sensitivity of a deep neural network policy for an agent.
Revised: This section offers guidance on testing the sensitivity of a deep neural network policy for an agent.

Original: We consider additive disturbance to a deep RL policy.
Revised: We consider additive disturbance to a deep RL policy.

Original: Our goal is to study the degradation of performance of a trained neural network policy in the presence of the “worst” possible disturbance in the parameter space of the policy; if this disturbance cannot alter the performance of the trained policy, we have some value for the policy parameters in the prescribed range that the decision strategy is acceptable.
Revised: Our goal is to study the degradation of performance of a trained neural network policy in the presence of the “worst” possible disturbance in the parameter space of the policy; if this disturbance cannot alter the performance of the trained policy, we have some value for the policy parameters in the prescribed range that the decision strategy is acceptable.

Original: We follow the model described above, where Σ ∆ denotes the uncertainty injected by the adversary.
Revised: We follow the model described above, where Σ ∆ denotes the uncertainty injected by the adversary.

Original: We arrive at the nominal system from u to y when the transfer matrix of Σ ∆ is zero.
Revised: We arrive at the nominal system from u to y when the transfer matrix of Σ ∆ is zero.

Original: We call Σ ∆ the adversary whose control, v ’s effect on the output z is to be minimized.
Revised: We call Σ ∆ the adversary whose control, v ’s effect on the output z is to be minimized.

Original: We quantify the effect of v on z in closed loop using a suitable cost function as a min-max criteria.
Revised: We quantify the effect of v on z in closed loop using a suitable cost function as a min-max criteria.

Original: This can be seen as an H ∞ norm on the system.
Revised: This can be seen as an H ∞ norm on the system.

Original: Suppose the local actions, p ( u k | x k ) , of the controller belong to the policy space π
Revised: Suppose the local actions, p ( u k | x k ) , of the controller belong to the policy space π

Original: =
Revised: =

Original: [ π 0 , ..., π T ] that maximize the expected sum of rewards
Revised: [ π 0 , ..., π T ] that maximize the expected sum of rewards

Original: [Equation]
Revised: [Equation]

Original: Therefore, the augmented reward for the closed-loop protagonist-adversary system becomes
Revised: Therefore, the augmented reward for the closed-loop protagonist-adversary system becomes

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) ( τ ) =
Revised: where (cid:96) ( τ ) =

Original: r ( x t , u t , t )
Revised: r ( x t , u t , t )

Original: + r ( x t f , u t f , t f ) , and (cid:96) (¯ τ ) =
Revised: + r ( x t f , u t f , t f ) , and (cid:96) (¯ τ ) =

Original: r ( x t , u t , v t , t )
Revised: r ( x t , u t , v t , t )

Original: + r ( x t f , u t f , v t f , t f ) .
Revised: + r ( x t f , u t f , v t f , t f ) .

Original: Essentially, (cid:96) (¯ τ ) =
Revised: Essentially, (cid:96) (¯ τ ) =

Original: (cid:96) (¯ τ ) − γ 2 α ( v t ) where α ( · ) can be chosen as a function of the adversarial disturbance v t 1 .
Revised: (cid:96) (¯ τ ) − γ 2 α ( v t ) where α ( · ) can be chosen as a function of the adversarial disturbance v t 1 .

Original: We chose α as the L 2 norm of the disturbance v t in our implementation.
Revised: We chose α as the L 2 norm of the disturbance v t in our implementation.

Original: γ is a sensitivity parameter that adjusts the strength of the adversary by increasing the penalty incurred by its actions.
Revised: γ is a sensitivity parameter that adjusts the strength of the adversary by increasing the penalty incurred by its actions.

Original: In (2), we carry out the optimization procedure by ﬁrst learning the optimal policy for the controller; we then ﬁx this optmal policy and carry out the minimization of the augmented reward function with the adversary in closed-loop as in (3).
Revised: In (2), we carry out the optimization procedure by ﬁrst learning the optimal policy for the controller; we then ﬁx this optmal policy and carry out the minimization of the augmented reward function with the adversary in closed-loop as in (3).

Original: As γ → ∞ in (3), the optimal closed-loop policy is for the agent to do nothing, since any action will incur a large penalty; as γ decreases, however, the adversary’s actions have a greater effect on the state of the closed-loop system.
Revised: As γ → ∞ in (3), the optimal closed-loop policy is for the agent to do nothing, since any action will incur a large penalty; as γ decreases, however, the adversary’s actions have a greater effect on the state of the closed-loop system.

Original: The (inverse of the) lowest value of γ for which the adversary’s policy causes unacceptable performance provides a measure of robustness of the control policy π θ ( u t | x t ) .
Revised: The (inverse of the) lowest value of γ for which the adversary’s policy causes unacceptable performance provides a measure of robustness of the control policy π θ ( u t | x t ) .

Original: For various values of γ , the state-of-the-art robot learning policies are non-robust to small perturbations as we show in Sec. 6.
Revised: For various values of γ , the state-of-the-art robot learning policies are non-robust to small perturbations as we show in Sec. 6.

Original: 4. R OBUST ZERO - SUM , TWO - PERSON GAMES
Revised: 4. R OBUST ZERO - SUM , TWO - PERSON GAMES

Original: To learn robust policies, we run an alternating optimization algorithm that maximizes the cost function with respect to the adversarial controller (modeled with the worst possible disturbance) and minimizes the cost function with respect to the protagonist’s policy.
Revised: To learn robust policies, we run an alternating optimization algorithm that maximizes the cost function with respect to the adversarial controller (modeled with the worst possible disturbance) and minimizes the cost function with respect to the protagonist’s policy.

Original: We consider a two-player, zero-sum Markov game framework for simultaneously learning policies for the protagonist and the adversary.
Revised: We consider a two-player, zero-sum Markov game framework for simultaneously learning policies for the protagonist and the adversary.

Original: We seek to learn saddle-point equilibrium strategies for the zero-sum game:
Revised: We seek to learn saddle-point equilibrium strategies for the zero-sum game:

Original: [Equation]
Revised: [Equation]

Original: where we have overloaded notation such that π ( x t )
Revised: where we have overloaded notation such that π ( x t )

Original: =
Revised: =

Original: π θ ( u t | x t ) and ¯ π ( x t ) =
Revised: π θ ( u t | x t ) and ¯ π ( x t ) =

Original: π θ ( v t | x t ) ; (cid:96) ( x t , u t , v t ) =
Revised: π θ ( v t | x t ) ; (cid:96) ( x t , u t , v t ) =

Original: r ( x t , u t ) + γα ( v t ) is the stage cost.
Revised: r ( x t , u t ) + γα ( v t ) is the stage cost.

Original: ¯ π ( x t ) denotes that the adversarial actions are drawn from outside of the action space of the protagonist’s policy.
Revised: ¯ π ( x t ) denotes that the adversarial actions are drawn from outside of the action space of the protagonist’s policy.

Original: Fixing a value of γ is equivalent to an assumption on the capability of the adversary or the magnitude of a worst possible disturbance.
Revised: Fixing a value of γ is equivalent to an assumption on the capability of the adversary or the magnitude of a worst possible disturbance.

Original: To validate this proposal, we develop locally robust controllers for a trajectory optimization problem from multiple initial states using (4) as a guiding cost; a neural network function approximator is then used to parameterize these local controllers using supervised learning.
Revised: To validate this proposal, we develop locally robust controllers for a trajectory optimization problem from multiple initial states using (4) as a guiding cost; a neural network function approximator is then used to parameterize these local controllers using supervised learning.

Original: We discuss this procedure in the next section.
Revised: We discuss this procedure in the next section.

Original: 4. R OBUST GPS
Revised: 4. R OBUST GPS

Original: GPS adds off-policy guiding samples to a sample set: this guides the policy toward spaces of high rewards.
Revised: GPS adds off-policy guiding samples to a sample set: this guides the policy toward spaces of high rewards.

Original: If p ( τ ) is the trajectory distribution induced by the locally linear Gaussian controller p ( u t | x t ) and ¯ p ( u t | x t ) denotes the previous local controller, GPS algorithms reduce the effect of visiting regions of low entropy by minimizing the KL divergence of the current local policy from the previous one as follows,
Revised: If p ( τ ) is the trajectory distribution induced by the locally linear Gaussian controller p ( u t | x t ) and ¯ p ( u t | x t ) denotes the previous local controller, GPS algorithms reduce the effect of visiting regions of low entropy by minimizing the KL divergence of the current local policy from the previous one as follows,

Original: [Equation]
Revised: [Equation]

Original: where H is the entropy term that favors broad distributions, η is a Lagrange multiplier and the ﬁrst term forces the actions p to be high in regions of high reward.
Revised: where H is the entropy term that favors broad distributions, η is a Lagrange multiplier and the ﬁrst term forces the actions p to be high in regions of high reward.

Original: The trajectory is optimized using optimal control principles under linear quadratic Gaussian assumptions [23].
Revised: The trajectory is optimized using optimal control principles under linear quadratic Gaussian assumptions [23].

Original: GPS minimizes the expected cost, E π θ ( x t , u t ) r ( x t , u t ) over the joint distribution of state and action pairs given by the marginals π θ ( τ ) =
Revised: GPS minimizes the expected cost, E π θ ( x t , u t ) r ( x t , u t ) over the joint distribution of state and action pairs given by the marginals π θ ( τ ) =

Original: p ( x 1 ) (cid:81) Tt =1 p ( x t +1 | x t , u t ) .
Revised: p ( x 1 ) (cid:81) Tt =1 p ( x t +1 | x t , u t ) .

Original: GPS algorithms optimize the cost J ( θ ) via a split process of trajectory optimization of local control laws and a standard supervised learning to generalize to high-dimensional policy space settings.
Revised: GPS algorithms optimize the cost J ( θ ) via a split process of trajectory optimization of local control laws and a standard supervised learning to generalize to high-dimensional policy space settings.

Original: A generic GPS algorithm is shown in Algorithm 1.
Revised: A generic GPS algorithm is shown in Algorithm 1.

Original: During the C-step, multiple local control laws, p i ( u t | x t ) , are generated for different initial states x i 1 ∼ p ( x 1 ) .
Revised: During the C-step, multiple local control laws, p i ( u t | x t ) , are generated for different initial states x i 1 ∼ p ( x 1 ) .

Original: The supervised learning stage (S-step) regresses the global policy π θ ( u t | x t ) to all the local actions computed in the C-step.
Revised: The supervised learning stage (S-step) regresses the global policy π θ ( u t | x t ) to all the local actions computed in the C-step.

Original: For unknown dynamics, one can ﬁt p ( x t +1 | x t , u t ) to sampled trajectories from the trajectory distribution under ¯ p ( τ ) .
Revised: For unknown dynamics, one can ﬁt p ( x t +1 | x t , u t ) to sampled trajectories from the trajectory distribution under ¯ p ( τ ) .

Original: To avoid divergence in dynamics, the difference between the current and previous trajectories are constrained by the KL divergence as in stepin algorithm 1.
Revised: To avoid divergence in dynamics, the difference between the current and previous trajectories are constrained by the KL divergence as in stepin algorithm 1.

Original: The KL divergence ¯ p from p in (5) will not optimize for a robust policy in the presence of modeling errors, changes in environment settings or disturbance as we show in the sensitivity section in subsection 4.1.
Revised: The KL divergence ¯ p from p in (5) will not optimize for a robust policy in the presence of modeling errors, changes in environment settings or disturbance as we show in the sensitivity section in subsection 4.1.

Original: To make the computed neural network policy robust to these uncertainties, we propose a zero-sum, two-person dynamic game scenario in the next section.
Revised: To make the computed neural network policy robust to these uncertainties, we propose a zero-sum, two-person dynamic game scenario in the next section.

Original: 1: for iteration k ∈ { 1 , . . .
Revised: 1: for iteration k ∈ { 1 , . . .

Original: , K } do 2:
Revised: , K } do 2:

Original: Generate samples D i
Revised: Generate samples D i

Original: = { τ i,j } by running p
Revised: = { τ i,j } by running p

Original: i ( u k | x k ) and p i ( v k | x k ) or π θ i ( u | x k ) and π θ i ( v | x k ) 3:
Revised: i ( u k | x k ) and p i ( v k | x k ) or π θ i ( u | x k ) and π θ i ( v | x k ) 3:

Original: Fit linear-Gaussian dynamics p
Revised: Fit linear-Gaussian dynamics p

Original: i ( x k +1 | x k , u k , v k ) using samples in D i 4:
Revised: i ( x k +1 | x k , u k , v k ) using samples in D i 4:

Original: Fit linearized protagonist policy π θ i ( u k | x k ) using samples in D i 5:
Revised: Fit linearized protagonist policy π θ i ( u k | x k ) using samples in D i 5:

Original: Regress global policies ¯ π θ i ( u k | x k ) , ¯ π θ i ( v k | x k ) with samples in D i 6:
Revised: Regress global policies ¯ π θ i ( u k | x k ) , ¯ π θ i ( v k | x k ) with samples in D i 6:

Original: C-step : p i ← arg min p u i max p v i (cid:104)
Revised: C-step : p i ← arg min p u i max p v i (cid:104)

Original: E p (¯ τ ) (cid:80)
Revised: E p (¯ τ ) (cid:80)

Original: Kk
Revised: Kk

Original: =1 l ( x k , u k , v k ) (cid:105) s.t.
Revised: =1 l ( x k , u k , v k ) (cid:105) s.t.

Original: D KL ( p u i (¯ τ ) (cid:107)
Revised: D KL ( p u i (¯ τ ) (cid:107)

Original: ¯
Revised: ¯

Original: π θ u
Revised: π θ u

Original: i (¯ τ )) ≤ (cid:15) 7:
Revised: i (¯ τ )) ≤ (cid:15) 7:

Original: S-step : π θ ← arg min θ max θ (cid:80) k,i,j D KL ( π θ ( u k | x k,i,j ) (cid:107) p u i ( u k | x k,i,j )) (via supervised learning) 8:
Revised: S-step : π θ ← arg min θ max θ (cid:80) k,i,j D KL ( π θ ( u k | x k,i,j ) (cid:107) p u i ( u k | x k,i,j )) (via supervised learning) 8:

Original: Adjust (cid:15) (see [16, § 4.2]) 9: end for
Revised: Adjust (cid:15) (see [16, § 4.2]) 9: end for

Original:  T WO -P LAYER Z ERO -S UM ITERATIVE DYNAMIC GPS
Revised:  T WO -P LAYER Z ERO -S UM ITERATIVE DYNAMIC GPS

Original: To guarantee robust performance during the training of policies of a stochastic system, we introduce the “worst” disturbance in the H ∞ paradigm to the search for a good guiding distribution problem.
Revised: To guarantee robust performance during the training of policies of a stochastic system, we introduce the “worst” disturbance in the H ∞ paradigm to the search for a good guiding distribution problem.

Original: We begin by augmenting the reward function with a term that allows for withstanding a disturbing input
Revised: We begin by augmenting the reward function with a term that allows for withstanding a disturbing input

Original: [Equation]
Revised: [Equation]

Original: (6) where γ 2 v T v allows us to introduce a quadratic weighting term in the disturbing input; γ denotes the robustness parameter.
Revised: (6) where γ 2 v T v allows us to introduce a quadratic weighting term in the disturbing input; γ denotes the robustness parameter.

Original: A zero-sum game follows explicitly: the protagonist is guided toward regions of high reward regions while adversary pulls in its own favorite direction – yielding a saddle-point solution.
Revised: A zero-sum game follows explicitly: the protagonist is guided toward regions of high reward regions while adversary pulls in its own favorite direction – yielding a saddle-point solution.

Original: This framework facilitates learning control decision strategies that are robust in the presence of disturbances and modeling errors – improving upon the generic optimal control policies that GPS and indeed deep RL algorithms guarantee.
Revised: This framework facilitates learning control decision strategies that are robust in the presence of disturbances and modeling errors – improving upon the generic optimal control policies that GPS and indeed deep RL algorithms guarantee.

Original: 5. T WO -P LAYER T RAJECTORY O PTIMIZATION
Revised: 5. T WO -P LAYER T RAJECTORY O PTIMIZATION

Original: We propose repeatedly solving an MPC-based ﬁnite-horizon trajectory optimization problem within the framework of DDP.
Revised: We propose repeatedly solving an MPC-based ﬁnite-horizon trajectory optimization problem within the framework of DDP.

Original: Speciﬁcally, we generalize a DDP variant – the iLQG algorithm of [22], to a two-player, zero-sum dynamic game as follows:
Revised: Speciﬁcally, we generalize a DDP variant – the iLQG algorithm of [22], to a two-player, zero-sum dynamic game as follows:

Original: • we iteratively approximate the nonlinear dynamics, ˙ x = f ( x t , u t , v t ) , starting with nominal control, ¯ u t ; t ∈ [ t 0 , t f ] , and nominal adversarial input ¯ v t ; t ∈ [ t 0 , t f ] which are assumed to be available.
Revised: • we iteratively approximate the nonlinear dynamics, ˙ x = f ( x t , u t , v t ) , starting with nominal control, ¯ u t ; t ∈ [ t 0 , t f ] , and nominal adversarial input ¯ v t ; t ∈ [ t 0 , t f ] which are assumed to be available.

Original: • we run the passive dynamics with ¯ u t and ¯ v t to generate a trajectory (¯ x t , ¯ u t , ¯ v t ) • discretizing time, we linearize the nonlinear system, ˙ x t , about (¯ x k , ¯ u k , ¯ v k ) , so that the new state and action pairs become
Revised: • we run the passive dynamics with ¯ u t and ¯ v t to generate a trajectory (¯ x t , ¯ u t , ¯ v t ) • discretizing time, we linearize the nonlinear system, ˙ x t , about (¯ x k , ¯ u k , ¯ v k ) , so that the new state and action pairs become

Original: [Equation]
Revised: [Equation]

Original: δ x k , δ u k , and δ v k are measured w.r.t the nominal vectors ¯ x k , ¯ u k , ¯ v k and are not necessarily small.
Revised: δ x k , δ u k , and δ v k are measured w.r.t the nominal vectors ¯ x k , ¯ u k , ¯ v k and are not necessarily small.

Original: The LQG approximation to the original optimal control problem and reward become
Revised: The LQG approximation to the original optimal control problem and reward become

Original: [Equation]
Revised: [Equation]

Original: where single and double subscripts in the augmented reward denote ﬁrst-order and second-order derivatives respectively, and f z k are the respective Jacobians
Revised: where single and double subscripts in the augmented reward denote ﬁrst-order and second-order derivatives respectively, and f z k are the respective Jacobians

Original: e.g. f x k = ∂f ( · ) ∂ x | k and f xx k
Revised: e.g. f x k = ∂f ( · ) ∂ x | k and f xx k

Original: = ∂∂ x ∂f ( · ) ∂ x | k at time k , E ( w t ) is an additive random noise term (folded into v t in our implementation); the value function is the cost-to-go given by the min-max of the control sequence
Revised: = ∂∂ x ∂f ( · ) ∂ x | k at time k , E ( w t ) is an additive random noise term (folded into v t in our implementation); the value function is the cost-to-go given by the min-max of the control sequence

Original: [Equation]
Revised: [Equation]

Original: Setting V ( x k f )
Revised: Setting V ( x k f )

Original: = (cid:96) k
Revised: = (cid:96) k

Original: f
Revised: f

Original: ( x k f ) , where k f is the ﬁnal time step, the dynamic programming problem transforms the min-max over an entire control sequence to a series of optimizations over a single control, which proceeds backward in time as
Revised: ( x k f ) , where k f is the ﬁnal time step, the dynamic programming problem transforms the min-max over an entire control sequence to a series of optimizations over a single control, which proceeds backward in time as

Original: [Equation]
Revised: [Equation]

Original: The Hamiltonian, (cid:96) (
Revised: The Hamiltonian, (cid:96) (

Original: [Equation]
Revised: [Equation]

Original: + V ( · ) , can be considered as a function of perturbations around the tuple { x k , u k , v k } .
Revised: + V ( · ) , can be considered as a function of perturbations around the tuple { x k , u k , v k } .

Original: Given the intractability of solving the Bellman partial differential equation above, we restrict our attention to the local neighborhood of the nominal trajectory by expanding a power series about the nominal, nonoptimal trajectory similar to [18].
Revised: Given the intractability of solving the Bellman partial differential equation above, we restrict our attention to the local neighborhood of the nominal trajectory by expanding a power series about the nominal, nonoptimal trajectory similar to [18].

Original: We proceed as follows:
Revised: We proceed as follows:

Original: • we maintain a second-order local model of the perturbed Q -coefﬁcients of the LQR problem, ( Q k , Q x k , Q u
Revised: • we maintain a second-order local model of the perturbed Q -coefﬁcients of the LQR problem, ( Q k , Q x k , Q u

Original: [Equation]
Revised: [Equation]

Original: Q xx k
Revised: Q xx k

Original: [Equation]
Revised: [Equation]

Original: Q ux k
Revised: Q ux k

Original: [Equation]
Revised: [Equation]

Original: Q
Revised: Q

Original: [Equation]
Revised: [Equation]

Original: ) 2 , deﬁned thus
Revised: ) 2 , deﬁned thus

Original: [Equation]
Revised: [Equation]

Original: • a second-order Taylor approximation of Q ( δ x k , δ u k , δ v k , k ) in the preceding equation yields
Revised: • a second-order Taylor approximation of Q ( δ x k , δ u k , δ v k , k ) in the preceding equation yields

Original: [Equation]
Revised: [Equation]

Original: • the best possible (protagonist) action and the worst possible (adversarial) action can be found by performing the respective arg min and arg max operations
Revised: • the best possible (protagonist) action and the worst possible (adversarial) action can be found by performing the respective arg min and arg max operations

Original: [Equation]
Revised: [Equation]

Original: so
Revised: so

Original: that we have
Revised: that we have

Original: the following linear controllers
Revised: the following linear controllers

Original: that
Revised: that

Original: minimize
Revised: minimize

Original: and
Revised: and

Original: maximize the quadratic
Revised: maximize the quadratic

Original: Q function respectively:
Revised: Q function respectively:

Original: [Equation]
Revised: [Equation]

Original: For nonlinear systems, the inverse of the 2nd partial derivatives of the Hamiltonian with respect to the controls must be strictly positive deﬁnite.
Revised: For nonlinear systems, the inverse of the 2nd partial derivatives of the Hamiltonian with respect to the controls must be strictly positive deﬁnite.

Original: When the inverse of the Hessians above are nonpositive-deﬁnite, we can circumvent this bottleneck by adding a suitably large positive quantity to
Revised: When the inverse of the Hessians above are nonpositive-deﬁnite, we can circumvent this bottleneck by adding a suitably large positive quantity to

Original: Q − 1 uu k and Q − 1 vv k
Revised: Q − 1 uu k and Q − 1 vv k

Original: [12, 5], by replacing the Hessian with an identity matrix (which gives the steepest descent) [23], or by multiplying by lowest eigenvalue of the matrix.
Revised: [12, 5], by replacing the Hessian with an identity matrix (which gives the steepest descent) [23], or by multiplying by lowest eigenvalue of the matrix.

Original: We ﬁnd that the protagonist and adversary in the above-equations have a local action containing a state feedback term, G , and an open-loop term, g , given by
Revised: We ﬁnd that the protagonist and adversary in the above-equations have a local action containing a state feedback term, G , and an open-loop term, g , given by

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: The tuple { g u k , G u k , g v k , G v k } can be computed efﬁciently as shown in (17).
Revised: The tuple { g u k , G u k , g v k , G v k } can be computed efﬁciently as shown in (17).

Original: We can construct linear Gaussian controllers with mean given by the deterministic optimal solutions and the covariance proportional to the curvatures of the respective Q functions:
Revised: We can construct linear Gaussian controllers with mean given by the deterministic optimal solutions and the covariance proportional to the curvatures of the respective Q functions:

Original: [Equation]
Revised: [Equation]

Original: [13] has shown that these types of distributions optimize an objective function with maximum entropy given by arg
Revised: [13] has shown that these types of distributions optimize an objective function with maximum entropy given by arg

Original: [Equation]
Revised: [Equation]

Original: ] subject to
Revised: ] subject to

Original: [Equation]
Revised: [Equation]

Original: while p ( v k | x k ) optimizes
Revised: while p ( v k | x k ) optimizes

Original: [Equation]
Revised: [Equation]

Original: ] subject to
Revised: ] subject to

Original: [Equation]
Revised: [Equation]

Original: where ¯ τ =
Revised: where ¯ τ =

Original: ( x it , u it , v it ) is the system’s trajectory evolution over all states, i , visited by both local controllers, and H is the differential entropy.
Revised: ( x it , u it , v it ) is the system’s trajectory evolution over all states, i , visited by both local controllers, and H is the differential entropy.

Original: Equation (9) produces a trajectory that follows the widest, highest-entropy distribution while minimizing the expected cost under linearized dynamics and quadratic cost; (10) produces an opposing trajectory to what p ( u k | x k ) does by maximizing the expected cost under locally linear quadratic assumptions about the dynamics.
Revised: Equation (9) produces a trajectory that follows the widest, highest-entropy distribution while minimizing the expected cost under linearized dynamics and quadratic cost; (10) produces an opposing trajectory to what p ( u k | x k ) does by maximizing the expected cost under locally linear quadratic assumptions about the dynamics.

Original: Note that the open-loop control strategies in (8) depend on the action of the other player.
Revised: Note that the open-loop control strategies in (8) depend on the action of the other player.

Original: Therefore, equations (8) ensure we have a cooperative game in which the protagonist and the adversary alternate between taking best possible and worst possible local actions during the trajectory optimization phase.
Revised: Therefore, equations (8) ensure we have a cooperative game in which the protagonist and the adversary alternate between taking best possible and worst possible local actions during the trajectory optimization phase.

Original: This helps maintain equilibrium around the system’s desired trajectory, while ensuring robustness in local policies.
Revised: This helps maintain equilibrium around the system’s desired trajectory, while ensuring robustness in local policies.

Original: Substituting (8) into (7) and equating coefﬁcients of δ x k , δ u k , δ v k to those of V ( x k + δ x k , k ) =
Revised: Substituting (8) into (7) and equating coefﬁcients of δ x k , δ u k , δ v k to those of V ( x k + δ x k , k ) =

Original: V ( x k , k ) + V x k δ x k + 

Revised: V ( x k , k ) + V x k δ x k +

Original: δ x Tk V
Revised: δ x Tk V

Original: xx
Revised: xx

Original: k
Revised: k

Original: δ x k , we obtain a quadratic value function at time k , through the backward pass given by (19) in the appendix.
Revised: δ x k , we obtain a quadratic value function at time k , through the backward pass given by (19) in the appendix.

Original: Say, the protagonist ﬁrst implements its strategy, then transmits its information to the adversary, who subsequently chooses its strategy; it follows that the adversary can choose a more favorable outcome since it knows what the protagonist’s choice of strategy is.
Revised: Say, the protagonist ﬁrst implements its strategy, then transmits its information to the adversary, who subsequently chooses its strategy; it follows that the adversary can choose a more favorable outcome since it knows what the protagonist’s choice of strategy is.

Original: It becomes obvious that the best action for the protagonist is to choose a control strategy that is an optimal response to the choice of the adversary determined from
Revised: It becomes obvious that the best action for the protagonist is to choose a control strategy that is an optimal response to the choice of the adversary determined from

Original: [Equation]
Revised: [Equation]

Original: Similarly, if the roles of the players are changed, the protagonist response to the adversary’s worst choice will be
Revised: Similarly, if the roles of the players are changed, the protagonist response to the adversary’s worst choice will be

Original: [Equation]
Revised: [Equation]

Original: Therefore, it does not matter that the order of play is predetermined.
Revised: Therefore, it does not matter that the order of play is predetermined.

Original: We end up with an iterative dynamic game , where each agent’s strategy depends on its previous actions.
Revised: We end up with an iterative dynamic game , where each agent’s strategy depends on its previous actions.

Original: The update rules for the Q coefﬁcients are determined using a Gauss-Newton approximation and is given in (15) in the appendices.
Revised: The update rules for the Q coefﬁcients are determined using a Gauss-Newton approximation and is given in (15) in the appendices.

Original: In the forward pass, we integrate the state equation, ˙ x , compute the protagonist’s deterministic optimal policy and update the trajectory as follows:
Revised: In the forward pass, we integrate the state equation, ˙ x , compute the protagonist’s deterministic optimal policy and update the trajectory as follows:

Original: [Equation]
Revised: [Equation]

Original: Compared to previous GPS algorithms, the local controllers not only produce locally linear Gaussian controllers that favors the widest and highest entropy, they also have robustness to disturbance and modeling errors built into them in the H ∞ sense.
Revised: Compared to previous GPS algorithms, the local controllers not only produce locally linear Gaussian controllers that favors the widest and highest entropy, they also have robustness to disturbance and modeling errors built into them in the H ∞ sense.

Original: We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c. We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c.
Revised: We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c. We arrive at a saddle point in the energy space of the cost function and we posit that the local controllers generated during the trajectory optimization phase become robust to external perturbations, modeling errors e.t.c.

Original: The next section shows how we generate the function
Revised: The next section shows how we generate the function

Original: V ( x k ) that guarantees saddle-point equilibria for our examples.
Revised: V ( x k ) that guarantees saddle-point equilibria for our examples.

Original: 5. E STIMATING D YNAMICS D ISTRIBUTION
Revised: 5. E STIMATING D YNAMICS D ISTRIBUTION

Original: The dynamics of the two player system is given by the tuple { x it , u it , v it , x it +1 } and we ﬁt the system dynamics using piecewise linear functions in the form of a mixture of N Gaussians as proposed in [1] and [14] over the vectors { x it , u it , v it , x it +1 } T , where the i th index represents the i -th trajectory rollout on the robot.
Revised: The dynamics of the two player system is given by the tuple { x it , u it , v it , x it +1 } and we ﬁt the system dynamics using piecewise linear functions in the form of a mixture of N Gaussians as proposed in [1] and [14] over the vectors { x it , u it , v it , x it +1 } T , where the i th index represents the i -th trajectory rollout on the robot.

Original: We build a Gaussian Mixture Model (GMM) to ﬁt piecewise linear dynamics so that within each GMM cluster, k i , we represent a linear Gaussian dynamics model as k i ( x i t
Revised: We build a Gaussian Mixture Model (GMM) to ﬁt piecewise linear dynamics so that within each GMM cluster, k i , we represent a linear Gaussian dynamics model as k i ( x i t

Original: +1 | x it , u it , v it ) and the marginal k i ( x it | u it , v it )
Revised: +1 | x it , u it , v it ) and the marginal k i ( x it | u it , v it )

Original: represents the portion of the state-actions space where our Gaussian model is valid.
Revised: represents the portion of the state-actions space where our Gaussian model is valid.

Original: In order to avoid the GMM not being a good separator of boundaries of complex modes, we follow [1], and use the GMM to generate a prior for the regression phase.
Revised: In order to avoid the GMM not being a good separator of boundaries of complex modes, we follow [1], and use the GMM to generate a prior for the regression phase.

Original: This enables us to obtain different linear modes at separate time steps based on the observed transitions, even when the states are dissimilar.
Revised: This enables us to obtain different linear modes at separate time steps based on the observed transitions, even when the states are dissimilar.

Original: The correct linear mode is obtained from the empirical covariance of { x t , u t , v t } with x t +1 in the current samples at time t .
Revised: The correct linear mode is obtained from the empirical covariance of { x t , u t , v t } with x t +1 in the current samples at time t .

Original: As in [1] and [14], we improve sample efﬁciency by reﬁtting the GMM at each iteration to all of the samples at all time steps from the current iteration and the previous 3 iterations and use this to construct a good prior for the dynamics.
Revised: As in [1] and [14], we improve sample efﬁciency by reﬁtting the GMM at each iteration to all of the samples at all time steps from the current iteration and the previous 3 iterations and use this to construct a good prior for the dynamics.

Original: We then obtain linear Gaussian dynamics by ﬁtting Gaussian distributions to samples { x it , u it , v it , x it +1 } which are then conditioned on [ x t , u t , v t ] T .
Revised: We then obtain linear Gaussian dynamics by ﬁtting Gaussian distributions to samples { x it , u it , v it , x it +1 } which are then conditioned on [ x t , u t , v t ] T .

Original: The prior allows us to build a normal-inverse Wishart prior on the conditioned Gaussians so that the maximum a posteriori estimates for mean µ and covariance Σ are given by
Revised: The prior allows us to build a normal-inverse Wishart prior on the conditioned Gaussians so that the maximum a posteriori estimates for mean µ and covariance Σ are given by

Original: [Equation]
Revised: [Equation]

Original: where Σ e and µ e are respectively the empirical covariance and mean of the dataset and Φ , µ 0 , m and n 0 are prior parameters so chosen: Φ = n 0 ¯Σ and µ 0 = ¯ µ .
Revised: where Σ e and µ e are respectively the empirical covariance and mean of the dataset and Φ , µ 0 , m and n 0 are prior parameters so chosen: Φ = n 0 ¯Σ and µ 0 = ¯ µ .

Original: As in [14], we set n 0 = m = 1 in order to ﬁt the prior to many samples than what is available at each time step.
Revised: As in [14], we set n 0 = m = 1 in order to ﬁt the prior to many samples than what is available at each time step.

Original: 5. S UPERVISED LEARNING OF GLOBAL NEURAL NETWORK POLICIES
Revised: 5. S UPERVISED LEARNING OF GLOBAL NEURAL NETWORK POLICIES

Original: The trajectories from the previous subsection are used to generate training data for global policies for the controller and adversary.
Revised: The trajectories from the previous subsection are used to generate training data for global policies for the controller and adversary.

Original: The local policies p ¯ τ ( u k | x k ) and p ¯ τ ( v k | x k ) will ideally be generated for all possible initial states x i 1 ∼ p ( x k ) .
Revised: The local policies p ¯ τ ( u k | x k ) and p ¯ τ ( v k | x k ) will ideally be generated for all possible initial states x i 1 ∼ p ( x k ) .

Original: Since the iLQG-based linearized dynamics will only be valid within a ﬁnite region of the state space; we used the KL-divergence constraint proposed in [16] to ensure the current protagonist policy does not diverge too much from the previous policy.
Revised: Since the iLQG-based linearized dynamics will only be valid within a ﬁnite region of the state space; we used the KL-divergence constraint proposed in [16] to ensure the current protagonist policy does not diverge too much from the previous policy.

Original: The learning problem involves imposing KL constraints on the cost function such that the protagonist controller distribution agree with the global policy π θ ( u t | x t ) by performing the following alternating optimization between two steps at each iteration i :
Revised: The learning problem involves imposing KL constraints on the cost function such that the protagonist controller distribution agree with the global policy π θ ( u t | x t ) by performing the following alternating optimization between two steps at each iteration i :

Original: [Equation]
Revised: [Equation]

Original: Essentially, p ( u k | x k ) above generates robust local policies; The ﬁrst step in (12) solves for a robust local policy p ( u k | x k ) via the min-max operation, by constraining p ( u k | x k ) against its global policy π i using the given KL divergence constraint; the second step projects the local linear Gaussian controller distribution onto the constraint set Π Θ , with respect to the divergence D ( p i , π ) .
Revised: Essentially, p ( u k | x k ) above generates robust local policies; The ﬁrst step in (12) solves for a robust local policy p ( u k | x k ) via the min-max operation, by constraining p ( u k | x k ) against its global policy π i using the given KL divergence constraint; the second step projects the local linear Gaussian controller distribution onto the constraint set Π Θ , with respect to the divergence D ( p i , π ) .

Original: The local policy that governs the agent’s dynamics is given by
Revised: The local policy that governs the agent’s dynamics is given by

Original: [Equation]
Revised: [Equation]

Original: Notice that the state is linearly dependent on the mean of the distribution p ( u k | x k ) and the covariance is independent of v k ; we therefore end up with a linear Gaussian controller for the robust guided policy search algorithm.
Revised: Notice that the state is linearly dependent on the mean of the distribution p ( u k | x k ) and the covariance is independent of v k ; we therefore end up with a linear Gaussian controller for the robust guided policy search algorithm.

Original: For linear Gaussian dynamics and policies, the iterative KL constraint during the S-step translates to minimizing the KL-divergence between policies i.e. ,
Revised: For linear Gaussian dynamics and policies, the iterative KL constraint during the S-step translates to minimizing the KL-divergence between policies i.e. ,

Original: [Equation]
Revised: [Equation]

Original: For the nonlinear cases that we treat in this work, the KL-divergence term in the S-step above is ﬂipped as proposed in [16] so that D KL ( π θ ( u k | x k ) (cid:107) p i ( u k | x k ) minimizes the augmented stage cost under p
Revised: For the nonlinear cases that we treat in this work, the KL-divergence term in the S-step above is ﬂipped as proposed in [16] so that D KL ( π θ ( u k | x k ) (cid:107) p i ( u k | x k ) minimizes the augmented stage cost under p

Original: i ( u k | x k ) w.r.t π θ ( u k | x k ) .
Revised: i ( u k | x k ) w.r.t π θ ( u k | x k ) .

Original: Therefore, the S-step minimizes,
Revised: Therefore, the S-step minimizes,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where x k,i,j is the j th sample from p i ( x k ) obtained by running p
Revised: where x k,i,j is the j th sample from p i ( x k ) obtained by running p

Original: i ( u k | x k ) on the real system, and
Revised: i ( u k | x k ) on the real system, and

Original: D i are the trajectory samples rolled out on the system.
Revised: D i are the trajectory samples rolled out on the system.

Original: Our robust GPS algorithm is thus given in algorithm 2.
Revised: Our robust GPS algorithm is thus given in algorithm 2.

Original: We follow the prior works in [1, 13, 14, 16] in computing the KL divergence term and we refer readers to these works for a more detailed treatment.
Revised: We follow the prior works in [1, 13, 14, 16] in computing the KL divergence term and we refer readers to these works for a more detailed treatment.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we present experiments to (i) conﬁrm our hypothesis that guided policy search methods, with carefully engineered complex high-dimensional policies, fail when exposed to the simplest of all perturbation signals; and (ii) answer the question of robustness using the trajectory optimization scheme and the robust guided policy framework we have presented.
Revised: In this section, we present experiments to (i) conﬁrm our hypothesis that guided policy search methods, with carefully engineered complex high-dimensional policies, fail when exposed to the simplest of all perturbation signals; and (ii) answer the question of robustness using the trajectory optimization scheme and the robust guided policy framework we have presented.

Original: We solve this under unknown dynamics.
Revised: We solve this under unknown dynamics.

Original: We answer both questions in this paper by using physics engines for policies that do not use visual features as feedback.
Revised: We answer both questions in this paper by using physics engines for policies that do not use visual features as feedback.

Original: Our validation examples are implemented in the MuJoCo physics engine [24] and the pybox2d game engine [6], aided by the publicly available GPS codebase [8].
Revised: Our validation examples are implemented in the MuJoCo physics engine [24] and the pybox2d game engine [6], aided by the publicly available GPS codebase [8].

Original: Highdimensional policy experiments are implemented on a PR2 robot, while low-dimensional policy experiments are implemented using a 2-DOF cart-pole swing-up experiment in the pybox2d game engine.
Revised: Highdimensional policy experiments are implemented on a PR2 robot, while low-dimensional policy experiments are implemented using a 2-DOF cart-pole swing-up experiment in the pybox2d game engine.

Original: The perturbation signal we consider are those that enter additively through the reward functions as described in subsection 4.1.
Revised: The perturbation signal we consider are those that enter additively through the reward functions as described in subsection 4.1.

Original: 6. S ENSITIVITY OF AN RL POLICY
Revised: 6. S ENSITIVITY OF AN RL POLICY

Original: We conducted simulated experiments demonstrating that guided policy search policies are sensitive to disturbance introduced into the action space of their policies.
Revised: We conducted simulated experiments demonstrating that guided policy search policies are sensitive to disturbance introduced into the action space of their policies.

Original: The 7-DoF robot result presented shortly previously appeared in our abstract that introduced robust GPS [21].
Revised: The 7-DoF robot result presented shortly previously appeared in our abstract that introduced robust GPS [21].

Original: The states x k are the joint angles, joint velocities, pose and velocity of the end effector as 3 points in 3-D.
Revised: The states x k are the joint angles, joint velocities, pose and velocity of the end effector as 3 points in 3-D.

Original: We assume the initial velocity of the 2-link and robot arm are zero.
Revised: We assume the initial velocity of the 2-link and robot arm are zero.

Original: Experimental tasks .
Revised: Experimental tasks .

Original: We simulated a 3D peg insertion task by a robot into a hole at the bottom of the slot Fig.
Revised: We simulated a 3D peg insertion task by a robot into a hole at the bottom of the slot Fig.

Original: The difﬁculty of this experiment stems from the discontinuity in dynamics from the contact between the peg and the walls.
Revised: The difﬁculty of this experiment stems from the discontinuity in dynamics from the contact between the peg and the walls.

Original: The 2-link arm swing-up experiment involves learning to balance the arm vertically about the origin of the cart (see right inset of Fig. 1).
Revised: The 2-link arm swing-up experiment involves learning to balance the arm vertically about the origin of the cart (see right inset of Fig. 1).

Original: The diffculty lies in the discontinuity of the dynamics along the vertical axis of the arm when it is upright.
Revised: The diffculty lies in the discontinuity of the dynamics along the vertical axis of the arm when it is upright.

Original: We initialized the linear-Gaussian controllers p i ( u k | x k ) , p i ( v k | x k ) in the neighborhood of the initial state x 1 using a PD control law for both the inverted pendulum task and the peg insertion task.
Revised: We initialized the linear-Gaussian controllers p i ( u k | x k ) , p i ( v k | x k ) in the neighborhood of the initial state x 1 using a PD control law for both the inverted pendulum task and the peg insertion task.

Original: Peg Insertion:
Revised: Peg Insertion:

Original: We implement the sensitivity algorithm for the peg insertion task of [8] with a robotic arm that requires dexterous manipulation.
Revised: We implement the sensitivity algorithm for the peg insertion task of [8] with a robotic arm that requires dexterous manipulation.

Original: The robot has 12 states consisting of joint angles
Revised: The robot has 12 states consisting of joint angles

Original: [Figure]
Revised: [Figure]

Original:  dv [Figure] s a r y c o [Figure]
Revised: a dv e r s a r y c o s t

Original: Optimal adversarial costs vs. γ -penalty
Revised: Optimal adversarial costs vs. γ -penalty

Original: [Figure]
Revised: [Figure]

Original: Optimal adversarial costs vs. γ -penalty [Figure]
Revised: penalty

Original: t a r ge t d i s t an c e [Figure] and angular velocities with two controller states.
Revised: Optimal adversarial costs vs. γ -penalty and angular velocities with two controller states.

Original: We train the protagonist’s policy using the GPS algorithm.
Revised: We train the protagonist’s policy using the GPS algorithm.

Original: We then pit an adversarial disturbance against the trained policy so that the adversary stays in closed-loop with the trained protagonist; The closed-loop cost function is given by
Revised: We then pit an adversarial disturbance against the trained policy so that the adversary stays in closed-loop with the trained protagonist; The closed-loop cost function is given by

Original: [Equation]
Revised: [Equation]

Original: (14) where γ represents the disturbance term, d x k denotes the end effector’s (EE) position at state x k and d (cid:63) denotes the EE’s position at the slot’s base.
Revised: (14) where γ represents the disturbance term, d x k denotes the end effector’s (EE) position at state x k and d (cid:63) denotes the EE’s position at the slot’s base.

Original: (cid:96) 12 ( ζ ) is a term that makes the peg reach the target at the hole’s base, precisely given by 12 ζ t ζ
Revised: (cid:96) 12 ( ζ ) is a term that makes the peg reach the target at the hole’s base, precisely given by 12 ζ t ζ

Original: + ( α + ζ 2 ) 12 .
Revised: + ( α + ζ 2 ) 12 .

Original: We set w u and w p to 10 − 6 andrespectively.
Revised: We set w u and w p to 10 − 6 andrespectively.

Original: For various values of γ , we check the sensitivity of the trained policy and its effect on the task performance by maximizing the cost function above w.r.t v k .
Revised: For various values of γ , we check the sensitivity of the trained policy and its effect on the task performance by maximizing the cost function above w.r.t v k .

Original: We run each sensitivity experiment for a total of 10 iterations.
Revised: We run each sensitivity experiment for a total of 10 iterations.

Original: Fig.
Revised: Fig.

Original: 2 shows that the adversary causes a sharp degradation in the protagonist’s performance for values of γ < 1 . 5 .
Revised: 2 shows that the adversary causes a sharp degradation in the protagonist’s performance for values of γ < 1 . 5 .

Original: This corresponds to when the GPS-trained policy gets destabilized and the arm struggles to reach the desired target.
Revised: This corresponds to when the GPS-trained policy gets destabilized and the arm struggles to reach the desired target.

Original: As values of γ ≥ 1 . 5 , however, we ﬁnd that the adversary has a reduced effect on task performance: the adversary’s effect decreases as γ gets larger.
Revised: As values of γ ≥ 1 . 5 , however, we ﬁnd that the adversary has a reduced effect on task performance: the adversary’s effect decreases as γ gets larger.

Original: Video of this result is available at https://goo.gl/YmmdhC.
Revised: Video of this result is available at https://goo.gl/YmmdhC.

Original: Arm Swing-up : Similar to the peg insertion task, we carry out a sensitivity evaluation procedure as we did for the robot arm with the peg insertion experiment with a 2D arm.
Revised: Arm Swing-up : Similar to the peg insertion task, we carry out a sensitivity evaluation procedure as we did for the robot arm with the peg insertion experiment with a 2D arm.

Original: The goal is to balance a 2D arm vertically about its origin.
Revised: The goal is to balance a 2D arm vertically about its origin.

Original: This agent has 7 states made up of two joint angles, two joint angle velocities and a 3D end effector point.
Revised: This agent has 7 states made up of two joint angles, two joint angle velocities and a 3D end effector point.

Original: The action space has two dimensions.
Revised: The action space has two dimensions.

Original: Contrary to the example in [8] that uses the Bregman alternating direction method of multipliers algorithm, we implement this experiment using the mirror descent GPS algorithm.
Revised: Contrary to the example in [8] that uses the Bregman alternating direction method of multipliers algorithm, we implement this experiment using the mirror descent GPS algorithm.

Original: We proceed as before: ﬁrst,[Figure] we optimize the optimal global policy using GPS on the agent; we then ﬁx the agent’s policy and pit various adversarial disturbances, controlled by the γ robustness term in order to evaluate its sensitivity.
Revised: We proceed as before: ﬁrst, we optimize the optimal global policy using GPS on the agent; we then ﬁx the agent’s policy and pit various adversarial disturbances, controlled by the γ robustness term in order to evaluate its sensitivity.

Original: We use a similar cost function as the one used for the peg insertion task.
Revised: We use a similar cost function as the one used for the peg insertion task.

Original: Fig.
Revised: Fig.

Original: 4 shows the evolution of the cost function as we vary the values of γ .
Revised: 3 shows the evolution of the cost function as we vary the values of γ .

Original: We notice that the augmented reward function gets larger as the adversary’s torque increases in magnitude and for lower values of γ , the augmented cost is relatively low stays the same.
Revised: We notice that the augmented reward function gets larger as the adversary’s torque increases in magnitude and for lower values of γ , the augmented cost is relatively low stays the same.

Original: The values of γ < 10 12 in Fig.
Revised: The values of γ < 10 12 in Fig.

Original: 4 represent the disturbance band where the protagonist’s learned policy becomes unstable and the arm never reaches the vertical position (see videos here: https://goo.gl/52rKnt).
Revised: 3 represent the disturbance band where the protagonist’s learned policy becomes unstable and the arm never reaches the vertical position (see videos here: https://goo.gl/52rKnt).

Original: This experiment further conﬁrms that the state-of-the-art reinforcement learning algorithms fail in the presence of additive disturbances to their parameter space making them brittle when used in situations that call for robustness.
Revised: This experiment further conﬁrms that the state-of-the-art reinforcement learning algorithms fail in the presence of additive disturbances to their parameter space making them brittle when used in situations that call for robustness.

Original: To mitigate these sensitivity errors, we implement the robust two-player, zero-sum game framework provided in 5 in order to develop more robust deep RL controllers and mitigate modeling errors and uncertainty. 
Revised: To mitigate these sensitivity errors, we implement the robust two-player, zero-sum game framework provided in 5 in order to develop more robust deep RL controllers and mitigate modeling errors and uncertainty. [Figure]

Original: 6. R OBUST RL WITH GPS
Revised: 6. R OBUST RL WITH GPS

Original: As proposed in section 5, our goal is to improve the robustness of the controller’s policy in the presence of modeling errors and uncertainties and transfer errors.
Revised: As proposed in section 5, our goal is to improve the robustness of the controller’s policy in the presence of modeling errors and uncertainties and transfer errors.

Original: We follow the formulation in sectionand generate v k from zero-mean, unit variance noise samples in every iteration.
Revised: We follow the formulation in sectionand generate v k from zero-mean, unit variance noise samples in every iteration.

Original: We employ various values of γ as a robustness parameter and we run the dynamic game during the trajectory optimization phase of the GPS algorithm.
Revised: We employ various values of γ as a robustness parameter and we run the dynamic game during the trajectory optimization phase of the GPS algorithm.

Original: Speciﬁcally, for the values of γ that the erstwhile policies in the previous subsection fail, we run the dynamic game algorithm to provide robustness in performnace at test time compared against the GPS algorithm.
Revised: Speciﬁcally, for the values of γ that the erstwhile policies in the previous subsection fail, we run the dynamic game algorithm to provide robustness in performnace at test time compared against the GPS algorithm.

Original: We run experiments on the peg insertion task to verify the algorithm.
Revised: We run experiments on the peg insertion task to verify the algorithm.

Original: Figure 4 shows the cost of running the robust GPS algorithm on the 7-DoF robot.
Revised: Figure 3 shows the cost of running the robust GPS algorithm on the 7-DoF robot.

Original: We see that the policies that show achieve optimal performance behavior are now less costly compared to vanilla GPS algorithm.
Revised: We see that the policies that show achieve optimal performance behavior are now less costly compared to vanilla GPS algorithm.

Original: For values of the sensitivity term γ that the algorithm erstwhile fails in, we now see smoother execution of the trajectory in trying to achieve our goal.
Revised: For values of the sensitivity term γ that the algorithm erstwhile fails in, we now see smoother execution of the trajectory in trying to achieve our goal.

Original: The modeling phase of the algorithm is also much less data consuming as our GMM algorithm now takes less samples before generalizing to the global model.
Revised: The modeling phase of the algorithm is also much less data consuming as our GMM algorithm now takes less samples before generalizing to the global model.

Original:  C ONCLUSION 
Revised:  C ONCLUSION AND FUTURE WORK

Original: We have evaluated the sensitivity of select deep reinforcement learning algorithms and shown that despite the most carefully designed policies, such policies implemented on real-world agents exhibit a potential for disastrous performances when unexpected such as when there exist modeling errors and discrepancy between training environment and real-world roll-outs (as evidenced by the results from the two dynamics the agent faces in our sensitivity experiment).
Revised: We have evaluated the sensitivity of select deep reinforcement learning algorithms and shown that despite the most carefully designed policies, such policies implemented on real-world agents exhibit a potential for disastrous performances when unexpected such as when there exist modeling errors and discrepancy between training environment and real-world roll-outs (as evidenced by the results from the two dynamics the agent faces in our sensitivity experiment).

Original: F UTURE WORK
Revised: We then test the dynamic trajectory optimization two-player algorithm on a robot motor task using Levine et al’s [14]’s guided policy search algorithm.

Original: We have presented an H ∞ framework in a two-player zero-sum dynamic gain in order to answer thequestion of robustness to learning the underlying dynamics and control strategy of a robot learning task. In order to evaluate our formulation in this paper, we have adopted and modiﬁed
Revised: In our implementation of the dynamic game algorithm, we focus on the robustness parameters that cause the robot’s policy to fail in the presence of the erstwhile sensysensitivity parameter.

Original: Levine et al’s [14]’s guided policy search algorithm  to test our assumptions.
Revised: We demonstrate that our two-player game framework allows agents operating under nonlinear dynamics to learn the underlying dynamics under signiﬁcantly more ﬁnite samples than vanilla GPS algorithm does – thus improving upon the Gaussian model mixture method used in [1] and [14]. Having agents that are robust to unmodeled nonlinearities, dynamics, and high frequency modes in a nonlinear dynamical system has long been a fundamental question that control theory strives to achieve.

Original: Our experiments show that learning robust policies under unknown nonlinear dynamicscan be made more efﬁcient with this game theory androbust control framework. We will later evaluate this framework on a real-world robot. We defer the implementation to a future work.
Revised: To the best of our knowledge, we are not aware of other works that addresses the robustness of deep policies that are trained end-to-end from a maximal robustness perspective. In future work, we hope to replace the crude Gaussian Mixture Model for the dynamics with a more sophisticated nonlinear model, and evaluate how the agent behaves in the presence of unknown dynamics.

Original: APPENDIX I
Revised: APPENDIX I

Original: The Q-coefﬁcients are estimated using LQR as follows:
Revised: The Q-coefﬁcients are estimated using LQR as follows:

Original: [Equation]
Revised: [Equation]

Original: where the subscript terms denote the partial derivatives with respect to the given matrix
Revised: where the subscript terms denote the partial derivatives with respect to the given matrix

Revised: [Equation]

Original: APPENDIX II
Revised: APPENDIX II

Original: The closed-form equations for the closed loop policy obtained after substituting (8) are given by
Revised: The closed-form equations for the closed loop policy obtained after substituting (8) are given by

Original: [Equation]
Revised: [Equation]

Original: Solving the system of equations in (16), we ﬁnd that 
Revised: Solving the system of equations in (16), we ﬁnd that [Equation] [Equation] δ

Original: [Equation]
Revised: [Equation]

Original: Suppose we let
Revised: Suppose we let

Original: [Equation]
Revised: [Equation]

Original: , so that
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: it follows that we can rewrite δ u (cid:63)k and δ v (cid:63)k as
Revised: it follows that we can rewrite δ u (cid:63)k and δ v (cid:63)k as

Original: [Equation]
Revised: [Equation]

Original: Plugging δ u (cid:63) and δ v (cid:63) back into the Q function expansion in (7), we ﬁnd that 
Revised: Plugging δ u (cid:63) and δ v (cid:63) back into the Q function expansion in (7), we ﬁnd that Q [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Comparing coefﬁcients, we obtain the following for the value function’s coefﬁcients
Revised: Comparing coefﬁcients, we obtain the following for the value function’s coefﬁcients

Original: [Equation]
Revised: [Equation]

Original: In practice, the K u k and K v k inverse terms above will result in numerical errors when the matrices are not positive deﬁnite since the DDP algorithm does not guarantee that the inverse of the Q functions will be positive deﬁnite.
Revised: In practice, the K u k and K v k inverse terms above will result in numerical errors when the matrices are not positive deﬁnite since the DDP algorithm does not guarantee that the inverse of the Q functions will be positive deﬁnite.

Original: To guarantee numerical stability and preserve the concave-convex properties, we add to K u k and K v k a sufﬁciently large positive quantity (greater than the lowest eigenvalue) in order to regularize the update equations [12, 5].
Revised: To guarantee numerical stability and preserve the concave-convex properties, we add to K u k and K v k a sufﬁciently large positive quantity (greater than the lowest eigenvalue) in order to regularize the update equations [12, 5].