Original: O BSERVATIONAL R OBUSTNESS AND I NVARIANCES IN R EINFORCEMENT L EARNING VIA L EXICOGRAPHIC O BJECTIVES
Revised: O BSERVATIONAL R OBUSTNESS AND I NVARIANCES IN R EINFORCEMENT L EARNING VIA L EXICOGRAPHIC O BJECTIVES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Policy robustness in
Revised: Policy robustness in

Original: Reinforcement Learning (RL) may not be desirable at any price; the alterations caused by robustness requirements from otherwise optimal policies should be explainable and quantifiable.
Revised: Reinforcement Learning (RL) may not be desirable at any price; the alterations caused by robustness requirements from otherwise optimal policies should be explainable and quantifiable.

Original: Policy gradient algorithms that have strong convergence guarantees are usually modified to obtain robust policies in ways that do not preserve algorithm guarantees, which defeats the purpose of formal robustness requirements.
Revised: Policy gradient algorithms that have strong convergence guarantees are usually modified to obtain robust policies in ways that do not preserve algorithm guarantees, which defeats the purpose of formal robustness requirements.

Original: In this work we study a notion of robustness in partially observable MDPs where state observations are perturbed by a noiseinduced stochastic kernel.
Revised: In this work we study a notion of robustness in partially observable MDPs where state observations are perturbed by a noiseinduced stochastic kernel.

Original: We characterise the set of policies that are maximally robust by analysing how the policies are altered by this kernel.
Revised: We characterise the set of policies that are maximally robust by analysing how the policies are altered by this kernel.

Original: We then establish a connection between such robust policies and certain properties of the noise kernel as well as with structural properties of the underlying MDPs, constructing sufficient conditions for policy robustness.
Revised: We then establish a connection between such robust policies and certain properties of the noise kernel as well as with structural properties of the underlying MDPs, constructing sufficient conditions for policy robustness.

Original: We use these notions to propose a robustness-inducing scheme, applicable to any policy gradient algorithm to formally trade off the reward achieved by a policy with its robustness level through lexicographic optimisation which preserves convergence properties of the orig
Revised: We use these notions to propose a robustness-inducing scheme, applicable to any policy gradient algorithm to formally trade off the reward achieved by a policy with its robustness level through lexicographic optimisation which preserves convergence properties of the orig

Original: - inal algorithm.
Revised: - inal algorithm.

Original: We test the the proposed approach on safety-critical RL environments and show how the proposed method helps achieve high robustness in observational noise problems
Revised: We test the the proposed approach on safety-critical RL environments and show how the proposed method helps achieve high robustness in observational noise problems

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Robustness in Reinforcement Learning (RL) (Morimoto & Doya, 2005) can be looked at from different perspectives: (1) distributional shifts in the training data with respect to the deployment stage
Revised: Robustness in Reinforcement Learning (RL) (Morimoto & Doya, 2005) can be looked at from different perspectives: (1) distributional shifts in the training data with respect to the deployment stage

Original: Satia & Lave Jr (1973); Heger (1994); Nilim & El Ghaoui (2005); Xu & Mannor (2006); (2) uncertainty in the model or observations (Pinto et al., 2017; Everett et al., 2021); (3) adversarial attacks against actions (Pattanaik et al., 2017; Fischer et al., 2019); and (4) sensitivity of neural networks (used as policy or value function approximators) towards input disturbances (Kos & Song, 2017;
Revised: Satia & Lave Jr (1973); Heger (1994); Nilim & El Ghaoui (2005); Xu & Mannor (2006); (2) uncertainty in the model or observations (Pinto et al., 2017; Everett et al., 2021); (3) adversarial attacks against actions (Pattanaik et al., 2017; Fischer et al., 2019); and (4) sensitivity of neural networks (used as policy or value function approximators) towards input disturbances (Kos & Song, 2017;

Original: Huang et al., 2017).
Revised: Huang et al., 2017).

Original: Robustness does not naturally emerge in most RL settings, since agents are typically only trained in a single, unchanging environment:
Revised: Robustness does not naturally emerge in most RL settings, since agents are typically only trained in a single, unchanging environment:

Original: There is a trade-off between how robust a policy is and how close it is to the set of optimal policies in its training environment, and in safety-critical applications we may need to provide formal guarantees for this trade-off.
Revised: There is a trade-off between how robust a policy is and how close it is to the set of optimal policies in its training environment, and in safety-critical applications we may need to provide formal guarantees for this trade-off.

Original: Motivation Consider a complex dynamical system where we need to synthesise a controller (policy) through a model-free approach.
Revised: Motivation Consider a complex dynamical system where we need to synthesise a controller (policy) through a model-free approach.

Original: When using a simulator for training we expect the deployment of the controller in the real system to be affected by different sources of noise, possibly not predictable or modelled ( e.g. for networked components we may have sensor faults, communication delays, etc ).
Revised: When using a simulator for training we expect the deployment of the controller in the real system to be affected by different sources of noise, possibly not predictable or modelled ( e.g. for networked components we may have sensor faults, communication delays, etc ).

Original: In safety-critical systems, robustness (in terms of successfully controlling the system under disturbances) should preserve formal guarantees, and plenty of effort has been put on developing formal convergence guarantees on policy gradient algorithms (Agarwal et al., 2021; Bhandari & Russo, 2019) which vanish when “robustifying” policies through regularisation or adversarial approaches.
Revised: In safety-critical systems, robustness (in terms of successfully controlling the system under disturbances) should preserve formal guarantees, and plenty of effort has been put on developing formal convergence guarantees on policy gradient algorithms (Agarwal et al., 2021; Bhandari & Russo, 2019) which vanish when “robustifying” policies through regularisation or adversarial approaches.

Original: Therefore, for such applications one would need a scheme to regulate the robustnessutility trade-off in RL policies that preserves the formal guarantees of original algorithms and retains sub-optimality conditions for the original problem.
Revised: Therefore, for such applications one would need a scheme to regulate the robustnessutility trade-off in RL policies that preserves the formal guarantees of original algorithms and retains sub-optimality conditions for the original problem.

Original: Lexicographic Reinforcement Learning Recently, lexicographic optimisation (Isermann, 1982;
Revised: Lexicographic Reinforcement Learning Recently, lexicographic optimisation (Isermann, 1982;

Original: Rentmeesters et al., 1996) has been applied to the multi-objective RL setting (Skalse et al., 2022b).
Revised: Rentmeesters et al., 1996) has been applied to the multi-objective RL setting (Skalse et al., 2022b).

Original: In an LRL setting with different reward-maximising objective functions { K i } 1 ≤ i ≤ n , some objec- tives may be more important than others, and so we may want to obtain policies that solve the multi-objective problem in a lexicographically prioritised way, i.e. , “find the policies that optimise objective i (reasonably well), and from those the ones that optimise objective i + 1 (reasonably well), and so on”.
Revised: In an LRL setting with different reward-maximising objective functions { K i } 1 ≤ i ≤ n , some objec- tives may be more important than others, and so we may want to obtain policies that solve the multi-objective problem in a lexicographically prioritised way, i.e. , “find the policies that optimise objective i (reasonably well), and from those the ones that optimise objective i + 1 (reasonably well), and so on”.

Original: There exist both value- and policy-based algorithms for LRL, and the approach is broadly applicable to (most) state of the art RL algorithms (Skalse et al., 2022b).
Revised: There exist both value- and policy-based algorithms for LRL, and the approach is broadly applicable to (most) state of the art RL algorithms (Skalse et al., 2022b).

Original: Previous Work
Revised: Previous Work

Original: In robustness against model uncertainty , the MDP may have noisy or uncertain reward signals or transition probabilities, as well as possible resulting distributional shifts in the training data (Heger, 1994; Xu & Mannor, 2006; Fu et al., 2018; Pattanaik et al., 2018;
Revised: In robustness against model uncertainty , the MDP may have noisy or uncertain reward signals or transition probabilities, as well as possible resulting distributional shifts in the training data (Heger, 1994; Xu & Mannor, 2006; Fu et al., 2018; Pattanaik et al., 2018;

Original: Pirotta et al., 2013; Abdullah et al., 2019), which connects to ideas on distributionally robust optimisation (Wiesemann et al., 2014; Van Parys et al., 2015).
Revised: Pirotta et al., 2013; Abdullah et al., 2019), which connects to ideas on distributionally robust optimisation (Wiesemann et al., 2014; Van Parys et al., 2015).

Original: One of the first examples is Heger (1994), where the author proposes using minimax approaches to learn Q functions that minimise the worst case total discounted cost in a general MDP setting.
Revised: One of the first examples is Heger (1994), where the author proposes using minimax approaches to learn Q functions that minimise the worst case total discounted cost in a general MDP setting.

Original: Derman et al.
Revised: Derman et al.

Original: (2020) propose a Bayesian approach to deal with uncertainty in the transitions.
Revised: (2020) propose a Bayesian approach to deal with uncertainty in the transitions.

Original: Another robustness sub-problem is studied in the form of adversarial attacks or disturbances by considering adversarial attacks on policies or action selection in RL agents (Gleave et al., 2020; Lin et al., 2017; Tessler et al., 2019; Pan et al., 2019; Tan et al., 2020; Klima et al., 2019).
Revised: Another robustness sub-problem is studied in the form of adversarial attacks or disturbances by considering adversarial attacks on policies or action selection in RL agents (Gleave et al., 2020; Lin et al., 2017; Tessler et al., 2019; Pan et al., 2019; Tan et al., 2020; Klima et al., 2019).

Original: Recently, Gleave et al.
Revised: Recently, Gleave et al.

Original: (2020) propose the idea that instead of modifying observations, one could attack RL agents by swapping the policy for an adversarial one at given times.
Revised: (2020) propose the idea that instead of modifying observations, one could attack RL agents by swapping the policy for an adversarial one at given times.

Original: For a detailed review on Robust RL see Moos et al.
Revised: For a detailed review on Robust RL see Moos et al.

Original: Our work focuses in the study of robustness versus observational disturbances , where agents observe a disturbed state measurement and use it as input for the policy (Kos & Song, 2017; Huang et al., 2017; Behzadan & Munir, 2017;
Revised: Our work focuses in the study of robustness versus observational disturbances , where agents observe a disturbed state measurement and use it as input for the policy (Kos & Song, 2017; Huang et al., 2017; Behzadan & Munir, 2017;

Original: Mandlekar et al., 2017; Zhang et al., 2020; 2021).
Revised: Mandlekar et al., 2017; Zhang et al., 2020; 2021).

Original: In particular Mandlekar et al.
Revised: In particular Mandlekar et al.

Original: (2017) consider both random and adversarial state perturbations, and introduce physically plausible generation of disturbances in the training of RL agents that make the resulting policy robust towards realistic disturbances.
Revised: (2017) consider both random and adversarial state perturbations, and introduce physically plausible generation of disturbances in the training of RL agents that make the resulting policy robust towards realistic disturbances.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020) propose a state-adversarial MDP framework, and utilise adversarial regularising terms that can be added to different deep RL algorithms to make the resulting policies more robust to observational disturbances, minimising the distance bound between disturbed and undisturbed policies through convex relaxations of neural networks to obtain robustness guarantees.
Revised: (2020) propose a state-adversarial MDP framework, and utilise adversarial regularising terms that can be added to different deep RL algorithms to make the resulting policies more robust to observational disturbances, minimising the distance bound between disturbed and undisturbed policies through convex relaxations of neural networks to obtain robustness guarantees.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) study how LSTM increases robustness with optimal state-perturbing adversaries.
Revised: (2021) study how LSTM increases robustness with optimal state-perturbing adversaries.

Original: 1. M AIN C ONTRIBUTIONS
Revised: 1. M AIN C ONTRIBUTIONS

Original: Most existing work on RL with observational disturbances proposes modifying RL algorithms (learning to deal with perturbations through linear combinations of regularising loss terms or adversarial terms) that come at the cost of explainability (in terms of sub-optimality bounds) and verifiability , since the induced changes in the new policies result in a loss of convergence guarantees.
Revised: Most existing work on RL with observational disturbances proposes modifying RL algorithms (learning to deal with perturbations through linear combinations of regularising loss terms or adversarial terms) that come at the cost of explainability (in terms of sub-optimality bounds) and verifiability , since the induced changes in the new policies result in a loss of convergence guarantees.

Original: Our main contributions are summarised in the following points.
Revised: Our main contributions are summarised in the following points.

Original: Structure of Robust Policy Sets 

Revised: Structure of Robust Policy Sets

Original: We consider general unknown stochastic disturbances and formulate a quantitative definition of observational robustness that allows us to characterise the sets of robust policies for any MDP in the form of operator-invariant sets.
Revised: We consider general unknown stochastic disturbances and formulate a quantitative definition of observational robustness that allows us to characterise the sets of robust policies for any MDP in the form of operator-invariant sets.

Original: We analyse how the structure of these sets depends on the MDP and noise kernel, and obtain an inclusion relation ( i.e. Inclusion
Revised: We analyse how the structure of these sets depends on the MDP and noise kernel, and obtain an inclusion relation ( i.e. Inclusion

Original: Theorem, Section 3) providing intuition into how we can search for robust policies more effectively.
Revised: Theorem, Section 3) providing intuition into how we can search for robust policies more effectively.

Original: Verifiable Robustness through LRL While LRL is developed for reward maximising objectives, through the proposed observational robustness definition we can cast robustness as a lexicographic objective, allowing us to retain policy optimality up to a specified tolerance while maximising robustness and yielding a mechanism to formally control the performance-robustness trade-off.
Revised: Verifiable Robustness through LRL While LRL is developed for reward maximising objectives, through the proposed observational robustness definition we can cast robustness as a lexicographic objective, allowing us to retain policy optimality up to a specified tolerance while maximising robustness and yielding a mechanism to formally control the performance-robustness trade-off.

Original: This preserves convergence guarantees of the original algorithm and yields formal bounds on policy suboptimality.
Revised: This preserves convergence guarantees of the original algorithm and yields formal bounds on policy suboptimality.

Original: We provide numerical examples on how this logic is applied to existing policy gradient
Revised: We provide numerical examples on how this logic is applied to existing policy gradient

Revised: [Figure]

Revised: Π ΠΠ D Π T Π Π ∗ π t { π ∈ Π : J ∗ − J ( π ) ≤ ϵ }

Original: [Figure]
Revised: [Figure]

Original: algorithms, compare with existing algorithms in previous work, and verify how the previously mentioned Inclusion Theorem helps to induce more robust policies while retaining algorithm optimality.
Revised: algorithms, compare with existing algorithms in previous work, and verify how the previously mentioned Inclusion Theorem helps to induce more robust policies while retaining algorithm optimality.

Original: Figure 1 represents a qualitative interpretation of the results in this work (the structure of the robust sets will become clear in following sections).
Revised: Figure 1 represents a qualitative interpretation of the results in this work (the structure of the robust sets will become clear in following sections).

Original: 1. P RELIMINARIES
Revised: 1. P RELIMINARIES

Original: Notation We use calligraphic letters A for collections of sets and ∆( A ) as the space of probability measures over A .
Revised: Notation We use calligraphic letters A for collections of sets and ∆( A ) as the space of probability measures over A .

Original: For two elements of a vector space we use ⟨· , ·⟩ as the inner product.
Revised: For two elements of a vector space we use ⟨· , ·⟩ as the inner product.

Original: We use 1 n as a column-vector of size n that has all entries equal to 1.
Revised: We use 1 n as a column-vector of size n that has all entries equal to 1.

Original: We say that an MDP is ergodic if for any policy the resulting Markov Chain (MC) is ergodic.
Revised: We say that an MDP is ergodic if for any policy the resulting Markov Chain (MC) is ergodic.

Original: We say that S is a n × n row-stochastic matrix if S ij ≥ 0 and each row of S sums to 1.
Revised: We say that S is a n × n row-stochastic matrix if S ij ≥ 0 and each row of S sums to 1.

Original: We use α, β, η for learning rates, µ for probability distributions and θ ∈ Θ for parameters.
Revised: We use α, β, η for learning rates, µ for probability distributions and θ ∈ Θ for parameters.

Original: Lexicographic Reinforcement Learning We propose using policy-based LRL (PB-LRL) to encode the idea that, when learning how to solve an RL task, robustness is important but not at any price , i.e. , we would like to solve the original objective reasonably well, and from those policies efficiently find the most robust one 2 .
Revised: Lexicographic Reinforcement Learning We propose using policy-based LRL (PB-LRL) to encode the idea that, when learning how to solve an RL task, robustness is important but not at any price , i.e. , we would like to solve the original objective reasonably well, and from those policies efficiently find the most robust one 2 .

Original: Consider a parameterised policy π θ with θ ∈ Θ , and two objective functions K 1 and K 2 .
Revised: Consider a parameterised policy π θ with θ ∈ Θ , and two objective functions K 1 and K 2 .

Original: PB-LRL uses a multi-timescale optimisation scheme to optimise θ faster for higher-priority objectives, iteratively updating the constraints induced by these priorities and encoding them via Lagrangian relaxation techniques (Bertsekas, 1997).
Revised: PB-LRL uses a multi-timescale optimisation scheme to optimise θ faster for higher-priority objectives, iteratively updating the constraints induced by these priorities and encoding them via Lagrangian relaxation techniques (Bertsekas, 1997).

Original: Let θ ′ ∈ arg max θ K 1 ( θ ) .
Revised: Let θ ′ ∈ arg max θ K 1 ( θ ) .

Original: Then,
Revised: Then,

Original: PB-LRL can be used to find parameters:
Revised: PB-LRL can be used to find parameters:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: This is done by computing the (estimated) gradient ascent update:
Revised: This is done by computing the (estimated) gradient ascent update:

Original: [Equation]
Revised: [Equation]

Original: where ˆ K ( θ ) := ( β 1 t + λβ 2 t ) · K 1 ( θ )+ β 2 t · K 2 ( θ ) , λ is a Langrange multiplier, β 1 t , β 2 t , η t are learning rates, and ˆ k 1 is an estimate of K 1 ( θ ′ ) .
Revised: where ˆ K ( θ ) := ( β 1 t + λβ 2 t ) · K 1 ( θ )+ β 2 t · K 2 ( θ ) , λ is a Langrange multiplier, β 1 t , β 2 t , η t are learning rates, and ˆ k 1 is an estimate of K 1 ( θ ′ ) .

Original: Typically, we set ϵ t → 0 , though we can use other tolerances too, e.g. , ϵ t = 0 .
Revised: Typically, we set ϵ t → 0 , though we can use other tolerances too, e.g. , ϵ t = 0 .

Original: 9 · ˆ k 1 .
Revised: 9 · ˆ k 1 .

Original: For more detail on the convergence proofs and particularities of PB-LRL we refer the reader to Skalse et al.
Revised: For more detail on the convergence proofs and particularities of PB-LRL we refer the reader to Skalse et al.

Original: (2022b).
Revised: (2022b).

Original:  O BSERVATIONALLY R OBUST R EINFORCEMENT L EARNING
Revised:  O BSERVATIONALLY R OBUST R EINFORCEMENT L EARNING

Original: We restrict the robustness problem considered in this work to the following version of a noiseinduced partially observable Markov Decision Process (Spaan, 2012).
Revised: We restrict the robustness problem considered in this work to the following version of a noiseinduced partially observable Markov Decision Process (Spaan, 2012).

Original: Definition 1.
Revised: Definition 1.

Original: An observationally-disturbed MDP (DOMDP) is (a POMDP) defined by the tuple ( X, U, P, R, T, γ ) where X is a finite set of states, U is a set of actions, P : U × X (cid:55)→ ∆( X ) is a probability measure of the transitions between states and R :
Revised: An observationally-disturbed MDP (DOMDP) is (a POMDP) defined by the tuple ( X, U, P, R, T, γ ) where X is a finite set of states, U is a set of actions, P : U × X (cid:55)→ ∆( X ) is a probability measure of the transitions between states and R :

Original: X × U × X
Revised: X × U × X

Original: (cid:55)→ R is a reward function.
Revised: (cid:55)→ R is a reward function.

Original: The map T : X (cid:55)→ ∆( X ) is a stochastic kernel induced by some unknown noise signal, such that T ( y | x ) is the probability of measuring y while the true state is x , and acts only on the state observations.
Revised: The map T : X (cid:55)→ ∆( X ) is a stochastic kernel induced by some unknown noise signal, such that T ( y | x ) is the probability of measuring y while the true state is x , and acts only on the state observations.

Original: At last γ ∈ [0 , 1] is a reward discount factor.
Revised: At last γ ∈ [0 , 1] is a reward discount factor.

Original: In a DOMDP 3 agents can measure the full state, but the measurement will be disturbed by some unknown random signal in the policy roll-out phase .
Revised: In a DOMDP 3 agents can measure the full state, but the measurement will be disturbed by some unknown random signal in the policy roll-out phase .

Original: Unlike the POMDP setting the agent has access to the true state x during learning of the policies (the simulator is noise-free), and no information about the noise kernel T or a way to estimate it.
Revised: Unlike the POMDP setting the agent has access to the true state x during learning of the policies (the simulator is noise-free), and no information about the noise kernel T or a way to estimate it.

Original: The difficulty of acting in such DOMDP is that the transitions are actually undisturbed and a function of the true state x , but agents will have to act based on disturbed states ˜ x ∼ T ( · | x ) .
Revised: The difficulty of acting in such DOMDP is that the transitions are actually undisturbed and a function of the true state x , but agents will have to act based on disturbed states ˜ x ∼ T ( · | x ) .

Original: We then need to construct policies that will be as robust as possible against noise without being able to construct noise estimates.
Revised: We then need to construct policies that will be as robust as possible against noise without being able to construct noise estimates.

Original: This is a setting that reflects many control problems; we can design a controller for ideal noise-less conditions, and we know that at deployment there will likely be noise, data corruption, adversarial perturbations, etc. , but we do not have certainty on the disturbance structure.
Revised: This is a setting that reflects many control problems; we can design a controller for ideal noise-less conditions, and we know that at deployment there will likely be noise, data corruption, adversarial perturbations, etc. , but we do not have certainty on the disturbance structure.

Original: Remark 1.
Revised: Remark 1.

Original: In Sections 2 and 3 we reason about the influence of T in the characterisation of robustness and robust policies.
Revised: In Sections 2 and 3 we reason about the influence of T in the characterisation of robustness and robust policies.

Original: However, when trying to learn robust policies we will to introduce uncertainty in the training phase.
Revised: However, when trying to learn robust policies we will to introduce uncertainty in the training phase.

Original: How to add this uncertainty will become clear in further sections.
Revised: How to add this uncertainty will become clear in further sections.

Original: A (memoryless) policy for the agent is a stochastic kernel π : X (cid:55)→ ∆( U ) .
Revised: A (memoryless) policy for the agent is a stochastic kernel π : X (cid:55)→ ∆( U ) .

Original: For simplicity, we overload notation on π , denoting by π ( x, u ) as the probability of taking action u at state x under the stochastic policy π in the MDP, i.e. , π ( x, u ) =
Revised: For simplicity, we overload notation on π , denoting by π ( x, u ) as the probability of taking action u at state x under the stochastic policy π in the MDP, i.e. , π ( x, u ) =

Original: Pr { u | x } .
Revised: Pr { u | x } .

Original: The value function of a policy π ,
Revised: The value function of a policy π ,

Original: V π : X (cid:55)→ R , is given by V π ( x 0 ) =
Revised: V π : X (cid:55)→ R , is given by V π ( x 0 ) =

Original: E [ (cid:80) ∞ t =

Revised: E [ (cid:80) ∞ t =

Original: γ t R ( x t , π ( x t ) , x t +1 )] .
Revised: γ t R ( x t , π ( x t ) , x t +1 )] .

Original: The action-value function of π ( Q -function) is given by Q π ( x, u )
Revised: The action-value function of π ( Q -function) is given by Q π ( x, u )

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: y ∈
Revised: y ∈

Original: X P ( x, u, y )
Revised: X P ( x, u, y )

Original: ( R ( x, u, y ) + γV π ( y )) .
Revised: ( R ( x, u, y ) + γV π ( y )) .

Original: It is well known that, under mild conditions (Sutton & Barto, 2018), the optimal value function can be obtained by means of the Bellman equation V ∗ ( x ) := max u (cid:80)
Revised: It is well known that, under mild conditions (Sutton & Barto, 2018), the optimal value function can be obtained by means of the Bellman equation V ∗ ( x ) := max u (cid:80)

Original: y ∈
Revised: y ∈

Original: X P ( x, u, y )
Revised: X P ( x, u, y )

Original: ( R ( x, u, y ) + γV ∗
Revised: ( R ( x, u, y ) + γV ∗

Original: y )) , and an optimal policy is guaranteed to exist such that π ∗ ( x ) :
Revised: y )) , and an optimal policy is guaranteed to exist such that π ∗ ( x ) :

Original: = arg max π V π ( x ) ∀ x ∈ X .
Revised: = arg max π V π ( x ) ∀ x ∈ X .

Original: We then define the objective function as J ( π ) :
Revised: We then define the objective function as J ( π ) :

Original: =
Revised: =

Original: E x 0 ∼ µ 

Revised: E x 0 ∼ µ

Original: [ V π ( x 0 )] with µ 0 being a distribution of initial states, and we use J ∗ := max π J ( π ) .
Revised: [ V π ( x 0 )] with µ 0 being a distribution of initial states, and we use J ∗ := max π J ( π ) .

Original: If a policy is parameterised by θ ∈ Θ we write π θ and J ( θ ) .
Revised: If a policy is parameterised by θ ∈ Θ we write π θ and J ( θ ) .

Original: Assumption 1.
Revised: Assumption 1.

Original: For any DOMDP and policy π , the resulting MC is irreducible and aperiodic.
Revised: For any DOMDP and policy π , the resulting MC is irreducible and aperiodic.

Original: Assumption 1 ensures that for any DOMDP and policy π , there exists a stationary probability distribution of states µ π ∈ ∆( X ) , and for every policy and state this probability is larger than zero.
Revised: Assumption 1 ensures that for any DOMDP and policy π , there exists a stationary probability distribution of states µ π ∈ ∆( X ) , and for every policy and state this probability is larger than zero.

Original: We now formalise a notion of observational robustness .
Revised: We now formalise a notion of observational robustness .

Original: Firstly, due to the presence of the stochastic kernel T , the policy we are applying is altered as we are applying a collection of actions in a possibly wrong state.
Revised: Firstly, due to the presence of the stochastic kernel T , the policy we are applying is altered as we are applying a collection of actions in a possibly wrong state.

Original: This behaviour can be formally captured by:
Revised: This behaviour can be formally captured by:

Original: [Equation]
Revised: [Equation]

Original: where ⟨ π, T ⟩ : X (cid:55)→ ∆( U ) is the disturbed policy, which averages the current policy given the error induced by the presence of the stochastic kernel.
Revised: where ⟨ π, T ⟩ : X (cid:55)→ ∆( U ) is the disturbed policy, which averages the current policy given the error induced by the presence of the stochastic kernel.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: ∆
Revised: ∆

Original: [Equation]
Revised: [Equation]

Original: ) is an averaging operator yielding the alteration of the policy due to noise.
Revised: ) is an averaging operator yielding the alteration of the policy due to noise.

Original: We can then define the robustness regret :
Revised: We can then define the robustness regret :

Original: [Equation]
Revised: [Equation]

Original: Definition 2 (Policy Robustness) .
Revised: Definition 2 (Policy Robustness) .

Original: We say that a policy π is κ - robust against a stochastic kernel T if ρ ( π, T ) ≤ κ .
Revised: We say that a policy π is κ - robust against a stochastic kernel T if ρ ( π, T ) ≤ κ .

Original: If π is 0 - robust we say it is maximally robust.
Revised: If π is 0 - robust we say it is maximally robust.

Original: We define the sets of κ -robust policies, Π κ :=
Revised: We define the sets of κ -robust policies, Π κ :=

Original: { π ∈ Π : ρ ( π, T ) ≤ κ } , with Π 0 being the set of maximally robust policies.
Revised: { π ∈ Π : ρ ( π, T ) ≤ κ } , with Π 0 being the set of maximally robust policies.

Original: One can motivate the characterisation and models above from a control perspective, where policies use as input discretised state measurements with possible sensor measurement errors.
Revised: One can motivate the characterisation and models above from a control perspective, where policies use as input discretised state measurements with possible sensor measurement errors.

Original: Formally ensuring robustness properties when learning RL policies will, in general, force the resulting policies to deviate from optimality in the undisturbed MDP.
Revised: Formally ensuring robustness properties when learning RL policies will, in general, force the resulting policies to deviate from optimality in the undisturbed MDP.

Original: With this motivation, we propose solving the problem of increasing robustness of RL policies through a hierarchical lexicographic approach, which naturally incorporates trade-offs during the policy design.
Revised: With this motivation, we propose solving the problem of increasing robustness of RL policies through a hierarchical lexicographic approach, which naturally incorporates trade-offs during the policy design.

Original: The first objective is to minimise the distance J ∗ − J ( π ) up to some tolerance.
Revised: The first objective is to minimise the distance J ∗ − J ( π ) up to some tolerance.

Original: Then, from the policies that satisfy this constraint, we want to steer the learning algorithm towards a maximally robust policy according to the metric defined in Definition 2.
Revised: Then, from the policies that satisfy this constraint, we want to steer the learning algorithm towards a maximally robust policy according to the metric defined in Definition 2.

Original: This can be formulated as the following problem, to be solved by means of
Revised: This can be formulated as the following problem, to be solved by means of

Original: LRL casting robustness as a valid lexicographic objective.
Revised: LRL casting robustness as a valid lexicographic objective.

Original: Problem 1.
Revised: Problem 1.

Original: For a DOMDP and a given tolerance level ϵ , derive a policy π ϵ that satisfies J ∗ −
Revised: For a DOMDP and a given tolerance level ϵ , derive a policy π ϵ that satisfies J ∗ −

Original: J ( π ϵ ) ≤ ϵ as a prioritised objective and is as robust as possible according to Definition 2.
Revised: J ( π ϵ ) ≤ ϵ as a prioritised objective and is as robust as possible according to Definition 2.

Original:  C HARACTERISATION OF R OBUST P OLICIES
Revised:  C HARACTERISATION OF R OBUST P OLICIES

Original: An important question to be addressed, before trying to synthesise robust policies through LRL, is what these robust policies look like, and how they are related to DOMDP properties.
Revised: An important question to be addressed, before trying to synthesise robust policies through LRL, is what these robust policies look like, and how they are related to DOMDP properties.

Original: The robustness notion in Definition 2 is intuitive and it allows us to classify policies.
Revised: The robustness notion in Definition 2 is intuitive and it allows us to classify policies.

Original: We begin by exploring what are the types of policies that are maximally robust, starting with the set of constant policies and set of fix point of the operator ⟨· , T ⟩ , whose formal descriptions are now provided.
Revised: We begin by exploring what are the types of policies that are maximally robust, starting with the set of constant policies and set of fix point of the operator ⟨· , T ⟩ , whose formal descriptions are now provided.

Original: Definition 3.
Revised: Definition 3.

Original: A policy π : X (cid:55)→ ∆( U ) is said to be constant if π ( x ) =
Revised: A policy π : X (cid:55)→ ∆( U ) is said to be constant if π ( x ) =

Original: π ( y ) for all x, y ∈ X , and the collection of all constant policies is denoted by ¯Π .
Revised: π ( y ) for all x, y ∈ X , and the collection of all constant policies is denoted by ¯Π .

Original: A policy π : X (cid:55)→ ∆( U ) is called a fixed point of the operator ⟨· , T ⟩ if π ( x ) = ⟨ π, T ⟩ ( x ) for all x ∈ X .
Revised: A policy π : X (cid:55)→ ∆( U ) is called a fixed point of the operator ⟨· , T ⟩ if π ( x ) = ⟨ π, T ⟩ ( x ) for all x ∈ X .

Original: The collection of all fixed points will be denoted by Π
Revised: The collection of all fixed points will be denoted by Π

Original: T .
Revised: T .

Original: In other words, a constant policy is any policy that yields the same action distribution for any state, and a fixed point policy is any policy whose action distributions are un-altered by the noise kernel.
Revised: In other words, a constant policy is any policy that yields the same action distribution for any state, and a fixed point policy is any policy whose action distributions are un-altered by the noise kernel.

Original: Observe furthermore that Π T only depends on the kernel T and the set 4 X .
Revised: Observe furthermore that Π T only depends on the kernel T and the set 4 X .

Original: We now present a proposition that links the two sets of policies in Definition 3 with our notion of robustness.
Revised: We now present a proposition that links the two sets of policies in Definition 3 with our notion of robustness.

Original: Proposition 1. Consider a DOMDP as in Definition 1, the robustness notion given in Definitionand the concepts in Definition 3, then we have that
Revised: Proposition 1. Consider a DOMDP as in Definition 1, the robustness notion given in Definitionand the concepts in Definition 3, then we have that

Original: [Equation]
Revised: [Equation]

Original: The importance of Proposition 1 is that it allows us to produce (approximately) maximally robust policies by computing the distance of a policy to either the set of constant policies or to the fix point of the operator ⟨· , T ⟩ , and this is at the core of the construction in Section 4.
Revised: The importance of Proposition 1 is that it allows us to produce (approximately) maximally robust policies by computing the distance of a policy to either the set of constant policies or to the fix point of the operator ⟨· , T ⟩ , and this is at the core of the construction in Section 4.

Original: However, before this, let us introduce another set that is sandwiched between Π 0 and Π T .
Revised: However, before this, let us introduce another set that is sandwiched between Π 0 and Π T .

Original: Let us assume we have a policy iteration algorithm that employs an action-value function Q π and policy π .
Revised: Let us assume we have a policy iteration algorithm that employs an action-value function Q π and policy π .

Original: The advantage function for π is defined as A π ( x, u ) :
Revised: The advantage function for π is defined as A π ( x, u ) :

Original: = Q π ( x, u ) − V π ( x ) and can be used as a maximisation objective to learn optimal policies (as in, e.g. , A2C (Sutton et al., 1999), A3C (Mnih et al., 2016)).
Revised: = Q π ( x, u ) − V π ( x ) and can be used as a maximisation objective to learn optimal policies (as in, e.g. , A2C (Sutton et al., 1999), A3C (Mnih et al., 2016)).

Original: We can similarly define the noise disadvantage (a form of negative advantage) of policy π as:
Revised: We can similarly define the noise disadvantage (a form of negative advantage) of policy π as:

Original: [Equation]
Revised: [Equation]

Original: which measures the difference of applying at state x an action according to the policy π with that of playing an action according to ⟨ π, T ⟩ and then continuing playing an action according to π .
Revised: which measures the difference of applying at state x an action according to the policy π with that of playing an action according to ⟨ π, T ⟩ and then continuing playing an action according to π .

Original: Our intuition says that if it happens to be the case that D π ( x, T ) = 

Revised: Our intuition says that if it happens to be the case that D π ( x, T ) =

Original: for all states in the DOMDP, then such a policy is maximally robust.
Revised: for all states in the DOMDP, then such a policy is maximally robust.

Original: And this is indeed the case, as shown in the next proposition.
Revised: And this is indeed the case, as shown in the next proposition.

Original: Proposition 2. Consider a DOMDP as in Definition 1 and the robustness notion as in Definition 2, then if a policy π is such that D π ( x, T )
Revised: Proposition 2. Consider a DOMDP as in Definition 1 and the robustness notion as in Definition 2.

Original:  = [-
-] for all x ∈ X , then π is maximally robust, i.e., let
Revised: If a policy π is such that D π ( x, T ) = [-
-]0 for all x ∈ X , then π is maximally robust, i.e., let

Original: [Equation]
Revised: [Equation]

Original: then we have that Π D ⊆
Revised: then we have that Π D ⊆

Original: Π 
Revised: Π 0 .

Original: So far we have shown that both the set of fixed points Π and the set of policies for which the disadvantage function is equal to zero Π D are contained in the set of maximally robust policies.
Revised: So far we have shown that both the set of fixed points Π and the set of policies for which the disadvantage function is equal to zero Π D are contained in the set of maximally robust policies.

Original: More interesting is the fact that the inclusion established in Proposition 1 and the one in Proposition 2 can be linked in a natural way.
Revised: More interesting is the fact that the inclusion established in Proposition 1 and the one in Proposition 2 can be linked in a natural way.

Original: We call this connection, which is the main result of this section, the Inclusion Theorem.
Revised: We call this connection, which is the main result of this section, the Inclusion Theorem.

Original: Theorem 1 (Inclusion Theorem) .
Revised: Theorem 1 (Inclusion Theorem) .

Original: For a DOMDP with noise kernel T , consider the sets Π , Π T , Π D and Π 0 .
Revised: For a DOMDP with noise kernel T , consider the sets Π , Π T , Π D and Π 0 .

Original: Then, the following inclusion relation holds:
Revised: Then, the following inclusion relation holds:

Original: [Equation]
Revised: [Equation]

Original: Additionally, the sets Π , Π T are convex for all MDPs and kernels T , but Π D , Π 0 may not be.
Revised: Additionally, the sets Π , Π T are convex for all MDPs and kernels T , but Π D , Π 0 may not be.

Original: Remark 2.

Original: The above inclusions are equalities for some MDPs.

Original: See Appendix A for examples.

Original: 3. C ONSIDERATIONS ON THE R OBUST P OLICY S ETS

Original: Let us reflect on the inclusion relations  of Theorem 1.
Revised: Let us reflect on the inclusion relations 5 of Theorem 1.

Original: The inclusions are in general not strict, and in fact the geometry of the sets (as well as whether some of the relations are in fact equalities) is highly dependent on the reward function, and in particular on the complexity (from an information-theoretic perspective) of the reward function.
Revised: The inclusions are in general not strict, and in fact the geometry of the sets (as well as whether some of the relations are in fact equalities) is highly dependent on the reward function, and in particular on the complexity (from an informationtheoretic perspective) of the reward function.

Original: As an intuition, less complex reward functions (more uniform) will make the inclusions above expand to the entire policy set, and more complex reward functions will make the relations collapse to equalities.
Revised: As an intuition, less complex reward functions (more uniform) will make the inclusions above expand to the entire policy set, and more complex reward functions will make the relations collapse to equalities.

Original: The following Corollary illustrates this observation.
Revised: The following Corollary illustrates this.

Original: Corollary 1.
Revised: Corollary 1.

Original: For any ergodic DOMDP there exist reward functions R and R
Revised: For any ergodic DOMDP there exist reward functions R and R

Original: such that the resulting
Revised: such that the resulting

Original: DOMDP satisfies: (i) Π D
Revised: DOMDP satisfies: (i) Π D

Original: = Π 

Revised: = Π

Original: =
Revised: =

Original: Π (any policy is max.
Revised: Π (any policy is max.

Original: robust)
Revised: robust)

Original: if R = R , (ii) Π T =
Revised: if R = R , (ii) Π T =

Original: Π D = Π(only fixed point policies are maximally robust)
Revised: Π D = Π(only fixed point policies are maximally robust)

Original: if R = R .
Revised: if R = R .

Original: We can now summarise the insights from Theorem 1 and Corollary 1 in the following conclusions:
Revised: We can now summarise the insights from Theorem 1 and Corollary 1 in the following conclusions:

Original: (1) The set Π is maximally robust, convex and independent of the DOMDP , (2)
Revised: (1) The set Π is maximally robust, convex and independent of the DOMDP , (2)

Original: The set Π T is maximally robust, convex, includes Π , and its properties only depend on T , (3)
Revised: The set Π T is maximally robust, convex, includes Π , and its properties only depend on T , (3)

Original: The set Π D includes Π T and is maximally robust, but its properties depend on the DOMDP .
Revised: The set Π D includes Π T and is maximally robust, but its properties depend on the DOMDP .

Original:  R OBUSTNESS THROUGH L EXICOGRAPHIC O BJECTIVES
Revised:  R OBUSTNESS THROUGH L EXICOGRAPHIC O BJECTIVES

Original: We have now characterised robustness in a DOMDP and explored the relation between the sets of policies that are robust according to the definition proposed.
Revised: We have now characterised robustness in a DOMDP and explored the relation between the sets of policies that are robust according to the definition proposed.

Original: We have seen in the Inclusion Theorem that several classes of policies are maximally robust, and our goal now is to connect these results with lexicographic optimisation.
Revised: We have seen in the Inclusion Theorem that several classes of policies are maximally robust, and our goal now is to connect these results with lexicographic optimisation.

Original: To be able to apply LRL results to our robustness problem we need to first cast robustness as a valid objective to be maximised, and then show that a stochastic gradient descent approach would indeed find a global maximum of the objective, therefore yielding a maximally robust policy.
Revised: To be able to apply LRL results to our robustness problem we need to first cast robustness as a valid objective to be maximised, and then show that a stochastic gradient descent approach would indeed find a global maximum of the objective, therefore yielding a maximally robust policy.

Original: Then, this robustness objective can be combined with a primary rewardmaximising objective K 1 ( θ ) =
Revised: Then, this robustness objective can be combined with a primary rewardmaximising objective K 1 ( θ ) =

Original: E x 0 ∼ µ 

Revised: E x 0 ∼ µ

Original: [ V π θ ( x 0 )] and any algorithm with certified convergence to produce a solution to Problem 1.
Revised: [ V π θ ( x 0 )] and any algorithm with certified convergence to produce a solution to Problem 1.

Original: We do not know T In the introduction, we emphasised the motivation for this work came partially from the fact that we may not know T in reality, or have a way to estimate it.
Revised: We do not know T In the introduction, we emphasised the motivation for this work came partially from the fact that we may not know T in reality, or have a way to estimate it.

Original: However, the theoretical results until now depend on T .
Revised: However, the theoretical results until now depend on T .

Original: Our proposed solution to this lies in the results of Theorem 1.
Revised: Our proposed solution to this lies in the results of Theorem 1.

Original: If we use an assumed generator ˜ T with the smallest possible fixed point set (i.e. the constant policy set), the robustness lexicographic objective will drive the policy towards the set of fixed points of ˜ T , which will be included in the fixed points of T (from Theorem 1).
Revised: If we use an assumed generator ˜ T with the smallest possible fixed point set (i.e. the constant policy set), the robustness lexicographic objective will drive the policy towards the set of fixed points of ˜ T , which will be included in the fixed points of T (from Theorem 1).

Original: We argue that this is reasonable since we expect that it will improve robustness for any noise structure.
Revised: We argue that this is reasonable since we expect that it will improve robustness for any noise structure.

Original: If we do have information about the noise generator, it may be sensible to pick a different ˜ T .
Revised: If we do have information about the noise generator, it may be sensible to pick a different ˜ T .

Original: For details on how T may relate to ˜ T see Appendix B.3.
Revised: For details on how T may relate to ˜ T see Appendix B.3.

Revised: A reasonable choice for the stochastic kernel ˜ T discussed in the above paragraph is the uniform kernel, following the Principle of Maximum Entropy (when no information about T is available, we consider the maximum entropy distribution).

Revised: In specific problems, other priors, adversarial noise, etc. , may be more appropriate.

Original: We propose now a valid lexicographic objective for which a minimising solution yields a maximally robust policy.
Revised: We propose now a valid lexicographic objective for which a minimising solution yields a maximally robust policy.

Original: One of the messages of the Inclusion Theorem is the fact that fixed points and constant policies are maximally robust, the latter being completely oblivious to the choice of ˜ T , a relevant feature to the design of robust policies.
Revised: One of the messages of the Inclusion Theorem is the fact that fixed points and constant policies are maximally robust, the latter being completely oblivious to the choice of ˜ T , a relevant feature to the design of robust policies.

Original: Consider the optimisation problem minimise θ
Revised: Consider the optimisation problem minimise θ

Original: [Equation]
Revised: [Equation]

Original: where we recall that π θ is a given parameterisation of the set of policies.
Revised: where we recall that π θ is a given parameterisation of the set of policies.

Original: Notice that the optimisation problem 5 projects the current policy onto the set of fixed points of the operator ⟨· , ˜ T ⟩ , and due to
Revised: Notice that the optimisation problem 5 projects the current policy onto the set of fixed points of the operator ⟨· , ˜ T ⟩ , and due to

Original: Assumption 1, which requires µ π θ ( x ) > 0 for all x ∈ X , the optimal solution is equal to zero if and only if there exists a value of the parameter θ for which the corresponding π θ is a fixed point of ⟨· , ˜ T ⟩ .
Revised: Assumption 1, which requires µ π θ ( x ) > 0 for all x ∈ X , the optimal solution is equal to zero if and only if there exists a value of the parameter θ for which the corresponding π θ is a fixed point of ⟨· , ˜ T ⟩ .

Original: In practice, the objectives are computed for a batch of trajectory sampled states X s ⊂ X , and averaged over| X s | ; we denote these approximations with a hat.
Revised: In practice, the objectives are computed for a batch of trajectory sampled states X s ⊂ X , and averaged over| X s | ; we denote these approximations with a hat.

Original: Computation of objectives By applying standard stochastic approximation arguments, we can prove that convergence is guaranteed for a SGD iteration using ∇ θ ˆ K ˜ T ( θ )( x ) =
Revised: By applying standard stochastic approximation arguments, we can prove that convergence is guaranteed for a SGD iteration using ∇ θ ˆ K ˜ T ( θ )( x ) =

Original: ( π θ ( x ) − π θ ( y )) ∇ θ π θ ( x ) , y ∼ ˜ T ( · | x ) (which is an unbiased estimator for the objective) to the optimal solution of problem 5.
Revised: ( π θ ( x ) − π θ ( y )) ∇ θ π θ ( x ) , y ∼ ˜ T ( · | x ) (which is an unbiased estimator for the objective) to the optimal solution of problem 5.

Original: For details and proof, see Appendix B.
Revised: For details and proof, see Appendix B.

Original: A reasonable choice for the stochastic kernel ˜ T discussed in the above paragraph is the uniform kernel, following the Principle of Maximum Entropy (when no information about T is available, we consider the maximum entropy distribution).

Original: In specific problems, other priors, adversarial noise, etc. , may be more appropriate.

Original: Remark 3.
Revised: Remark 2.

Original: The gradient approximation ∇ θ ˆ K ˜ T ( θ )( x ) is not the true gradient of K ˜ T .
Revised: The gradient approximation ∇ θ ˆ K ˜ T ( θ )( x ) is not the true gradient of K ˜ T .

Original: However, this approximation is sufficient to ensure convergence of the policy π θ to a fixed point of the operator ⟨· , ˜ T ⟩ , provided we have a fully parameterised policy.
Revised: However, this approximation is sufficient to ensure convergence of the policy π θ to a fixed point of the operator ⟨· , ˜ T ⟩ , provided we have a fully parameterised policy.

Original: Such an approximation is also easy to compute from sampled points x ∈ X both on- and off-policy.
Revised: Such an approximation is also easy to compute from sampled points x ∈ X both on- and off-policy.

Original: Other types of policy parameterisations may also yield a fixed point of ⟨· , ˜ T ⟩ if it is such that we can make the policy state independent, i.e., if there is a parameter θ for which π θ ( x )
Revised: Other types of policy parameterisations may also yield a fixed point of ⟨· , ˜ T ⟩ if it is such that we can make the policy state independent, i.e., if there is a parameter θ for which π θ ( x )

Original: =
Revised: =

Original: π θ ( y ) for all x, y ∈ X .
Revised: π θ ( y ) for all x, y ∈ X .

Original: This is the case, for example, when considering general neural network architectures if we set the weights to zero (but not the bias).
Revised: This is the case, for example, when considering general neural network architectures if we set the weights to zero (but not the bias).

Original: Assumption 2 (Learning Rates) .
Revised: Assumption 2 (Learning Rates) .

Original: We assume all learning rates α t ( x, u ) ∈ [0 , 1] satisfy the conditions (cid:80) ∞ t =1 α t ( x, u ) = ∞ and (cid:80) ∞ t =1 α t ( x, u ) 2 < ∞ .
Revised: We assume all learning rates α t ( x, u ) ∈ [0 , 1] satisfy the conditions (cid:80) ∞ t =1 α t ( x, u ) = ∞ and (cid:80) ∞ t =1 α t ( x, u ) 2 < ∞ .

Original: 1: input

Original: , ˜ T , ϵ 2: initialise θ , critic (if using), λ , { β1 t , β 2 t , η } 3: set t= 0 , x t ∼ µ 0 4:while t< maxiterations do 5: perform u t ∼ πθ ( x t ) 6: observe r t , x t +1 , sample y ∼ ˜ T ( ·| x ) 7: if ˆ K1 ( θ ) notconverged thenˆ k 1 ←ˆ K 1 ( θ ) 8: update critic (if using) 9: update θ and λ using equation10: output θ
Revised: β η } 3: set t x t ∼ µ 0 4: t max do 5: perform u t ∼ π ( x t ) 6: observe r t , x t +1 , sample y ∼ ˜ T ( · x ) 7: if ˆ K θ ) not then k 1 ← K 10: output θ

Original: Now, the convergence of PB-LRL algorithms is guaranteed as long as the original policy gradient algorithm (such as PPO (Liu et al., 2019) or A2C (Konda & Tsitsiklis, 2000; Bhatnagar et al., 2009)) for each single objective converges Skalse et al.
Revised: Now, the convergence of PB-LRL algorithms is guaranteed as long as the original policy gradient algorithm (such as PPO (Liu et al., 2019) or A2C (Konda & Tsitsiklis, 2000; Bhatnagar et al., 2009)) for each single objective converges Skalse et al.

Original: (2022b).
Revised: (2022b).

Original: We can then combine Lemma 1 with these results to guarantee that Lexicographically Robust Policy Gradient (LRPG), Algorithm 1, converges to a policy that maximise robustness while remaining (approximately) optimal with respect to R .
Revised: We can then combine Lemma 1 with these results to guarantee that Lexicographically Robust Policy Gradient (LRPG), Algorithm 1, converges to a policy that maximise robustness while remaining (approximately) optimal with respect to R .

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider a DOMDP as in Definition 1 and let π θ be a parameterised policy.
Revised: Consider a DOMDP as in Definition 1 and let π θ be a parameterised policy.

Original: Take K 1 ( θ ) =
Revised: Take K 1 ( θ ) =

Original: E x 0 ∼ µ 

Revised: E x 0 ∼ µ

Original: [ V π θ ( x 0 )] to be computed through a chosen algorithm ( e.g. , A2C, PPO) that optimises K 1 ( θ ) , and let K 2 ( θ )
Revised: [ V π θ ( x 0 )] to be computed through a chosen algorithm ( e.g. , A2C, PPO) that optimises K 1 ( θ ) , and let K 2 ( θ )

Original: = − K ˜ T ( θ ) .
Revised: = − K ˜ T ( θ ) .

Original: Given an ϵ > 0 , if the iteration θ ← proj Θ (cid:2) θ + ∇ θ ˆ K 1 (cid:3) is guaranteed to converge to a parameter set θ ∗ that maximises K 1 , and hence J (locally or globally), then LRPG converges a.s. under PB-LRL conditions to parameters θ ϵ that satisfy:
Revised: Given an ϵ > 0 , if the iteration θ ← proj Θ (cid:2) θ + ∇ θ ˆ K 1 (cid:3) is guaranteed to converge to a parameter set θ ∗ that maximises K 1 , and hence J (locally or globally), then LRPG converges a.s. under PB-LRL conditions to parameters θ ϵ that satisfy:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: where Θ ′ =
Revised: where Θ ′ =

Original: Θ if θ ∗ is globally optimal and a compact local neighbourhood of θ ∗ otherwise.
Revised: Θ if θ ∗ is globally optimal and a compact local neighbourhood of θ ∗ otherwise.

Original: We reflect again on Figure 1.
Revised: We reflect again on Figure 1.

Original: The main idea behind LRPG is that by formally expanding the set of acceptable policies with respect to K 1 , we may find robust policies more effectively while guaranteeing a minimum performance in terms of expected rewards.
Revised: The main idea behind LRPG is that by formally expanding the set of acceptable policies with respect to K 1 , we may find robust policies more effectively while guaranteeing a minimum performance in terms of expected rewards.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We verify the theoretical results of LRPG in a series of experiments on discrete state/action safetyrelated environments (Chevalier-Boisvert et al., 2018).
Revised: We verify the theoretical results of LRPG in a series of experiments on discrete state/action safetyrelated environments (Chevalier-Boisvert et al., 2018).

Original: Minigrid-LavaGap , Minigrid-LavaCrossing are safe exploration tasks where the agent needs to navigate an environment with cliff-like regions 
Revised: Minigrid-LavaGap , Minigrid-LavaCrossing are safe exploration tasks where the agent needs to navigate an environment with cliff-like regions and receives a reward of 1 when it finds a target.

Revised: Minigrid-DynamicObstacles is a dynamic obstacleavoidance environment where the agent is penalised for hitting an obstacle, and gets a positive

Original: [Figure]
Revised: [Figure]

Original: and receives a rewardof 1 when it finds a target.
Revised: reward when finding a target.

Original: Minigrid-DynamicObstacles is a dynamic obstacleavoidance environment where the agent is penalised for hitting an obstacle, and gets a positive reward when finding a target.

Original: Minigrid-LavaGap is small enough to be fully observable, and the other two environments are partially observable.
Revised: Minigrid-LavaGap is small enough to be fully observable, and the other two environments are partially observable.

Original: In all cases observations consist of a 7 × 7 field of view in front of the agent, with 3 channels encoding the color and state of objects in the environment.
Revised: In all cases observations consist of a 7 × 7 field of view in front of the agent, with 3 channels encoding the color and state of objects in the environment.

Original: We use A2C (Sutton & Barto, 2018) and PPO (Schulman et al., 2017) for our implementations of
Revised: We use A2C (Sutton & Barto, 2018) and PPO (Schulman et al., 2017) for our implementations of

Original: LRPG which we denote by LR-PPO and LR-A2C, respectively.
Revised: LRPG which we denote by LR-PPO and LR-A2C, respectively.

Original: In all cases, the lexicographic tolerance was set to ϵ = 0 . 99ˆ
Revised: In all cases, the lexicographic tolerance was set to ϵ = 0 . 99ˆ

Original: k 1 to deviate as little as possible from the primary objective.
Revised: k 1 to deviate as little as possible from the primary objective.

Original: Sampling ˜T
Revised: Sampling ˜T

Original: To simulate ˜ T we disturb x as ˜ x
Revised: To simulate ˜ T we disturb x as ˜ x

Original: = x + ξ for (1) a uniform bounded noise signal ξ ∼ U [ − b,b ] ( ˜ T u ) with b = 2 ( 1 . 5 for LavaCrossing) and (2) and a Gaussian noise ( ˜ T g ) such that ξ ∼ N (0 , 0 . 5) .
Revised: = x + ξ for (1) a uniform bounded noise signal ξ ∼ U [ − b,b ] ( ˜ T u ) with b = 2 ( 1 . 5 for LavaCrossing) and (2) and a Gaussian noise ( ˜ T g ) such that ξ ∼ N (0 , 0 . 5) .

Original: We test the resulting policies against a noiseless environment ( ∅ ), a kernel T 1 = ˜ T u and a kernel T 2 = ˜ T g .
Revised: We test the resulting policies against a noiseless environment ( ∅ ), a kernel T 1 = ˜ T u and a kernel T 2 = ˜ T g .

Original: The main point of these combinations is to also test the policies when the true noise T is similar to ˜ T .
Revised: The main point of these combinations is to also test the policies when the true noise T is similar to ˜ T .

Original: General Robustness Results.
Revised: General Robustness Results.

Original: Firstly, we investigate the robustness of four algorithms where we do not have a Q function.
Revised: Firstly, we investigate the robustness of four algorithms where we do not have a Q function.

Original: If we do not have an estimator for the critic Q π , Proposition 1 suggests that minimising the distance between π and ⟨ π, T ⟩ can serve as a proxy to minimise the robustness regret.
Revised: If we do not have an estimator for the critic Q π , Proposition 1 suggests that minimising the distance between π and ⟨ π, T ⟩ can serve as a proxy to minimise the robustness regret.

Original: We consider the algorithms:
Revised: We consider the algorithms:

Original: Vanilla PPO (noiseless).
Revised: Vanilla PPO (noiseless).

Original: LR-PPO with a Gaussian noise kernel ( K gT ).
Revised: LR-PPO with a Gaussian noise kernel ( K gT ).

Original: LR-PPO with a uniform noise kernel ( K uT ).
Revised: LR-PPO with a uniform noise kernel ( K uT ).

Original: SA-PPO from Zhang et al.
Revised: SA-PPO from Zhang et al.

Original: In these experiments, we use PPO with a neural policies and value functions; the architectures and hyper-parameters used in each case can be found in Appendix C.
Revised: In these experiments, we use PPO with a neural policies and value functions; the architectures and hyper-parameters used in each case can be found in Appendix C.

Original: The results are summarised in the left-hand side of Table 1.
Revised: The results are summarised in the left-hand side of Table 1.

Original: Each entry is the median of 10 independent training processes, with reward values measured as the mean of 100 independent trajectories.
Revised: Each entry is the median of 10 independent training processes, with reward values measured as the mean of 100 independent trajectories.

Original: Robustness through Disadvantage Objectives.
Revised: Robustness through Disadvantage Objectives.

Original: If we have an estimator for the critic Q π we can obtain robustness without inducing regularity in the policy using D π , yielding a larger policy subspace to steer towards, and hopefully achieving policies closer to optimal.
Revised: If we have an estimator for the critic Q π we can obtain robustness without inducing regularity in the policy using D π , yielding a larger policy subspace to steer towards, and hopefully achieving policies closer to optimal.

Original: With the goal of diving deeper into the results of Theorem 1, we consider the objective:
Revised: With the goal of diving deeper into the results of Theorem 1, we consider the objective:

Original: [Equation]
Revised: [Equation]

Original: We aim to test the hypothesis introduced through this work: by setting K 2 = K D and thus aiming to minimise the disadvantage D , we may obtain policies that yield better robustness with similar expected rewards.
Revised: We aim to test the hypothesis introduced through this work: by setting K 2 = K D and thus aiming to minimise the disadvantage D , we may obtain policies that yield better robustness with similar expected rewards.

Original: Observe that π D ∈ Π D = ⇒
Revised: Observe that π D ∈ Π D = ⇒

Original: K D ( π D ) = 0 .
Revised: K D ( π D ) = 0 .

Original: To test this, we compare the following algorithms on the same environments:
Revised: To test this, we compare the following algorithms on the same environments:

Original: Vanilla A2C (noiseless).
Revised: Vanilla A2C (noiseless).

Original: LR-A2C with K gT .
Revised: LR-A2C with K gT .

Original: LR-A2C with K u T . 4.
Revised: LR-A2C with K u T . 4.

Original: LR-A2C with K 2 = K D .
Revised: LR-A2C with K 2 = K D .

Original: We use A2C in this case since the structure of the original cost functions are simpler than PPO, and hence easier to compare between the scenarios above, and we modified A2C to retain a Q function as a critic.
Revised: We use A2C in this case since the structure of the original cost functions are simpler than PPO, and hence easier to compare between the scenarios above, and we modified A2C to retain a Q function as a critic.

Original: With each objective function resulting in gradient descent steps that pull the policy towards different maximally robust sets ( K T → Π T and K D → Π D respectively), we would expect to obtain increasing robustness for K
Revised: With each objective function resulting in gradient descent steps that pull the policy towards different maximally robust sets ( K T → Π T and K D → Π D respectively), we would expect to obtain increasing robustness for K

Original: D .
Revised: D .

Original: The results are presented in the right-hand side of Table 1.
Revised: The results are presented in the right-hand side of Table 1.

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Experiments.
Revised: Experiments.

Original: We applied LRPG on PPO and A2C algorithms, for a set of discrete action, discrete state grid environments.
Revised: We applied LRPG on PPO and A2C algorithms, for a set of discrete action, discrete state grid environments.

Original: These environments are particularly sensitive to robustness problems; the rewards are sparse, and applying a sub-optimal action at any step of the trajectory often leads to terminal states with zero (or negative) reward.
Revised: These environments are particularly sensitive to robustness problems; the rewards are sparse, and applying a sub-optimal action at any step of the trajectory often leads to terminal states with zero (or negative) reward.

Original: LRPG successfully induces lower robustness regrets in the tested scenarios, and the use of K D as an objective (even though we did not prove the convergence of a gradient based method with such objective) yields a better compromise between robustness and rewards.
Revised: LRPG successfully induces lower robustness regrets in the tested scenarios, and the use of K D as an objective (even though we did not prove the convergence of a gradient based method with such objective) yields a better compromise between robustness and rewards.

Original: When compared to recent observational robustness methods, LRPG obtains similar robustness results while preserving the original guarantees of the chosen algorithm (it even outperforms in some cases, although this is probably highly problem dependent, so we do not claim an improvement for every DOMDP).
Revised: When compared to recent observational robustness methods, LRPG obtains similar robustness results while preserving the original guarantees of the chosen algorithm (it even outperforms in some cases, although this is probably highly problem dependent, so we do not claim an improvement for every DOMDP).

Original: Further Considerations on LRPG.
Revised: Further Considerations on LRPG.

Original: The characterisation of robustness as a policy being invariant to a stochastic operator may be useful for other versions of robustness in RL.
Revised: The characterisation of robustness as a policy being invariant to a stochastic operator may be useful for other versions of robustness in RL.

Original: For example, in robustness against transition probability disturbances (or distributional shifts), one may consider distribution ambiguity sets and exploit distributionally robust optimisation ideas to investigate policy invariances.
Revised: For example, in robustness against transition probability disturbances (or distributional shifts), one may consider distribution ambiguity sets and exploit distributionally robust optimisation ideas to investigate policy invariances.

Original: In this case, investigating the structure of maximally robust policies may yield a mechanism to design RL algorithms that are generally robust to model uncertainties.
Revised: In this case, investigating the structure of maximally robust policies may yield a mechanism to design RL algorithms that are generally robust to model uncertainties.

Original: We leave these considerations as future work.

Original: Shortcomings The motivation for LRPG comes partially from the situation where, when deploying a model free controller in a complex dynamical system, we may not have a feasible way of estimating the noise generator.
Revised: Shortcomings The motivation for LRPG comes partially from the situation where, when deploying a model free controller in a complex dynamical system, we may not have a feasible way of estimating the noise generator.

Original: There is an alternative approach for robust RL (exploited in most of the literature), which consists on taking a disturbance structure ( e.g. adversarial noise) and training directly to optimise the rewards in the disturbed MDP.
Revised: There is an alternative approach for robust RL (exploited in most of the literature), which consists on taking a disturbance structure ( e.g. adversarial noise) and training directly to optimise the rewards in the disturbed MDP.

Original: Apart from LRPG preserving formal guarantees, there is no clear answer over what approach is more rational, or more effective in general.
Revised: Apart from LRPG preserving formal guarantees, there is no clear answer over what approach is more rational, or more effective in general.

Original: The choice would depend on the problem at hand, the possible existence of an adversary, the requirement (or lack thereof) for formal guarantees, etc .
Revised: The choice would depend on the problem at hand, the possible existence of an adversary, the requirement (or lack thereof) for formal guarantees, etc .

Original: We cannot claim that our approach is better in every way; we simply show through this work that it is a useful approach for learning policies in specific problems where, for example, we need to control dynamical system where the noise sources are unknown and we need to retain certain formal guarantees of the algorithms used.
Revised: We cannot claim that our approach is better in every way; we simply show through this work that it is a useful approach for learning policies in specific problems where, for example, we need to control dynamical system where the noise sources are unknown and we need to retain certain formal guarantees of the algorithms used.

Original: Robustness, Complexity and Invariances.
Revised: Robustness, Complexity and Invariances.

Original: Sections 2 and 3 discuss at large the structure, shape and dependence of the maximally robust policy sets.
Revised: Sections 2 and 3 discuss at large the structure, shape and dependence of the maximally robust policy sets.

Original: These insights help derive optimisation objectives to use in LRPG, but there is more to be said about how policy robustness is affected by the underlying
Revised: These insights help derive optimisation objectives to use in LRPG, but there is more to be said about how policy robustness is affected by the underlying

Original: MDP properties.
Revised: MDP properties.

Original: We hint at this in the proof of Corollary 1.
Revised: We hint at this in the proof of Corollary 1.

Original: More regular ( less complex in entropy terms, or more symmetric ) reward functions ( e.g. , reward functions with smaller variance across the actions R ( x, · , y ) ) seem to induce larger robust policy sets.
Revised: More regular ( less complex in entropy terms, or more symmetric ) reward functions ( e.g. , reward functions with smaller variance across the actions R ( x, · , y ) ) seem to induce larger robust policy sets.

Original: In other words, for a fixed policy, a more complex reward function yields larger robustness regrets as soon as any noise is introduced in the system.
Revised: In other words, for a fixed policy, a more complex reward function yields larger robustness regrets as soon as any noise is introduced in the system.

Original: This raises questions on how to use these principles to derive more robust policies in a comprehensive way, but we leave these questions for future work.
Revised: This raises questions on how to use these principles to derive more robust policies in a comprehensive way, but we leave these questions for future work.

Original: Additionally, one couldquickly extend these ideasand use LRL toinduce other kinds of invariances to policies. For example, one could use LRL to obtain policies that generalise to a subclass of reward functions (connecting to previous thoughts on complexity). One advantage of this approach would be that the resulting policy gradient algorithm would retain the original algorithm guarantees.
Revised: Additionally, one could extend these ideas to use LRL to obtain policies that generalise to a subclass of reward functions.

Original: A E XAMPLES AND F URTHER C ONSIDERATIONS
Revised: A E XAMPLES AND F URTHER C ONSIDERATIONS

Original: We provide here two examples to show how we can obtain limit scenarios Π 

Revised: We provide here two examples to show how we can obtain limit scenarios Π

Original: = Π (any policy is maximally robust) or Π 0 =
Revised: = Π (any policy is maximally robust) or Π 0 =

Original: Π T (Example 1), and how for some MDPs the third inclusion in
Revised: Π T (Example 1), and how for some MDPs the third inclusion in

Original: Theorem 1 is strict (Example 2).
Revised: Theorem 1 is strict (Example 2).

Original: Example 1 Consider the simple MDP in Figure 3.
Revised: Example 1 Consider the simple MDP in Figure 3.

Original: First, consider the reward function
Revised: First, consider the reward function

Original: R 1 ( x 1 , · , · ) = 10 , R 1 ( x 2 , · , · ) = 0 .
Revised: R 1 ( x 1 , · , · ) = 10 , R 1 ( x 2 , · , · ) = 0 .

Original: This produces a “dummy” MDP where all policies have the same reward sum.
Revised: This produces a “dummy” MDP where all policies have the same reward sum.

Original: Then, ∀ T, π , V ⟨ π,T ⟩ = V π , and therefore we have Π D = Π 

Revised: Then, ∀ T, π , V ⟨ π,T ⟩ = V π , and therefore we have Π D = Π

Original: = Π .
Revised: = Π .

Original: [Figure]
Revised: [Figure]

Original: Now, consider the reward function R 2 ( x 1 , u 1 , · ) = 10 , R 2 ( · , · , · ) = 0 elsewhere.
Revised: Now, consider the reward function R 2 ( x 1 , u 1 , · ) = 10 , R 2 ( · , · , · ) = 0 elsewhere.

Original: Take a non- constant policy π , i.e. , π ( x 1 ) ̸
Revised: Take a non- constant policy π , i.e. , π ( x 1 ) ̸

Original: = π ( x 2 ) .
Revised: = π ( x 2 ) .

Original: In the example DOMDP (assuming the initial state is drawn uniformly from X 0 = { x 1 , x 2 } ) one can show that at any time in the trajectory, there is a stationary probability
Revised: In the example DOMDP (assuming the initial state is drawn uniformly from X 0 = { x 1 , x 2 } ) one can show that at any time in the trajectory, there is a stationary probability

Original: Pr { x t = x 1 } = 12 .
Revised: Pr { x t = x 1 } = 12 .

Original: Let us abuse notation and write π ( x i ) =
Revised: Let us abuse notation and write π ( x i ) =

Original: ( π ( x i , u 1 ) π ( x i , u 2 ) )
Revised: ( π ( x i , u 1 ) π ( x i , u 2 ) )

Original: ⊤ and R ( x i ) =
Revised: ⊤ and R ( x i ) =

Original: ( R ( x i , u 1 , · ) R ( x i , u 2 , · ) )
Revised: ( R ( x i , u 1 , · ) R ( x i , u 2 , · ) )

Original: ⊤ .
Revised: ⊤ .

Original: For the given reward structure we have R ( x 2 ) =
Revised: For the given reward structure we have R ( x 2 ) =

Original: ( 0 0 ) ⊤ , and therefore:
Revised: ( 0 0 ) ⊤ , and therefore:

Original: [Equation]
Revised: [Equation]

Original: Since the transitions of the MDP are independent of the actions, following the same principle as in equation 7: J ⟨ π, T ⟩ = 12 ⟨ R ( x 1 ) , ⟨· , T ⟩ ( π )( x 1 ) ⟩ γ 1 − γ .
Revised: Since the transitions of the MDP are independent of the actions, following the same principle as in equation 7: J ⟨ π, T ⟩ = 12 ⟨ R ( x 1 ) , ⟨· , T ⟩ ( π )( x 1 ) ⟩ γ 1 − γ .

Original: For any noise map ⟨· , T ⟩ ̸
Revised: For any noise map ⟨· , T ⟩ ̸

Original: = Id , for the two-state policy it holds that π / ∈ Π T = ⇒ ⟨ π, T ⟩ ̸
Revised: = Id , for the two-state policy it holds that π / ∈ Π T = ⇒ ⟨ π, T ⟩ ̸

Original: = π .
Revised: = π .

Original: Therefore ⟨ π, T ⟩ ( x 1 ) ̸
Revised: Therefore ⟨ π, T ⟩ ( x 1 ) ̸

Original: = π ( x 1 ) and:
Revised: = π ( x 1 ) and:

Original: [Equation]
Revised: [Equation]

Original: which implies that π / ∈ Π 0 .
Revised: which implies that π / ∈ Π 0 .

Original: Example 2 Consider the same MDP in Figure 3 with reward function R ( x 1 , u 1 , · ) =
Revised: Example 2 Consider the same MDP in Figure 3 with reward function R ( x 1 , u 1 , · ) =

Original: R ( x 2 , u 1 , · ) = 10 , and a reward of zero for all other transitions.
Revised: R ( x 2 , u 1 , · ) = 10 , and a reward of zero for all other transitions.

Original: Take a policy π ( x 1 )
Revised: Take a policy π ( x 1 )

Original: =
Revised: =

Original: (1 0) , π ( x 2 ) =
Revised: (1 0) , π ( x 2 ) =

Original: The policy yields a reward of 10 in state x 1 and a reward of 0 in state x 2 .
Revised: The policy yields a reward of 10 in state x 1 and a reward of 0 in state x 2 .

Original: Again we assume the initial state is drawn uniformly from X 

Revised: Again we assume the initial state is drawn uniformly from X

Original: = { x 1 , x 2 } .
Revised: = { x 1 , x 2 } .

Original: Then, observe:
Revised: Then, observe:

Original: [Equation]
Revised: [Equation]

Original: Define now noise map T ( · | x 1 ) =
Revised: Define now noise map T ( · | x 1 ) =

Original: ( 12 12 ) and T ( · | x 2 ) =
Revised: ( 12 12 ) and T ( · | x 2 ) =

Original: Observe this noise map yields a policy with non-zero disadvantage, D π ( x 1 , T ) =
Revised: Observe this noise map yields a policy with non-zero disadvantage, D π ( x 1 , T ) =

Original: 5 γ 1 − γ − (cid:0) 5 γ 1 − γ − 2 .
Revised: 5 γ 1 − γ − (cid:0) 5 γ 1 − γ − 2 .

Original: 5 (cid:1) = 2 . 5 and similarly
Revised: 5 (cid:1) = 2 . 5 and similarly

Original: D π ( x 2 , T ) = − 2 .
Revised: D π ( x 2 , T ) = − 2 .

Original: 5 , therefore π / ∈ Π D .
Revised: 5 , therefore π / ∈ Π D .

Original: However, the policy is maximally robust :
Revised: However, the policy is maximally robust :

Original: [Equation]
Revised: [Equation]

Original: Therefore, π ∈ Π 0 .
Revised: Therefore, π ∈ Π 0 .

Original: B T HEORETICAL R ESULTS B. A UXILIARY R ESULTS
Revised: B T HEORETICAL R ESULTS B. A UXILIARY R ESULTS

Original: Theorem 3 (Stochastic Approximation with Non-Expansive Operator) .
Revised: Theorem 3 (Stochastic Approximation with Non-Expansive Operator) .

Original: Let { ξ t } be a random se- quence with ξ t ∈ R n defined by the iteration:
Revised: Let { ξ t } be a random se- quence with ξ t ∈ R n defined by the iteration:

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: The step sizes α t satisfy Assumption 2. 2.
Revised: The step sizes α t satisfy Assumption 2. 2.

Original: F : R n (cid:55)→ R n is a ∥ · ∥ ∞ non-expansive map.
Revised: F : R n (cid:55)→ R n is a ∥ · ∥ ∞ non-expansive map.

Original: That is, for any ξ 1 , ξ 2 ∈ R n , ∥ F ( ξ 1 ) − F ( ξ 2 ) ∥ ∞
Revised: That is, for any ξ 1 , ξ 2 ∈ R n , ∥ F ( ξ 1 ) − F ( ξ 2 ) ∥ ∞

Original: ≤ ∥ ξ 1 − ξ 2 ∥ ∞ .
Revised: ≤ ∥ ξ 1 − ξ 2 ∥ ∞ .

Original: { M t } is a martingale difference sequence with respect to the increasing family of σ − fields
Revised: { M t } is a martingale difference sequence with respect to the increasing family of σ − fields

Original: [Equation]
Revised: [Equation]

Original: Then, the sequence ξ t → ξ ∗ almost surely where ξ ∗ is a fixed point such that F ( ξ ∗ ) = ξ ∗ .
Revised: Then, the sequence ξ t → ξ ∗ almost surely where ξ ∗ is a fixed point such that F ( ξ ∗ ) = ξ ∗ .

Original: Proof.
Revised: Proof.

Original: See Borkar & Soumyanatha (1997).
Revised: See Borkar & Soumyanatha (1997).

Original: Theorem 4 (PB-LRL Convergence) .
Revised: Theorem 4 (PB-LRL Convergence) .

Original: Let M be a multi-objective MDP with objectives
Revised: Let M be a multi-objective MDP with objectives

Original: K i , i ∈ { 1 , ..., m } of the same form.
Revised: K i , i ∈ { 1 , ..., m } of the same form.

Original: Assume a policy π is twice differentiable in parameters θ , and if using a critic V i assume it is continuously differentiable on w i .
Revised: Assume a policy π is twice differentiable in parameters θ , and if using a critic V i assume it is continuously differentiable on w i .

Original: Suppose that if PB-LRL is run for T steps, there exists some limit point w ∗
Revised: Suppose that if PB-LRL is run for T steps, there exists some limit point w ∗

Original: i ( θ ) when θ is held fixed under conditions C on M , π and V i .
Revised: i ( θ ) when θ is held fixed under conditions C on M , π and V i .

Original: If lim T →∞ E t [ θ ] ∈ Θ ϵ 1 for m = 1 , then for any m ∈ N we have lim T →∞ E t [ θ ] ∈ Θ ϵm where ϵ depends on the representational power of the parameterisations of π , V i .
Revised: If lim T →∞ E t [ θ ] ∈ Θ ϵ 1 for m = 1 , then for any m ∈ N we have lim T →∞ E t [ θ ] ∈ Θ ϵm where ϵ depends on the representational power of the parameterisations of π , V i .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We refer the interested reader to Skalse et al.
Revised: We refer the interested reader to Skalse et al.

Original: (2022b) for a full proof, and here attempt to provide the intuition behind the result in the form of a proof sketch.
Revised: (2022b) for a full proof, and here attempt to provide the intuition behind the result in the form of a proof sketch.

Original: Let us begin by briefly recalling the general problem statement: we wish to take a multi-objective
Revised: Let us begin by briefly recalling the general problem statement: we wish to take a multi-objective

Original: MDP M with m objectives, and obtain a lexicographically optimal policy (one that optimises the first objective, and then subject to this optimises the second objective, and so on).
Revised: MDP M with m objectives, and obtain a lexicographically optimal policy (one that optimises the first objective, and then subject to this optimises the second objective, and so on).

Original: More precisely, for a policy π parameterised by θ , we say that π is (globally) lexicographically ϵ -optimal if θ ∈ Θ ϵm , where Θ ϵ 0 =
Revised: More precisely, for a policy π parameterised by θ , we say that π is (globally) lexicographically ϵ -optimal if θ ∈ Θ ϵm , where Θ ϵ 0 =

Original: Θ is the set of all policies in M , Θ ϵi +1 :
Revised: Θ is the set of all policies in M , Θ ϵi +1 :

Original: = { θ ∈ Θ ϵi | max θ
Revised: = { θ ∈ Θ ϵi | max θ

Original: ′ ∈
Revised: ′ ∈

Original: Θ ϵi K
Revised: Θ ϵi K

Original: i ( θ ′ )
Revised: i ( θ ′ )

Original: − K i ( θ ) ≤ ϵ i } , and R m − 1 ∋ ϵ ≽ 0 .
Revised: − K i ( θ ) ≤ ϵ i } , and R m − 1 ∋ ϵ ≽ 0 .

Original: The basic idea behind policy-based lexicographic reinforcement learning (PB-LRL) is to use a multitimescale approach to first optimise θ using K 1 , then at a slower timescale optimise θ using K 2 while adding the condition that the loss with respect to K 1 remains bounded by its current value, and so on.
Revised: The basic idea behind policy-based lexicographic reinforcement learning (PB-LRL) is to use a multitimescale approach to first optimise θ using K 1 , then at a slower timescale optimise θ using K 2 while adding the condition that the loss with respect to K 1 remains bounded by its current value, and so on.

Original: This sequence of constrained optimisations problems can be solved using a Lagrangian relaxation (Bertsekas, 1999), either in series or – via a judicious choice of learning rates – simultaneously, by exploiting a separation in timescales (Borkar, 2008).
Revised: This sequence of constrained optimisations problems can be solved using a Lagrangian relaxation (Bertsekas, 1999), either in series or – via a judicious choice of learning rates – simultaneously, by exploiting a separation in timescales (Borkar, 2008).

Original: In the simultaneous case, the parameters of the critic w i (if using an actor-critic algorithm, if not this part of the argument may be safely ignored) for each objective are updated on the fastest timescale, then the parameters θ , and finally (i.e., most slowly) the Lagrange multipliers for each of the remaining constraints.
Revised: In the simultaneous case, the parameters of the critic w i (if using an actor-critic algorithm, if not this part of the argument may be safely ignored) for each objective are updated on the fastest timescale, then the parameters θ , and finally (i.e., most slowly) the Lagrange multipliers for each of the remaining constraints.

Original: The proof proceeds via induction on the number of objectives, using a standard stochastic approximation argument (Borkar, 2008).
Revised: The proof proceeds via induction on the number of objectives, using a standard stochastic approximation argument (Borkar, 2008).

Original: In particular, due to the learning rates chosen, we may consider those more slowly updated parameters fixed for the purposes of analysing the convergence of the more quickly updated parameters.
Revised: In particular, due to the learning rates chosen, we may consider those more slowly updated parameters fixed for the purposes of analysing the convergence of the more quickly updated parameters.

Original: In the base case where m = 1 , we have (by assumption) that lim T →∞ E t [ θ ] ∈ Θ ϵ 1 .
Revised: In the base case where m = 1 , we have (by assumption) that lim T →∞ E t [ θ ] ∈ Θ ϵ 1 .

Original: This is simply the standard (non-lexicographic) RL setting.
Revised: This is simply the standard (non-lexicographic) RL setting.

Original: Before continuing to the inductive step, Skalse et al.
Revised: Before continuing to the inductive step, Skalse et al.

Original: (2022b) observe that because gradient descent on Kconverges to globally optimal stationary point when m = 1 then K 1 must be globally invex (where the opposite implication is also true) (Ben-Israel & Mond, 1986a)
Revised: (2022b) observe that because gradient descent on Kconverges to globally optimal stationary point when m = 1 then K 1 must be globally invex (where the opposite implication is also true) (Ben-Israel & Mond, 1986a)

Original: The reason this observation is useful is that because each of the objectives K i shares the same functional form, they are all invex, and furthermore, invexity is conserved under linear combinations and the addition of scalars, meaning that the Lagrangian formed in the relaxation of each constrained optimisation problem is also invex.
Revised: The reason this observation is useful is that because each of the objectives K i shares the same functional form, they are all invex, and furthermore, invexity is conserved under linear combinations and the addition of scalars, meaning that the Lagrangian formed in the relaxation of each constrained optimisation problem is also invex.

Original: As a result, if we assume that lim T →∞ E t [ θ ] ∈ Θ ϵi as our inductive hypothesis, then the stationary point of the Lagrangian for optimising objective K i +1 is a global optimum, given the constraints that it does not worsen performance on K 1 , . . .
Revised: As a result, if we assume that lim T →∞ E t [ θ ] ∈ Θ ϵi as our inductive hypothesis, then the stationary point of the Lagrangian for optimising objective K i +1 is a global optimum, given the constraints that it does not worsen performance on K 1 , . . .

Original: , K i .
Revised: , K i .

Original: Via
Revised: Via

Original: Slater’s condition (Slater, 1950) and standard saddle-point arguments (Bertsekas, 1999; Paternain et al., 2019), we therefore have that lim T →∞ E t
Revised: Slater’s condition (Slater, 1950) and standard saddle-point arguments (Bertsekas, 1999; Paternain et al., 2019), we therefore have that lim T →∞ E t

Original: [ θ ] ∈ Θ ϵi +1 , completing the inductive step, and thus the overall inductive argument.
Revised: [ θ ] ∈ Θ ϵi +1 , completing the inductive step, and thus the overall inductive argument.

Original: This concludes the proof that lim T →∞ E t [ θ ] ∈ Θ ϵm .
Revised: This concludes the proof that lim T →∞ E t [ θ ] ∈ Θ ϵm .

Original: We refer the reader to Skalse et al.
Revised: We refer the reader to Skalse et al.

Original: (2022b) for a discussion of the error ϵ ,
Revised: (2022b) for a discussion of the error ϵ ,

Original: but intuitively it corresponds to a combination of the representational power of θ , the critic parameters w i (if used), and the duality gap due to the Lagrangian relaxation (Paternain et al., 2019).
Revised: but intuitively it corresponds to a combination of the representational power of θ , the critic parameters w i (if used), and the duality gap due to the Lagrangian relaxation (Paternain et al., 2019).

Original: In cases where the representational power of the various parameters is sufficiently high, then it can be shown that ϵ
Revised: In cases where the representational power of the various parameters is sufficiently high, then it can be shown that ϵ

Original: =
Revised: =

Original: Lemma 1.
Revised: Lemma 1.

Original: Let π θ be a fully-parameterised policy in a DOMDP, and α t a learning rate satisfying
Revised: Let π θ be a fully-parameterised policy in a DOMDP, and α t a learning rate satisfying

Original: Assumption 2.
Revised: Assumption 2.

Original: Consider the following approximated gradient for objective K ˜ T ( π ) and sampled point x ∈ X :
Revised: Consider the following approximated gradient for objective K ˜ T ( π ) and sampled point x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: Then, the following iteration with x ∈ X and some initial θ 0 ,
Revised: Then, the following iteration with x ∈ X and some initial θ 0 ,

Original: [Equation]
Revised: [Equation]

Original: 10) yields θ → ˜ θ almost surely where ˜ θ satisfies K ˜ T
Revised: 10) yields θ → ˜ θ almost surely where ˜ θ satisfies K ˜ T

Original: (˜ θ ) = 0 .
Revised: (˜ θ ) = 0 .

Original: Proof.
Revised: Proof.

Original: See Appendix B.2.
Revised: See Appendix B.2.

Original: B. P ROOFS
Revised: B. P ROOFS

Original: We now present the proofs for the statements through the work.
Revised: We now present the proofs for the statements through the work.

Original: Proposition 1.
Revised: Proposition 1.

Original: If a policy π ∈ Π is a fixed point of the operator ⟨· , T ⟩ , then it holds that ⟨ π, T ⟩ = π .
Revised: If a policy π ∈ Π is a fixed point of the operator ⟨· , T ⟩ , then it holds that ⟨ π, T ⟩ = π .

Original: Therefore, one can compute the robustness of the policy π to obtain ρ ( π, T ) =
Revised: Therefore, one can compute the robustness of the policy π to obtain ρ ( π, T ) =

Original: J ( π ) − J ( ⟨ π, T ⟩ ) =
Revised: J ( π ) − J ( ⟨ π, T ⟩ ) =

Original: J ( π ) − J ( π ) = 

Revised: J ( π ) − J ( π ) =

Original: = ⇒ π ∈ Π 0 .
Revised: = ⇒ π ∈ Π 0 .

Original: Therefore, Π T ⊆
Revised: Therefore, Π T ⊆

Original: Π 0 .
Revised: Π 0 .

Original: For a discrete state and action spaces, the space of stochastic kernels K : X (cid:55)→ ∆( X ) is equivalent to the space of row-stochastic | X | × | X | matrices, therefore one can write T ( y | x ) ≡ T xy as the xy − th entry of the matrix T .
Revised: For a discrete state and action spaces, the space of stochastic kernels K : X (cid:55)→ ∆( X ) is equivalent to the space of row-stochastic | X | × | X | matrices, therefore one can write T ( y | x ) ≡ T xy as the xy − th entry of the matrix T .

Original: Then, the representation of a constant policy as an X × U matrix can be written as π = 1 | X | v ⊤ , where 1 | X | where v ∈ ∆( U ) is any probability distribution over the action space.
Revised: Then, the representation of a constant policy as an X × U matrix can be written as π = 1 | X | v ⊤ , where 1 | X | where v ∈ ∆( U ) is any probability distribution over the action space.

Original: Observe that, applying the operator ⟨ π, T ⟩ to a constant policy yields:
Revised: Observe that, applying the operator ⟨ π, T ⟩ to a constant policy yields:

Original: [Equation]
Revised: [Equation]

Original: By the Perron-Frobenius Theorem (Horn & Johnson, 2012), since T is row-stochastic it has at least one eigenvalue eig( T ) = 1 , and this admits a (strictly positive) eigenvector T 1 | X | = 1 | X | .
Revised: By the Perron-Frobenius Theorem (Horn & Johnson, 2012), since T is row-stochastic it has at least one eigenvalue eig( T ) = 1 , and this admits a (strictly positive) eigenvector T 1 | X | = 1 | X | .

Original: Therefore, substituting this in equation 11:
Revised: Therefore, substituting this in equation 11:

Original: [Equation]
Revised: [Equation]

Original: Proposition 2.
Revised: Proposition 2.

Original: Recall the definition in equation 2 and that the noise disadvantage function of a policy π is given by equation 4.
Revised: Recall the definition in equation 2 and that the noise disadvantage function of a policy π is given by equation 4.

Original: We want to show that D π ( x, T ) = 

Revised: We want to show that D π ( x, T ) =

Original: = ⇒ ρ ( π, T ) = 0 .
Revised: = ⇒ ρ ( π, T ) = 0 .

Original: Taking
Revised: Taking

Original: D π ( x, T ) =
Revised: D π ( x, T ) =

Original: one has a policy that produces an disadvantage of zero when noise kernel T is applied.
Revised: one has a policy that produces an disadvantage of zero when noise kernel T is applied.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Now define the value of the disturbed policy
Revised: Now define the value of the disturbed policy

Original: [Equation]
Revised: [Equation]

Original: and take:
Revised: and take:

Original: [Equation]
Revised: [Equation]

Original: We will now show that V π ( x ) =
Revised: We will now show that V π ( x ) =

Original: V ⟨ π,T ⟩ ( x ) , for all x ∈ X .
Revised: V ⟨ π,T ⟩ ( x ) , for all x ∈ X .

Original: Observe, from equation 12 using
Revised: Observe, from equation 12 using

Original: V π ( x ) =
Revised: V π ( x ) =

Original: E u ∼⟨ π,T ⟩ ( x ) [ Q π ( x, u )] , we have ∀ x ∈ X :
Revised: E u ∼⟨ π,T ⟩ ( x ) [ Q π ( x, u )] , we have ∀ x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: Now, taking the sup norm at both sides of equation 13 we get
Revised: Now, taking the sup norm at both sides of equation 13 we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Observe that for the right hand side of equation 14, we have
Revised: Observe that for the right hand side of equation 14, we have

Original: [Equation]
Revised: [Equation]

Original: u )
Revised: u )

Original: [Equation]
Revised: [Equation]

Original: V π ( y ) − V ⟨ π,T ⟩ ( y ) (cid:3)(cid:13)(cid:13) ∞ ≤ ∥ V π ( x ) − V ⟨ π,T ⟩ ( x ) ∥ ∞ .
Revised: V π ( y ) − V ⟨ π,T ⟩ ( y ) (cid:3)(cid:13)(cid:13) ∞ ≤ ∥ V π ( x ) − V ⟨ π,T ⟩ ( x ) ∥ ∞ .

Original: Therefore, since γ < 1 , ∥ V
Revised: Therefore, since γ < 1 , ∥ V

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, ∥ V π ( x ) − V ⟨ π,T ⟩ ( x ) ∥ ∞
Revised: Finally, ∥ V π ( x ) − V ⟨ π,T ⟩ ( x ) ∥ ∞

Original: = 
 =
Revised: = 
 =

Original: ⇒
Revised: ⇒

Original: V π ( x ) − V ⟨ π,T ⟩ ( x ) = 0 ∀ x ∈ X , and V π ( x ) −
Revised: V π ( x ) − V ⟨ π,T ⟩ ( x ) = 0 ∀ x ∈ X , and V π ( x ) −

Original: V ⟨ π,T ⟩ ( x ) = 

Revised: V ⟨ π,T ⟩ ( x ) =

Original: ∀ x ∈ X =
Revised: ∀ x ∈ X =

Original: ⇒ J ( π ) =
Revised: ⇒ J ( π ) =

Original: J ( ⟨ π, T ⟩ ) =
Revised: J ( ⟨ π, T ⟩ ) =

Original: ⇒ ρ
Revised: ⇒ ρ

Original: ( π, T )
Revised: ( π, T )

Original: = 0 .
Revised: = 0 .

Original: Inclusion Theorem 1.
Revised: Inclusion Theorem 1.

Original: Combining Proposition 1 and Proposition 2, we simply need to show that Π T ⊂ Π D .
Revised: Combining Proposition 1 and Proposition 2, we simply need to show that Π T ⊂ Π D .

Original: Take π to be a fixed point of ⟨ π, T ⟩ .
Revised: Take π to be a fixed point of ⟨ π, T ⟩ .

Original: Then ⟨ π, T ⟩ = π , and from the definition in equation 4:
Revised: Then ⟨ π, T ⟩ = π , and from the definition in equation 4:

Original: [Equation]
Revised: [Equation]

Original: Therefore, π ∈ Π D , which completes the sequence of inclusions.
Revised: Therefore, π ∈ Π D , which completes the sequence of inclusions.

Original: To show convexity of Π , Π T , first for a constant policy π ∈ Π , recall that we can write π = 1 v ⊤ , where v ∈ ∆( U ) is any probability distribution over the action space.
Revised: To show convexity of Π , Π T , first for a constant policy π ∈ Π , recall that we can write π = 1 v ⊤ , where v ∈ ∆( U ) is any probability distribution over the action space.

Original: Now take π 1 , π 2 ∈ Π .
Revised: Now take π 1 , π 2 ∈ Π .

Original: For any α ∈ [0 , 1] , απ 1 + (1 − α ) π 2 =
Revised: For any α ∈ [0 , 1] , απ 1 + (1 − α ) π 2 =

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: + (
Revised: + (

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ) v 2 ) ⊤ ∈ Π .
Revised: ) v 2 ) ⊤ ∈ Π .

Original: At last, for the set Π T , assume there exist two different policies π 1 , π 2 both fixed points of ⟨· , T ⟩ .
Revised: At last, for the set Π T , assume there exist two different policies π 1 , π 2 both fixed points of ⟨· , T ⟩ .

Original: Then, for any α ∈ [0 , 1] , ⟨ ( απ 1 + (1 − α ) π 2 ) , T ⟩ = αTπ 1 + (1 − α )
Revised: Then, for any α ∈ [0 , 1] , ⟨ ( απ 1 + (1 − α ) π 2 ) , T ⟩ = αTπ 1 + (1 − α )

Original: Tπ 

Revised: Tπ

Original: = απ 

Revised: = απ

Original: + (1 − α )
Revised: + (1 − α )

Original: π 2 .
Revised: π 2 .

Original: Therefore, any affine combination of fixed points is also a fixed point.
Revised: Therefore, any affine combination of fixed points is also a fixed point.

Original: Corollary 1.
Revised: Corollary 1.

Original: For statement (i), let R ( · , · , · ) =
Revised: For statement (i), let R ( · , · , · ) =

Original: c for some constant c ∈ R .
Revised: c for some constant c ∈ R .

Original: Then, J ( π ) =
Revised: Then, J ( π ) =

Original: E x 0 ∼ µ 

Revised: E x 0 ∼ µ

Original: [ (cid:80) t γ t r t | π ] = cγ 1 − γ , which does not depend on the policy π .
Revised: [ (cid:80) t γ t r t | π ] = cγ 1 − γ , which does not depend on the policy π .

Original: For any noise kernel T and policy π , J ( π ) − J ⟨ π, T ⟩ = 0 = ⇒ π ∈ Π 0 .
Revised: For any noise kernel T and policy π , J ( π ) − J ⟨ π, T ⟩ = 0 = ⇒ π ∈ Π 0 .

Original: For statement (ii) assume ∃ π ∈ Π 0 : π / ∈ Π T .
Revised: For statement (ii) assume ∃ π ∈ Π 0 : π / ∈ Π T .

Original: Then, ∃ x ∗ ∈ X and u ∗ ∈
Revised: Then, ∃ x ∗ ∈ X and u ∗ ∈

Original: U such that π ( x ∗ , u ∗ ) ̸
Revised: U such that π ( x ∗ , u ∗ ) ̸

Original: = ⟨ π, T ⟩ ( x ∗ , u ∗ ) .
Revised: = ⟨ π, T ⟩ ( x ∗ , u ∗ ) .

Original: Let:
Revised: Let:

Original: R ( x, u, x ′ ) :
Revised: R ( x, u, x ′ ) :

Original: = (cid:26) c if x = x ∗ and u = u ∗ otherwise .
Revised: = (cid:26) c if x = x ∗ and u = u ∗ otherwise .

Original: Then, E [ R ( x, π ( x ) , x ′ ] <
Revised: Then, E [ R ( x, π ( x ) , x ′ ] <

Original: E
Revised: E

Original: [ R ( x, ⟨ π, T ⟩ ( x ) , x ′ ] and since the MDP is ergodic x is visited infinitely often and J ( π ) − J ( ⟨ π, T ⟩ ) > 0 =
Revised: [ R ( x, ⟨ π, T ⟩ ( x ) , x ′ ] and since the MDP is ergodic x is visited infinitely often and J ( π ) − J ( ⟨ π, T ⟩ ) > 0 =

Original: ⇒ π / ∈ Π 0 , which contradicts the assumption.
Revised: ⇒ π / ∈ Π 0 , which contradicts the assumption.

Original: Therefore, Π 0 \ Π T = ∅ = ⇒ Π
Revised: Therefore, Π 0 \ Π T = ∅ = ⇒ Π

Original: =
Revised: =

Original: Π T .
Revised: Π T .

Original: Lemma 1.
Revised: Lemma 1.

Original: We make use of standard results on stochastic approximation with non-expansive operators (specifically, Theorem 3 in the appendix) Borkar & Soumyanatha (1997).
Revised: We make use of standard results on stochastic approximation with non-expansive operators (specifically, Theorem 3 in the appendix) Borkar & Soumyanatha (1997).

Original: First, observe that for a fully parameterised policy, one can assume to have a tabular representation such that π θ ( x, u ) = θ xu , and ∇ θ π θ ( x ) ≡ Id .
Revised: First, observe that for a fully parameterised policy, one can assume to have a tabular representation such that π θ ( x, u ) = θ xu , and ∇ θ π θ ( x ) ≡ Id .

Original: We can then write the stochastic gradient descent problem in terms of the policy.
Revised: We can then write the stochastic gradient descent problem in terms of the policy.

Original: Let y ∼ ˜ T ( · | x ) .
Revised: Let y ∼ ˜ T ( · | x ) .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: We now need to verify that the necessary conditions for applying Theorem 3 hold.
Revised: We now need to verify that the necessary conditions for applying Theorem 3 hold.

Original: First, α t satisfies
Revised: First, α t satisfies

Original: Assumption 2.
Revised: Assumption 2.

Original: Second, making use of the property ∥ ˜ T ∥ ∞
Revised: Second, making use of the property ∥ ˜ T ∥ ∞

Original: = 1 for any row-stochastic matrix ˜ T , for any two policies π 1 , π 2 ∈ Π :
Revised: = 1 for any row-stochastic matrix ˜ T , for any two policies π 1 , π 2 ∈ Π :

Original: [Equation]
Revised: [Equation]

Original: Therefore, the operator ⟨· , ˜ T ⟩ is non-expansive with respect to the sup-norm.
Revised: Therefore, the operator ⟨· , ˜ T ⟩ is non-expansive with respect to the sup-norm.

Original: For the final condition, we have
Revised: For the final condition, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, the difference π t ( y ) − ⟨ π t , ˜ T ⟩ ( x ) is a martingale difference for all x .
Revised: Therefore, the difference π t ( y ) − ⟨ π t , ˜ T ⟩ ( x ) is a martingale difference for all x .

Original: One can then apply
Revised: One can then apply

Original: Theorem 3 with ξ t ( x ) ≡ π t ( x ) , F ( · ) ≡ ⟨· , ˜ T ⟩ and M t +1 ≡ π t ( y ) − ⟨ π t , ˜ T ⟩ ( x ) to conclude that π t ( x ) → ˜ π ( x ) almost surely.
Revised: Theorem 3 with ξ t ( x ) ≡ π t ( x ) , F ( · ) ≡ ⟨· , ˜ T ⟩ and M t +1 ≡ π t ( y ) − ⟨ π t , ˜ T ⟩ ( x ) to conclude that π t ( x ) → ˜ π ( x ) almost surely.

Original: Finally from assumption 1, for any policy all states x ∈ X are visited infinitely often, therefore π t ( x ) → ˜ π ( x ) ∀ x ∈ X = ⇒ π t →
Revised: Finally from assumption 1, for any policy all states x ∈ X are visited infinitely often, therefore π t ( x ) → ˜ π ( x ) ∀ x ∈ X = ⇒ π t →

Original: ˜ π and ˜ π satisfies ⟨ ˜ π, ˜ T ⟩ = ˜ π , and K ˜ T (˜ π )
Revised: ˜ π and ˜ π satisfies ⟨ ˜ π, ˜ T ⟩ = ˜ π , and K ˜ T (˜ π )

Original: = 0 .
Revised: = 0 .

Original: Theorem 2.
Revised: Theorem 2.

Original: We apply the results from Skalse et al.
Revised: We apply the results from Skalse et al.

Original: (2022b) in Theorem 4.
Revised: (2022b) in Theorem 4.

Original: Essentially, Skalse et al.
Revised: Essentially, Skalse et al.

Original: (2022b) prove that for a policy gradient algorithm to lexicographically optimise a policy for multiple objectives, it is a sufficient condition that the stochastic gradient descent algorithm finds optimal parameters for each of the objectives independently.
Revised: (2022b) prove that for a policy gradient algorithm to lexicographically optimise a policy for multiple objectives, it is a sufficient condition that the stochastic gradient descent algorithm finds optimal parameters for each of the objectives independently.

Original: From Lemma 1 we know that a policy gradient algorithm using the gradient estimate in equation 9 converges to a maximally robust policy, i.e. a set of parameters θ ′ =
Revised: From Lemma 1 we know that a policy gradient algorithm using the gradient estimate in equation 9 converges to a maximally robust policy, i.e. a set of parameters θ ′ =

Original: arg max θ K ˜ T .
Revised: arg max θ K ˜ T .

Original: Additionally, by assumption, the chosen algorithm for Kconverges to an optimal point θ ∗ .
Revised: Additionally, by assumption, the chosen algorithm for Kconverges to an optimal point θ ∗ .

Original: While the two objective functions are not of the same form – as in Skalse et al. (2022b) – the fact they are both invex (Ben-Israel & Mond, 1986b) either locally or globally depending on the form of K 1 , implies that ˆ K is also invex and hence that the stationary point θ ϵ computed by LRPG satisfies equation 6.
Revised: While the two objective functions are not of the same form – as in Skalse et al. (2022b) – the fact they are both invex (Ben-Israel & Mond, 1986b) either locally or globally depending on the form of K 1 , implies that ˆ K is also invex and hence that the stationary point θ ϵ computed by LRPG satisfies equation 6.

Original: B. O N A DVERSARIAL D ISTURBANCES AND OTHER N OISE K ERNELS
Revised: B. O N A DVERSARIAL D ISTURBANCES AND OTHER N OISE K ERNELS

Original: A problem that remains open after this work is what constitutes an appropriate choice of ˜ T , and what can we expect by restricting a particular class of ˜ T .
Revised: A problem that remains open after this work is what constitutes an appropriate choice of ˜ T , and what can we expect by restricting a particular class of ˜ T .

Original: We first discuss adversarial examples, and then general considerations on ˜ T versus T .
Revised: We first discuss adversarial examples, and then general considerations on ˜ T versus T .

Original: Adversarial Noise As mentioned in the introduction, much of the previous work focuses on adversarial disturbances.
Revised: Adversarial Noise As mentioned in the introduction, much of the previous work focuses on adversarial disturbances.

Original: We did not directly address this in the results of this work since our motivation lies in the scenarios where the disturbance is not adversarial and is unknown.
Revised: We did not directly address this in the results of this work since our motivation lies in the scenarios where the disturbance is not adversarial and is unknown.

Original: However, following the results of Section 3, we are able to reason about adversarial disturbances.
Revised: However, following the results of Section 3, we are able to reason about adversarial disturbances.

Original: Consider an adversarial map T adv to be
Revised: Consider an adversarial map T adv to be

Original: [Equation]
Revised: [Equation]

Original: with X ad ( x ) ⊆
Revised: with X ad ( x ) ⊆

Original: X being a set of admissible disturbance states for x , and d ( · , · ) is a distance measure between distributions ( e.g. 2-norm).
Revised: X being a set of admissible disturbance states for x , and d ( · , · ) is a distance measure between distributions ( e.g. 2-norm).

Original: Proposition 3.
Revised: Proposition 3.

Original: Constant policies are a fixed point of T adv , and are the only fixed points if for all pairs x 0 , x k there exists a sequence { x 0 , ..., x k } ⊆
Revised: Constant policies are a fixed point of T adv , and are the only fixed points if for all pairs x 0 , x k there exists a sequence { x 0 , ..., x k } ⊆

Original: X such that x i ∈
Revised: X such that x i ∈

Original: X ad ( x i ) .
Revised: X ad ( x i ) .

Original: Proof.
Revised: Proof.

Original: First, it is straight-forward that if π ∈ Π = ⇒ ⟨ π, T adv ⟩ ( x ) =
Revised: First, it is straight-forward that if π ∈ Π = ⇒ ⟨ π, T adv ⟩ ( x ) =

Original: π ( x ) .
Revised: π ( x ) .

Original: To show they are the only fixed points, assume that there is a non-constant policy π ′ that is a fixed point of T ad .
Revised: To show they are the only fixed points, assume that there is a non-constant policy π ′ that is a fixed point of T ad .

Original: Then, there exists x, z such that π ′ ( x ) ̸ = π ′ ( z ) .
Revised: Then, there exists x, z such that π ′ ( x ) ̸ = π ′ ( z ) .

Original: However, by assumption, we can construct a sequence { x, ..., z } ⊆
Revised: However, by assumption, we can construct a sequence { x, ..., z } ⊆

Original: X that connects x and z and every state in the sequence is in the admissible set of the previous one.
Revised: X that connects x and z and every state in the sequence is in the admissible set of the previous one.

Original: Assume without loss of generality that this sequence is { x, y, z } .
Revised: Assume without loss of generality that this sequence is { x, y, z } .

Original: Then, if π ′ is a fixed point, ⟨ π ′ , T adv ⟩ ( x ) = π ′ ( x ) , ⟨ π ′ , T adv ⟩ ( y ) =
Revised: Then, if π ′ is a fixed point, ⟨ π ′ , T adv ⟩ ( x ) = π ′ ( x ) , ⟨ π ′ , T adv ⟩ ( y ) =

Original: π ′ ( y ) and ⟨ π ′ , T adv ⟩ ( z ) =
Revised: π ′ ( y ) and ⟨ π ′ , T adv ⟩ ( z ) =

Original: π ′ ( z ) .
Revised: π ′ ( z ) .

Original: However, π ′ ( x ) ̸ = π ′ ( z ) , so either π ′ ( x ) ̸ = π ′ ( y )
Revised: However, π ′ ( x ) ̸ = π ′ ( z ) , so either π ′ ( x ) ̸ = π ′ ( y )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: d ( π ′ ( y ) , π ′ ( z ))
Revised: d ( π ′ ( y ) , π ′ ( z ))

Original: ̸
Revised: ̸

Original: = 0 , therefore π ′ cannot be a fixed point of T adv .
Revised: = 0 , therefore π ′ cannot be a fixed point of T adv .

Original: The main difference between an adversarial operator and the random noise considered throughout this work is that T adv is not a linear operator , and additionally, it is time varying (since the policy is being modified at every time step of the PG algorithm).
Revised: The main difference between an adversarial operator and the random noise considered throughout this work is that T adv is not a linear operator , and additionally, it is time varying (since the policy is being modified at every time step of the PG algorithm).

Original: Therefore, including it as a LRPG objective would invalidate the assumptions required for LRPG to retain formal guarantees of the original PG algorithm used, and it is not guaranteed that the resulting policy gradient algorithm would converge.
Revised: Therefore, including it as a LRPG objective would invalidate the assumptions required for LRPG to retain formal guarantees of the original PG algorithm used, and it is not guaranteed that the resulting policy gradient algorithm would converge.

Original: Assumption of Noise Kernel A question emerging from Section 4 is how to choose ˜ T , and how the choice influences the resulting policy robustness towards any other true T .
Revised: Assumption of Noise Kernel A question emerging from Section 4 is how to choose ˜ T , and how the choice influences the resulting policy robustness towards any other true T .

Original: In general, for any arbitrary policy landscape with respect to utility in a given MDP, there is no way of bounding the distance of resulting policies for two different noise kernels T 1 , T 2 .
Revised: In general, for any arbitrary policy landscape with respect to utility in a given MDP, there is no way of bounding the distance of resulting policies for two different noise kernels T 1 , T 2 .

Original: As a counter-example, consider a
Revised: As a counter-example, consider a

Original: MDP where there are 2 possible optimal policies π ∗ 1 , π ∗ 2 , and take these two policies to be maximally different, i.e. ∥ π ∗ 1 ( x ) − π ∗ 2 ( x ) ∥ ∞ = 1 ∀ x ∈ X .
Revised: MDP where there are 2 possible optimal policies π ∗ 1 , π ∗ 2 , and take these two policies to be maximally different, i.e. ∥ π ∗ 1 ( x ) − π ∗ 2 ( x ) ∥ ∞ = 1 ∀ x ∈ X .

Original: Then, when using LRPG to obtain a robust policy, a slight deviation in the choice of ˜ T can cause the gradient descent scheme to deviate from converging to π ∗ 1 to converging to π ∗ 2 , yielding in principle a completely different policy.
Revised: Then, when using LRPG to obtain a robust policy, a slight deviation in the choice of ˜ T can cause the gradient descent scheme to deviate from converging to π ∗ 1 to converging to π ∗ 2 , yielding in principle a completely different policy.

Original: However, what remains bounded is the optimality of the policy : Through LRPG guarantees we know that, for both cases, the resulting policy will be at most ϵ far from the optimal sum of rewards.
Revised: However, what remains bounded is the optimality of the policy : Through LRPG guarantees we know that, for both cases, the resulting policy will be at most ϵ far from the optimal sum of rewards.

Original: We can, however, state the following.
Revised: We can, however, state the following.

Original: Take T to be any arbitrary noise kernel, and ˜ T a kernel that satisfies Π ˜ T ≡ Π .
Revised: Take T to be any arbitrary noise kernel, and ˜ T a kernel that satisfies Π ˜ T ≡ Π .

Original: Let π to be a policy resulting from a LRPG algorithm.
Revised: Let π to be a policy resulting from a LRPG algorithm.

Original: Assume, for a distance metric d , that min π ′ ∈ Π ˜ T d ( π, π ′ ) ≤ a for some a < 1 .
Revised: Assume, for a distance metric d , that min π ′ ∈ Π ˜ T d ( π, π ′ ) ≤ a for some a < 1 .

Original: Then, it holds for any T that min
Revised: Then, it holds for any T that min

Original: π ′ ∈ Π T d ( π, π ′ ) ≤
Revised: π ′ ∈ Π T d ( π, π ′ ) ≤

Original: a : That is, the resulting policy is at most a far from the set of fixed points (and therefore a maximally robust policy) with respect to the true T .
Revised: a : That is, the resulting policy is at most a far from the set of fixed points (and therefore a maximally robust policy) with respect to the true T .

Original: This is the key argument behind our choices for ˜ T :
Revised: This is the key argument behind our choices for ˜ T :

Original: A priori, the most sensible choice is a kernel that has no other fixed point than the set of constant policies.
Revised: A priori, the most sensible choice is a kernel that has no other fixed point than the set of constant policies.

Original: C E XPERIMENTS M ETHODOLOGY
Revised: C E XPERIMENTS M ETHODOLOGY

Original: We use in the experiments well-tested implementations of A2C and PPO adapted from Zhang (2018) to include the computation of the lexicographic parameters in equation 1.
Revised: We use in the experiments well-tested implementations of A2C and PPO adapted from Zhang (2018) to include the computation of the lexicographic parameters in equation 1.

Original: Since all the environments use a pixel representation of the observation, we use a shared representation for the value function and policy, where the first component is a convolutional network, implemented as in Zhang (2018).
Revised: Since all the environments use a pixel representation of the observation, we use a shared representation for the value function and policy, where the first component is a convolutional network, implemented as in Zhang (2018).

Original: The hyper-parameters of the neural representations are presented in Table 2.
Revised: The hyper-parameters of the neural representations are presented in Table 2.

Original: [Table]
Revised: [Table]

Original: The actor and critic layers, for both algorithms, are a fully connected layer with 256 features as input and the corresponding output.
Revised: The actor and critic layers, for both algorithms, are a fully connected layer with 256 features as input and the corresponding output.

Original: We used in all cases an Adam optimiser.
Revised: We used in all cases an Adam optimiser.

Original: We optimised the parameters for each (vanilla) algorithm through a quick parameter search, and apply the same parameters for the Lexicographically Robust versions.
Revised: We optimised the parameters for each (vanilla) algorithm through a quick parameter search, and apply the same parameters for the Lexicographically Robust versions.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: For the implementation of the LRPG versions of the algorithms, in all cases we allow the algorithm to iterate for 1 / 3 of the total steps before starting to compute the robustness objectives.
Revised: For the implementation of the LRPG versions of the algorithms, in all cases we allow the algorithm to iterate for 1 / 3 of the total steps before starting to compute the robustness objectives.

Original: In other words, we use ˆ K ( θ )
Revised: In other words, we use ˆ K ( θ )

Original: =
Revised: =

Original: K 1 ( θ ) until t = 13 max steps , and from this point we resume the lexicographic robustness computation as described in Algorithm 1.
Revised: K 1 ( θ ) until t = 13 max steps , and from this point we resume the lexicographic robustness computation as described in Algorithm 1.

Original: This is due to the structure of the environments simulated.
Revised: This is due to the structure of the environments simulated.

Original: The rewards (and in particular the positive rewards) are very sparse in the environments considered.
Revised: The rewards (and in particular the positive rewards) are very sparse in the environments considered.

Original: Therefore, when computing the policy gradient steps, the loss for the primary objective is practically zero until the environment is successfully solved at least once.
Revised: Therefore, when computing the policy gradient steps, the loss for the primary objective is practically zero until the environment is successfully solved at least once.

Original: If we implement the combined lexicographic loss from the first time step, many times the algorithm would converge to a (constant) policy without exploring for enough steps, leading to convergence towards a maximally robust policy that does not solve the environment.
Revised: If we implement the combined lexicographic loss from the first time step, many times the algorithm would converge to a (constant) policy without exploring for enough steps, leading to convergence towards a maximally robust policy that does not solve the environment.

Original: Noise Kernels.
Revised: Noise Kernels.

Original: We consider two types of noise; a normal distributed noise ˜ T g and a uniform distributed noise ˜ T u .
Revised: We consider two types of noise; a normal distributed noise ˜ T g and a uniform distributed noise ˜ T u .

Original: For the environments LavaGap and DynamicObstacles, the kernel ˜ T u produces a disturbed state ˜ x
Revised: For the environments LavaGap and DynamicObstacles, the kernel ˜ T u produces a disturbed state ˜ x

Original: = x + ξ where ∥ ξ ∥ ∞ ≤ 2 , and for LavaCrossing ∥ ξ ∥ ∞ ≤ 1 . 5 .
Revised: = x + ξ where ∥ ξ ∥ ∞ ≤ 2 , and for LavaCrossing ∥ ξ ∥ ∞ ≤ 1 . 5 .

Original: The normal distributed noise is in all cases
Revised: The normal distributed noise is in all cases

Original: N (0 , 0 . 5) .
Revised: N (0 , 0 . 5) .

Original: The maximum norm of the noise is quite large, but this is due to the structure of the observations in these environments.
Revised: The maximum norm of the noise is quite large, but this is due to the structure of the observations in these environments.

Original: The pixel values are encoded as integers 0 − 9 , where each integer represents a different feature in the environment (empty space, doors, lava, obstacle, goal...).
Revised: The pixel values are encoded as integers 0 − 9 , where each integer represents a different feature in the environment (empty space, doors, lava, obstacle, goal...).

Original: Therefore, any noise ∥ ξ ∥ ∞ ≤ 0 .
Revised: Therefore, any noise ∥ ξ ∥ ∞ ≤ 0 .

Original: 5 would most likely not be enough to confuse the agent.
Revised: 5 would most likely not be enough to confuse the agent.

Original: On the other hand, too large noise signals are unrealistic and produce pathological environments.
Revised: On the other hand, too large noise signals are unrealistic and produce pathological environments.

Original: All the policies are then tested against two “true” noise kernels, T 1 = ˜ T u and
Revised: All the policies are then tested against two “true” noise kernels, T 1 = ˜ T u and

Original: T 2 =
Revised: T 2 =

Original: ˜ T g .
Revised: ˜ T g .

Original: The main reason for this is to test both the scenarios where we assume a wrong noise kernel, and the case where we are training the agents with the correct kernel.
Revised: The main reason for this is to test both the scenarios where we assume a wrong noise kernel, and the case where we are training the agents with the correct kernel.

Original: LRPG Parameters.
Revised: LRPG Parameters.

Original: The LRL parameters are initialised in all cases as β 10 = 2 , β 20 = 1 , λ =and η = 0 .
Revised: The LRL parameters are initialised in all cases as β 10 = 2 , β 20 = 1 , λ =and η = 0 .

Original: The LRL tolerance is set to ϵ t = 0 .
Revised: The LRL tolerance is set to ϵ t = 0 .

Original: 99 ˆ k 1 to ensure we never deviate too much from the original objective, since the environments have very sparse rewards.
Revised: 99 ˆ k 1 to ensure we never deviate too much from the original objective, since the environments have very sparse rewards.

Original: We use a first order approximation to compute the LRL weights from the original LMORL implementation.
Revised: We use a first order approximation to compute the LRL weights from the original LMORL implementation.

Original: Comparison with SA-PPO.
Revised: Comparison with SA-PPO.

Original: One of the baselines included is the State-Adversarial PPO algorithm proposed in Zhang et al.
Revised: One of the baselines included is the State-Adversarial PPO algorithm proposed in Zhang et al.

Original: We altered our implementation of PPO to incorporate the adversarial optimisation for the disturbances as described by Zhang et al.

Original: The implementation includes an extra parameter that multiplies the regularisation objective, k ppo .
Revised: The implementation includes an extra parameter that multiplies the regularisation objective, k ppo .

Original: Since we were not able to find indications on the best parameter for  discrete action environments, we implemented k ppo ∈ { 0 . 1 , 1 , 2 } and picked the best result for each entry in Table 1.
Revised: Since we were not able to find indications on the best parameter for [Table] discrete action environments, we implemented k ppo ∈ { 0 . 1 , 1 , 2 } and picked the best result for each entry in Table 1.

Original: Larger values seemed to de-stabilise the learning in some cases.
Revised: Larger values seemed to de-stabilise the learning in some cases.

Original: The rest of the parameters are kept as in the vanilla PPO implementation.
Revised: The rest of the parameters are kept as in the vanilla PPO implementation.

Revised: C. E XTENDED R ESULTS : A DVERSARIAL D ISTURBANCES

Revised: Even though we do not use an adversarial attacker or disturbance in our reasoning through this work, we implemented a policy-based state-adversarial noise disturbance to test the benchmark algorithms against, and evaluate how well each of the methods reacts to such adversarial disturbances.

Revised: Adversarial Disturbance

Revised: We implement a bounded policy-based adversarial attack, where at each state x we maximise for the KL divergence between the disturbed and undisturbed state, such that the adversarial operator is:

Revised: [Equation]

Revised: The optimisation problem is solved at every point by using a Stochastic Gradient Langevin Dynamics (SGLD) optimiser.

Revised: The results are presented in Table 5.

Revised: This type of adversarial attack with SGLD optimiser was proposed in Zhang et al.

Revised: As one can see, the adversarial disturbance is quite successful at severely lowering the obtained rewards in all scenarios.

Revised: Additionally, as expected SA-PPO was the most effective at minimizing the disturbance effect (as it is trained with adversarial disturbances), although LRPG produces reasonably robust policies against this type of disturbances as well.

Revised: At last, A2C appears to be much more sensitive to adversarial disturbances than PPO, indicating that the policies produced by PPO are by default more robust than A2C.