S
AFE R EINFORCEMENT L EARNING WITH N ATURAL L ANGUAGE C ONSTRAINTS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we tackle the problem of learning control policies for tasks when provided with constraints in natural language.
In contrast to instruction following, language here is used not to specify goals, but rather to describe situations that an agent must avoid during its exploration of the environment.
Specifying constraints in natural language also differs from the predominant paradigm in safe reinforcement learning, where safety criteria are enforced by hand-deﬁned cost functions.
While natural language allows for easy and ﬂexible speciﬁcation of safety constraints and budget limitations, its ambiguous nature presents a challenge when mapping these speciﬁcations into representations that can be used by techniques for safe reinforcement learning.
To address this, we develop a model that contains two components: (1) a constraint interpreter to encode natural language constraints into vector representations capturing spatial and temporal information on forbidden states, and (2) a policy network that uses these representations to output a policy with minimal constraint violations.
Our model is end-to-end differentiable and we train it using a recently proposed algorithm for constrained policy optimization.
To empirically demonstrate the effectiveness of our approach, we create a new benchmark task for autonomous navigation with crowd-sourced freeform text specifying three different types of constraints.
Our method outperforms several baselines by achieving 6-7 times higher returns and 76% fewer constraint violations on average.
Dataset and code to reproduce our experiments are available at https://sites.google.com/view/polco-hazard-world/ .
I NTRODUCTION
Reinforcement learning (RL) has shown great promise in a variety of control problems including robot navigation (Anderson et al., 2018;
Misra et al., 2018) and robotic control (Levine et al., 2016;
Rajeswaran et al., 2017), where the main goal is to optimize for scalar returns.
However, as RL is increasingly deployed in many real-world problems, it is imperative to ensure the safety of both agents and their surroundings, which requires accounting for constraints that may be orthogonal to maximizing returns.
While there exist several safe RL algorithms (Achiam et al., 2017; Chow et al., 2019; Yang et al., 2020b) in the literature, a major limitation they share is the need to manually specify constraint costs and budget limitations.
In many real-world problems, safety criteria tend to be abstract and quite challenging to deﬁne, making their speciﬁcation ( e.g., as logical rules or mathematical constraints) an expensive task requiring domain expertise.
On the other hand, natural language provides an intuitive and easily-accessible medium for specifying constraints – not just for experts or system developers, but also for potential end users of the RL agent.
For example, instead of specifying a safety constraint in the form of “ if water not in previously visited states then do not visit lava ”, one can simply say “ Do not visit the lava before visiting the water .”
The key challenge lies in training the RL agent to interpret natural language and accurately adhere to the constraints during exploration and execution.
In this paper, we develop a novel framework for safe reinforcement learning that can handle natural language constraints.
This setting is different from traditional instruction following, in which text instructions are used to specify goals for the agent ( e.g., “ reach the key ” or “ go forward two steps ”).
To effectively learn a safe policy that obeys text constraints, we propose a model consisting of two key modules.
First, we use a constraint interpreter to encode language constraints into intermediate vector and matrix representations–this captures spatial information of forbidden states and the long-term dependency of the past states.
Second, we design a policy network that operates on a combination of these intermediate representations and state observations and is trained using a constrained policy optimization algorithm
( e.g., PCPO (Yang et al., 2020b)).
This allows our agent to map the abstract safety criteria (in language) into cost representations that are amendable for safe
RL.
We call our approach Policy Optimization with Language COnstraints (POLCO).
Since there do not exist standard benchmarks for safe RL with language constraints, we construct a new navigation task called Hazard World .
Hazard World is a 2D grid world environment with diverse, free-form text representing three types of constraints: (1) budgetary constraints that limit resource usage or the frequency of being in undesirable states ( e.g., “ The lava is really hot, so it will hurt you a lot.
Please only walk on it 3 times ”), (2) relational constraints that specify forbidden states in relation to surrounding entities in the environment ( e.g., “ There should always be at least 3 squares between you and water ”), and (3) sequential constraints that depend on past events ( e.g., “ Grass will surround your boots and protect you from dangerous lava. ’).
Fig.
1 provides a sample situation from the task.
In summary, we make the following key contributions.
First, we formulate the problem of safe RL with safety criteria speciﬁed in natural language.
Second, we propose POLCO, a new policy architecture and two-stage safe RL algorithm that ﬁrst encodes natural language constraints into quantitative representations and then uses these representations to learn a constraint-satisfying policy.
Third, we introduce a new safe RL dataset (Hazard
World) containing three broad classes of abstract safety criteria, all described in diverse free-form text.
Finally, we empirically compare POLCO against other baselines in Hazard World.
We show that POLCO outperforms other baselines by achieving 6-7 times higher returns and 76% fewer constraint violations on average over three types of constraints.
We also perform extensive evaluations and analyses of our model and provide insights for future improvements.
R ELATED W ORK
Policy optimization with constraints.
Learning constraint-satisfying policies has been explored in prior work in the context of safe RL (see Garcia & Fernandez (2015) for a survey).
Typically, the agent learns policies either by (1) exploring the environment to identify forbidden behaviors (Achiam et al., 2017; Tessler et al., 2018;
Chow et al., 2019;
Yang et al., 2020b; Stooke et al., 2020), or (2) through expert demonstration data to recognize the safe trajectories (Ross et al., 2011;
Rajeswaran et al., 2017; Gao et al., 2018;
Yang et al., 2020a).
Critically, these works all require a human to specify the cost constraints manually.
In contrast, we use natural language to describe the cost constraints, which allows for easier and more ﬂexible speciﬁcations of safety constraint.
Instruction following without constraints.
Instruction following for 2D and 3D navigation has been explored in the context of deep RL (MacMahon et al., 2006; Vogel & Jurafsky, 2010; Chen &
Mooney, 2011; Artzi & Zettlemoyer, 2013; Kim & Mooney, 2013; Andreas & Klein, 2015; Thomason et al., 2020; Luketina et al., 2019; Tellex et al., 2020).
Prior work either focuses on providing a dataset with a real-life visual urban or household environment ( e.g., Google street view)
(Bisk et al., 2018;
Chen et al., 2019; Anderson et al., 2018; de Vries et al., 2018); or proposes a computational model to learn multi-modal representations that fuse 2D or 3D images with goal instructions (Janner et al., 2018;
Blukis et al., 2018; Fried et al., 2018; Liu et al., 2019; Jain et al., 2019; Gaddy & Klein, 2019; Hristov et al., 2019; Fu et al., 2019).
These work use text to specify goals, not environmental hazards.
In contrast, we use language to describe the constraints that the agent must obey.
Constraints in natural language.
Misra et al.
(2018) propose two datasets called LANI and CHAI to study spatial and temporal reasoning, as well as perception and planning.
Their dataset contains a few trajectory constraints , which specify goal locations ( e.g., “ go past the house by the right side of the apple ”).
However, (1) their ‘constraints’ are actually goal instructions describing where the reward is–not where forbidden states are, and (2) their model ﬁrst recognizes (sub)goals from language and then uses these language-deﬁned goals as reward functions.
Prakash et al.
(2020) also propose a method to convert constraints in synthetic language to reward functions.
These approaches do not guarantee safety since they do not explicitly model the constraints.
In our work, we deﬁne ‘constraints’ as restrictions in state space, or speciﬁcations of forbidden states and learn safe policies.
The table below provides a quick summary of related work in instruction following and safe RL.
We differentiate these methods on several key axes: (1) whether they propose a new dataset, algorithm or model, (2) whether they optimize reward functions or satisfy cost constraints, and (3) whether they have text or 3D view simulation components.
We also brieﬂy describe their main objectives.
[Table]
P ROBLEM F ORMULATION
Our goal is to learn a constraint-satisfying policy that imposes safety or other application-speciﬁc constraints using text descriptions.
We formulate our problem as a partially observable constrained
Markov decision process (PO-CMDP) (Kaelbling et al., 1998; Altman, 1999) with text, which is deﬁned by the tuple < S , O , A , T, Z, R, C, X > .
Here S is the set of states, O is the set of observations, A is the set of actions, T is the conditional probability T ( s (cid:48) | s, a ) of the next state s (cid:48) given the current state s and action a , Z is the conditional probability
Z
( o | s ) of the observation o given the state s, R : S ×
A → R is the reward function, C : S ×
A → R is the true underlying constraint speciﬁcation, and X is the set of constraint speciﬁcations in natural language.
The reward function encodes the beneﬁt of using action a in state s , while the cost speciﬁcation encodes penalties for constraint violations.
The agent does not know C and only observes constraints from X .
We assume that there exists a mapping from a language constraint x ∈ X to a cost speciﬁcation C.
A policy π : O → P ( A ) is a mapping from observation O to the distributions of actions A .
Let γ ∈ (0 , 1) denote a discount factor, µ : S → [0 , 1] denote the initial state distribution, and τ denote the trajectory τ =
( o 0 , a 0 , o 1 , · · · ) induced by a policy π.
We seek a policy π that maximizes the cumulative discounted reward J R while keeping the cumulative discounted cost J C below the cost constraint threshold h C ( x ) :
[Equation]
where τ ∼ π is shorthand for indicating that the distribution over trajectories depends on π : s 0 ∼ µ, o t ∼ Z ( ·| s t ) , a t ∼ π ( ·| o t ) , s t +1 ∼ T ( ·| s t , a t ) , and we use C ( s t , a t ; x ) and h C ( x ) to emphasize that they are speciﬁed by x .
And we will drop x for notation simplicity.
L EARNING P OLICIES WITH N ATURAL L ANGUAGE C ONSTRAINTS 4. M ODEL
We now describe our proposed neural model in POLCO, illustrated in Fig.
The model consists of two parts – (1) the constraint interpreter processes the text to form safety criteria (constraint mask and threshold) and (2) the policy network subsequently produces an action.
For simplicity, we assume state s and observation o to be 2D matrices, although the model can easily be extended to other input representations ( e.g., we describe an extension to 3D scene inputs in the appendix).
Constraint interpreter.
Fig.
3 illustrates the constraint interpreter.
The constraint interpreter itself consists of two parts – (1) a constraint mask module and a (2) constraint threshold module .
(1) The constraint mask module uses the observation o t and the text x to predict a binary constraint mask , denoted by ˆ M C , a prediction of the true constraint mask M C .
Each cell in ˆ M C will contain a one if there is a cost entity ( i.e., the forbidden states mentioned in the text) in i th row and j th column of the observation o t (denoted by o t ( i, j ) ).
Otherwise, the cell contains a zero.
We use ˆ M C to identify the cost entity in texts while preserving its spatial information for the policy network.
The constraint threshold module uses an LSTM to obtain the text vector representation, followed by a dense layer to produce ˆ h C , a prediction of the true constraint threshold h C . Full details of the constraint interpreter can be found in Appendix A.2.
For sequential constraints with long-term dependency of the past states, ˆ M C changes based on the states visited.
For example, in Fig. 3(b), after the agent visits ‘ water ’, ˆ M C starts to locate the cost entity ( i.e., ‘ grass ’).
Hence, we will need memory to keep track of the visited states.
For this, we use an LSTM that takes the vector representation produced by a convolutional neural network (CNN) as an input and predicts ˆ M C , as illustrated in Fig. 3(b).
Using ˆ M C and ˆ h C allows us to specify safety criteria in text while encoding them into the policy network.
Policy network.
The policy network produces an action given the safety criteria processed by the constraint interpreter.
The input to the network is the environment embedding (a tensor of size n × n × m ) that encodes the observation o t which is of size n × n.
This tensor is concatenated with the constraint mask ˆ M C and a cost budget mask , denoted by ˆ M B , a prediction of the true M B .
ˆ M B is a n × n matrix, where each element takes the value of (cid:80) t (cid:48) t
=0 ˆ C ( s t , a t ; x ) − ˆ h C ( i.e., the number of cost violations until t (cid:48) th step) if there is a cost entity in o t ( i, j ) , or zero otherwise.
{+Note that at training and test time, we estimate the cumulative cost (cid:80) t (cid:48) t =
+}
ˆ C ( s t , a t ; x ) using ˆ M C and the agent’s current location at time t .
This allows the agent to understand the constraint satisfaction at each step and use this information to accordingly plan for safe trajectories.
Note that ˆ M C is produced by the constraint interpreter, not by the policy network itself.
This allows us to take advantage of the factorization of the policy network and to separately design a constraint mask module that produces an appropriate ˆ M C for different types of constraints.
After applying both constraint mask ˆ M C and cost budget mask ˆ M B to the environment embedding, we then feed the output into CNN to obtain a vector representation.
While the constraint threshold prediction ˆ h C (an important metric to measure safety) is an input to the constraint budget mask ˆ M B , empirically we observe its signal can be weakened by the downstream convolution operation (see Table 2(d)).
To alleviate this issue, we further concatenate ˆ h C with the vector representation outputted from the CNN.
Finally, we pass the concatenated representation to a dense layer to produce [Figure] an action a t .
This policy architecture enables us to encode the language constraint into the policy network and hence learn a constraint-satisfying policy.
4. T RAINING
Initialize a policy π 0 = π ( ·| θ 0 ) and a buffer B
Run π 0 and store trajectories in B
Train Θ 1 and Θ 2 using Eq.
(2) and Eq.
Stage 2 (Policy learning)
Empty B
For k = 0 , 1 , 2 , · · · do
Run π k = π ( ·| θ k ) and store trajectories in B
Obtain θ k +1 using Eq.
(4) and Eq.
Empty B
Training this model with RL would require signiﬁcant amount of data and computation, especially when the constraint interpreter model needs to be learned from scratch.
To improve data efﬁciency, we decompose our training procedure into two stages (Algorithm 1): (1) a pre-training phase for the constraint interpreter, which is followed by (2) a policy optimization phase to learn the rest of the policy network.
Once all the model components are trained, there is no further change to the parameters.
During execution, the agent is set in a random environment and provided a randomly sampled constraint to follow ( e.g., “ Don’t touch grass more than two times ”).
We then measure the ability of the policy to complete the task without violating the constraints.
Importantly, our model architecture does allow for end-to-end differentiability via tricks like Gumbel Softmax (one can treat ˆ h C as a continuous variable).
However, in this version, we do not train end-to-end for computational reasons.
Stage 1 (Interpreter pre-training).
For each type of constraint, we split the collected text into training and test sets using an 80-20 ratio, denoted by D train and D test , respectively.
We then form an ofﬂine pre-training dataset D inter by using a random policy to obtain trajectories over observations o t along with corresponding text constraints x from the training set D train .
For the observations in D inter , we compute the corresponding ground-truth M C and h C values to use as supervision.
Utilizing this dataset we ﬁrst train the constraint mask module in the constraint interpreter by minimizing the following binary cross-entropy loss:
[Equation]
where Θ 1 is the parameter of the constraint mask module, M C ( i, j ; o t , x ) denotes the ground-truth (binary) mask label in i th row and j th column of the n × n environment given o t and x, and ˆ
M C ( i, j ; o t , x ) is the corresponding binary probability prediction of constraint mask model (here we emphasize that M C is speciﬁed by o t and x ).
For the constraint threshold module, we minimize the following mean-square-error loss using the data from D train :
[Equation]
where Θ 2 is the parameter of the constraint threshold module.
Unlike RL, empirically we only need a smaller dataset to train both models in the constraint interpreter, as the models tend to converge relatively quickly.
Stage 2 (Policy learning).
We then use a state-of-the-art safe RL algorithm–projection-based constrained policy optimization (PCPO)
(Yang et al., 2020b)–to train the policy network.
During training, the agent interacts with the environment to obtain rewards and penalty costs from trained ˆ M C for computing J R ( π ) and J C ( π ) .
PCPO is an iterative method that performs two key steps in each iteration:
PCPO Step 1 (Optimize the reward).
The PCPO algorithm performs one step of trust region policy optimization (TRPO (Schulman et al., 2015a)) to maximize the reward advantage function A πR ( s, a ) over a KL-divergence neighborhood of π k :
[Equation]
where d π k is the state visitation frequency induced by the policy π at k th update.
PCPO Step 2 (Project to satisfy the cost constraint).
Next, PCPO projects π k + 12 onto the set of policies satisfying the cost constraint to minimize the distance function D ( e.g., KL divergence):
[Equation]
where γ is the discounted factor and A πC ( s, a ) is the cost advantage function.
Finally, during test time, we evaluate our model in D test .
While our method does require pre-training of the constraint interpreter with ground-truth constraint maps, we envision in future work the dependence on this phase can be reduced by leveraging techniques such as curriculum learning or pre-trained language models like BERT (Devlin et al., 2019).
E XPERIMENTS
Task.
We construct a new task Hazard World for our experimental evaluation (Fig. 1).
In this task, for each episode, the agent starts at a random location within a procedurally generated environment and receives an abstract constraint speciﬁed by natural language, sampled from a pool of all available constraints.
The agent’s objective is to collect all the reward entities and navigate safely ( i.e., avoid cost entities) by adhering to the constraint.
The agent needs to satisfy the constraints during both training and testing, unlike prior work (Misra et al., 2018) that allows the agent to explore the entire state space during training without any constraints.
We implement Hazard World on top of the 2D GridWorld layout of BabyAI (Chevalier-Boisvert et al., 2018a;b).
We randomly place three reward entities on the map: ‘ ball ,’ ‘ box ,’ and ‘ key ,’ with rewards of 1, 2, and 3, respectively.
We also randomly place numerous cost entities on the map: ‘ lava ,’ ‘ water ,’ and ‘ grass ’.
A textual constraint is then imposed upon one of these cost entities ( e.g., “ Touch water fewer than once ”).
Note that to construct the Hazard World task we can generate multiple maps given the same text constraint.
The environment s t is of size 13 × 13, including the surrounding walls, and the agent’s observation o t is of size 7 × 7.
Hazard World contains three types of constraints – (1) Budgetary constraints , which limit on the number of times a set of states can be visited ( i.e., up to h C ), (2) Relational constraints , which deﬁne a minimal distance that must be maintained between the agent and a set of cost entities, and (3) Sequential constraints , which restrict the set of safe states, given that some condition has been met.
For relational and sequential constraints, we set budget h C as zero for simplicity.
[Table]
We collect free-form text in English for the constraints using Amazon Mechanical Turk (AMT).
To generate a sample constraint, we provided workers with a description of
Hazard World, the cost entity to be avoided, and one of three possible additional pieces of information, depending on the constraint type: the cost budget (budgetary), the minimum safe distance (relational), or the other cost entity impacted by past events (sequential).
Workers were then asked to instruct another person to safely navigate Hazard World.
After collecting all responses from AMT, we ﬁltered the data to remove off-topic responses.
For examples of text and AMT worker prompts, we refer the reader to Appendix A.1.
Table 1 provides statistics on the text. [Table]
Train-test split.
For each type of constraint, we split the collected text into training and test sets using an 80-20 ratio, denoted by D and
Stage 1 (Interpreter pre-training).
To form D inter for pre-training, we ﬁrst sample a text constraint x from D train and generate a map.
We then use a random policy to obtain the trajectories with rollout length 200 over observations o t along with corresponding text constraints x.
In total, we repeat this procedure 5,000 times to form D inter ( i.e., each constraint is paired with 6 .
35 maps on average).
We train one constraint interpreter for each type of constraint, which ensures that the constraint interpreter can robustly produce M C and h C .
During testing of the cost constraint interpreter, we use the same procedure and generate 5,000 constraint-map pairs.
Stage 2 (Policy learning).
For training POLCO, we use text constraints from D train paired with randomly generated maps.
Note that for each type of constraints, we train a single network with different values of h C .
During testing, we randomly sample constraints from D test , followed by randomly generating the map.
In total, we obtain 5,000 constraint-map pairs to test the model ( i.e., on average one constraint is paired with 25 .
38 maps).
Please read Appendix A.2 for more details.
Baselines.
The baselines are combinations of the following models and training algorithms: (1) Models.
We consider a baseline model from Walsman et al.
(2018), which does not have the
M C , M B and h C representations.
This model simply takes a concatenation of the observations and text representations as input and produces an action.
We call this model constraint-fusion (CF).
(2) Algorithms.
(a) TRPO (Schulman et al., 2015a).
In TRPO, the agent ignores all constraints and only optimizes the reward.
(b) Fixed-point policy optimization (FPO) (Achiam et al., 2017).
In FPO, the reward objective is combined with the weighted cost objective ( i.e., receive the penalty of 1 if the agent visits the cost entity).
We include it to demonstrate that simply treating the cost penalty as a negative reward will hinder the exploration of the agent.
Finally, we also include a random walk (RW) baseline to serve as a lower bound on the reward and cost performance.
POLCO is a combination of our policy network (PN) and the PCPO algorithm.
Evaluation metrics.
Our primary evaluation metrics are (1) the average value of the reward ( i.e.,
J R ( π ) ), (2) the average cost violations ∆ C :
=
J C ( π ) − h C , and (3) the difference between the reward and the cost violations ∆ RC :
= J R ( π ) − α · max(0 , ∆ C ) , where α represents a tradeoff between reward and cost violation.
We set α to 1 for simplicity, but α may depend on the task.
Note that we want to enforce ∆ C to be non-positive while maximizing J R ( π ) ( i.e., maximize ∆ RC ).
Overall performance.
Table 2(a-c) details the performance of POLCO and the baselines over three types of constraints on the test set.
We observe from Table 2(a) that POLCO achieves the best ∆ RC over different choices of h C , outperforming other baseline methods.
For instance, when h C =
the constraint violation of the policy network trained with TRPO is 3.21 times larger than that of POLCO.
This implies that simply optimizing the reward as in Misra et al.
(2018) cannot guarantee safety.
We also observe that PN trained with TRPO achieves better reward performance compared when with CF and TRPO, which indicates that our proposed policy network is easier to optimize than the CF model.
CF with PCPO also fails to improve the reward since merely combining the observations and the text is not sufﬁcient to learn an effective representation for parsing the [Figure] [Figure]
constraints. Finally, the PN agent trained with FPO has a low reward, because simply treating the cost penalty as the negative reward hinders the agent’s exploration.
Tables 2(b-c) also show that POLCO achieves the best ∆ RC for the more complex relational and sequential constraints.
For the relational case, although the CF agent trained with PCPO satisﬁes the constraints, it has a relatively low reward.
For the sequential constraints, even though the PN agent trained with TRPO obtains a higher reward than POLCO, it also has 1.5 times higher cost violations.
Ablation studies.
We perform ablation studies on the proposed POLCO policy network to examine the importance of each component – M C , M B and h C embedding.
The performance over different h C in terms of the reward and cost violation is shown in Table. 2(d).
To eliminate the prediction error of the constraint interpreter, here we use the ground-truth M C and h C instead.
We observe that the full model achieves the best performance in all cases, averaging 5.12% more reward and 2.22% fewer cost violations.
Without M C , the agent cannot recognize the cost entities.
This causes the agent to incur 66.67% higher ∆ C compared with the full model.
In addition, ∆ C in the full model is almost zero.
This suggests that with the h C embedding and the M B mask the agent is able to better understand the cost satisfaction at every step and hence plan for safer trajectories.
In contrast, without these components the agent fails to understand the text constraints and improve the reward.
Constraint interpreter.
Next, we examine the performance of the constraint interpreter.
The result is illustrated in Fig. 4(a).
Please read the caption for more details.
The rule-based baseline here is similar to a n -gram model, in which we compute the likelihood of the cost entity given the text.
Please read Appendix A.2 for more details.
We observe that our model achieves superior performance in all cases.
We further visualize the predicted M C for the sequential constraints shown in
Fig. 4(b).
We observe that after visiting ‘ grass ’, M C changes from near zero prediction probability to predict cost entities precisely.
This shows that our constraint interpreter model can effectively maintain the long-term dependency of the past states and produce an accurate M C .
Generalization to different environments.
Finally, we examine whether POLCO is robust to covariate shift in the environment distribution ( i.e., a new environment containing only a subset of all cost entities).
We ﬁrst train POLCO and the other baselines in Hazard World and then test them on the commonly-used Lavawall (Chevalier-Boisvert et al., 2018a) task as shown in Fig. 5(a).
We select free-form constraints from Hazard World imposed over the ‘ lava ’ cost entity to be the training set.
Fig. 5(b) shows the reward and cost violation over different h C of different agents in the budgetaryconstrained Lavawall experiment.
We observe that in experiments with a larger h C , generally the agents are less constrained during navigation and yield higher reward and lower cost violations.
Still, in most cases the baseline agents have substantially higher cost violations or obtain minimal reward.
This shows that the POLCO agents generalize better to unseen environments, obtaining higher reward and lower constraint violations.
C ONCLUSION
We addressed the problem of safe reinforcement learning when safety constraints are speciﬁed by natural language.
We proposed the two-stage POLCO algorithm that both learns to understand the abstract language constraints and effectively solves for a safe policy.
To demonstrate the effectiveness of POLCO, we created a new benchmark navigation task via crowd-sourcing which contains a variety of text constraints.
Experiments showed that POLCO achieves superior reward performance, lower constraint violation and better generalizability than other safe RL approaches.
Future directions include (1) applying POLCO to 3D navigation tasks with high-dimensional (image) observations, (2) improving representation learning to handle more complex language constraints, and (3) applying POLCO to large-scale safety-critical problems ( e g., robotics).
A A PPENDIX
Outline.
The appendix is outlined as follows.
Section A.1 details the dataset and the procedure of collecting the dataset.
Section A.2 describes the parameters of the constraint interpreter and the policy network and the PCPO training details.
Section A.3 provides the learning curves of training the policy network.
Finally, we ﬁll the Machine Learning Reproducibility Checklist in Section A.5. [Table]
The water should only be stepped [Table] max [Table]
hurts a lot, but you have special shoes that you can use to walk on it, but only up to 5 times, remember!
[Table]
Four
[Table]
most number of
[Table]
touch
[Table]
You
[Table]
all.
You
[Table]
die otherwise.
[Table]
Water
[Table]
hurt you if
[Table]
less
[Table]
[Table]
Any block within one
[Table]
of a
[Table]
cannot be touched.
The explosion radius of grass is three, so stay at least that distance away from grass.
Waters are dangerous, so do not cross them.
[Table]
Make sure you
[Table]
walk on water
[Table]
walking on grass.
[Table]
touch the water or water will become risky.
You
[Table]
touch the water ﬁrst
[Table]
but the lava is dangerous so do not touch it after.
Avoid lava since you can only walk on it once.
After that the lava will hurt you.
Water will trigger grass to become dangerous.
[Table]
Just avoid the perimeter when collecting the objects, and you’ll be safe.
Your directions are as follows: if you’re facing a block with a water block in front of it, walk ﬁve blocks ahead ...(
[Table]
words
[Table]
A. D ATASET
At a high level, Hazard World applies the instruction following paradigm to safe reinforcement learning.
Concretely, this means that safety constraints in our environment are speciﬁed via language.
Our dataset is thus comprised of two components: the environment, made up of the objects that the agent interacts with, and the constraint, which imposes a restriction on which environmental states can be visited.
The environment is procedurally generated.
For each episode, Hazard World places the agent at a randomized start location and ﬁlls the environment with objects.
Hazard World then randomly samples one constraint out of all possible constraints and assigns this constraint to the environment.
We collected natural language constraints in a two-step process.
In the ﬁrst step, or the data generation step, we prompted workers on Amazon Mechanical Turk with scenarios shown in Fig. 6.
Workers are provided the minimum necessary information to deﬁne the constraint and asked to describe the situation to another person.
For example, to generate a so-called budgetary constraint, workers are given the cost entity to avoid ( ‘lava’, ‘grass’ or ‘water’ ) and the budget ( i.e., h C , a number 0 through 5).
The workers use this information to write an instruction for another person.
This allows us to ensure that the texts we collected are free-form.
These generations form our language constraints.
In the second step, or the data validation step, we employed an undergraduate student to remove invalid constraints.
We deﬁne a constraint as invalid if (a) the constraint is off-topic or (b) the constraint does not clearly describe states that should be avoided.
Examples of valid and invalid constraints are included in Table 3.
Finally, we randomly split the dataset into 80% training and 20% test sets.
In Hazard World and Lawawall, the agent has 4 actions in total:
a ∈ A = { right , left , up , down } .
The transition dynamics T is deterministic.
[Figure]
A. A RCHITECTURES , P ARAMETERS , AND T RAINING D ETAILS
Policy network in POLCO.
The architecture of the policy network is shown in Fig.
The environment embedding for the observation o t is of the size 7 × 7 × 3.
This embedding is further concatenated with the cost constraint mask M C and the cost budget mask M B .
This forms the input with the size 7 × 7 × 5.
We then use convolutions, followed by dense layers to get a vector with the size 5.
This vector is further concatenated with the h C embedding.
Finally, we use dense layers to the categorical distribution four classes (
.e.
right
Constraint interpreter in POLCO.
The architecture of the constraint interpreter is shown in Fig.
For the constraint mask module, the input is the text with w words.
We then use an embedding network, followed by an LSTM to obtain the text embedding with the size 5.
The text embedding is duplicated to get a tensor with the size 7 × 7 × 5.
This tensor is concatenated with the observation of size 7 × 7 × 3 , creating a tensor with the size 7 × 7 × 8.
In addition, we use a convolution, followed by dense layers and a reshaping to get the cost constraint mask M C .
Next, we use a heuristic to compute ˆ C tot := (cid:80) t (cid:48) t =0 C ( s t , a t ; x ) from M C .
At execution time, we give our constraint interpreter access to the agent’s actions.
We initialize ˆ C tot = 0 .
Per timestep,
[Table] our agent either turns or moves forward. If the agent moves forward and the square in front of the agent contains a cost entity according to M C , we increment ˆ C tot . For the constraint threshold module, we use the same architecture to get the text embedding.
We then use dense layers to predict the value of h C .
Rule-based baseline.
The rule-based baseline is similar to a n -gram model, in which we compute the likelihood of the cost entity given the text.
Speciﬁcally, in the training set we count the number of word-cost entity pairs.
For example, for each map, given the language constraints “ The water should only be stepped on a max of 5 times ” and the ground-truth cost entity ( e.g., ‘ water ’), the pair ‘ the ’-‘ water ’ is added one and so forth.
During testing, for a given text and map, we compute the following probability for each possible cost entity:
[Equation]
[Figure]
[Equation]
Finally, we select the one with the maximum value to predict the cost entity.
Details of the algorithm–PCPO.
We use a KL divergence projection in PCPO to project the policy onto the cost constraint set since it has a better performance than L 2 norm projection.
We use GAEλ approach (Schulman et al., 2015b) to estimate A πR ( s, a ) and A πC ( s, a ) .
We use neural network baselines with the same architecture and activation functions as the policy networks.
The hyperparameters of training POLCO are in Table 4.
We conduct the experiments on the machine with
Intel Core i7-4770HQ CPU.
The experiments are implemented in rllab (Duan et al., 2016), a tool for developing RL algorithms.
Baseline model–Constraint Fusion (CF).
Our baseline is adapted from Walsman et al.
The model is illustrated in Fig.
An LSTM takes the text x as an input and produces a vector representation.
The CNN takes the environment embedding of o t as an input and produces a vector representation.
These two vector representations are concatenated, followed by a MLP to produce an action a t .
We do not consider other baselines in Janner et al.
(2018) and Misra et al.
This is because that their models are designed to learn a multi-modal representation ( e.g., processing a 3D vision) and follow goal instructions.
In contrast, our work focuses on learning a constraint-satisfying policy.
The parameters of the baseline is shown in Fig.
We use the same CNN parameters as in our policy network to process o t .
Then, we use the same LSTM parameters as in our constraint mask module to get a vector representation with size 5.
Note that we use almost the same number of the parameters to ensure that POLCO does not have an advantage over CF.
Finally, we use dense layers to the categorical distribution with four classes.
We then sample an action from this distribution.
A. A DDITIONAL E XPERIMENTS Learning curves of training the policy network.
The learning curves of the undiscounted constraint cost, the discounted reward, and the number of steps over policy updates are shown for all tested algorithms and the constrains in Fig.
Overall, we observe that
(1) POLCO improves the reward performance while satisfying the cost constraints during training in all cases, (2) the policy network trained with TRPO has substantial cost constraint violations during training, (3) the policy network trained with FPO is overly restricted, hindering the reward improvement.
[Figure]
In contrast, the network trained with TRPO suffers from violating constraints and trained with FPO cannot effectively improve
A. POLCO FOR P IXEL O BSERVATIONS /3D E GO - CENTRIC O BSERVATIONS
To deal with pixel observations o t , we can still use the proposed architecture to process o t as shown in Fig. To predict the cost constraint mask ˆ M C , we use the object segmentation method to get the bounding box of each object in the scene. As a result, the area of that bounding box will be one if there is a cost entity ( i.e., the forbidden states mentioned in the text).
Otherwise, the bounding box contains a zero.
For ˆ M B , we can use a similar approach to compute the cumulative cost violations at each step.
In addition, to deal with navigation environments with 3D ego-centric observations, we propose shifting the o t , ˆ M C and ˆ M B matrices to be the ﬁrst-person view.
The bounding box approach for image case can still be applied here. We leave this proposal to future work. A. T HE M ACHINE L EARNING R EPRODUCIBILITY C HECKLIST (V ERSION 1.2, M AR .
For all models and algorithms presented, indicate if you include 2 :
• A clear description of the mathematical setting, algorithm, and/or model: – Yes , please see the problem formulation in Section 3, the algorithm and the model in Section 4.
• An analysis of the complexity (time, space, sample size) of any algorithm:
– Yes , we provide discussions in Section 4 and Appendix A.2 for showing a training infrastructure.
• A link to a downloadable source code, with speciﬁcation of all dependencies, including external libraries: – Yes , please see our abstract.
For any theoretical claim, check if you include:
• A statement of the result: – Not applicable .
A clear explanation of any assumptions: – Not applicable .
• A complete proof of the claim: – Not applicable .
For all ﬁgures and tables that present empirical results, indicate if you include:
• A complete description of the data collection process, including sample size: – Yes , please see Appendix A.1 and Table 1 for sample size.
• A link to a downloadable version of the dataset or simulation environment: – Yes , please see our abstract.
• An explanation of any data that were excluded, description of any pre-processing step: – Yes , please see Appendix A.1.
• An explanation of how samples were allocated for training / validation / testing: – Yes , please see section 5 and
Appendix A. •
The range of hyper-parameters considered, method to select the best hyper-parameter conﬁguration, and speciﬁcation of all hyper-parameters used to generate results: – Yes , please see Appendix A.2.
• The exact number of evaluation runs: – Yes , please see section 5.
• A description of how experiments were run: – Yes , please see section 5.
• A clear deﬁnition of the speciﬁc measure or statistics used to report results: – Yes , please see Section 5.
• Clearly deﬁned error bars: – Not applicable.
• A description of results with central tendency ( e.g., mean) variation ( e.g., stddev): – Not applicable.
• A description of the computing infrastructure used: – Yes , please see Appendix A.2.