SCERL: A Benchmark for intersecting language and Safe Reinforcement Learning
Abstract
The issue of safety and robustness is a critical focus for AI research.
Two linesof research are so far distinct, namely (i) safe reinforcement learning , where anagent needs to interact with the world under safety constraints, and (ii) textualreinforcement learning , where agents need to perform robust reasoning and modeling of the state of the environment by interacting with it using text (promptsand commands).
In this paper, we propose Safety-Constrained Environments for
Reinforcement Learning (SCERL), a benchmark to bridge the gap between thesetwo research directions.
The contribution of this benchmark is safety-relevantenvironments with i) a sample set of 20 games built on new logical rules to represent physical safety issues; ii) added monitoring of safety violations and iii) amechanism to further generate a more diverse set of games with safety constraintsand their corresponding metrics of safety types and difﬁculties.
This paper showsselected baseline results on the benchmark.
SCERL benchmark and its ﬂexibleframework aims at providing a set of tasks to demonstrate language-based safetychallenges to inspire the research community to further explore safety applicationsin a text-based domain.
Introduction
Safety has emerged as an important issue for machine learning applications in real-life, with multipleframeworks to categorise the types of safety [Garcıa and Fernández, 2015].
We present a newbenchmark called Safety Constrained Environments for Reinforcement Learning (SCERL) forsafe RL tasks with natural language, as depicted in Figure 1.
SCERL is a sandbox environment thatdirectly measures the physical safety aspect of the agent learning process with contributions are asfollows:
• Text-based safety constraints and goals
• A sample set of games from easy to difﬁcult with different safety goals and constraints• Automatic generation of games with unsafe items and potential goals
• Monitoring of the agent performance and safety events  Related Work
Real-life decision-making problems are associated with natural language; thus, the intersectionbetween RL and natural language has attracted the attention of the research community [Luketinaet al., 2019, Osborne et al., 2021]. Although there are multiple safe RL and text-based RL benchmark,there has not been an integrated benchmark combining physical safety issues together with naturallanguage interactions [Yang et al., 2021, Mahmood et al., 2018, Brunke et al., 2021]. There is aneed to incorporate safety constraint types into a text-based RL benchmark that can drive furtherdevelopment of language and safe Reinforcement Learning.
SCERL: a safety-focused framework and benchmark for text-basedReinforcement Learning3.1 Our safety gameset
SCERL has been developed from the core of TextWorld [Côté et al., 2018] by generating set of gamesrepresenting safety constraints for language-instructed agents.
We have introduced a schema forsafety annotation which includes constraints, goals and additional scripts to generate safety games. There is a monitoring script which gives information on safety violation and yields different levelsof language-assisted warnings.
The safety conditions are sourced from real life examples of safetyconstraints such as reports of incidents and summary reports of hazards.
These unsafe conditions wereincluded in the logic of the game to create safety constraints.
For example, we introduce conditionsrelating to ﬁre risks and chemical risks:
• Electric or hot item : ﬁre hazard if not turned off or being attended by the agent.
• Chemical items : dangerous if not kept in a locked cabinet or a designated area.
• Other mechanical risks : such as open drawers can pose risks of harming the agent.
3.2 New schema for safety annotation
There is a variety of goal and constraint speciﬁcations to provide different challenges for an RL agentto learn from a range of tasks and safety constraints.
In this benchmark, users can introduce safetyrestrictions under two forms: soft penalties and terminating penalties.
Additionally, the user canspecify the goals of the games, the goal of which may or may not directly involve unsafe items.
Example Baselines and Additional features for Language-assisted warningand safety penalty monitoring4.1 Game design
The games are designed to include constraints that make the agent refrain from taking certain actionswhich may change the state of an object to an unsafe one.
For example, keeping the fridge open orleaving ﬁre risk objects like candles and the induction cook-top unattended.
The difﬁculty level (easy,medium and hard) of these games is decided from the number and complexity of the constraints,objects and rooms involved.
Our categorisation of difﬁculty follows the room and object valuesused in [Murugesan et al., 2021]
[Côté et al., 2018]; however the games can be generated with upto 8 rooms, 600 objects, and 100 unsafe objects (with one unsafe object having one to multiplesafety constraints).
For testing the agents, a subset of games were used from the baseline whichhad objectives like avoid eating rotten egg , where the agent is penalised if it uses the rotten egg butrewarded if it cooks and eats the big and small eggs.
It is also rewarded for putting the rotten egg inthe trashcan (hard game).
The challenge for the agent is to determine the safety relating to objects ofthe same type.
Second example, is regular eating egg game where the objective is to cook and eat anegg while avoiding the unsafe states of the stove being turned on and the fridge left open.
Anotherexample is the packing lunchbox game where the objective is to pack the cooked egg in a lunchbox.
4.2 Example Baselines
To test the current baselines of the benchmark, we have selected two state-of-the-art agents[Narasimhan et al., 2015, Ammanabrolu and Hausknecht, 2020, Murugesan et al., 2021].
Thespeciﬁc hyperparameters and computing resources are speciﬁed in the Supplementary.
• Text-based agent (Simple agent) :
LSTM-A2C from [Narasimhan et al., 2015] whichchooses actions based on the observed text. • Knowledge-aware and commonsense agent: KG-2AC [Ammanabrolu and Hausknecht ,2020] which encodes the state of the world as a knowledge graph from the game observations.
We leverage the Numberbatch embedding based on ConceptNet following the setup of Murugesan et al.
Overall the agents violated safety constraints at the beginning of the training but learnt to reduce therisks.
However, their performance has a high variation and the number of episodes it takes for theagents to converge (Figure 2) is well beyond the range of 80-100 episodes (of 50 steps per episode)reported in [Murugesan et al., 2021].
[Figure]
benchmark’s
The mode of observations and warning appear to inﬂuence agent learning.
For safe packing lunchbox,both agents improve the mean score and reduce the standard deviation of return with more information;however the standard deviation remains large - which suggests that the improvement is not consistent.
Table 1 shows example results on a subset of games.
The results show that both agents performssub-optimally, far from the 100 score if performed optimally, across the different observation modes.
This gives further scope for developing new language-assisted safe Reinforcement Learning agentsthat can tackle these challenges more effectively.
[Table]
4.4 Monitoring safety with the benchmark
The benchmark also has a mechanism of monitoring the frequency of constraint violation (by lookingat actions taken and consequent object states) which gives an insight into the training process of theagent.
Figure 3 shows two of the example game-sets reﬂecting the avoid eating rotten egg, whichcan have a max score of 30 and regular eating egg challenge.
The training progress showed that theagent learnt to achieve the eating-egg goal while reducing both turning on the stove and leaving thefridge open with every action contributing the following average scores per episode - turn on stove:-1.50, open fridge: -1.58 and eat egg: 4.60.
In the rotten egg game, the agent ended up developing apolicy of collecting rewards from putting the rotten egg in the trashcan rather than cooking the eggs.
[Figure]
Conclusion
In this benchmark we have presented a dataset of games and a ﬂexible framework to bridge the gapbetween the two research areas of safe reinforcement learning and textual reinforcement learning.
SCERL is a ﬂexible framework to provide a set of tasks to demonstrate physical safety challenges forreinforcement learning agents and aims to help the research community explore safety applicationsin a text-based domain.
Currently the work is limited to the domestic setting and can be expandedto further context such as factory or commercial applications.
Furthermore, the underlying logicand rule sets can be further expanded to incorporate a more extensive range of safety constraints.
The benchmark provides a ﬂexible architect to introduce further features, and direction for futuredevelopment can include further autogeneration and other types of safety aligned to human risk-basedconstraints, such as commonsense-based moral and physical safety.
Baseline Algorithmic and Hyperparameters
In the paper we included two agents as described in Murugesan et al.
[2021]:• Text-based agent (Simple agent):
LSTM-A2C from Narasimhan et al.
[2015] which chooses actionsbased on the observed text.• Knowledge-aware and commonsense agent: KG 2AC Ammanabrolu and Hausknecht [2020] whichencodes the state of the world as a knowledge graph from the game observations.
We leverage the
Numberbatch embedding based on ConceptNet following the setup of Murugesan et al.
The Hyperparameters used in the experiments are described in Table
[Table]
Data documentation and intended uses
The data’s intended uses are toward practical examples of safety problems that can beneﬁt the Reinforcement 153 Learning community.
3.1 Benchmark workﬂows
This subsection outlines the workﬂow of creating a new batch of games.
Figure 2 shows the process includes 156 these components:
In this benchmark, users can introduce safety restrictions under two forms: soft penalties and terminating 158 penalties.
Additionally, the user can specify the goals of the games, the goal of which may or may not directlyinvolve unsafe items.
To deﬁne the safety constraints relating to an object in the game, the user can deﬁne unsafe 160 conditions relating to location, the object properties and actions on the object as follows and as described in
These contributions were associated with new engineering features as follows:
• A new feature in the game generation function to automatically source safety constraints from a json
ﬁle including both soft or hard constraints
• new logical predicates/properties and actions added to the game logic ﬁles such as "turn on", "turn
off", "stained", "broken" and "unattended"
• new logical rules to link the newly added actions/properties
The objective of the game set is to present a set of challenges to the agent which needs an awareness about safety 169 in order to be solved.
SCERL game set is a set of 50 games which include various environments with differentsafety constraints.
The objective in all these games is to navigate through an environment (E) with minimum 171 safety constraint (C) violations to ﬁnally accomplish a goal (G).
Safety constraints (C): these are conditions in a game which when met will result in a penalty or warning being 173 issued by the environment.
For example: Leaving the washing machine open in an environment where theobjective is to wash dirty laundry will result into a penalty.
Goal (G): it refers to the ﬁnal task which the agent needs to perform to win the game.
For example: Cooking an 176 omelette.
The games in the SCERL game set were created to offer a big range of safety related challenges whichapply to a vast variety of objects.
Game generation process: Textworld was modiﬁed to generate the 178 safety-aware games in SCERL.
The modiﬁcation included two major steps:
Introducing new entity types which don’t exist in
[Table]
The safety conditions can be deﬁned directly in the gameset, similar to a quest (a state-action pair with a 189 penalty/reward) creation in the original TextWorld package.
In this benchmark, we provide an additionalmechanism to provide safety constraints as described in Table
[Table] [Table]
The
[Table]
of such games is to achieve the safety goal and to place the ob- jects present in the game in their right position while ensuring that none of the safety constraints are vio- lated.
These games are more difﬁcult because of the inclusion of the safety goal which usually involves the agent performing an ac- tion that leads to a change in the state of the desired object.
For example, the egg becoming cooked from raw.
Increased number of rooms, unsafe objects and constraints also add to the difﬁculty.
4.1 Example of Interacting with the Game
This section provides some example of how to interact with the environment.
The environment can be used as a 195 gym environment or in the original TextWorld setup.
In order to gain extra information about penalty scoresfrom unsafe conditions, the user can wrap the environment in our custom-built Monitor Wrapper as 197 demonstrated below.
All of the other commands with the environment remains the same.
The code below shows how a sequence of winning command play out in the game.
The printout of the state and 200 scores come directly from the Monitor Wrapper.
The code below shows the case when the agent violates one of the safety constraints.
This results in a negative 202 score.
In this case, since it does not have any concurrent rewards (such as by residual rewards from an ongoingwinning event), the overall score reﬂects the safety penalty.