Original: WordCraft: An Environment for Benchmarking Commonsense Agents
Revised: WordCraft: An Environment for Benchmarking Commonsense Agents

Original: Abstract
Revised: Abstract

Original: The ability to quickly solve a wide range of realworld tasks requires a commonsense understanding of the world.
Revised: The ability to quickly solve a wide range of realworld tasks requires a commonsense understanding of the world.

Original: Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge.
Revised: Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge.

Original: This is partly due to the lack of lightweight simulation environments that sufﬁciently reﬂect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.
Revised: This is partly due to the lack of lightweight simulation environments that sufﬁciently reﬂect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.

Original: To  enable research onbenchmarking agents with commonsense knowledge, we propose WordCraft, an RL environment based on LittleAlchemy2.
Revised: To better enable research on agents making use of commonsense knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2.

Original: This  environment issmall and fast to run, but built upon entities and relations inspired by realworld semantics.
Revised: This lightweight environment is fast to run and built upon entities and relations inspired by realworld semantics.

Original: We evaluate several representation learning methods on this benchmarks and propose a new method for integrating knowledge graphs within an RL agent.
Revised: We evaluate several representation learning methods on this new benchmark and propose a new method for integrating knowledge graphs with an RL agent.

Original: 1. Introduction
Revised: 1. Introduction

Original: Recent progress in Reinforcement Learning (RL) has been impressive (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019), but it has mostly been fueled by tabula-rasa learning.
Revised: Recent progress in Reinforcement Learning (RL), while impressive (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019) has mostly focused on tabula-rasa learning, rather than exploitation of prior world knowledge.

Original: For example, while Go is an extremely difﬁcult game to master, its rules and thus environment dynamics are simple, not heavily relying on knowledge about concepts that can be encountered in the real world.
Revised: For example, while Go is an extremely difﬁcult game to master, its rules and thus environment dynamics are simple and not heavily reliant on knowledge about concepts that can be encountered in the real world.

Original: Focusing on tabula-rasa learning conﬁnes state-of-the-art approaches to simulation environments that can be solved without requiring transfer of commonsense, world or domain knowledge.
Revised: Focusing on tabula-rasa learning conﬁnes state-of-the-art approaches to simulation environments that can be solved without the transfer of commonsense, world, or domain knowledge.

Original: Many real-world applications ( e g. personal assistants and household robots) require agents that can learn fast and generalise well to novel situations, something that is likely not possiblewithout commonsense and general knowledge to
Revised: Many real-world applications ( e g. personal assistants and household robots) require agents that can learn fast and generalise well to novel situations, which is likely not possible to

Original: Learning
Revised: Reinforcement

Original: ICML 2020, Copyright author

Original: [Table]
Revised: [Table]

Original:  about the world.
Revised: without the ability to reason with commonsense and general knowledge about the world.

Original: Consider, for example, an agent tasked with solving common household problems who has never seen a dirty ashtray.
Revised: Consider, for example, an agent tasked with performing common household chores that has never seen a dirty ashtray.

Original: If presented with this new object, the agent needs to know that a reasonable set of actions  include cleaning the ashtray, but not feeding it to the cat.
Revised: When presented with this new object, the agent needs to know that a reasonable set of actions involving this object include cleaning the ashtray, but not feeding it to the cat.

Original: Humans communicate much of such knowledge through written language, for example,for example Wikipedia encodes such knowledgeexplicitly or implicitly through corpus statistics.
Revised: Humans encode a large amount of commonsense knowledge in written language. For example, Wikipedia encodes such knowledge implicitly through corpus statistics, and at times explicitly in writing.

Original: Outside of RL, learning to represent and utilise such knowledge has improved considerably over the last years.
Revised: Outside of RL, learning to represent and utilise prior knowledge has improved considerably over recent years.

Original: For example, a common approach is to pretrain neural language models and ﬁne-tuning them on downstream tasks (Devlin et al., 2019; Raffel et al., 2019).
Revised: For example, a common approach is to pre-train neural language models and ﬁne-tune them on downstream tasks (Devlin et al., 2019; Raffel et al., 2019).

Original: Recent works highlight that such pre-trained models capture several aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019; Hewitt & Manning, 2019; Reif et al., 2019).
Revised: Recent works highlight that such pre-trained models capture many aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019; Hewitt & Manning, 2019; Reif et al., 2019).

Original: Another  approach is to represent commonsense knowledge explicitly in Knowledge Graphs (KGs),such as  ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017).
Revised: Another popular approach is to represent commonsense knowledge explicitly as Knowledge Graphs (KGs), as done by ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017).

Original: We refer to Nickel et al.
Revised: Such KGs have been used in commonsense reasoning tasks (Lin et al., 2019).

Original: for an overview of knowledge base construction methods. However, the question  how to best utilisegeneral or commonsense knowledge (either in texts or KGs) for downstream RL environments has not received the same level of attention, partly due  scarce simulation environments that directly beneﬁt from transferring such  knowledge.
Revised: However, the question of how to best utilise commonsense knowledge (provided in language corpora or KGs) for downstream RL tasks has not received the same level of attention, partly due to scarce simulation environments that directly beneﬁt from transferring such prior knowledge.

Original: Robotics environments such as Mujoco require knowledgethat is not  expressed in natural language such as  intuitive understanding of physics (Todorov et al., 2012; Yu et al., 2019).
Revised: Robotics environments such as MuJoCo require knowledge not readily expressed in natural language, such as an intuitive understanding of physics (Todorov et al., 2012; Yu et al., 2019).

Original: Environments like Go or Chess (Silver et al., 2017; 2018), while very hard to master, are based on simple transition dynamics and largely disconnected from entities and concepts that one would encounter in the real world.
Revised: Rule-based game environments such as Go or Chess (Silver et al., 2017; 2018) , while very hard to master, are based on simple transition dynamics and largely disconnected from entities and concepts encountered in the real world.

Original: On the other hand,  environments such as  (Vinyals et al., 2019; Berner et al., 2019) that contain many complex interactions with  large number of entities (many of which have real-world analogues), commonsense knowledge could be helpful.
Revised: On the other hand, commonsense knowledge could be helpful in environments such as StarCraft II (Vinyals et al., 2019; Berner et al., 2019), which contain many complex interactions with a large number of entities (many of which have real-world analogues).

Original: However, running such environment is very costly and learning to ground knowledge from external sources is very challenging as natural language annotations of states and actions  are not readily available.
Revised: However, running such environments is very costly, and learning to ground knowledge from external sources remains challenging, as natural language annotations of states and actions in such games are not readily available.

Original: To study agents with commonsense knowledge, we present
Revised: To study agents with commonsense knowledge, we present

Original: WordCraft, a fast RL environment based on the popular game Little Alchemy 2 .
Revised: WordCraft, a fast RL environment based on the popular game Little Alchemy 2.

Original: In this game, the player is tasked to craft over 700 different entities by combining already discovered entities.
Revised: In this game, the player is tasked with crafting over 700 different entities by combining previously discovered entities.

Original: For example, combining “water” and “earth” createsthe new entity “mud”.
Revised: For example, combining “water” and “earth” creates “mud”.

Original: Learning to play the game in a way that generalizes to unseen concepts requires commonsenseand general knowledge about the world, yet the environment is quick to run which allows us to train agents for many steps (around 8,000 steps per second).
Revised: Learning policies that generalize to unseen entities and combinations requires commonsense knowledge about the world. The environment runs quickly (around 8,000 steps per second on a single machine), enabling fast experimentation cycles.

Original: As the entities in the game are described by words, they can be easily grounded withthe external knowledge sources.
Revised: As the entities in the game correspond to words, they can be easily grounded with external knowledge sources.

Original: In addition, we introduce a method for conditioning agents trained in WordCraft onknowledge graphs, which can be utilized for incorporating commonsense KGs  such as ConceptNet.
Revised: In addition, we introduce a method for conditioning agents trained in WordCraft on KGs encoding prior knowledge, such as ConceptNet and other commonsense KGs.

Original: 2. Background
Revised: 2. Background

Original: We employ a standard formulation of the RL problem: the agent-environment interface is framed as a Markov Decision Problem (MDP) (Sutton & Barto, 2018) deﬁned by the tuple ( S, A, T, R, γ ) , where s ∈ S is a state, 
Revised: We formulate the RL problem as a Markov Decision Process (MDP) (Sutton & Barto, 2018) deﬁned by the tuple ( S, A, T, R, γ ) , where s ∈ S is a state, a ∈ A is an action,

Original: a ∈ A is an action, T ( s, a ) → s (cid:48) a transition function, R ( s, a ) → R a reward function, and γ a discount factor.
Revised: T ( s, a ) → s (cid:48) is the transition function, R ( s, a ) → R is the reward function, and γ , the discount factor.

Original: The goal is to ﬁnd a policy π that maximises the expected cumulative return (cid:80) ∞ k =

Revised: The goal is to ﬁnd a policy π that maximises the expected sum of future discounted rewards: (cid:80) ∞ k =

Original: γ k r k +1 .
Revised: γ k r k +1 .

Original: We represent knowledge about compounds and ingredients as a Knowledge Graph, i.e. a collection of (cid:104) s, p, o (cid:105) triples, each encoding a relationship of type p (predicate) between the subject s and the object o of the triple.
Revised: We represent knowledge about entities and the relations among them as a knowledge graph (KG): a collection of (cid:104) s, p, o (cid:105) triples, each encoding a relationship of type p (predicate) between the subject s and the object o of the triple.

Original: In order tobe able to work with incomplete knowledge graph and infer missing relationships, we use the ComplEx  model (Trouillon et al., 2016)with the loss function and regulariser proposed by Lacroix et al.
Revised: In order to work with incomplete knowledge graphs and infer missing relationships, we use the ComplEx link prediction model (Trouillon et al., 2016)

Original: (see Appendix A for details).
Revised: (see Appendix A for details).

Original: 3. WordCraft
Revised: 3. WordCraft

Original: To benchmark commonsense knowledge of RL agents, we present WordCraft  based on Little Alchemy 2 1 which is a simple associative game where, starting from a set of four basic ingredients, the player must create as many new items as possible.
Revised: In order to benchmark the ability of RL agents to make use of commonsense knowledge, we present WordCraft , based on Little Alchemy 2 1 . Little Alchemy 2 is a simple association game: Starting from a set of four basic items, the player must create as many different items as possible.

Original: Each new item can be created by combiningan existing two  items.
Revised: Each non-starter item can be created by combining two other items.

Original: For example, combining “moon” and “butterﬂy” yields “moth”, whereas combining “human” and “medusa” yields “statue”.
Revised: For example, combining “moon” and “butterﬂy” yields “moth”, and combining “human” and “medusa” yields “statue”.

Original: There arein total 700  items (including rare and compound words) and 3277 permissible item combinations (referred to as valid recipes throughout this paper)
Revised: There are 700 total items (including rare and compound words) and 3,417 permissible item combinations (referred to as valid recipes throughout this paper)

Original: (IGN, 2020).
Revised: (IGN, 2020).

Original: Solving the game without trying every possible combination of ingredients requires general knowledge of the world and the relationships between  concepts.
Revised: Efﬁciently solving this game without trying every possible combination of items requires using knowledge about relations between common concepts.

Original: WordCraft is based on entities and valid recipes from Little Alchemy 2, but otherwise signiﬁcantly simpliﬁes the problem. First,there is no graphical interface, the agent operates by selecting entities in a table.
Revised: WordCraft is a simpliﬁed version of Little Alchemy 2, sharing the same entities and valid recipes: First, the interface is text-based rather than graphical.

Original: Second, instead of one long open-ended task, WordCraft contains a large number of smaller tasks.
Revised: Second, instead of a single open-ended task, WordCraft consists of a large number of simpler tasks.

Original: Each tasks is created by randomly sampling a goal entity, entities that can be combined towards this goal and distractors entities.
Revised: Each task is created by randomly sampling a goal entity, valid constituent entities, and distractor entities.

Original: The agent must choose which entities to combine in order to create the goal entity.
Revised: The agent must choose which entities to combine in order to create the goal entity.

Original: The task difﬁculty can be varied by increasing the number of distractor items or increasing the number of intermediary entities that need to be created in order to reach the goal entity (we refer to this variant as depth- n , with n − 1 intermediate entities).
Revised: The task difﬁculty can be adjusted by increasing the number of distractors or increasing the number of intermediate entities that must be created in order to reach the goal entity (we refer to this variant as depth- n , with n − 1 intermediate entities).

Original: The training and testing split is created by selecting a set of test entities, goals or valid recipes which are not seen or required to complete the task during training.
Revised: The training and testing split is created by selecting a set of either goal entities or valid recipes that are not used during training.

Original: A relatively large number of entities and valid recipes enables the study of generalization to unseen goal entities.
Revised: The relatively large number of entities and valid recipes enables the study of generalization to unseen goals.

Original: The environment is deterministic and fully-observable.
Revised: The environment is deterministic and fully-observable.

Original: At each  step, the state s consists of the goal entity x g , a set of k currently available entities x table = { x i } ki =0  and the currently selected entity x s, 1 .
Revised: At each time step, the state s consists of the goal entity x g , a set of k currently selectable entities x table = { x i } ki =0 , and the currently selected entity x s, 1 .

Original: The action a corresponds to choosing one of the available entities as the second selected entity x s, 2 ∈ x table .
Revised: The action a corresponds to choosing one of the available entities as the second selected entity x s, 2 ∈ x table .

Original: When no entity is currently selected, x s, 1 is set to None.
Revised: When no entity is currently selected, x s, 1 is set to a placeholder empty entity.

Original: Ifselected ( x s, 1 , x s, 2 ) corresponds toone of the valid recipes, a new entity is added to the table x table and x s, 1 is set to None.
Revised: If ( x s, 1 , x s, 2 ) corresponds to a valid recipe, the corresponding resulting entity is added to the table x table and x s, 1 is set to the empty entity.

Original: The episode terminates when the newly created entity is x g or after a maximum number of steps is reached.
Revised: The episode terminates when the newly created entity is x g or after a maximum number of steps is reached.

Original: The maximum number of stepsis kept low to discourage brute-force solutions (for depth- 1 , trying out all possible pairs is O ( k 2 ) ).
Revised: We keep the maximum number of steps low to discourage bruteforce solutions ( O ( k 2 ) for depth- 1 tasks).

Original: The reward can be sparse (with r = 1 or 0 received only at the end of the episode) or shaped (with a penalty for choosing a wrong pair of items  and  reward for creating intermediary ingredients).
Revised: The reward can be sparse (with r = 1 or 0 received only at the end of the episode) or shaped (with a penalty for choosing a wrong pair of items or creating irrelevant entities and a reward for creating intermediate ingredients).

Original: Compared to other environments that combine language and RL (Janner et al., 2018; Chevalier-Boisvert et al., 2019; Cˆot´e et al., 2018; Das et al., 2018), WordCraft contains a much larger number of objects, with the relationships between the objects reﬂecting the structure of the real world.
Revised: Compared to other environments that combine language and RL (Janner et al., 2018; Chevalier-Boisvert et al., 2019; Cˆot´e et al., 2018; Das et al., 2018), WordCraft contains a much larger number of objects, with object relationships that reﬂect the structure of the real world.

Original: For example, the underlying knowledge graph of TextWorld (which is perhaps the closest comparison), containsabout 99 different objects and 10 types of relations (Zelinka et al., 2019).
Revised: For example, the underlying KG of TextWorld (which is perhaps the closest comparison), contains 99 different objects and 10 types of relations (Zelinka et al., 2019).

Original: Furthermore, WordCraft is lightweight and fast to train.
Revised: Furthermore, the environment does not contain movement-related actions.

Original: It allows  us to focus on incorporating commonsense knowledge without requiring  the agentto learn to ground states to text or dealing with a combinatorially large action space (e.g. as in text games).Furthermore, the environment does not contain movement actions and the observation space is small, yet generalizing to novel entities is challenging.
Revised: The state space is limited to combinations of entity words, and the action space, to table positions, yet generalization remains challenging. This allows allowing us to study commonsense reasoning in isolation, without requiring that the agent learn to ground states to text or deal with a combinatorially large action space (as seen in text games).

Original: 4. Guiding Agents with a Knowledge Graph
Revised: 4. Guiding Agents with a Knowledge Graph

Original: We propose an agent architecture that makes use of information from an external knowledge graph to guide the agent’s policy.
Revised: We propose an agent architecture that makes use of information from an external KG to guide the agent’s policy.

Original: At a high level, the model consists of a self-attentionbased actor-critic network and an external knowledge-graph link prediction model.
Revised: At a high level, the model consists of a self-attention-based actor-critic network and an external KG link prediction model.

Original: Given that the recipes in WordCraftare based on real-world semantics among common entities, conditioning on commonsense knowledge  in a knowledge graph should enable agents to learn more efﬁciently by constraining their search space to policies biased toward interactions with underlying commonsense semantics.
Revised: Given that the recipes in WordCraft are based on real-world semantics among common entities, conditioning on commonsense knowledge present in a KG should enable agents to learn more efﬁciently by constraining their search space to policies biased toward interactions with underlying commonsense semantics.

Original: 4.1. Self-attention Actor-Critic Network
Revised: 4.1. Self-attention Actor-Critic Network

Original: In order to handle a variable number of selectable entities at each time step and ensureorder invariance of selection decisions with respect tothese entities in the environment, our policy network makes use of a scaled dot-product attention mechanism introduced in (Vaswani et al., 2017).
Revised: In order to handle a variable number of selectable entities at each time step and ensure selection decisions are invariant to the table position of entities, our policy network makes use of the scaled dot-product attention mechanism introduced in (Vaswani et al., 2017).

Original: The self-attention policy network encodes the features x g and current selection features x s, 1:2 into attention weights over the set of selectable entities.
Revised: The self-attention policy network encodes the features x g and current selection features x s, 1: into attention weights over the set of selectable entities.

Original: These attention weights α i act as the logits to the action distribution over the set of selectable entities e i at each time step.
Revised: These attention weights α i act as the logits to the action distribution over the set of selectable entities e i at each time step:

Original: [Equation]
Revised: [Equation]

Original:  where x table are learned representations of selectable entities.
Revised: j e α j , where x table are learned representations of selectable entities.

Original: In order to predict value estimates, we ﬁrst apply a linear transform W to x table and encode the state as an attentionweighted sum of these linearly transformed entity features.
Revised: In order to predict value estimates, we ﬁrst apply a linear transform W to x table and encode the state as an attentionweighted sum of these linearly transformed entity features.

Original: This encoding is passed through a fully-connected layer to yield value estimates V ( x ) =
Revised: This encoding is passed through a fully-connected layer to yield value estimates V ( x ) =

Original: MLP ( α (cid:62) W ( x table )) .
Revised: MLP ( α (cid:62) W ( x table )) .

Original: 4.2. Link Prediction Model
Revised: 4.2. Link Prediction Model

Original: We assume access to a link prediction model L trained on an external knowledge graph that contains information relevant to the RL task.
Revised: We assume access to a link prediction model L trained on an external knowledge graph that contains information relevant to the RL task.

Original: While our model makes use of ComplEx for link prediction (Trouillon et al., 2016), in general this choice can be substituted by any other link prediction model.
Revised: While our model makes use of ComplEx for link prediction (Trouillon et al., 2016), in general this choice can be substituted by any other link prediction model.

Original: We chose ComplEx for its reliable performance in practice.
Revised: We chose ComplEx for its reliable performance in practice.

Original: The link prediction model is trained on a set of subjectobject-predicate relation triplets of the form ( s, p, o ) to predict higher likelihood scores for true relation triplets.
Revised: The link prediction model is trained on a set of subjectobject-predicate relation triplets of the form ( s, p, o ) to predict higher likelihood scores for true relation triplets.

Original: In the context of WordCraft, each recipe { e 1 , e 2 } −→ e 3 reduces to four relevant relation triplets: ( e 1 , combinesWith , e 2 ) , ( e 2 , combinesWith , e 1 ) , ( e 1 , componentOf , e 3 ) , and ( e 2 , componentOf , e 2 ) .
Revised: In the context of WordCraft, each recipe { e 1 , e 2 } −→ e 3 reduces to four relevant relation triplets: ( e 1 , combinesWith , e 2 ) , ( e 2 , combinesWith , e 1 ) , ( e 1 , componentOf , e 3 ) , and ( e 2 , componentOf , e 2 ) .

Original: These triplets, when aggregated across all recipes, forms the recipe graph.
Revised: These triplets, when aggregated across all recipes, forms the recipe graph.

Original: In our experiments, we train ComplEx on a subgraph of the recipe graph containing all nodes to ensure full entity coverage.
Revised: In our experiments, we train ComplEx on a subgraph of the recipe graph containing all nodes to ensure full entity coverage.

Original: At each time step, we compute relation score vectors u and v between the goal and selected entities and each e i on the table and e s,j in the selection: u i =
Revised: At each time step, we compute relation score vectors u and v between the goal and selected entities and each e i on the table and e s,j in the selection: u i =

Original: L ( e i , combinesWith , e s,j ) , v i =
Revised: L ( e i , combinesWith , e s,j ) , v i =

Original: L ( e i , componentOf , e g ) .
Revised: L ( e i , componentOf , e g ) .

Original: These scores are then mixed component-wise with the attention weights, before passing the mixed weights into the softmax policy head, thereby guiding the agent toward relevant predictions from the knowledge graph model.
Revised: These scores are then mixed component-wise with the attention weights, before passing the mixed weights into the softmax policy head, thereby guiding the agent toward relevant predictions from the KG model.

Original: Details on learning the mixing coefﬁcients are included in the Appendix.
Revised: Details on learning the mixing coefﬁcients are provided in Appendix B.1.

Original: 5. Experiments
Revised: 5. Experiments

Original: All of our experiments focus on zero-shot generalization performance.
Revised: Our experiments focus on zero-shot generalization performance.

Original: We split the set of all valid recipes into train (80%) and test (20%) sets.
Revised: We split the set of all valid recipes into train (80%) and test (20%) sets.

Original: Train tasks are generated such that task goals does not involve any recipes in the test set.
Revised: Train tasks are generated such that task goals do not involve any recipes in the test set.

Original: The model is trained using IMPALA/TorchBeast (Espeholt et al., 2018; K ¨ uttler et al., 2019).
Revised: The model is trained using the TorchBeast implementation of IMPALA (Espeholt et al., 2018; K ¨ uttler et al., 2019).

Original: 5.1. WordCraft Benchmarks
Revised: 5.1. WordCraft Benchmarks

Original: To demonstrate how well the task capturesthe real-world relationships between entities,when evaluating the algorithm from Section 4.1, we will be using GloVe embeddings (Pennington et al., 2014).
Revised: To demonstrate how well the task captures real-world relationships between entities, we represent entities as GloVe embeddings (Pennington et al., 2014) when evaluating the model in Section 4.1.

Original: If semantically similar entities have similar uses in this environment, we expect GloVe to help generalization.
Revised: We expect GloVe to help generalization if semantically similar entities have similar uses in this environment.

Original: We compare the zero-shot performance of our agent model without the link prediction module on depth-1 and depth-2 tasks with 1, 2, and 8 distractors.
Revised: We assess the zero-shot performance of our agent model without a link prediction module on depthtasks with 1 and 8 distractors.

Original: We also collect a human baseline on the same difﬁculty settings of WordCraft.
Revised: We also collect a human baseline at the same difﬁculty settings of WordCraft.

Original: This human baseline serves as an estimate of the zero-shot performance that can be achieved using commonsense and general knowledge (see the Appendix  fordetails about the procedure for human evaluation).
Revised: This human baseline serves as an estimate of the zero-shot performance that can be achieved using commonsense and general knowledge (see the Appendix C.5 for the description of human evaluation protocol).

Original: [Figure]
Revised: [Figure]

Revised: 3M

Revised: Steps 0.

Original: M ean ep i s ode r e t u r n
Revised: M ean ep i s ode r e t u r n

Revised: Self-attention

Original: [Figure]
Revised: [Figure]

Revised: GloVe

Original: [Figure]
Revised: [Figure]

Original: ean ep i
Revised: 1 distractor Self-attention + random

Original: [Figure]
Revised: [Figure]

Original: ode r
Revised: 1 distractor

Original: [Figure] t [Figure] r n
Revised: Self

Original: [Figure]
Revised: [Figure]

Original: Agents trained with GloVe embeddings generalize much better to tasks with recipes that were not part of any training task (see Figure 2).
Revised: As seen in Figure 2, agents trained with GloVe embeddings generalize much better to tasks with recipes that were not part of any training task.

Original: This margin of improvement persists in harder task settings, even as the test success rate falls.
Revised: This margin of improvement persists in harder task settings, even as the test success rate falls.

Original: 5.2. Knowledge Graph Benchmarks
Revised: 5.2. Knowledge Graph Benchmarks

Original: We benchmark variants of our full agent with the link prediction module, and compare zero-shot performance under different combinations of link prediction scores used to modulate the self-attention weights: both combinesWith and componentOf scores, or each score in isolation.
Revised: We benchmark variants of our full agent with the link prediction module and compare zero-shot performance under different combinations of link prediction scores used to modulate the self-attention weights: both combinesWith and componentOf scores, or each score in isolation.

Original: In these experiments, L is trained on the same train set of recipes used to generate the training tasks.
Revised: In these experiments, L is trained on the same train set of recipes used to generate the training tasks.

Original: Our results show that access to a full KG model drastically improves the agent’s zero-shot performance, as expected (see Figure 3).
Revised: Our results show that access to a full KG model drastically improves the agent’s zero-shot performance (see Figure 3).

Original: However, while  partial-KG-model agents reach an equivalent zero-shot success rate as an agent without any KG model in fewer training steps, they ultimately reach comparable levels of test performance as training progresses (see Appendix for a more detailed discussion).
Revised: However, while a partial-KG-model agent reaches an equivalent zero-shot success rate as an agent without any KG model in fewer training steps, they ultimately reach comparable levels of test performance as training progresses (see Appendix for a more detailed discussion).

Original: 6. Related Work
Revised: 6. Related Work

Original: Recently, several new environments that incorporate a degree of commonsense dynamics and text-base corpora have been proposed.
Revised: Recently, several new environments incorporating a degree of commonsense dynamics and text-base corpora have been proposed.

Original: For example, TextWorld (Cˆot´e et al., 2018) is a framework that enables to create novel textual RL environments as well as interfacing with existing commercial ones (such as Zork games).
Revised: For example, TextWorld (Cˆot´e et al., 2018) is a framework that enables creation of novel textual RL environments as well as interfacing with existing text-based games, such as Zork.

Original: Chevalier-Boisvert et al.
Revised: Chevalier-Boisvert et al.

Original: K¨uttler et al.
Revised: K¨uttler et al.

Original: (2020) respectively propose BabyAI and NLE, grid-world environments that evaluate agents on procedural, goal-conditioned tasks with varying degrees of commonsense structure and available textual corpora.
Revised: (2020) respectively propose BabyAI and NLE, gridworld environments that evaluate agents on procedurallygenerated, goal-conditioned tasks with varying degrees of commonsense structure and available textual corpora.

Original: These environments areinteresting and challenging because of multiple reasons: complex dynamics, large action spaces, sparse rewards,  long planning horizons.
Revised: While these environments are challenging in their complex dynamics, large action spaces, sparse rewards, and long planning horizons, these same aspects confound the ability to isolate

Original: However, these are confounding elements do not easily enable us to isolate the problem of conditioning on external knowledge sources.
Revised: [Figure] the problem of conditioning on external knowledge sources.

Original: Another line of  related work conditions policies ontextual knowledge and/or knowledge bases, recently surveyed in Luketina et al.
Revised: A comprehensive survey of a related line of research that conditions policies on knowledge in textual corpora or knowledge bases can be found in Luketina et al.

Original: Fulda et al.
Revised: Fulda et al.

Original: ; Zahavy et al.
Revised: ; Zahavy et al.

Original: (2018) demonstrate that word embeddings can be an effective modelling tool for policies, while Branavan et al.
Revised: (2018) demonstrate that word embeddings can be an effective modeling tool for policies, while Branavan et al.

Original:  Narasimhan et al.
Revised: ; Narasimhan et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: (2020) present work that directly conditions policies on domain knowledge encoded in textual corpora.
Revised: (2020) present work that directly conditions policies on domain knowledge encoded in textual corpora.

Original: Marzoev et al.
Revised: Marzoev et al.

Original: ; Hill et al.
Revised: ; Hill et al.

Original: (2020) show that language models can enable generalising to unseen subgoal-instructions, which indicates thatat least some linguistic knowledge can be successfully transferred to RL.
Revised: (2020) show that language models can enable generalization to unseen instructions, which indicates that linguistic knowledge can be successfully transferred to aid downstream RL tasks.

Revised: Models integrating KGs with agents have been studied in the literature on text-based games (Ammanabrolu & Riedl, 2019; Adhikari et al., 2020).

Original: Closest to our work, Murugesan et al.
Revised: Closest to our work, Murugesan et al.

Original: (2020) concurrently propose to solve a set of TextWorld tasks using a model that encodes a combination of domain knowledge, via ConceptNet, andenvironment state into a knowledge graph, noting that while usable, knowledge graphs can  providea too strong  prior.
Revised: (2020) concurrently propose solving a set of TextWorld tasks using a model that incorporates a combination of commonsense knowledge, encoded in ConceptNet, and a learned KG representing the current environment state. They ﬁnd that while at times beneﬁcial, KGs can also provide too strong a prior.

Original: 7. Future Work
Revised: 7. Future Work

Original: There are multiple avenues that we wish to further explore.
Revised: There are multiple avenues that we plan to further explore.

Original:  WordCraftallows users toplay Little Alchemy 2 as if the game had a very short horizon, however, the full game, in which the user a set of basic elements, and needs to discover as many entities as possible could be an interesting problem for RL as well.
Revised: Extending WordCraft to the longer horizon setting of the original Little Alchemy 2, in which the user must discover as many entities as possible, could be an interesting setting to study commonsense-driven exploration.

Original: In addition, we are interested in including additional modalities such as images, as well as integrating alchemy into existing gridworld environments.
Revised: We also plan to introduce additional modalities such as images and control into WordCraft.

Original: We are also interested in extending the environment to directly condition on textual knowledge sources.

Original: Furthermore, we believe the architecture we presented could have a strong impact in  more complex RL tasks  withaccess to large corpora of task-speciﬁc knowledge, such as NLE.
Revised: Furthermore, we believe the ideas in this work could beneﬁt more complex RL tasks associated with large corpora of task-speciﬁc knowledge, such as NLE.

Original: Doing so requires automatically constructing knowledge graphs from available corpora, or further research on agents thatcan retrieve and  condition on natural language texts.
Revised: This path of research entails further investigation of methods for automatically constructing knowledge graphs from available corpora as well as agents that retrieve and directly condition on natural language texts in such corpora.

Original: A. Background
Revised: A. Background

Original: Knowledge Graphs Knowledge graphs naturally represent the ontology of concepts underlying commonsense knowledge about the world.
Revised: Knowledge Graphs Knowledge graphs naturally represent the ontology of concepts underlying commonsense knowledge about the world.

Original: In a knowledge graph, nodes represent concepts, and edges between two nodes represent a relationship between the corresponding concepts.
Revised: In a knowledge graph, nodes represent concepts, and edges between two nodes represent a relationship between the corresponding concepts.

Original: Several projects provide freely available commonsense knowledge graphs, including Freebase, DBPedia, and ConceptNet (Speer et al., 2017).

Original: Such large-scale knowledge graphs hold an enormous amount of useful prior knowledge about the world, that should be useful for guiding an RL agent in any environment with real-world semantics.

Original: Let E and R denote a set of entities and a set of relation types, respectively.
Revised: Let E and R denote a set of entities and a set of relation types, respectively.

Original: Formally, a KG G ⊆
Revised: Formally, a KG G ⊆

Original: E × R × E is a set of subject-predicate-object triples (cid:104) s, p, o (cid:105) , with s, o ∈ E and p ∈ R , where each triple encodes a relationship of type p between the subject s and the object o .
Revised: E × R × E is a set of subject-predicate-object triples (cid:104) s, p, o (cid:105) , with s, o ∈ E and p ∈ R , where each triple encodes a relationship of type p between the subject s and the object o .

Original: There are several examples of large-scale KGs, both from academia (such as YAGO (Suchanek et al., 2007) and DBpedia (Auer et al., 2007)) and industry (Such asMicrosoft’s Bing Knowledge
Revised: There are several examples of large-scale KGs encoding commonsense knowledge, both from academia (such as YAGO (Suchanek et al., 2007), DBpedia (Auer et al., 2007)) and industry (such as

Original:  Graph and the Google Knowledge Graph (Noy et al., 2019)).
Revised: Microsoft’s Bing Knowledge Graph and the Google Knowledge Graph (Noy et al., 2019)).

Revised: Such large-scale knowledge graphs hold an enormous amount of useful prior knowledge about the world, that should be useful for guiding an RL agent in any environment with real-world semantics.

Original: Especially relevant to this work are ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), two KGs encoding common-sense knowledge.
Revised: Especially relevant to this work are ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), two KGs encoding common-sense knowledge.

Original: Representation Learning of KGs Similarly to language, an effective way of enabling statistical learning on KGs consists in learning continuous, distributed representations, also referred to as embeddings , for all entities in a KG – we refer to Nickel et al.
Revised: Representation Learning of KGs An effective way of enabling statistical learning on KGs consists of learning continuous, distributed representations, also referred to as embeddings , for all entities in a KG. We refer to Nickel et al.

Original:  for a recent survey on this topic.
Revised: (2016) for a recent survey on this topic.

Original: In this work, we use the ComplEx (Trouillon et al., 2016)  model, with the loss functions and regularisers proposed by
Revised: In this work, we use the ComplEx (Trouillon et al., 2016) link prediction model, with the loss functions and regularizers proposed by

Original: Lacroix et al.
Revised: Lacroix et al.

Original: Given a subject-predicate-object triple (cid:104) s,
Revised: Given a subject-predicate-object triple (cid:104) s,

Original: p, o (cid:105) ∈
Revised: p, o (cid:105) ∈

Original: E × R × E , ComplEx deﬁnes the score φ ( s, p, o ) of such a triple as:
Revised: E × R × E , ComplEx deﬁnes the score φ ( s, p, o ) of such a triple as:

Original: [Equation]
Revised: [Equation]

Original:  where (cid:104)· , · , ·(cid:105) denotes the tri-linear dot product, e s , e p , e o ∈ C k denote the complex-valued k -dimensional representations of s , p , and o , x denote the complex conjugate of x ∈ C k , Re ( x ) is the real part of x .
Revised: o [Equation] [Equation] where (cid:104)· , · , ·(cid:105) denotes the tri-linear dot product, e s , e p , e o ∈ C k denote the complex-valued k -dimensional representations of s , p , and o , x denote the complex conjugate of x ∈ C k , Re ( x ) is the real part of x .

Original: In order to learn the embedding representations Θ for all entities and relation types in a KG G , we follow Lacroix et al.
Revised: In order to learn the embedding representations Θ for all entities and relation types in a KG G , we follow Lacroix et al.

Original: and minimise the following objective via gradient-based optimisation:
Revised: and minimise the following objective via gradient-based optimisation:

Original: [Equation]
Revised: [Equation]

Original: regularised via the weighted Nuclear p -norm regulariser proposed by Lacroix et al.
Revised: regularized via the weighted nuclear p -norm regularizer proposed by Lacroix et al.

Original: B. Model Details B.1. Learning Mixing Coefﬁcients
Revised: B. Model Details B.1. Learning Mixing Coefﬁcients

Original: The link-prediction scores between goal and table entities for the componentOf relation and between each selected entity and table entity for the combinesWith relation are combined component-wise with the self-attention weights
Revised: The link-prediction scores between goal and table entities for the componentOf relation and between each selected entity and table entities for the combinesWith relation are combined component-wise with the self-attention weights

Original: α i using mixing coefﬁcients to yield ﬁnal policy logits β :
Revised: α i using mixing coefﬁcients to yield ﬁnal policy logits β :

Original: [Equation]
Revised: [Equation]

Original: The mixing coefﬁcients λ are computed by passing the selfattention-weighted table encoding through a fully-connected layer:
Revised: The mixing coefﬁcients λ are computed by passing the selfattention-weighted table encoding through a fully-connected layer:

Original: [Equation]
Revised: [Equation]

Original: C. Experiments C.1. Training Parameters
Revised: C. Experiments C.1. Self-Attention Actor-Critic Architecture

Original: [Table]
Revised: used

Original: Batch size Unroll length
Revised: C.2. Training Parameters

Original: [Table]
Revised: [Table]

Original: Entropy regularization
Revised: C.3. Model Parameters

Original: [Table]
Revised: [Table]

Original: Reward clipping none
Revised: C.4. Extended Discussion

Original: Total steps 3,000, C.2. Model Parameters Parameter
Revised: In addition to initial benchmarks conducted in 5.

Original: Value
Revised: [Table]

Original: Self-attention key size Self-attention value size ComplEx embedding size Simple-MLP hidden size
Revised: we also compare the performance of the self-attention actor-critic agent described in 4.1 to a simple, single-layer MLP that predicts the policy logits and value estimates from the concatenation of all GloVe word embeddings of goal, selection, and table entities at each time step.

Original: C.3. Extended Discussion
Revised: These results are presented in Fig. [Table]

Original: We  benchmark the performance of the self-attention actorcritic agentdescribed in Section 4.1 with and without access to an instance of ComplEx trained on a subgraph of the full recipe graph.
Revised: We further benchmark the performance of the self-attention actor-critic agent with and without access to an instance of ComplEx trained on a subgraph of the full recipe graph.

Original: We compare agents trained with access to ComplEx trained on a partial graph and those trained on the full graph.
Revised: We compare agents trained with access to ComplEx trained on a partial graph and those trained on the full graph.

Original: The partial KG models are trained on  the relations within the set of training recipes.
Revised: The partial KG models are trained on only the relations present in the set of training recipes.

Original: As the method for incorporating ComplEx’s predicted relation scores into the policy prediction performs a weighted sum of each score with the self-attention weights, we investigate the effect of only using different subsets of the scores.
Revised: As the method for incorporating ComplEx’s predicted relation scores into the policy prediction performs a weighted sum of each score with the self-attention weights, we investigate the effect of only using different subsets of the scores.

Original: The subset of relation scores that are considered by the policy seem to vary across task depth and distractor settings.
Revised: As seen in Fig. 6 and Fig.

Original: [Figure]
Revised: [Table]

Original: ean ep i s ode
Revised: the subset of relation scores that are helpful to policy learning varies across task depth and distractor settings. We generally ﬁnd that access to a full KG model drastically improves the agent’s zero-shot success rate, as might be expected. In fact, one might expect that having access

Original: [Figure]
Revised: [Table]

Original: r n

Original: [Figure]
Revised: [Figure]

Original: We generally ﬁnd that access to a full KG model drastically improves the agent’s zero-shot success rate, as might be expected.

Original: In fact, one might expect that having access to the ground-truth recipe graph as encoded by ComplEx should enable the agent to achieve 100% zero-shot success rates across all task settings.
Revised: to the ground-truth recipe graph as encoded by ComplEx should enable the agent to achieve 100% zero-shot success rates across all task settings.

Original: In order to test the extent to which the full-KG ComplEx scores are predictive of the best actions, we also assessed the performance of agents whose ﬁnal policy logit mixture only assigned positive weights to different subsets of the relation scores.
Revised: In order to test the extent to which the full-KG ComplEx scores are predictive of the best actions [Figure] we also assessed the performance of agents whose ﬁnal policy logit mixture only assigned positive weights to different subsets of the relation scores.

Original: Such full-KG-only agents that choose items only based on the combinesWith relation score will still often choose the same entity twice, believing the entity combines with itself.
Revised: Such full-KG-only agents that choose items only based on the combinesWith relation score will still often choose the same entity twice, believing the entity combines with itself.

Original: This is likely because the recipe graph includes several entities that combine with themselves, while the set of all self-combination triplets are not included in the training set.
Revised: This is likely because the recipe graph includes several entities that combine with themselves, while the set of all self-combination triplets are not included in the training set [Figure] [Figure]

Original: Full-KG-only agents that choose items only based on the componentOf relation score will tend to choose the same item twice, as this approach will always assign the highest action probability on selecting the entity on the table that
Revised: Full-KG-only agents that choose items only based on the componentOf relation score will tend to choose the same item twice, as this approach will always assign the highest action probability to selecting the entity on the table that

Original: ComplEx predicts to be most likely to be a component of the goal entity.
Revised: ComplEx predicts to be most likely to be a component of the goal entity.

Original: Interestingly, partial-KG-agents do not always do much better than a KG-free agent.
Revised: Interestingly, partial-KG-agents do not always do much better than a KG-free agent.

Original: This implies that as the KGfree agent sees moreand more training tasks, it becomes comparable with ComplEx in predicting unseen relations among entities.
Revised: This implies that as the KGfree agent sees more training tasks, it becomes comparable to ComplEx in predicting unseen relations among entities.

Original: Early on in training, the relation scores predicted by ComplEx seem to help the agent achieve higher zero-shot success rates in fewer training steps compared to KG-free agents (see Figure 6). [Figure] [Figure]
Revised: Early on in training, the relation scores predicted by ComplEx seem to help the agent achieve higher zero-shot success rates in fewer training steps compared to KG-free agents (see Figure 7).

Original: C.4. Human Evaluation Protocol
Revised: C.5. Human Evaluation Protocol

Original: We estimate  generalization ability of humans players on depth-1 tasks with 1 and 8 distractors.
Revised: We estimate the generalization ability of humans players on depth-1 tasks with 1 and 8 distractors.

Original: Each difﬁculty setting is tested on a different individual, and we test one individual per setting.
Revised: Each difﬁculty setting is tested on a different individual, and we test one individual per setting.

Original: The selected individuals were not familiar with the game and got to train on 40 randomly sampled training tasks before being evaluated on 40 testing tasks.
Revised: The selected individuals were not familiar with the game and got to train on 40 randomly sampled training tasks before being evaluated on 40 testing tasks.

Original: The train and test split was the same as in the RL experiments.
Revised: The train and test split was the same as in the RL experiments.

Original: Surprisingly, we found that changing the number of distractors from 1 to 8 did not signiﬁcantly inﬂuence human performance.
Revised: Surprisingly, we found that changing the number of distractors from 1 to 8 did not signiﬁcantly inﬂuence human performance.

Revised: We plan to conduct a larger-scale human performance evaluation.

Original: D. Example recipes
Revised: D. Example recipes

Original: Resulting entity Entity combination(s) airplane bird metal; bird, steel alcohol fruit, time; juice, time batter ﬂour, milk cereal wheat, milk catnip cat, plant charcoal ﬁre, wood dew water, grass; fog, grass farmer human, ﬁeld; human, plant; human, pitchfork geyser steam, earth glacier ice, mountain hay bale hay, hay iced tea ice, tea ivy plant, wall juice water, fruit; pressure, fruit kite wind, paper; sky, paper lake water, pond; river, dam milk farmer, cow; cow, human milk shake milk, ice cream narwhal unicorn, ocean; unicorn, water; sea, unicorn oasis desert, water paper wood, pressure pasta ﬂour, egg rainbow rain, sun; rain, light reindeer Santa, wild animal; livestock, Santa sand castle sand, castle Santa human, Christmas tree scythe blade, grass; blade, wheat telescope glass, sky; glass, star; glass, space umbrella tool, rain; rain, fabric volcano lava, earth; lava, mountain; ﬁre, mountain wallet leather, money watch human, clock x-ray light, bone; light, skeleton yogurt milk, bacteria zombie corpse, life
Revised: [Table]