Original: WordCraft: An Environment for Benchmarking Commonsense Agents
Revised: WordCraft: An Environment for Benchmarking Commonsense Agents

Original: Abstract
Revised: Abstract

Original: The ability to quickly solve a wide range of realworld tasks requires a commonsense understanding of the world.
Revised: The ability to quickly solve a wide range of realworld tasks requires a commonsense understanding of the world.

Original: Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge.
Revised: Yet, how to best extract such knowledge from natural language corpora and integrate it with reinforcement learning (RL) agents remains an open challenge.

Original: This is partly due to the lack of lightweight simulation environments that sufﬁciently reﬂect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.
Revised: This is partly due to the lack of lightweight simulation environments that sufﬁciently reﬂect the semantics of the real world and provide knowledge sources grounded with respect to observations in an RL environment.

Original: To better enable research on agents making use of commonsense knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2.
Revised: To better enable research on agents making use of commonsense knowledge, we propose WordCraft, an RL environment based on Little Alchemy 2.

Original: This lightweight environment is fast to run and built upon entities and relations inspired by realworld semantics.
Revised: This lightweight environment is fast to run and built upon entities and relations inspired by realworld semantics.

Original: We evaluate several representation learning methods on this new benchmark and propose a new method for integrating knowledge graphs with an RL agent.
Revised: We evaluate several representation learning methods on this new benchmark and propose a new method for integrating knowledge graphs with an RL agent.

Original: 1. Introduction
Revised: 1. Introduction

Original: Recent progress in Reinforcement Learning (RL), while impressive (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019) has mostly focused on tabula-rasa learning, rather than exploitation of prior world knowledge.
Revised: Recent progress in Reinforcement Learning (RL), while impressive (Silver et al., 2016; Vinyals et al., 2019; Berner et al., 2019) has mostly focused on tabula-rasa learning, rather than exploitation of prior world knowledge.

Original: For example, while Go is an extremely difﬁcult game to master, its rules and thus environment dynamics are simple and not heavily reliant on knowledge about concepts that can be encountered in the real world.
Revised: For example, while Go is an extremely difﬁcult game to master, its rules and thus environment dynamics are simple and not heavily reliant on knowledge about concepts that can be encountered in the real world.

Original: Focusing on tabula-rasa learning conﬁnes state-of-the-art approaches to simulation environments that can be solved without the transfer of commonsense, world, or domain knowledge.
Revised: Focusing on tabula-rasa learning conﬁnes state-of-the-art approaches to simulation environments that can be solved without the transfer of commonsense, world, or domain knowledge.

Original: Many real-world applications ( e g. personal assistants and household robots) require agents that can learn fast and generalise well to novel situations, which is likely not possible to
Revised: Many real-world applications ( e g. personal assistants and household robots) require agents that can learn fast and generalise well to novel situations, which is likely not possible to

Original: Reinforcement
Revised: Reinforcement

Original: [Table]
Revised: [Table]

Original: without the ability to reason with commonsense and general knowledge about the world.
Revised: without the ability to reason with commonsense and general knowledge about the world.

Original: Consider, for example, an agent tasked with performing common household chores that has never seen a dirty ashtray.
Revised: Consider, for example, an agent tasked with performing common household chores that has never seen a dirty ashtray.

Original: When presented with this new object, the agent needs to know that a reasonable set of actions involving this object include cleaning the ashtray, but not feeding it to the cat.
Revised: When presented with this new object, the agent needs to know that a reasonable set of actions involving this object include cleaning the ashtray, but not feeding it to the cat.

Original: Humans encode a large amount of commonsense knowledge in written language.
Revised: Humans encode a large amount of commonsense knowledge in written language.

Original: For example, Wikipedia encodes such knowledge implicitly through corpus statistics, and at times explicitly in writing.
Revised: For example, Wikipedia encodes such knowledge implicitly through corpus statistics, and at times explicitly in writing.

Original: Outside of RL, learning to represent and utilise prior knowledge has improved considerably over recent years.
Revised: Outside of RL, learning to represent and utilise prior knowledge has improved considerably over recent years.

Original: For example, a common approach is to pre-train neural language models and ﬁne-tune them on downstream tasks (Devlin et al., 2019; Raffel et al., 2019).
Revised: For example, a common approach is to pre-train neural language models and ﬁne-tune them on downstream tasks (Devlin et al., 2019; Raffel et al., 2019).

Original: Recent works highlight that such pre-trained models capture many aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019; Hewitt & Manning, 2019; Reif et al., 2019).
Revised: Recent works highlight that such pre-trained models capture many aspects of commonsense (Da & Kasai, 2019) and relational knowledge (Petroni et al., 2019), in addition to linguistic knowledge (Jawahar et al., 2019; Hewitt & Manning, 2019; Reif et al., 2019).

Original: Another popular approach is to represent commonsense knowledge explicitly as Knowledge
Revised: Another popular approach is to represent commonsense knowledge explicitly as Knowledge

Original: Graphs (KGs), as done by ATOMIC (Sap et al., 2019) and
Revised: Graphs (KGs), as done by ATOMIC (Sap et al., 2019) and

Original: ConceptNet (Speer et al., 2017).
Revised: ConceptNet (Speer et al., 2017).

Original: Such KGs have been used in commonsense reasoning tasks (Lin et al., 2019).
Revised: Such KGs have been used in commonsense reasoning tasks (Lin et al., 2019).

Original: However, the question of how to best utilise commonsense knowledge (provided in language corpora or KGs) for downstream RL tasks has not received the same level of attention, partly due to scarce simulation environments that directly beneﬁt from transferring such prior knowledge.
Revised: However, the question of how to best utilise commonsense knowledge (provided in language corpora or KGs) for downstream RL tasks has not received the same level of attention, partly due to scarce simulation environments that directly beneﬁt from transferring such prior knowledge.

Original: Robotics environments such as MuJoCo require knowledge not readily expressed in natural language, such as an intuitive understanding of physics (Todorov et al., 2012; Yu et al., 2019).
Revised: Robotics environments such as MuJoCo require knowledge not readily expressed in natural language, such as an intuitive understanding of physics (Todorov et al., 2012; Yu et al., 2019).

Original: Rule-based game environments such as Go or Chess (Silver et al., 2017; 2018)
Revised: Rule-based game environments such as Go or Chess (Silver et al., 2017; 2018)

Original: , while very hard to master, are based on simple transition dynamics and largely disconnected from entities and concepts encountered in the real world.
Revised: , while very hard to master, are based on simple transition dynamics and largely disconnected from entities and concepts encountered in the real world.

Original: On the other hand, commonsense knowledge could be helpful in environments such as StarCraft II (Vinyals et al., 2019; Berner et al., 2019), which contain many complex interactions with a large number of entities (many of which have real-world analogues).
Revised: On the other hand, commonsense knowledge could be helpful in environments such as StarCraft II (Vinyals et al., 2019; Berner et al., 2019), which contain many complex interactions with a large number of entities (many of which have real-world analogues).

Original: However,running such environments is very costly, and learning to ground knowledge from external sources remains challenging, as natural language annotations of states and actions in such games are not readily available.
Revised: However, such environments are costly to run, and learning to ground knowledge from external sources remains challenging, as natural language annotations of states and actions in such games are not readily available.

Original: To study agents with commonsense knowledge, we present
Revised: To study agents with commonsense knowledge, we present

Original: WordCraft, a fast RL environment based on the popular game Little Alchemy 2.
Revised: WordCraft, a fast RL environment based on the popular game Little Alchemy 2.

Original: In this game, the player is tasked with crafting over 700 different entities by combining previously discovered entities.
Revised: In this game, the player is tasked with crafting over 700 different entities by combining previously discovered entities.

Original: For example, combining “water” and “earth” creates “mud”.
Revised: For example, combining “water” and “earth” creates “mud”.

Original: Learning policies that generalize to unseen entities and combinations requires commonsense knowledge about the world.
Revised: Learning policies that generalize to unseen entities and combinations requires commonsense knowledge about the world.

Original: The environment runs quickly (around 8,000 steps per second on a single machine), enabling fast experimentation cycles.
Revised: The environment runs quickly (around 8,000 steps per second on a single machine), enabling fast experimentation cycles.

Original: As the entities in the game correspond to words, they can be easily grounded with external knowledge sources.
Revised: As the entities in the game correspond to words, they can be easily grounded with external knowledge sources.

Original: In addition, we introduce a method for conditioning agents trained in WordCraft on
Revised: In addition, we introduce a method for conditioning agents trained in WordCraft on

Original: KGs encoding prior knowledge, such as ConceptNet and other commonsense KGs.
Revised: KGs encoding prior knowledge, such as ConceptNet and other commonsense KGs.

Original: 2. Background
Revised: 2. Background

Original: We formulate the RL problem as a Markov Decision Process (MDP) (Sutton & Barto, 2018) deﬁned by the tuple ( S, A, T, R, γ ) , where s ∈ S is a state, a ∈ A is an action,
Revised: We formulate the RL problem as a Markov Decision Process (MDP) (Sutton & Barto, 2018) deﬁned by the tuple ( S, A, T, R, γ ) , where s ∈ S is a state, a ∈ A is an action,

Original: T ( s, a ) → s (cid:48) is the transition function, R ( s, a ) → R is the reward function, and γ , the discount factor.
Revised: T ( s, a ) → s (cid:48) is the transition function, R ( s, a ) → R is the reward function, and γ , the discount factor.

Original: The goal is to ﬁnd a policy π that maximises the expected sum of future discounted rewards: (cid:80) ∞ k =

Revised: The goal is to ﬁnd a policy π that maximises the expected sum of future discounted rewards: (cid:80) ∞ k =

Original: γ k r k +1 .
Revised: γ k r k +1 .

Original: We represent knowledge about entities and the relations among them as a knowledge graph (KG): a collection of (cid:104) s, p, o (cid:105) triples, each encoding a relationship of type p (predicate) between the subject s and the object o of the triple.
Revised: We represent knowledge about entities and the relations among them as a knowledge graph (KG): a collection of (cid:104) s, p, o (cid:105) triples, each encoding a relationship of type p (predicate) between the subject s and the object o of the triple.

Original: In order to work with incomplete knowledge graphs and infer missing relationships, we use the ComplEx link prediction model (Trouillon et al., 2016)
Revised: In order to work with incomplete knowledge graphs and infer missing relationships, we use the ComplEx link prediction model (Trouillon et al., 2016)

Original: (see Appendix A for details).
Revised: (see Appendix A for details).

Original: 3. WordCraft
Revised: 3. WordCraft

Original: In order to benchmark the ability of RL agents to make use of commonsense knowledge, we present WordCraft , based on Little Alchemy 2 1 .
Revised: In order to benchmark the ability of RL agents to make use of commonsense knowledge, we present WordCraft , based on Little Alchemy 2 1 .

Original: Little Alchemy 2 is a simple association game: Starting from a set of four basic items, the player must create as many different items as possible.
Revised: Little Alchemy 2 is a simple association game: Starting from a set of four basic items, the player must create as many different items as possible.

Original: Each non-starter item can be created by combining two other items.
Revised: Each non-starter item can be created by combining two other items.

Original: For example, combining “moon” and “butterﬂy” yields “moth”, and combining “human” and “medusa” yields “statue”.
Revised: For example, combining “moon” and “butterﬂy” yields “moth”, and combining “human” and “medusa” yields “statue”.

Original: There are 700 total items (including rare and compound words) and 3,417 permissible item combinations (referred to as valid recipes throughout this paper)
Revised: There are 700 total items (including rare and compound words) and 3,417 permissible item combinations (referred to as valid recipes throughout this paper)

Original: (IGN, 2020).
Revised: (IGN, 2020).

Original: Efﬁciently solving this game without trying every possible combination of items requires using knowledge about relations between common concepts.
Revised: Efﬁciently solving this game without trying every possible combination of items requires using knowledge about relations between common concepts.

Original: WordCraft is a simpliﬁed version of Little Alchemy 2, sharing the same entities and valid recipes: First, the interface is text-based rather than graphical.
Revised: WordCraft is a simpliﬁed version of Little Alchemy 2, sharing the same entities and valid recipes: First, the interface is text-based rather than graphical.

Original: Second, instead of a single open-ended task, WordCraft consists of a large number of simpler tasks.
Revised: Second, instead of a single open-ended task, WordCraft consists of a large number of simpler tasks.

Original: Each task is created by randomly sampling a goal entity, valid constituent entities, and distractor entities.
Revised: Each task is created by randomly sampling a goal entity, valid constituent entities, and distractor entities.

Original: The agent must choose which entities to combine in order to create the goal entity.
Revised: The agent must choose which entities to combine in order to create the goal entity.

Original: The task difﬁculty can be adjusted by increasing the number of distractors or increasing the number of intermediate entities that must be created in order to reach the goal entity (we refer to this variant as depth- n , with n − 1 intermediate entities).
Revised: The task difﬁculty can be adjusted by increasing the number of distractors or increasing the number of intermediate entities that must be created in order to reach the goal entity (we refer to this variant as depth- n , with n − 1 intermediate entities).

Original: The training and testing split is created by selecting a set of either goal entities or valid recipes that are not used during training.
Revised: The training and testing split is created by selecting a set of either goal entities or valid recipes that are not used during training.

Original: The relatively large number of entities and valid recipes enables the study of generalization to unseen goals.
Revised: The relatively large number of entities and valid recipes enables the study of generalization to unseen goals.

Original: The environment is deterministic and fully-observable.
Revised: The environment is deterministic and fully-observable.

Original: At each time step, the state s consists of the goal entity x g , a set of k currently selectable entities x table = { x i } ki =0 , and the currently selected entity x s, 1 .
Revised: At each time step, the state s consists of the goal entity x g , a set of k currently selectable entities x table = { x i } ki =0 , and the currently selected entity x s, 1 .

Original: The action a corresponds to choosing one of the available entities as the second selected entity x s, 2 ∈ x table .
Revised: The action a corresponds to choosing one of the available entities as the second selected entity x s, 2 ∈ x table .

Original: When no entity is currently selected, x s, 1 is set to a placeholder empty entity.
Revised: When no entity is currently selected, x s, 1 is set to a placeholder empty entity.

Original: If ( x s, 1 , x s, 2 ) corresponds to a valid recipe, the corresponding resulting entity is added to the table x table and x s, 1 is set to the empty entity.
Revised: If ( x s, 1 , x s, 2 ) corresponds to a valid recipe, the corresponding resulting entity is added to the table x table and x s, 1 is set to the empty entity.

Original: The episode terminates when the newly created entity is x g or after a maximum number of steps is reached.
Revised: The episode terminates when the newly created entity is x g or after a maximum number of steps is reached.

Original: We keep the maximum number of steps low to discourage bruteforce solutions ( O ( k 2 ) for depth- 1 tasks).
Revised: We keep the maximum number of steps low to discourage bruteforce solutions ( O ( k 2 ) for depth- 1 tasks).

Original: The reward can be sparse (with r = 1 or 0 received only at the end of the episode) or shaped (with a penalty for choosing a wrong pair of items or creating irrelevant entities and a reward for creating intermediate ingredients).
Revised: The reward can be sparse (with r = 1 or 0 received only at the end of the episode) or shaped (with a penalty for choosing a wrong pair of items or creating irrelevant entities and a reward for creating intermediate ingredients).

Original: Compared to other environments that combine language and RL (Janner et al., 2018; Chevalier-Boisvert et al., 2019;
Revised: Compared to other environments that combine language and RL (Janner et al., 2018; Chevalier-Boisvert et al., 2019;

Original: Cˆot´e et al., 2018; Das et al., 2018), WordCraft contains a much larger number of objects, with object relationships that reﬂect the structure of the real world.
Revised: Cˆot´e et al., 2018; Das et al., 2018), WordCraft contains a much larger number of objects, with object relationships that reﬂect the structure of the real world.

Original: For example, the underlying KG of TextWorld (which is perhaps the closest comparison), contains 99 different objects and 10 types of relations (Zelinka et al., 2019).
Revised: For example, the underlying KG of TextWorld (which is perhaps the closest comparison), contains 99 different objects and 10 types of relations (Zelinka et al., 2019).

Original: Furthermore, the environment does not contain movement-related actions.
Revised: Furthermore, the environment does not contain movement-related actions.

Original: The state space is limited to combinations of entity words, and the action space, to table positions, yet generalization remains challenging.
Revised: The state space is limited to combinations of entity words, and the action space, to table positions, yet generalization remains challenging.

Original: This allows allowing us to study commonsense reasoning in isolation, without requiring that the agent learn to ground states to text or deal with a combinatorially large action space (as seen in text games).
Revised: This allows allowing us to study commonsense reasoning in isolation, without requiring that the agent learn to ground states to text or deal with a combinatorially large action space (as seen in text games).

Original: 4. Guiding Agents with a Knowledge Graph
Revised: 4. Guiding Agents with a Knowledge Graph

Original: We propose an agent architecture that makes use of information from an external KG to guide the agent’s policy.
Revised: We propose an agent architecture that makes use of information from an external KG to guide the agent’s policy.

Original: At a high level, the model consists of a self-attention-based actor-critic network and an external KG link prediction model.
Revised: At a high level, the model consists of a self-attention-based actor-critic network and an external KG link prediction model.

Original: Given that the recipes in WordCraft are based on real-world semantics among common entities, conditioning on commonsense knowledge present in a KG should enable agents to learn more efﬁciently by constraining their search space to policies biased toward interactions with underlying commonsense semantics.
Revised: Given that the recipes in WordCraft are based on real-world semantics among common entities, conditioning on commonsense knowledge present in a KG should enable agents to learn more efﬁciently by constraining their search space to policies biased toward interactions with underlying commonsense semantics.

Original: 4.1. Self-attention Actor-Critic Network
Revised: 4.1. Self-attention Actor-Critic Network

Original: In order to handle a variable number of selectable entities at each time step and ensure selection decisions are invariant to the table position of entities, our policy network makes use of the scaled dot-product attention mechanism introduced in (Vaswani et al., 2017).
Revised: In order to handle a variable number of selectable entities at each time step and ensure selection decisions are invariant to the table position of entities, our policy network makes use of the scaled dot-product attention mechanism introduced in (Vaswani et al., 2017).

Original: The self-attention policy network encodes the features x g and current selection features x s, 1: into attention weights over the set of selectable entities.
Revised: The self-attention policy network encodes the features x g and current selection features x s, 1: into attention weights over the set of selectable entities.

Original: These attention weights α i act as the logits to the action distribution over the set of selectable entities e i at each time step:
Revised: These attention weights α i act as the logits to the action distribution over the set of selectable entities e i at each time step:

Original: [Equation]
Revised: [Equation]

Original: j e α j , where x table are learned representations of selectable entities.
Revised: j e α j , where x table are learned representations of selectable entities.

Original: In order to predict value estimates, we ﬁrst apply a linear transform W to x table and encode the state as an attentionweighted sum of these linearly transformed entity features.
Revised: In order to predict value estimates, we ﬁrst apply a linear transform W to x table and encode the state as an attentionweighted sum of these linearly transformed entity features.

Original: This encoding is passed through a fully-connected layer to yield value estimates V ( x ) =
Revised: This encoding is passed through a fully-connected layer to yield value estimates V ( x ) =

Original: MLP ( α (cid:62) W ( x table )) .
Revised: MLP ( α (cid:62) W ( x table )) .

Original: 4.2. Link Prediction Model
Revised: 4.2. Link Prediction Model

Original: We assume access to a link prediction model L trained on an external knowledge graph that contains information relevant to the RL task.
Revised: We assume access to a link prediction model L trained on an external knowledge graph that contains information relevant to the RL task.

Original: While our model makes use of ComplEx for link prediction (Trouillon et al., 2016), in general this choice can be substituted by any other link prediction model.
Revised: While our model makes use of ComplEx for link prediction (Trouillon et al., 2016), in general this choice can be substituted by any other link prediction model.

Original: We chose ComplEx for its reliable performance in practice.
Revised: We chose ComplEx for its reliable performance in practice.

Original: The link prediction model is trained on a set of subjectobject-predicate relation triplets of the form ( s, p, o ) to predict higher likelihood scores for true relation triplets.
Revised: The link prediction model is trained on a set of subjectobject-predicate relation triplets of the form ( s, p, o ) to predict higher likelihood scores for true relation triplets.

Original: In the context of WordCraft, each recipe { e 1 , e 2 } −→ e 3 reduces to four relevant relation triplets: ( e 1 , combinesWith , e 2 ) , ( e 2 , combinesWith , e 1 ) , ( e 1 , componentOf , e 3 ) , and ( e 2 , componentOf , e 2 ) .
Revised: In the context of WordCraft, each recipe { e 1 , e 2 } −→ e 3 reduces to four relevant relation triplets: ( e 1 , combinesWith , e 2 ) , ( e 2 , combinesWith , e 1 ) , ( e 1 , componentOf , e 3 ) , and ( e 2 , componentOf , e 2 ) .

Original: These triplets, when aggregated across all recipes, forms the recipe graph.
Revised: These triplets, when aggregated across all recipes, forms the recipe graph.

Original: In our experiments, we train ComplEx on a subgraph of the recipe graph containing all nodes to ensure full entity coverage.
Revised: In our experiments, we train ComplEx on a subgraph of the recipe graph containing all nodes to ensure full entity coverage.

Original: At each time step, we compute relation score vectors u and v between the goal and selected entities and each e i on the table and e s,j in the selection: u i =
Revised: At each time step, we compute relation score vectors u and v between the goal and selected entities and each e i on the table and e s,j in the selection: u i =

Original: L ( e i , combinesWith , e s,j ) , v i =
Revised: L ( e i , combinesWith , e s,j ) , v i =

Original: L ( e i , componentOf , e g ) .
Revised: L ( e i , componentOf , e g ) .

Original: These scores are then mixed component-wise with the attention weights, before passing the mixed weights into the softmax policy head, thereby guiding the agent toward relevant predictions from the KG model.
Revised: These scores are then mixed component-wise with the attention weights, before passing the mixed weights into the softmax policy head, thereby guiding the agent toward relevant predictions from the KG model.

Original: Details on learning the mixing coefﬁcients are provided in Appendix B.1.
Revised: Details on learning the mixing coefﬁcients are provided in Appendix B.1.

Original: 5. Experiments
Revised: 5. Experiments

Original: Our experiments focus on zero-shot generalization performance.
Revised: Our experiments focus on zero-shot generalization performance.

Original: We split the set of all valid recipes into train (80%) and test (20%) sets.
Revised: We split the set of all valid recipes into train (80%) and test (20%) sets.

Original: Train tasks are generated such that task goals do not involve any recipes in the test set.
Revised: Train tasks are generated such that task goals do not involve any recipes in the test set.

Original: The model is trained using the TorchBeast implementation of
Revised: The model is trained using the TorchBeast implementation of

Original: IMPALA (Espeholt et al., 2018; K ¨ uttler et al., 2019).
Revised: IMPALA (Espeholt et al., 2018; K ¨ uttler et al., 2019).

Original: 5.1. WordCraft Benchmarks
Revised: 5.1. WordCraft Benchmarks

Original: To demonstrate how well the task captures real-world relationships between entities, we represent entities as GloVe embeddings (Pennington et al., 2014) when evaluating the model in Section 4.1.
Revised: To demonstrate how well the task captures real-world relationships between entities, we represent entities as GloVe embeddings (Pennington et al., 2014) when evaluating the model in Section 4.1.

Original: We expect GloVe to help generalization if semantically similar entities have similar uses in this environment.
Revised: We expect GloVe to help generalization if semantically similar entities have similar uses in this environment.

Original: We assess the zero-shot performance of our agent model without a link prediction module on depthtasks with 1 and 8 distractors.
Revised: We assess the zero-shot performance of our agent model without a link prediction module on depthtasks with 1 and 8 distractors.

Original: We also collect a human baseline at the same difﬁculty settings of WordCraft.
Revised: We also collect a human baseline at the same difﬁculty settings of WordCraft.

Original: This human baseline serves as an estimate of the zero-shot performance that can be achieved using commonsense and general knowledge (see the Appendix
Revised: This human baseline serves as an estimate of the zero-shot performance that can be achieved using commonsense and general knowledge (see the Appendix

Original: C.5 for the description of human evaluation protocol).
Revised: C.5 for the description of human evaluation protocol).

Original: [Figure]
Revised: [Figure]

Original: 3M

Original: Steps 0.

Original: M ean ep i s ode re t u r n
Revised: ean ep i s ode r

Original: Self-attention

Original: [Figure]
Revised: [Figure]

Original: GloVe
Revised: t u r

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 1 distractor Self-attention + random
Revised: ean ep i

Original: [Figure]
Revised: [Figure]

Original: 1 distractor
Revised: ode r e t

Original: Self
Revised: u r n

Original: [Figure]
Revised: [Figure]

Original: As seen in Figure 2, agents trained with GloVe embeddings generalize much better to tasks with recipes that were not part of any training task.
Revised: As seen in Figure 2, agents trained with GloVe embeddings generalize much better to tasks with recipes that were not part of any training task.

Original: This margin of improvement persists in harder task settings, even as the test success rate falls.
Revised: This margin of improvement persists in harder task settings, even as the test success rate falls.

Original: 5.2. Knowledge Graph Benchmarks
Revised: 5.2. Knowledge Graph Benchmarks

Original: We benchmark variants of our full agent with the link prediction module and compare zero-shot performance under different combinations of link prediction scores used to modulate the self-attention weights: both combinesWith and componentOf scores, or each score in isolation.
Revised: We benchmark variants of our full agent with the link prediction module and compare zero-shot performance under different combinations of link prediction scores used to modulate the self-attention weights: both combinesWith and componentOf scores, or each score in isolation.

Original: In these experiments, L is trained on the same train set of recipes used to generate the training tasks.
Revised: In these experiments, L is trained on the same train set of recipes used to generate the training tasks.

Original: Our results show that access to a full KG model drastically improves the agent’s zero-shot performance (see Figure 3).
Revised: Our results show that access to a full KG model drastically improves the agent’s zero-shot performance (see Figure 3).

Original: However, while a partial-KG-model agent reaches an equivalent zero-shot success rate as an agent without any KG model in fewer training steps, they ultimately reach comparable levels of test performance as training progresses (see
Revised: However, while a partial-KG-model agent reaches an equivalent zero-shot success rate as an agent without any KG model in fewer training steps, they ultimately reach comparable levels of test performance as training progresses (see

Original: Appendix for a more detailed discussion).
Revised: Appendix for a more detailed discussion).

Original: 6. Related Work
Revised: 6. Related Work

Original: Recently, several new environments incorporating a degree of commonsense dynamics and text-base corpora have been proposed.
Revised: Recent work proposes several new environments incorporating a degree of commonsense dynamics and text-base corpora.

Original: For example, TextWorld (Cˆot´e et al., 2018) is a framework that enables creation of novel textual RL environments as well as interfacing with existing text-based games, such as Zork.
Revised: For example, TextWorld (Cˆot´e et al., 2018) is a framework that enables creation of novel textual RL environments as well as interfacing with existing text-based games, such as Zork.

Original: Chevalier-Boisvert et al.
Revised: Chevalier-Boisvert et al.

Original: K¨uttler et al.
Revised: K¨uttler et al.

Original: (2020) respectively propose BabyAI and NLE, gridworld environments that evaluate agents on procedurallygenerated, goal-conditioned tasks with varying degrees of commonsense structure and available textual corpora.
Revised: (2020) respectively propose BabyAI and NLE, grid-world environments that evaluate agents on procedurally-generated, goal-conditioned tasks with varying degrees of commonsense structure and available textual corpora.

Original: While these environments are challenging in their complex dynamics, large action spaces, sparse rewards, and long planning horizons, these same aspects confound the ability to isolate[Figure] the problem of conditioning on external knowledge sources.
Revised: While these environments are challenging in their complex dynamics, large action spaces, sparse rewards, and long planning horizons, these same aspects confound the ability to isolate the problem of conditioning on external knowledge sources.

Original: A comprehensive survey of a related line of research that conditions policies on knowledge in textual corpora or knowledge bases can be found in Luketina et al.
Revised: Luketina et al. (2019) presents a comprehensive survey of a related line of research conditioning policies on knowledge in textual corpora or knowledge bases.

Original: Fulda et al.
Revised: Fulda et al.

Original: ; Zahavy et al.
Revised: Zahavy et al.

Original: (2018) demonstrate that word embeddings can be an effective modeling tool for policies, while Branavan et al.
Revised: (2018) demonstrate that word embeddings can be an effective modeling tool for policies, while Branavan et al.

Original: ; Narasimhan et al.
Revised: ; Narasimhan et al.

Original: ; Zhong et al.
Revised: ; Zhong et al.

Original: (2020) present work that directly conditions policies on domain knowledge encoded in textual corpora.
Revised: (2020) present work that directly conditions policies on domain knowledge encoded in textual corpora.

Original: Marzoev et al.
Revised: Marzoev et al.

Original: ; Hill et al.
Revised: ; Hill et al.

Original: (2020) show that language models can enable generalization to unseen instructions, which indicates that linguistic knowledge can be successfully transferred to aid downstream RL tasks.
Revised: (2020) show that language models can enable generalization to unseen instructions, which indicates that linguistic knowledge can be successfully transferred to aid downstream RL tasks.

Original: Models integrating KGs with agents have been studied in the literature on text-based games (Ammanabrolu & Riedl, 2019; Adhikari et al., 2020).
Revised: Models integrating KGs with agents have been studied in the literature on text-based games (Ammanabrolu & Riedl, 2019; Adhikari et al., 2020).

Original: Closest to our work, Murugesan et al.
Revised: Closest to our work, Murugesan et al.

Original: (2020) concurrently propose solving a set of
Revised: (2020) concurrently propose solving a set of

Original: TextWorld tasks using a model that incorporates a combination of commonsense knowledge, encoded in ConceptNet, and a learned KG representing the current environment state.
Revised: TextWorld tasks using a model that incorporates a combination of commonsense knowledge, encoded in ConceptNet, and a learned KG representing the current environment state.

Original: They ﬁnd that while at times beneﬁcial, KGs can also provide too strong a prior.
Revised: They ﬁnd that while at times beneﬁcial, KGs can also provide too strong a prior.

Original: 7. Future Work
Revised: 7. Future Work

Original: There are multiple avenues that we plan to further explore.
Revised: There are multiple avenues that we plan to further explore.

Original: Extending WordCraft to the longer horizon setting of the original Little Alchemy 2, in which the user must discover as many entities as possible, could be an interesting setting to study commonsense-driven exploration.
Revised: Extending WordCraft to the longer horizon setting of the original Little Alchemy 2, in which the user must discover as many entities as possible, could be an interesting setting to study commonsense-driven exploration.

Original: We also plan to introduce additional modalities such as images and control into WordCraft.
Revised: We also plan to introduce additional modalities such as images and control into WordCraft.

Original: Furthermore, we believe the ideas in this work could beneﬁt more complex RL tasks associated with large corpora of task-speciﬁc knowledge, such as NLE.
Revised: Furthermore, we believe the ideas in this work could beneﬁt more complex RL tasks associated with large corpora of task-speciﬁc knowledge, such as NLE.

Original: This path of research entails further investigation of methods for automatically constructing knowledge graphs from available corpora as well as agents that retrieve and directly condition on natural language texts in such corpora.
Revised: This path of research entails further investigation of methods for automatically constructing knowledge graphs from available corpora as well as agents that retrieve and directly condition on natural language texts in such corpora.

Original: A. Background
Revised: A. Background

Original: Knowledge Graphs Knowledge graphs naturally represent the ontology of concepts underlying commonsense knowledge about the world.
Revised: Knowledge Graphs Knowledge graphs naturally represent the ontology of concepts underlying commonsense knowledge about the world.

Original: In a knowledge graph, nodes represent concepts, and edges between two nodes represent a relationship between the corresponding concepts.
Revised: In a knowledge graph, nodes represent concepts, and edges between two nodes represent a relationship between the corresponding concepts.

Original: Let E and R denote a set of entities and a set of relation types, respectively.
Revised: Let E and R denote a set of entities and a set of relation types, respectively.

Original: Formally, a KG G ⊆
Revised: Formally, a KG G ⊆

Original: E × R × E is a set of subject-predicate-object triples (cid:104) s, p, o (cid:105) , with s, o ∈ E and p ∈ R , where each triple encodes a relationship of type p between the subject s and the object o .
Revised: E × R × E is a set of subject-predicate-object triples (cid:104) s, p, o (cid:105) , with s, o ∈ E and p ∈ R , where each triple encodes a relationship of type p between the subject s and the object o .

Original: There are several examples of large-scale KGs encoding commonsense knowledge, both from academia (such as YAGO (Suchanek et al., 2007), DBpedia (Auer et al., 2007)) and industry (such as
Revised: There are several examples of large-scale KGs encoding commonsense knowledge, both from academia (such as YAGO (Suchanek et al., 2007), DBpedia (Auer et al., 2007)) and industry (such as

Original: Microsoft’s Bing Knowledge Graph and the Google Knowledge Graph (Noy et al., 2019)).
Revised: Microsoft’s Bing Knowledge Graph and the Google Knowledge Graph (Noy et al., 2019)).

Original: Such large-scale knowledge graphs hold an enormous amount of useful prior knowledge about the world, that should be useful for guiding an RL agent in any environment with real-world semantics.
Revised: Such large-scale knowledge graphs hold an enormous amount of useful prior knowledge about the world, that should be useful for guiding an RL agent in any environment with real-world semantics.

Original: Especially relevant to this work are ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), two KGs encoding common-sense knowledge.
Revised: Especially relevant to this work are ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017), two KGs encoding common-sense knowledge.

Original: Representation Learning of KGs An effective way of enabling statistical learning on KGs consists of learning continuous, distributed representations, also referred to as embeddings , for all entities in a KG.
Revised: Representation Learning of KGs An effective way of enabling statistical learning on KGs consists of learning continuous, distributed representations, also referred to as embeddings , for all entities in a KG.

Original: We refer to Nickel et al.
Revised: We refer to Nickel et al.

Original: (2016) for a recent survey on this topic.
Revised: (2016) for a recent survey on this topic.

Original: In this work, we use the ComplEx (Trouillon et al., 2016) link prediction model, with the loss functions and regularizers proposed by
Revised: In this work, we use the ComplEx (Trouillon et al., 2016) link prediction model, with the loss functions and regularizers proposed by

Original: Lacroix et al.
Revised: Lacroix et al.

Original: Given a subject-predicate-object triple (cid:104) s,
Revised: Given a subject-predicate-object triple (cid:104) s,

Original: p, o (cid:105) ∈
Revised: p, o (cid:105) ∈

Original: E × R × E , ComplEx deﬁnes the score φ ( s, p, o ) of such a triple as:
Revised: E × R × E , ComplEx deﬁnes the score φ ( s, p, o ) of such a triple as:

Original: [Equation]
Revised: [Equation]

Original: o
Revised: o

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:104)· , · , ·(cid:105) denotes the tri-linear dot product, e s , e p , e o ∈ C k denote the complex-valued k -dimensional representations of s , p , and o , x denote the complex conjugate of x ∈ C k , Re ( x ) is the real part of x .
Revised: where (cid:104)· , · , ·(cid:105) denotes the tri-linear dot product, e s , e p , e o ∈ C k denote the complex-valued k -dimensional representations of s , p , and o , x denote the complex conjugate of x ∈ C k , Re ( x ) is the real part of x .

Original: In order to learn the embedding representations Θ for all entities and relation types in a KG G , we follow Lacroix et al.
Revised: In order to learn the embedding representations Θ for all entities and relation types in a KG G , we follow Lacroix et al.

Original: and minimise the following objective via gradient-based optimisation:
Revised: and minimise the following objective via gradient-based optimisation:

Original: [Equation]
Revised: [Equation]

Original: regularized via the weighted nuclear p -norm regularizer proposed by Lacroix et al.
Revised: regularized via the weighted nuclear p -norm regularizer proposed by Lacroix et al.

Original: B. Model Details B.1. Learning Mixing Coefﬁcients
Revised: B. Model Details B.1. Learning Mixing Coefﬁcients

Original: The link-prediction scores between goal and table entities for the componentOf relation and between each selected entity and table entities for the combinesWith relation are combined component-wise with the self-attention weights
Revised: The link-prediction scores between goal and table entities for the componentOf relation and between each selected entity and table entities for the combinesWith relation are combined component-wise with the self-attention weights

Original: α i using mixing coefﬁcients to yield ﬁnal policy logits β :
Revised: α i using mixing coefﬁcients to yield ﬁnal policy logits β :

Original: [Equation]
Revised: [Equation]

Original: The mixing coefﬁcients λ are computed by passing the selfattention-weighted table encoding through a fully-connected layer:
Revised: The mixing coefﬁcients λ are computed by passing the selfattention-weighted table encoding through a fully-connected layer:

Original: [Equation]
Revised: [Equation]

Original: C. Experiments C.1. Self-Attention Actor-Critic Architecture
Revised: C. Experiments C.1. Self-Attention Actor-Critic Architecture

Original: used
Revised: used

Original: C.2. Training Parameters
Revised: C.2. Training Parameters

Original: [Table]
Revised: [Table]

Original: C.3. Model Parameters
Revised: C.3. Model Parameters

Original: [Table]
Revised: [Table]

Original: C.4. Extended Discussion
Revised: C.4. Extended Discussion

Original: In addition to initial benchmarks conducted in 5.
Revised: In addition to initial benchmarks conducted in 5.

Original: [Table]
Revised: [Table]

Original: we also compare the performance of the self-attention actor-critic agent described in 4.1 to a simple, single-layer MLP that predicts the policy logits and value estimates from the concatenation of all GloVe word embeddings of goal, selection, and table entities at each time step.
Revised: we also compare the performance of the self-attention actor-critic agent described in 4.1 to a simple, single-layer MLP that predicts the policy logits and value estimates from the concatenation of all GloVe word embeddings of goal, selection, and table entities at each time step.

Original: These results are presented in Fig.
Revised: These results are presented in Fig.

Original: [Table]
Revised: [Table]

Original: We further benchmark the performance of the self-attention actor-critic agent with and without access to an instance of
Revised: We further benchmark the performance of the self-attention actor-critic agent with and without access to an instance of

Original: ComplEx trained on a subgraph of the full recipe graph.
Revised: ComplEx trained on a subgraph of the full recipe graph.

Original: We compare agents trained with access to ComplEx trained on a partial graph and those trained on the full graph.
Revised: We compare agents trained with access to ComplEx trained on a partial graph and those trained on the full graph.

Original: The partial
Revised: The partial

Original: KG models are trained on only the relations present in the set of training recipes.
Revised: KG models are trained on only the relations present in the set of training recipes.

Original: As the method for incorporating ComplEx’s predicted relation scores into the policy prediction performs a weighted sum of each score with the self-attention weights, we investigate the effect of only using different subsets of the scores.
Revised: As the method for incorporating ComplEx’s predicted relation scores into the policy prediction performs a weighted sum of each score with the self-attention weights, we investigate the effect of only using different subsets of the scores.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 6 and Fig.
Revised: 6 and Fig.

Original: [Table]
Revised: [Table]

Original: the subset of relation scores that are helpful to policy learning varies across task depth and distractor settings.
Revised: the subset of relation scores that are helpful to policy learning varies across task depth and distractor settings.

Original: We generally ﬁnd that access to a full KG model drastically improves the agent’s zero-shot success rate, as might be expected.
Revised: We generally ﬁnd that access to a full KG model drastically improves the agent’s zero-shot success rate, as might be expected.

Original: In fact, one might expect that having access
Revised: In fact, one might expect that having access

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: to the ground-truth recipe graph as encoded by ComplEx should enable the agent to achieve 100% zero-shot success rates across all task settings.
Revised: to the ground-truth recipe graph as encoded by ComplEx should enable the agent to achieve 100% zero-shot success rates across all task settings.

Original: In order to test the extent to which the full-KG ComplEx scores are predictive of the best actions
Revised: In order to test the extent to which the full-KG ComplEx scores are predictive of the best actions

Original: [Figure]
Revised: [Figure]

Original: we also assessed the performance of agents whose ﬁnal policy logit mixture only assigned positive weights to different subsets of the relation scores.
Revised: we also assessed the performance of agents whose ﬁnal policy logit mixture only assigned positive weights to different subsets of the relation scores.

Original: Such full-KG-only agents that choose items only based on the combinesWith relation score will still often choose the same entity twice, believing the entity combines with itself.
Revised: Such full-KG-only agents that choose items only based on the combinesWith relation score will still often choose the same entity twice, believing the entity combines with itself.

Original: This is likely because the recipe graph includes several entities that combine with themselves, while the set of all self-combination triplets are not included in the training set
Revised: This is likely because the recipe graph includes several entities that combine with themselves, while the set of all self-combination triplets are not included in the training set

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Full-KG-only agents that choose items only based on the componentOf relation score will tend to choose the same item twice, as this approach will always assign the highest action probability to selecting the entity on the table that
Revised: Full-KG-only agents that choose items only based on the componentOf relation score will tend to choose the same item twice, as this approach will always assign the highest action probability to selecting the entity on the table that

Original: ComplEx predicts to be most likely to be a component of the goal entity.
Revised: ComplEx predicts to be most likely to be a component of the goal entity.

Original: Interestingly, partial-KG-agents do not always do much better than a KG-free agent.
Revised: Interestingly, partial-KG-agents do not always do much better than a KG-free agent.

Original: This implies that as the KGfree agent sees more training tasks, it becomes comparable to ComplEx in predicting unseen relations among entities.
Revised: This implies that as the KGfree agent sees more training tasks, it becomes comparable to ComplEx in predicting unseen relations among entities.

Original: Early on in training, the relation scores predicted by ComplEx seem to help the agent achieve higher zero-shot success rates in fewer training steps compared to KG-free agents (see Figure 7).
Revised: Early on in training, the relation scores predicted by ComplEx seem to help the agent achieve higher zero-shot success rates in fewer training steps compared to KG-free agents (see Figure 7).

Original: C.5. Human Evaluation Protocol
Revised: C.5. Human Evaluation Protocol

Original: We estimate the generalization ability of humans players on depth-1 tasks with 1 and 8 distractors.
Revised: We estimate the generalization ability of humans players on depth-1 tasks with 1 and 8 distractors.

Original: Each difﬁculty setting is tested on a different individual, and we test one individual per setting.
Revised: Each difﬁculty setting is tested on a different individual, and we test one individual per setting.

Original: The selected individuals were not familiar with the game and got to train on 40 randomly sampled training tasks before being evaluated on 40 testing tasks.
Revised: The selected individuals were not familiar with the game and got to train on 40 randomly sampled training tasks before being evaluated on 40 testing tasks.

Original: The train and test split was the same as in the RL experiments.
Revised: The train and test split was the same as in the RL experiments.

Original: Surprisingly, we found that changing the number of distractors from 1 to 8 did not signiﬁcantly inﬂuence human performance.
Revised: Surprisingly, we found that changing the number of distractors from 1 to 8 did not signiﬁcantly inﬂuence human performance.

Original: We plan to conduct a larger-scale human performance evaluation.
Revised: We plan to conduct a larger-scale human performance evaluation.

Original: D. Example recipes
Revised: D. Example recipes

Original: [Table]
Revised: [Table]