Original: I NVESTIGATING H UMAN P RIORS FOR P LAYING V IDEO G AMES
Revised: I NVESTIGATING H UMAN P RIORS FOR P LAYING V IDEO G AMES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning algorithms have recently achieved impressive results on a range of video games, yet they remain much less efﬁcient than an average human player at learning a new game.
Revised: Deep reinforcement learning algorithms have recently achieved impressive results on a range of video games, yet they remain much less efﬁcient than an average human player at learning a new game.

Original: What makes humans so good at solving these video games?
Revised: What makes humans so good at solving these video games?

Original: Here, we study one aspect critical to human gameplay – their use of strong priors that enables efﬁcient decision making and problem solving.
Revised: Here, we study one aspect critical to human gameplay – their use of strong priors that enables efﬁcient decision making and problem solving.

Original: We created a sample video game and conducted various experiments to quantify the kinds of prior knowledge humans bring in while playing such games.
Revised: We created a sample video game and conducted various experiments to quantify the kinds of prior knowledge humans bring in while playing such games.

Original: We do this by modifying the video game environment to systematically remove different types of visual information that could be used by humans as priors.
Revised: We do this by modifying the video game environment to systematically remove different types of visual information that could be used by humans as priors.

Original: We ﬁnd that human performance degrades drastically once prior information has been removed, while that of an RL agent does not change.
Revised: We ﬁnd that human performance degrades drastically once prior information has been removed, while that of an RL agent does not change.

Original: Interestingly, we also ﬁnd that general priors about objects that humans learn when they are as little as 2 months old are some of the most critical priors that help in human gameplay.
Revised: Interestingly, we also ﬁnd that general priors about objects that humans learn when they are as little as 2 months old are some of the most critical priors that help in human gameplay.

Original: Based on these ﬁndings, we then propose a taxonomy of object priors people employ when solving video games that can potentially serve as a benchmark for future reinforcement learning algorithms aiming to incorporate human-like representations in their systems.
Revised: Based on these ﬁndings, we then propose a taxonomy of object priors people employ when solving video games that can potentially serve as a benchmark for future reinforcement learning algorithms aiming to incorporate human-like representations in their systems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider the following scenario: you are tasked to play an unfamiliar computer game shown in
Revised: Consider the following scenario: you are tasked to play an unfamiliar computer game shown in

Original: Figure 1(a).
Revised: Figure 1(a).

Original: No manual or instructions are provided.
Revised: No manual or instructions are provided.

Original: You do not know  the goal  or even what sprite on the game screen is your character.
Revised: You don’t know what the goal is or which game sprite is controlled by you.

Original: How quickly can you ﬁnish this game?
Revised: How quickly can you ﬁnish this game?

Original: We recruited forty subjects to play this game and found that subjects solved the game comfortably (taking justactions, corresponding to < 1 minute of gameplay, c.f. 
Revised: We recruited forty subjects to play this game and found that subjects solved the game quite easily (taking just 1600 actions and 1 minute of gameplay, c.f. Figure 1(c)).

Original: Figure 1(c)).
Revised: This is not overly surprising as one could easily guess that the goal of the game is to move the robot sprite towards the princess by stepping on the brick-like objects and using ladders to reach the higher platforms while also avoiding the angry purple sprite and the ﬁre object.

Original: Now consider the scenario in which the same game is rendered with new textures as shown in Figure 1(b).
Revised: Now consider a second scenario in which this same game is re-rendered with new textures, getting rid of semantic cues, as shown in Figure 1(b).

Original: How easy is it to play this game?
Revised: How would human performance change?

Original: A separate group of fortyhuman subjects took twice as many actions  to solve the second game  ascompared to the ﬁrst game (Figure 1(c)). Even though the placement of different object sprites and the reward structure is preserved across games, humans ﬁnd it harder to complete the second game.
Revised: We recruited another forty subjects to play this game and found that the average number of actions taken by players to solve the second game is twice as many as the ﬁrst game (Figure 1(c)). This game is clearly much harder for humans.

Original: In a second suite of experiments, we trained a state-of-the-art reinforcement learning (RL) agent (ICM-A3C; Pathak et al.
Revised: How would a reinforcement learning agent perform on the two games? We trained a state-of-the-art RL agent (ICM-A3C; Pathak et al.

Original: on both these games and found that the RL agent  took close to four million steps to solve boththe games (Figure 1(c)).In stark contrast to humans, the RL agents were virtually unaffected by rendering differences between the two games. Why is it the case that humans become signiﬁcantly worse in playing the second game, whereas the RL agents are indifferent?
Revised: on both these games and found that the RL agent was virtually unaffected – it took close to four million steps to solve both games (Figure 1(c)).

Original: In the ﬁrst game, it is easy for humans to infer that the goal of the game is to move the robot sprite towards the princess by stepping on the brick-like objects and using ladders to reach the higher platforms while also avoiding the angry purple sprite and the ﬁre object.

Original: On the other hand, in the second game, it is impossible for a human to infer just by looking at the game screen, which part of the screen shows the player sprite, let alone ﬁguring out the goal of the game.

Original: In contrast to human players, thecurrent state-of-the-art RL agents come tabula rasa i.e. without any prior knowledge about the world and thus both these games were more or less the same  from the perspective of theRL agent leading to no change in performance across games.
Revised: Since the RL agent came tabula rasa i.e. without any prior knowledge about the world, both these games carried the same amount of information from the perspective of the agent leading to no change in its performance.

Original: This simple experiment highlights the importance of prior knowledge that humans draw upon to quickly solve tasks given to them.
Revised: This simple experiment highlights the importance of prior knowledge that humans draw upon to quickly solve tasks given to them (Lake et al., 2016; Tsividis et al., 2017).

Original: Building RL algorithms that can quickly solve tasks (i.e., sample efﬁcient algorithms) is an active area of research, and further progress is inevitable. In addition to developing better optimization methods, we believe that instead ofalways initializing learning from scratch, either incorporating prior knowledge directly or constructing mechanisms for condensing experience into re-usable knowledge (i.e., learning priors) might be critical for building RL agents with human like efﬁciency. What priors should we incorporate in our RL agents? For inspiration, it is natural to study what priors humans use.
Revised: While the form of prior information tested above may be obvious, people bring in a wealth of prior information about the physical world that goes beyond simple knowledge about platforms, ladders, princess, monsters, etc.

Original: Developmental psychologists have been documentingthe prior knowledge that children draw upon in learning about the world (Spelke & Kinzler, 2007; Carey, 2009).
Revised: Developmental psychologists have been documenting prior knowledge that children draw upon in learning about the world (Spelke & Kinzler, 2007; Carey, 2009).

Original: However, these studies have not explicitly quantiﬁed how vital are different priors for problem-solving.
Revised: However, these studies have not explicitly quantiﬁed how vital are different priors for problem-solving.

Original: In this work, we have systematically quantiﬁed the importance of various priors humans bring to use while solving one particular kind of problems – video games.
Revised: In this work, we have systematically quantiﬁed the importance of various priors humans bring to bear while solving one particular kind of problem – video games.

Original: We chose video games as the task for our investigation because it is easy to systematically change the game to include or mask different kinds of knowledge, run large-scale human studies, and video games such as ATARI are a popular choice in the reinforcement learning community.
Revised: We chose video games as the task for our investigation because it is easy to systematically change the game to include or mask different kinds of knowledge, run large-scale human studies, and video games such as ATARI are a popular choice in the reinforcement learning community.

Original: One of the ﬁndings of our investigation is that while knowledge of the form that ladders are to be climbed, keys are used to open doors, jumping on spikes is dangerous is important for humans to quickly solve games, more general priors of the form that the objects are subgoals for exploration and things that look the same behave the same are even more critical.
Revised: One of the ﬁndings of our investigation is that while knowledge of the form that ladders are to be climbed, keys are used to open doors, jumping on spikes is dangerous is important for humans to quickly solve games, more general priors of the form that the objects are subgoals for exploration and things that look the same behave the same are even more critical.

Original: Although we use video games as our experimental test bed, such priors are more generally applicable even outside the domain of video games.
Revised: Although we use video games as our experimental test bed, such priors are more generally applicable even outside the domain of video games.

Original:  M ETHOD
Revised:  M ETHOD

Original: To investigate the aspects of visual information that enable humans to efﬁciently solve video games, we designed a browser based platform game consisting of a human sprite that could be controlled, platforms, ladders, slimy pink sprites that killed the agent, spikes that were dangerous to jump on, a key, and a door (see Figure 2 (a)).
Revised: To investigate the aspects of visual information that enable humans to efﬁciently solve video games, we designed a browser based platform game consisting of a human sprite that could be controlled, platforms, ladders, slimy pink sprites that killed the agent, spikes that were dangerous to jump on, a key, and a door (see Figure 2 (a)).

Original: The human sprite could be moved with help of arrow keys and the agent obtained a reward of +1 when it reached the door after taking the key thereby terminating the game.
Revised: The human sprite could be moved with help of arrow keys and the agent obtained a reward of +1 when it reached the door after taking the key thereby terminating the game.

Original: The game was reset whenever the agent touched the enemy, jumped on the spike, or fell below the lowermost platform.
Revised: The game was reset whenever the agent touched the enemy, jumped on the spike, or fell below the lowermost platform.

Original: We made this game to resemble the exploration challenges faced in the classic ATARI game of Montezuma’s revenge that has proven to be very challenging for state of deep reinforcement learning techniques (Bellemare et al., 2016; Mnih et al., 2015).
Revised: We made this game to resemble the exploration challenges faced in the classic ATARI game of Montezuma’s revenge that has proven to be very challenging for state of deep reinforcement learning techniques (Bellemare et al., 2016; Mnih et al., 2015).

Original: We systematically created different versions of this game by re-rendering various entities such as ladders, enemies, keys, platforms etc.
Revised: We systematically created different versions of this game by re-rendering various entities such as ladders, enemies, keys, platforms etc.

Original: using alternate textures (see Figure 2).
Revised: using alternate textures (see Figure 2).

Original: These textures were chosen to mask various forms of prior knowledge that are described in the experiments section.
Revised: These textures were chosen to mask various forms of prior knowledge that are described in the experiments section.

Original: Our experiment style draws inspiration from the neuroscience literature wherein researchers study aspects about the human brain by performing lesion studies (M ¨ uller & Knight (2006), Shi & Davis (1999)).
Revised: Our experiment style draws inspiration from the neuroscience literature wherein researchers study aspects about the human brain by performing lesion studies (M ¨ uller & Knight (2006), Shi & Davis (1999)).

Original: For the purposes of our experiment, since it was not possible to go directly inside participant’s brain in order to study the importance of various priors, we did the next best thing possiblemask those priors.
Revised: For the purposes of our experiment, since it was not possible to go directly inside participant’s brain in order to study the importance of various priors, we did the next best thing possiblemask those priors.

Original: For each version of the game created, we quantiﬁed human performance by recruiting 120 participants from Amazon Mechanical Turk.
Revised: For each version of the game created, we quantiﬁed human performance by recruiting 120 participants from Amazon Mechanical Turk.

Original: Each participant was instructed to use the arrow keys to move and ﬁnish the game as soon as possible.
Revised: Each participant was instructed to use the arrow keys to move and ﬁnish the game as soon as possible.

Original: No information about the goals or the reward structure of the game was communicated to the participants.
Revised: No information about the goals or the reward structure of the game was communicated to the participants.

Original: Each participant was paid $1 for successfully ﬁnishing the game.
Revised: Each participant was paid $1 for successfully ﬁnishing the game.

Original: The maximum time for allowed for playing the game was set to 30 minutes.
Revised: The maximum time for allowed for playing the game was set to 30 minutes.

Original: For each participant we recorded the ( x, y ) position of the player at every step of the game, the total time taken by the participant to ﬁnish the game (i.e. achieve the reward of +1) and the total number of deaths prior to ﬁnishing the game.
Revised: For each participant we recorded the ( x, y ) position of the player at every step of the game, the total time taken by the participant to ﬁnish the game (i e. achieve the reward of +1) and the total number of deaths prior to ﬁnishing the game.

Original: We used this data to quantify participant’s performance.
Revised: We used this data to quantify participant’s performance.

Original: Note that, we didn’t repeat participants and thus no participant played more than one instance of the game.
Revised: Note that, we didn’t repeat participants and thus no participant played more than one instance of the game.

Original:  Q UANTIFYING THE IMPORTANCE OF OBJECT PRIORS
Revised:  Q UANTIFYING THE IMPORTANCE OF OBJECT PRIORS

Original: The ﬁrst version of the game is shown in Figure 2(a),game link.
Revised: The ﬁrst version of the game is shown in Figure 2(a),game link.

Original: From a single glance at the game, human players can employ their prior knowledge to interpret that the game agent can climb on ladders, it is supported by platforms, the pink slimy sprite is dangerous, spikes are to be avoided and probably the goal of the game is to take the key to open the door.
Revised: From a single glance at the game, human players can employ their prior knowledge to interpret that the game agent can climb on ladders, it is supported by platforms, the pink slimy sprite is dangerous, spikes are to be avoided and probably the goal of the game is to take the key to open the door.

Original: As expected such interpretations enable humans to quickly solve the game.
Revised: As expected such interpretations enable humans to quickly solve the game.

Original: Figure 3(a) shows that the average time taken to complete the game is 1 . 8 minutes (blue bar) and the average number of deaths (orange bar) and game states visited by humans (yellow bar) are quite small.
Revised: Figure 3(a) shows that the average time taken to complete the game is 1 . 8 minutes (blue bar) and the average number of deaths (orange bar) and game states visited by humans (yellow bar) are quite small.

Original: 3. S EMANTICS
Revised: 3. S EMANTICS

Original: To study importance of prior knowledge about object semantics, we rendered objects and ladders with blocks of uniform color as shown in Figure 2(b),game link.
Revised: To study importance of prior knowledge about object semantics, we rendered objects and ladders with blocks of uniform color as shown in Figure 2(b),game link.

Original: Thus, in this game manipulation, the appearance of objects conveys no information about their semantics.
Revised: Thus, in this game manipulation, the appearance of objects conveys no information about their semantics.

Original: Results in Figure 3(b), show that human players take more than twice the time, have higher number of deaths, and explore signiﬁcantly larger number of states (p-value < 0.01 for all measures) as compared to the original version of the game clearly demonstrating that lesioning semantics hurts human performance.
Revised: Results in Figure 3(b), show that human players take more than twice the time, have higher number of deaths, and explore signiﬁcantly larger number of states (p-value < 0.01 for all measures) as compared to the original version of the game clearly demonstrating that lesioning semantics hurts human performance.

Original: A natural next question is how do humans make use of semantic information?
Revised: A natural next question is how do humans make use of semantic information?

Original: One hypothesis is that knowledge of semantics enables humans to infer the latent reward structure of the game.
Revised: One hypothesis is that knowledge of semantics enables humans to infer the latent reward structure of the game.

Original: If this indeed is the case, then in the original game players should ﬁrst visit the key and then go to the door, while in the version of the game without semantics, players should not exhibit any such bias.
Revised: If this indeed is the case, then in the original game players should ﬁrst visit the key and then go to the door, while in the version of the game without semantics, players should not exhibit any such bias.

Original: We found that in the original game, where key and door were both visible, almost all 120 participants reached the key ﬁrst, while in the version with masked semantics only 42 out of 120 participants reached the key before the door (see Figure 4(a)).
Revised: We found that in the original game, where key and door were both visible, almost all 120 participants reached the key ﬁrst, while in the version with masked semantics only 42 out of 120 participants reached the key before the door (see Figure 4(a)).

Original: Further investigation into time taken by human players to reach the door after taking the key revealed that they take signiﬁcantly larger amount of time when the semantics are masked (see Figure 4(b)).
Revised: Further investigation into time taken by human players to reach the door after taking the key revealed that they take signiﬁcantly larger amount of time when the semantics are masked (see Figure 4(b)).

Original: This provides further evidence that humans are unable to infer the reward structure and consequently signiﬁcantly increase their exploration when semantics are masked.
Revised: This provides further evidence that humans are unable to infer the reward structure and consequently signiﬁcantly increase their exploration when semantics are masked.

Original: Note that, to rule out the possibility that increase in time is simply due to the fact players take more time to ﬁnish the game without semantics, the time to reach the door after taking the key was normalized by the total amount of the time that was spent by the player to complete the game.
Revised: Note that, to rule out the possibility that increase in time is simply due to the fact players take more time to ﬁnish the game without semantics, the time to reach the door after taking the key was normalized by the total amount of the time that was spent by the player to complete the game.

Original: To further quantify importance of semantics, instead of simply masking, we reversed the semantics.
Revised: To further quantify importance of semantics, instead of simply masking, we reversed the semantics.

Original: This condition further deteriorated human performance and the results are detailed in the appendix.
Revised: This condition further deteriorated human performance and the results are detailed in the appendix.

Original: 3. O BJECTS AS S UBGOALS FOR E XPLORATION
Revised: 3. O BJECTS AS S UBGOALS FOR E XPLORATION

Original: While blocks of uniform color in game shown in Figure 2(b) convey no semantics, they are distinct from the background and seem to attract human attention.
Revised: While blocks of uniform color in game shown in Figure 2(b) convey no semantics, they are distinct from the background and seem to attract human attention.

Original: It is possible that humans infer these distinct entities (or objects) as subgoals, which results in more efﬁcient exploration than random search.
Revised: It is possible that humans infer these distinct entities (or objects) as subgoals, which results in more efﬁcient exploration than random search.

Original: This leads to the hypothesis that humans have a prior to treat visually distinct entities as subgoals to guide exploration.
Revised: This leads to the hypothesis that humans have a prior to treat visually distinct entities as subgoals to guide exploration.

Original: In order to test this, we modiﬁed the game to cover each space on the platform with a block of different color to hide where the objects are (see Figure 2(c), game link).
Revised: In order to test this, we modiﬁed the game to cover each space on the platform with a block of different color to hide where the objects are (see Figure 2(c), game link).

Original: Note that most colored blocks are placebos and do not correspond to any object and the actual objects have the same color and form as in the previous version of the game without semantics (i.e. Figure 2(b)).
Revised: Note that most colored blocks are placebos and do not correspond to any object and the actual objects have the same color and form as in the previous version of the game without semantics (i.e. Figure 2(b)).

Original: If the prior knowledge about entities that are visibly distinct are interesting to explore is critical, this manipulation in the game structure should lead to a signiﬁcant change in human performance.
Revised: If the prior knowledge about entities that are visibly distinct are interesting to explore is critical, this manipulation in the game structure should lead to a signiﬁcant change in human performance.

Original: Results in Figure 3(c) show that masking where objects are leads to drastic deterioration in performance.
Revised: Results in Figure 3(c) show that masking where objects are leads to drastic deterioration in performance.

Original: The average time taken by human players to solve the game is nearly four times, number of deaths is nearly six times and humans explore four times as many game states as compared to the original game (Figure 3(c)).
Revised: The average time taken by human players to solve the game is nearly four times, number of deaths is nearly six times and humans explore four times as many game states as compared to the original game (Figure 3(c)).

Original: When compared to game version in which only semantic information was removed, the time taken, number of deaths, and number of states are all signiﬁcantly greater (p-value < 0.01).
Revised: When compared to game version in which only semantic information was removed, the time taken, number of deaths, and number of states are all signiﬁcantly greater (p-value < 0.01).

Original: When only semantics are masked, after encountering one object the human player is aware of what possible locations might be interesting to explore next.
Revised: When only semantics are masked, after encountering one object the human player is aware of what possible locations might be interesting to explore next.

Original: However, when objects are also masked it is unclear what to explore next.
Revised: However, when objects are also masked it is unclear what to explore next.

Original: This effect can be seen by the increase in normalized time taken to reach the door from the key as compared to the game where only semantics are masked (Figure 4(b)).
Revised: This effect can be seen by the increase in normalized time taken to reach the door from the key as compared to the game where only semantics are masked (Figure 4(b)).

Original: All these results suggest that knowing that visibly distinct entities are interesting and can be used as subgoals for exploration is a more important prior than knowledge of semantics.
Revised: All these results suggest that knowing that visibly distinct entities are interesting and can be used as subgoals for exploration is a more important prior than knowledge of semantics.

Original: 3. A FFORDANCE
Revised: 3. A FFORDANCE

Original: Uptil now, we manipulated objects in ways that made inferring the underlying reward structure of the game non-trivial.
Revised: Uptil now, we manipulated objects in ways that made inferring the underlying reward structure of the game non-trivial.

Original: However, in these games it was obvious for humans that platforms can support agent sprites, ladders could be climbed to reach different platforms (even when the ladders were colored in uniform red in games shown in Figure 2(b,c), the connectivity pattern revealed what ladders were) and black parts of the game constitutes free space .
Revised: However, in these games it was obvious for humans that platforms can support agent sprites, ladders could be climbed to reach different platforms (even when the ladders were colored in uniform red in games shown in Figure 2(b,c), the connectivity pattern revealed what ladders were) and black parts of the game constitutes free space .

Original: Such knowledge about the use of an entity is referred to as affordance of the entity( Gibson (2014)).
Revised: Such knowledge about the use of an entity is referred to as affordance of the entity( Gibson (2014)).

Original: Note that we have purposefully constructed a difference between entities such as key, door, enemy, spike which cannot directly be used by the agent but convey the task structure and entities such as platforms, ladders and free space which do not necessarily convey the reward structure but are used to explore the environment.
Revised: Note that we have purposefully constructed a difference between entities such as key, door, enemy, spike which cannot directly be used by the agent but convey the task structure and entities such as platforms, ladders and free space which do not necessarily convey the reward structure but are used to explore the environment.

Original: In the next set of experiments we manipulated the game to mask the affordance prior.
Revised: In the next set of experiments we manipulated the game to mask the affordance prior.

Original: One way to mask affordances is to render the free space with random textures, which are visually similar to textures used for rendering ladders and platforms.
Revised: One way to mask affordances is to render the free space with random textures, which are visually similar to textures used for rendering ladders and platforms.

Original: Such rendering makes it difﬁcult for humans to infer what parts of the game screen belong to platforms or ladders (see Figure 2(d), game link).
Revised: Such rendering makes it difﬁcult for humans to infer what parts of the game screen belong to platforms or ladders (see Figure 2(d), game link).

Original: Note that in this game manipulation, objects and their semantics are clearly observable.
Revised: Note that in this game manipulation, objects and their semantics are clearly observable.

Original: When tasked to play this game, humans require signiﬁcantly more time, visit larger number of states, and die more often (p-value < 0.01) as compared to the original game.
Revised: When tasked to play this game, humans require signiﬁcantly more time, visit larger number of states, and die more often (p-value < 0.01) as compared to the original game.

Original: On the other hand, there is no signiﬁcant different in performance of humans in this game when compared to performance of humans in the game without semantics i.e. Figure 2(b), implying that the affordance prior is as important as the semantics prior in our setup.
Revised: On the other hand, there is no signiﬁcant different in performance of humans in this game when compared to performance of humans in the game without semantics i.e. Figure 2(b), implying that the affordance prior is as important as the semantics prior in our setup.

Original: 3. T HINGS THAT LOOK SIMILAR BEHAVE SIMILAR
Revised: 3. T HINGS THAT LOOK SIMILAR BEHAVE SIMILAR

Original: In the previous game, although we masked affordance information, once the player realizes that it is possible to stand on a particular texture and climb a speciﬁc texture, it is easy to use color/texture similarity to identify other platforms and ladders in the game.
Revised: In the previous game, although we masked affordance information, once the player realizes that it is possible to stand on a particular texture and climb a speciﬁc texture, it is easy to use color/texture similarity to identify other platforms and ladders in the game.

Original: Similarly, in the game with masked semantics (Figure 2(b)), visual similarity can be used to identify other enemies and spikes.
Revised: Similarly, in the game with masked semantics (Figure 2(b)), visual similarity can be used to identify other enemies and spikes.

Original: These considerations suggest that a general prior of the form that things that look the same act the same might help humans efﬁciently explore environments where semantics or affordances are hidden.
Revised: These considerations suggest that a general prior of the form that things that look the same act the same might help humans efﬁciently explore environments where semantics or affordances are hidden.

Original: We tested this hypothesis by modifying the masked affordance game in a way that none of the platforms and ladders had the same visual signature (Figure 2(e), game link).
Revised: We tested this hypothesis by modifying the masked affordance game in a way that none of the platforms and ladders had the same visual signature (Figure 2(e), game link).

Original: Such rendering prevented human players from using the similarity prior.
Revised: Such rendering prevented human players from using the similarity prior.

Original: Figure 3(e)) shows that performance of humans was signiﬁcantly worse in comparison to the original game (Figure 2(a)), the game with masked semantics (Figure 2(b)) and the game with masked affordances (Figure 2(d))
Revised: Figure 3(e)) shows that performance of humans was signiﬁcantly worse in comparison to the original game (Figure 2(a)), the game with masked semantics (Figure 2(b)) and the game with masked affordances (Figure 2(d))

Original: (p-value < 0.01).
Revised: (p-value < 0.01).

Original: When compared to the game with no object information (Figure 2(c)), the time to complete the game and the number of states explored by players were similar, but the number of deaths was signiﬁcantly lower (p-value < 0.01).
Revised: When compared to the game with no object information (Figure 2(c)), the time to complete the game and the number of states explored by players were similar, but the number of deaths was signiﬁcantly lower (p-value < 0.01).

Original: These results suggests that visual similarity is the second most important prior used by humans in game play after the knowledge of directing exploration towards objects.
Revised: These results suggests that visual similarity is the second most important prior used by humans in game play after the knowledge of directing exploration towards objects.

Original: In order to gain insight into how this prior knowledge effects humans, we investigated the exploration pattern of human players.
Revised: In order to gain insight into how this prior knowledge effects humans, we investigated the exploration pattern of human players.

Original: In the game when all information is visible we expect that progress of humans would be uniform in time.
Revised: In the game when all information is visible we expect that progress of humans would be uniform in time.

Original: In the case when affordances are removed, the human players would initially take sometime to ﬁgure out what visual pattern corresponds to what entity and then quickly make progress in the game.
Revised: In the case when affordances are removed, the human players would initially take sometime to ﬁgure out what visual pattern corresponds to what entity and then quickly make progress in the game.

Original: Finally, in the case when the similarity prior is removed, we would expect human players to be unable to generalize any knowledge across the game and take large amounts of time exploring the environment even towards the end.
Revised: Finally, in the case when the similarity prior is removed, we would expect human players to be unable to generalize any knowledge across the game and take large amounts of time exploring the environment even towards the end.

Original: We investigated if this indeed was true by computing the time taken by each player to reach different vertical distances in the game for the ﬁrst time.
Revised: We investigated if this indeed was true by computing the time taken by each player to reach different vertical distances in the game for the ﬁrst time.

Original: Note that the door is on the top of the game, so the moving up corresponds to getting closer to solving the game.
Revised: Note that the door is on the top of the game, so the moving up corresponds to getting closer to solving the game.

Original: The results of this analysis are shown in Figure 4(c).
Revised: The results of this analysis are shown in Figure 4(c).

Original: The x-axis shows the height reached by the player and the y-axis show the average time taken by the players.
Revised: The x-axis shows the height reached by the player and the y-axis show the average time taken by the players.

Original: As the ﬁgure shows, the results conﬁrm our hypothesis.
Revised: As the ﬁgure shows, the results conﬁrm our hypothesis.

Original: 3. H OW TO INTERACT WITH OBJECTS
Revised: 3. H OW TO INTERACT WITH OBJECTS

Original: Until now we have analyzed prior knowledge used by humans to interpret the visual structure in the game.
Revised: Until now we have analyzed prior knowledge used by humans to interpret the visual structure in the game.

Original: However, interpretation of visual structure is only useful if the player understands what to do with the interpretation.
Revised: However, interpretation of visual structure is only useful if the player understands what to do with the interpretation.

Original: Humans seem to possess prior knowledge about how to interact with different objects.
Revised: Humans seem to possess prior knowledge about how to interact with different objects.

Original: For e.g., monsters can be avoided by jumping over them, ladders can be climbed by pressing the up key repeatedly etc.
Revised: For e.g., monsters can be avoided by jumping over them, ladders can be climbed by pressing the up key repeatedly etc.

Original: Deep reinforcement learning agents on the other hand do not possess such priors and must learn how to interact with objects by mere hit and trial.
Revised: Deep reinforcement learning agents on the other hand do not possess such priors and must learn how to interact with objects by mere hit and trial.

Original: To test how critical is such prior knowledge, we created a version of the game in which the ladders couldn’t be climbed by simply pressing the up key.
Revised: To test how critical is such prior knowledge, we created a version of the game in which the ladders couldn’t be climbed by simply pressing the up key.

Original: Instead, the ladders were zigzag in nature and in order to climb the ladder players had to press the up key, followed by alternating presses between the right and left key.
Revised: Instead, the ladders were zigzag in nature and in order to climb the ladder players had to press the up key, followed by alternating presses between the right and left key.

Original: Note that the ladders in this version looked like normal ladders, so players couldn’t infer the properties of the ladder by simply looking at them (see Figure 2(f), game link).
Revised: Note that the ladders in this version looked like normal ladders, so players couldn’t infer the properties of the ladder by simply looking at them (see Figure 2(f), game link).

Original: As shown in Figure 3(f), changing the property of the ladder increases the time taken, number of deaths, and states explored when compared to the original game (p-value < 0.01).
Revised: As shown in Figure 3(f), changing the property of the ladder increases the time taken, number of deaths, and states explored when compared to the original game (p-value < 0.01).

Original: The time spent by the agent in different parts of the game visualized in the original game (top row of Figure 4(d)) and this game reveals (bottom row of Figure 4(d)) that humans spend signiﬁcantly more amount of time in the ﬁrst ladder in the modiﬁed version of the game.
Revised: The time spent by the agent in different parts of the game visualized in the original game (top row of Figure 4(d)) and this game reveals (bottom row of Figure 4(d)) that humans spend signiﬁcantly more amount of time in the ﬁrst ladder in the modiﬁed version of the game.

Original: However, once they learn about how to use the ladder they are able to quickly climb the second ladder.
Revised: However, once they learn about how to use the ladder they are able to quickly climb the second ladder.

Original: When compared to game versions without semantics (Figure 2(b)) and without affordance (Figure 2(d)), we note that the number of deaths and states explored are signiﬁcantly lesser ( p < 0 . 01 ).
Revised: When compared to game versions without semantics (Figure 2(b)) and without affordance (Figure 2(d)), we note that the number of deaths and states explored are signiﬁcantly lesser ( p < 0 . 01 ).

Original: This ﬁnding suggests that while prior knowledge about object properties plays a critical role in human gameplay, knowledge about semantics and affordances may be more important than this prior.
Revised: This ﬁnding suggests that while prior knowledge about object properties plays a critical role in human gameplay, knowledge about semantics and affordances may be more important than this prior.

Original:  T AXONOMY OF OBJECT PRIORS
Revised:  T AXONOMY OF OBJECT PRIORS

Original: In previous sections, we studied how different priors about objects affect human performance one at a time.
Revised: In previous sections, we studied how different priors about objects affect human performance one at a time.

Original: We next sought to quantify human performance when all object priors investigated so far are simultaneously masked.
Revised: We next sought to quantify human performance when all object priors investigated so far are simultaneously masked.

Original: This led to creation of the game shown in Figure 5(a) that hid all information about objects, semantics, affordance, and similarity(game link).
Revised: This led to creation of the game shown in Figure 5(a) that hid all information about objects, semantics, affordance, and similarity(game link).

Original: As shown in Figure 5(b), human performance was extremely poor in this version of the game.
Revised: As shown in Figure 5(b), human performance was extremely poor in this version of the game.

Original: The average time taken to solve the game increased to 20 minutes and the average number of deaths rose sharply to 40 .
Revised: The average time taken to solve the game increased to 20 minutes and the average number of deaths rose sharply to 40 .

Original: Remarkably, the exploration trajectory of humans is now almost completely random (refer to Figure 5(c)) with the number of unique states visited by the human players increasing by a factor of 9.
Revised: Remarkably, the exploration trajectory of humans is now almost completely random (refer to Figure 5(c)) with the number of unique states visited by the human players increasing by a factor of 9.

Original: Due to difﬁcult in completing the game, we noticed a high dropout of human participants before they ﬁnished the game.
Revised: Due to difﬁcult in completing the game, we noticed a high dropout of human participants before they ﬁnished the game.

Original: We had to increase the pay to $2.25 to incentivize participants to not quit.
Revised: We had to increase the pay to $2.25 to incentivize participants to not quit.

Original: Many participants noted that they could only solve the game by memorizing it.
Revised: Many participants noted that they could only solve the game by memorizing it.

Original: Even though we preserved priors related to physics (e.g. objects fall down) and motor control (e.g. pressing left key moves the agent sprite to the left), just by rendering the game in a way that makes it impossible to use prior knowledge about how to visually interpret the game screen, makes the game extremely hard to play.
Revised: Even though we preserved priors related to physics (e.g. objects fall down) and motor control (e.g. pressing left key moves the agent sprite to the left), just by rendering the game in a way that makes it impossible to use prior knowledge about how to visually interpret the game screen, makes the game extremely hard to play.

Original: To further test the limits of human ability, we designed a harder game where we also reversed gravity and randomly re-mapped the key presses to how it affect’s the motion of agent’s sprite.
Revised: To further test the limits of human ability, we designed a harder game where we also reversed gravity and randomly re-mapped the key presses to how it affect’s the motion of agent’s sprite.

Original: We, the creators of the game, having played previous version of the game hundred of times had an extremely hard time trying to complete this version of the game.
Revised: We, the creators of the game, having played previous version of the game hundred of times had an extremely hard time trying to complete this version of the game.

Original: This game placed us in the shoes of reinforcement learning (RL) agents that start off without the immense prior knowledge that humans possess.
Revised: This game placed us in the shoes of reinforcement learning (RL) agents that start off without the immense prior knowledge that humans possess.

Original: While improvements in performance of RL agents with better algorithms and better computational resources is inevitable, our results make a strong case for developing algorithms to incorporate prior knowledge as a way for improving the performance of artiﬁcial agents.
Revised: While improvements in performance of RL agents with better algorithms and better computational resources is inevitable, our results make a strong case for developing algorithms to incorporate prior knowledge as a way for improving the performance of artiﬁcial agents.

Original: While there are many possible directions on how to incorporate priors in RL and more generally
Revised: While there are many possible directions on how to incorporate priors in RL and more generally

Original: AI agents, it is informative to study how humans acquire such priors.
Revised: AI agents, it is informative to study how humans acquire such priors.

Original: Studies in developmental psychology suggest that human infants as young as two months old possess primitive notion of objects and expect them to move as connected and bounded wholes that allows them to perceive object boundaries and therefore possibly distinguish them from background ( Spelke (1990); Spelke & Kinzler (2007)).
Revised: Studies in developmental psychology suggest that human infants as young as two months old possess primitive notion of objects and expect them to move as connected and bounded wholes that allows them to perceive object boundaries and therefore possibly distinguish them from background ( Spelke (1990); Spelke & Kinzler (2007)).

Original: At this stage, infants do not reason about object categories.
Revised: At this stage, infants do not reason about object categories.

Original: By the age of 3months, infants start exhibiting categorization behavior based on similarity and familiarity.
Revised: By the age of 3months, infants start exhibiting categorization behavior based on similarity and familiarity.

Original: (Mandler (1998), Mareschal & Quinn (2001)).
Revised: (Mandler (1998), Mareschal & Quinn (2001)).

Original: The ability to recognize individual objects rapidly and accurately emerges comparatively late in development (usually by the time babies are 18-24 months old, Pereira & Smith (2009)).
Revised: The ability to recognize individual objects rapidly and accurately emerges comparatively late in development (usually by the time babies are 18-24 months old, Pereira & Smith (2009)).

Original: Similarly, while young infants exhibit some knowledge about affordances early during development, the ability to distinguish a walkable step from a cliff emerges only by the time they are 18 months old (Kretch & Adolph (2013)).
Revised: Similarly, while young infants exhibit some knowledge about affordances early during development, the ability to distinguish a walkable step from a cliff emerges only by the time they are 18 months old (Kretch & Adolph (2013)).

Original: These results in infant development suggest that starting with a primitive notion of objects, infants gradually learn about visual similarity and eventually about object semantics and affordances.
Revised: These results in infant development suggest that starting with a primitive notion of objects, infants gradually learn about visual similarity and eventually about object semantics and affordances.

Original: It is quite interesting to note that the order in which infants increase their knowledge matches the importance of different object priors such as existence of objects as sub-goals for exploration, visual similarity, object semantics, and affordances.
Revised: It is quite interesting to note that the order in which infants increase their knowledge matches the importance of different object priors such as existence of objects as sub-goals for exploration, visual similarity, object semantics, and affordances.

Original: Based on these results, we suggest a possible taxonomy and ranking of object priors in Figure 6.
Revised: Based on these results, we suggest a possible taxonomy and ranking of object priors in Figure 6.

Original: We here put object properties at the bottom as in the context of our problem, knowledge about how to interact with speciﬁc objects can be only learnt once recognition is performed.
Revised: We here put object properties at the bottom as in the context of our problem, knowledge about how to interact with speciﬁc objects can be only learnt once recognition is performed.

Original:  P RIOR KNOWLEDGE IS NOT ALWAYS DESIRABLE
Revised:  P RIOR KNOWLEDGE IS NOT ALWAYS DESIRABLE

Original: For many interesting real world tasks, for pragmatic reasons it is often only possible to provide agents with a terminal reward when they succeed and they receive no external rewards otherwise.
Revised: For many interesting real world tasks, for pragmatic reasons it is often only possible to provide agents with a terminal reward when they succeed and they receive no external rewards otherwise.

Original: Success in such scenarios critically depends on the agent’s ability to explore its environment and then quickly learn from its success (i.e. exploitation).
Revised: Success in such scenarios critically depends on the agent’s ability to explore its environment and then quickly learn from its success (i.e. exploitation).

Original: While understanding what enables an agent to efﬁciently exploit is an interesting question, without a good exploration strategy no exploitation is possible.
Revised: While understanding what enables an agent to efﬁciently exploit is an interesting question, without a good exploration strategy no exploitation is possible.

Original: It therefore naturally follows that agents that can efﬁciently explore their environment will be good at completing tasks with sparse rewards.
Revised: It therefore naturally follows that agents that can efﬁciently explore their environment will be good at completing tasks with sparse rewards.

Original: In this vein, our results demonstrate the importance of prior knowledge in helping humans explore efﬁciently in these sparse reward environments.
Revised: In this vein, our results demonstrate the importance of prior knowledge in helping humans explore efﬁciently in these sparse reward environments.

Original: However, that being said, being equipped with strong prior knowledge may not be beneﬁcial with regards to reward optimization in all kinds of environments.
Revised: However, that being said, being equipped with strong prior knowledge may not be beneﬁcial with regards to reward optimization in all kinds of environments.

Original: To illustrate this, we again recruited participants from Mechanical Turk ( n = 30 ) to play a short game that simply consisted of a player and a princess at a short distance away from the player (Figure 7.a).
Revised: To illustrate this, we again recruited participants from Mechanical Turk ( n = 30 ) to play a short game that simply consisted of a player and a princess at a short distance away from the player (Figure 7.a).

Original: Unknown to the participants, the game consisted of 10 hidden rewards (shown in yellow for illustration purposes) and the participants were given a bonus upon discovering them.
Revised: Unknown to the participants, the game consisted of 10 hidden rewards (shown in yellow for illustration purposes) and the participants were given a bonus upon discovering them.

Original: As shown in (Figure 7.b), human players do not explore this environment and end up with suboptimal rewards.
Revised: As shown in (Figure 7.b), human players do not explore this environment and end up with suboptimal rewards.

Original: Upon entering the game, the players saw the princess and mostly inferred that as the goal and immediately reached her, thereby terminating the game.
Revised: Upon entering the game, the players saw the princess and mostly inferred that as the goal and immediately reached her, thereby terminating the game.

Original: In contrast, a random agent (30 seeds of episode count=1 to simulate human experiments) ends up obtaining almost 4 times the rewards than human players.
Revised: In contrast, a random agent (30 seeds of episode count=1 to simulate human experiments) ends up obtaining almost 4 times the rewards than human players.

Original: Research in developmental psychology has also demonstrated such instances wherein children have been shown to be better learners than adults in some cases (Lucas et al.
Revised: Research in developmental psychology has also demonstrated such instances wherein children have been shown to be better learners than adults in some cases (Lucas et al.

Original: Thus, while incorporating prior knowledge in RL agents has many potential beneﬁts, it is also important to consider if that could lead to inﬂexibility in an algorithm leading to inefﬁcient exploration.
Revised: Thus, while incorporating prior knowledge in RL agents has many potential beneﬁts, it is also important to consider if that could lead to inﬂexibility in an algorithm leading to inefﬁcient exploration.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: While there is no doubt that the performance of recent deep RL algorithms is impressive, there is much to be learned from human cognition if our goal is to enable RL agents to solve sparse reward tasks with human-like efﬁciency.
Revised: While there is no doubt that the performance of recent deep RL algorithms is impressive, there is much to be learned from human cognition if our goal is to enable RL agents to solve sparse reward tasks with human-like efﬁciency.

Original: Humans have the amazing ability to bring to bear their past knowledge (i e., priors) to solve new tasks quickly.
Revised: Humans have the amazing ability to bring to bear their past knowledge (i e., priors) to solve new tasks quickly.

Original: Our work takes one of the ﬁrst steps to quantify the importance of various priors that humans employ in solving sparse reward tasks and in understanding how prior knowledge makes humans good at reinforcement learning tasks.
Revised: Our work takes one of the ﬁrst steps to quantify the importance of various priors that humans employ in solving sparse reward tasks and in understanding how prior knowledge makes humans good at reinforcement learning tasks.

Original: We believe that our results will inspire researchers to think about different mechanisms of incorporating prior knowledge in the design of RL agents instead of starting from scratch.
Revised: We believe that our results will inspire researchers to think about different mechanisms of incorporating prior knowledge in the design of RL agents instead of starting from scratch.

Original: We also hope that our experimental platform of video games, available in open-source, will fuel more detailed studies investigating human priors and a benchmark for quantifying the efﬁcacy of different mechanisms of incorporating prior knowledge into RL agents.
Revised: We also hope that our experimental platform of video games, available in open-source, will fuel more detailed studies investigating human priors and a benchmark for quantifying the efﬁcacy of different mechanisms of incorporating prior knowledge into RL agents

Original: A F URTHER EXPERIMENTS ON SEMANTICS A. R EVERSING SEMANTIC INFORMATION
Revised: A F URTHER EXPERIMENTS ON SEMANTICS A. R EVERSING SEMANTIC INFORMATION

Original: In Section 3.1, we masked semantic information by recoloring objects with plain colors.
Revised: In Section 3.1, we masked semantic information by recoloring objects with plain colors.

Original: An alternate mechanism to manipulate the semantic prior is by reversing the semantic of different entities (i.e. objects that people associate as good are bad, and vice versa)).
Revised: An alternate mechanism to manipulate the semantic prior is by reversing the semantic of different entities (i.e. objects that people associate as good are bad, and vice versa)).

Original: We created this version by replacing the pink enemy and spikes by coins and ice-cream sprite respectively which have a positive connotation, the ladder by ﬁre, the key and the door by spikes and slimes which have negative connotations (Figure 8(a)).
Revised: We created this version by replacing the pink enemy and spikes by coins and ice-cream sprite respectively which have a positive connotation, the ladder by ﬁre, the key and the door by spikes and slimes which have negative connotations (Figure 8(a)).

Original: As shown in Figure 8(b), participants took longer to solve this game compared to the original version with average time taken equal to 6 minutes (p-value < 0.05).
Revised: As shown in Figure 8(b), participants took longer to solve this game compared to the original version with average time taken equal to 6 minutes (p-value < 0.05).

Original: The average number of deaths was also signiﬁcantly greater and participants explored more compared to the original version (p-value < 0.01 for both).
Revised: The average number of deaths was also signiﬁcantly greater and participants explored more compared to the original version (p-value < 0.01 for both).

Original: Interestingly, participants also took longer to solve this game when compared to the masked semantics version (p-value < 0.05) implying that when we reverse semantic information, humans ﬁnd the game even tougher to solve.
Revised: Interestingly, participants also took longer to solve this game when compared to the masked semantics version (p-value < 0.05) implying that when we reverse semantic information, humans ﬁnd the game even tougher to solve.

Original: This experiment further demonstrates that in absence of semantics (or reversal of semantics as in this case), human players performance in video games drops signiﬁcantly.
Revised: This experiment further demonstrates that in absence of semantics (or reversal of semantics as in this case), human players performance in video games drops signiﬁcantly.

Original: B P HYSICS AND MOTOR CONTROL PRIORS
Revised: B P HYSICS AND MOTOR CONTROL PRIORS

Original: In addition to prior knowledge about objects, humans also bring in rich prior knowledge about intuitive physics as well as bring strong motor control priors when they approach a new task (Hespos et al.
Revised: In addition to prior knowledge about objects, humans also bring in rich prior knowledge about intuitive physics as well as bring strong motor control priors when they approach a new task (Hespos et al.

Original: (2009), Baillargeon (2004), Wolpert & Ghahramani (2000), Baillargeon (1994)).
Revised: (2009), Baillargeon (2004), Wolpert & Ghahramani (2000), Baillargeon (1994)).

Original: Here, we have taken some initial steps to explore the importance of such priors in context of human gameplay.
Revised: Here, we have taken some initial steps to explore the importance of such priors in context of human gameplay.

Original: B. G RAVITY
Revised: B. G RAVITY

Original: One of the most obvious knowledge that we have about the physical world is with regards to gravity i.e. things fall from up to down.
Revised: One of the most obvious knowledge that we have about the physical world is with regards to gravity i.e. things fall from up to down.

Original: To mask this prior, we created a version of the game in which the whole game window was rotated 90 ◦ .
Revised: To mask this prior, we created a version of the game in which the whole game window was rotated 90 ◦ .

Original: In this way, the gravity was reversed from left to right (as opposed to up to down).
Revised: In this way, the gravity was reversed from left to right (as opposed to up to down).

Original: As shown in Figure 9, participants spent more time to solve this game compared to the original version with average time taken close to 3 minutes (p-value < 0.01).
Revised: As shown in Figure 9, participants spent more time to solve this game compared to the original version with average time taken close to 3 minutes (p-value < 0.01).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).
Revised: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).

Original: B. N ON - UNIFORM GRAVITY
Revised: B. N ON - UNIFORM GRAVITY

Original: In the previous game, although we manipulated the gravity prior by reversing the gravity, participants still had access to more general notions about gravity such as gravity in the game is uniform and constant.
Revised: In the previous game, although we manipulated the gravity prior by reversing the gravity, participants still had access to more general notions about gravity such as gravity in the game is uniform and constant.

Original: We hypothesized that such a general prior about gravity might guide human exploration in an environment even when the gravity is reversed.
Revised: We hypothesized that such a general prior about gravity might guide human exploration in an environment even when the gravity is reversed.

Original: To test this, we modiﬁed the original game such that different platforms in the game had different gravity.
Revised: To test this, we modiﬁed the original game such that different platforms in the game had different gravity.

Original: This meant that some platforms had a very strong gravity so that the agent sprite couldn’t jump on these platforms, some platforms had a very weak gravity so that the agent sprite could jump signiﬁcantly higher, and some platforms had moderate gravity.
Revised: This meant that some platforms had a very strong gravity so that the agent sprite couldn’t jump on these platforms, some platforms had a very weak gravity so that the agent sprite could jump signiﬁcantly higher, and some platforms had moderate gravity.

Original: Thus, in this version, participants had to learn about the dynamics of the game (related to gravity and jumping) from scratch.
Revised: Thus, in this version, participants had to learn about the dynamics of the game (related to gravity and jumping) from scratch.

Original: As shown in Figure 9, participants took a signiﬁcantly longer time to solve this game compared to the version with reversed gravity with average time taken close to 5 minutes (p-value < 0.01).
Revised: As shown in Figure 9, participants took a signiﬁcantly longer time to solve this game compared to the version with reversed gravity with average time taken close to 5 minutes (p-value < 0.01).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the version with reverse gravity (p-value < 0.01).
Revised: The average number of deaths and number of states explored was also signiﬁcantly larger than the version with reverse gravity (p-value < 0.01).

Original: This suggests that similar to our results on object priors, general priors related to physics (such as uniform gravity) serve a prominent role in guiding efﬁcient human gameplay.
Revised: This suggests that similar to our results on object priors, general priors related to physics (such as uniform gravity) serve a prominent role in guiding efﬁcient human gameplay.

Original: B. M USCLE MEMORY
Revised: B. M USCLE MEMORY

Original: Human players also come with knowledge of the form such as pressing arrow keys moves the agent sprite in the corresponding directions (i.e. pressing up makes the agent sprite jump, pressing left makes the agent sprite go left and so forth).
Revised: Human players also come with knowledge of the form such as pressing arrow keys moves the agent sprite in the corresponding directions (i.e. pressing up makes the agent sprite jump, pressing left makes the agent sprite go left and so forth).

Original: We created a version of the game in which we reversed the arrow key controls.
Revised: We created a version of the game in which we reversed the arrow key controls.

Original: Thus, pressing left arrow key made the agent sprite go right, pressing right key moved the sprite left, pressing down key made the player jump (or go up the stairs), and pressing up key made the player go down the stairs.
Revised: Thus, pressing left arrow key made the agent sprite go right, pressing right key moved the sprite left, pressing down key made the player jump (or go up the stairs), and pressing up key made the player go down the stairs.

Original: Participants again took longer to solve this game compared to the original version with average time taken close to 3 minutes (refer to Figure 9).
Revised: Participants again took longer to solve this game compared to the original version with average time taken close to 3 minutes (refer to Figure 9).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).
Revised: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).

Original: Interestingly, the performance of players when the gravity was reversed and key controls were reversed is similar with no signiﬁcant difference between the two conditions.
Revised: Interestingly, the performance of players when the gravity was reversed and key controls were reversed is similar with no signiﬁcant difference between the two conditions.

Original: C P ERFORMANCE OF RL AGENT ON VARIOUS GAME MANIPULATIONS
Revised: C P ERFORMANCE OF RL AGENT ON VARIOUS GAME MANIPULATIONS

Original: In this section, we investigated how the RL agent (ICM-A3C; Pathak et al.
Revised: In this section, we investigated how the RL agent (ICM-A3C; Pathak et al.

Original: performed in each of the lesioned settings we investigated with humans.
Revised: performed in each of the lesioned settings we investigated with humans.

Original: While deep RL agents don’t come with any prior knowledge, they can at least ﬁnd and exploit regularities in data.
Revised: While deep RL agents don’t come with any prior knowledge, they can at least ﬁnd and exploit regularities in data.

Original: Thus the experiments in this section can help shed light on how statistical regularities in the data inﬂuence deep RL agents.
Revised: Thus the experiments in this section can help shed light on how statistical regularities in the data inﬂuence deep RL agents.

Original: To do this, we systematically created different versions of the game in Figure 1(a) to mask semantics, concept of object, affordance as well as similarity (refer to Figure 10) and ran 10 random seeds of the RL agent on each of the game versions.
Revised: To do this, we systematically created different versions of the game in Figure 1(a) to mask semantics, concept of object, affordance as well as similarity (refer to Figure 10) and ran 10 random seeds of the RL agent on each of the game versions.

Original: Note that, we modiﬁed the game in Figure 1(a) and not
Revised: Note that, we modiﬁed the game in Figure 1(a) and not

Original: Figure 2(a) to run the RL experiments as the game in Figure 2(a) was too hard for the RL agent to solve.
Revised: Figure 2(a) to run the RL experiments as the game in Figure 2(a) was too hard for the RL agent to solve.

Original: As shown in Figure 11, the RL agent is unaffected by the removal of semantics, concept of objects as well as affordance information – there is no signiﬁcant difference between the group means of the RL agent on these games and the original version.
Revised: As shown in Figure 11, the RL agent is unaffected by the removal of semantics, concept of objects as well as affordance information – there is no signiﬁcant difference between the group means of the RL agent on these games and the original version.

Original: The performance of the RL agent on game without object information (Figure 10(c)) is especially interesting because this prior information is extremely critical to human gameplay.
Revised: The performance of the RL agent on game without object information (Figure 10(c)) is especially interesting because this prior information is extremely critical to human gameplay.

Original: Interestingly, the RL agent is affected by removal of similarity information as it takes nearly two times to solve that version of the game implying that
Revised: Interestingly, the RL agent is affected by removal of similarity information as it takes nearly two times to solve that version of the game implying that

Original: RL agents do exploit visual similarity in the data.
Revised: RL agents do exploit visual similarity in the data.

Original: Future work aims to investigate how this visual similarity is automatically learned.
Revised: Future work aims to investigate how this visual similarity is automatically learned.

Original: The RL agent performs similarly on all games except for the one
Revised: The RL agent performs similarly on all games except for the one