Original: I NVESTIGATING H UMAN P RIORS FOR P LAYING V IDEO G AMES
Revised: I NVESTIGATING H UMAN P RIORS FOR P LAYING V IDEO G AMES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep reinforcement learning algorithms have recently achieved impressive results on a range of video games, yet they remain much less efﬁcient than an average human player at learning a new game.

Original: What makes humans so good at solving these video games?
Revised: What makes humans so good at solving seemingly complex video games?

Original: Here, we study one aspect critical to human gameplay – their use of strong priors that enables efﬁcient decision making and problem solving. We created a sample video game and conducted various experiments to quantify the kinds of prior knowledge humans bring in while playing such games.
Revised: Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efﬁcient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors.

Original: We do this by modifying the video game environment to systematically remove different types of visual information that could be used by humans as priors.
Revised: We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors.

Original: We ﬁnd that human performance degrades drastically once prior information has been removed, while that ofan RL agent does not change. Interestingly, we also ﬁnd that general priors about objects that humans learn when they are as little as 2 months old are some of the most critical priors that help in  human gameplay. Based on these ﬁndings, we then propose a taxonomy of object priors people employ when solving video games that can potentially serve as a benchmark for future reinforcement learning algorithms aiming to incorporate human-like representations in their systems.
Revised: We ﬁnd that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efﬁcient game-play.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Revised: While deep Reinforcement Learning (RL) methods have shown impressive performance on a variety of video games (Mnih et al., 2015), they remain woefully inefﬁcient compared to human players, taking millions of action inputs to solve even the simplest Atari games.

Revised: Much research is currently focused on improving sample efﬁciency of RL algorithms (Oh et al., 2017; Gu et al., 2016).

Revised: However, there is an orthogonal issue that is often overlooked: RL agents attack each problem tabula rasa , whereas humans come in with a wealth of prior knowledge about the world, from physics to semantics to affordances.

Original: Consider the following scenario: you are tasked to play an unfamiliar computer game shown in Figure 1(a).
Revised: Consider the following motivating example: you are tasked with playing an unfamiliar computer game shown in Figure 1(a).

Original: No manual or instructions are provided. You don’t  knowwhat the goal is or which game sprite is controlled by you.
Revised: No manual or instructions are provided; you don’t even know which game sprite is controlled by you.

Revised: Indeed, the only feedback you are ever given is “terminal”, i.e. once you successfully ﬁnish the game.

Original: How quickly can you  ﬁnish this game? 
Revised: Would you be able to successfully ﬁnish this game? How long would it take?

Original: We recruited forty  subjects to play this game and found that subjects solved the game quite easily (taking just 1600 actions and 1 minute of gameplay, c.f. Figure 1(c)).
Revised: We recruited forty human subjects to play this game and found that subjects ﬁnished it quite easily, taking just under 1 minute of game-play or 3000 action inputs.

Original: This is not overly surprising as one could easily guess that the  goalof the game is to move the robot sprite towards the princess by stepping on the brick-like objects and using ladders to reach the higher platforms whilealso avoiding the angry purple sprite and the ﬁre object.
Revised: This is not overly surprising as one could easily guess that the game’s goal is to move the robot sprite towards the princess by stepping on the brick-like objects and using ladders to reach the higher platforms while avoiding the angry pink and the ﬁre objects.

Original: Now consider a second scenario in which this same  game is re-rendered with new textures, getting rid of semantic  cues, as shown in Figure 1(b).
Revised: Now consider a second scenario in which this same simple game is re-rendered with new textures, getting rid of semantic and affordance (Gibson, 2014) cues, as shown in Figure 1(b).

Original: How would human performance change?
Revised: How would human performance change?

Original: We recruited another forty subjects to play this game and found that theaverage number of actions taken by players to solve the second game is twiceas many as the ﬁrst game (Figure 1(c)).
Revised: We recruited another forty subjects to play this game and found that, on average, it took the players more than twice the time ( 2 minutes) and action inputs ( 6500 ) to complete the game.

Original: This game is clearly much harder for humans.
Revised: The second game is clearly much harder for humans, likely because it is now more difﬁcult to guess the game structure and goal, as well as to spot obstacles.

Original: How would a reinforcement learning agent perform on the two games? We trained a state-of-the-art
Revised: For comparison, we can also examine how modern RL algorithms perform on these games.

Original:  RL agent (ICM-A3C; Pathak et al.
Revised: This is not so simple, as most standard RL approaches expect very dense rewards (e.g. continuously updated game-score (Mnih et al., 2015)), whereas we provide only a terminal reward, to mimic how most humans play video games. In such sparse reward scenarios, standard methods like A3C (Mnih et al., 2016) are too sample-inefﬁcient and were too slow to ﬁnish the games. Hence, we used a curiosity-based RL algorithm speciﬁcally tailored to sparse-reward settings (Pathak et al., 2017), which was able to solve both games.

Original: on both these games and found that the RL agent was virtually unaffected – it took close to four million steps to solve both games (Figure 1(c)).
Revised: Unlike humans, RL did not show much difference between the two games, taking about 4 million action inputs to solve each one.

Original: Since the RL agent came tabula rasa i.e. without any prior knowledge about the world, both these games carried  the same amount of information from the perspective of the agent leading to no change in its performance.
Revised: This should not be surprising: since RL did not have any prior knowledge about the world, both these games carried roughly the same amount of information from the perspective of the agent.

Original: This simple  experiment highlights the importance of prior knowledge that humans draw upon to quickly solve tasks given to them (Lake et al., 2016; Tsividis et al., 2017).
Revised: This simple motivating experiment highlights the importance of prior knowledge that humans draw upon to quickly solve tasks given to them (Lake et al., 2016; Tsividis et al., 2017).

Original: While the form of prior information tested above may be obvious, people bring in a wealth of prior information about the physical world that goes beyond simple knowledge about platforms, ladders, princess, monsters, etc.

Original: Developmental psychologists have been documenting  prior knowledge that children draw upon in learning about the world (Spelke & Kinzler, 2007; Carey, 2009).
Revised: Developmental psychologists have begun documenting the prior knowledge that children draw upon in learning about the world (Spelke & Kinzler, 2007; Carey, 2009).

Original: However, these studies have not explicitly quantiﬁed how vital are different priors for problem-solving.
Revised: However, these studies have not explicitly quantiﬁed the relative importance of the various priors for problem-solving.

Original: In this work, wehave systematically quantiﬁed the importance of various priors humans bring to bear while solving one particular kind of problem – video games.
Revised: In this work, we systematically quantify the importance of different types of priors humans bring to bear while solving one particular kind of problem – video games.

Original: We chose video games as the task for our investigation because it is  easy to systematically change the game to include or mask different kinds of knowledge, run large-scale human studies, and video games such as ATARI are a popular choice in the reinforcement learning community.
Revised: We chose video games as the task for our investigation because it is relatively easy to methodically change the game to include or mask different kinds of knowledge and run large-scale human studies. Furthermore, video games, such as ATARI, are a popular choice in the reinforcement learning community.

Revised: The paper consists of a series of ablation studies on a specially-designed game environment, systematically masking out various types of visual information that could be used by humans as priors.

Original: One of  the ﬁndings of our investigation is that while knowledge of the form that ladders are to be climbed, keys are used to open doors, jumping on spikes is dangerous is important for humans to quickly solve games, more general priors of the form that the  objectsare subgoals for exploration and things that look the same behave the same are even more critical.Although we use video games as our experimental test bed, such priors are more generally applicable even outside the domain of video games.
Revised: The full game (unlike the motivating example above) was designed to be sufﬁciently complex and difﬁcult for humans to easily measure changes in performance between different testing conditions. We ﬁnd that removal of some prior knowledge causes a drastic degradation in the performance of human players from 1 minute to over 20 minutes. Another key ﬁnding of our investigation is that while speciﬁc knowledge, such as “ladders are to be climbed”, “keys are used to open doors”, “jumping on spikes is dangerous”, is important for humans to quickly solve games, more general priors about the importance of objects and visual consistency are even more critical.

Original:  M ETHOD
Revised:  M ETHOD

Original: To investigate the aspects of visual information that enable humans to efﬁciently solve video games, we designed a browser based platform game consisting of a human sprite that could be controlled, platforms, ladders, slimy pink sprites that killed the agent, spikes that were dangerous to jump on, a key, and a door (see Figure 2 (a)).
Revised: To investigate the aspects of visual information that enable humans to efﬁciently solve video games, we designed a browser-based platform game consisting of an agent sprite, platforms, ladders, angry pink object that kills the agent, spikes that are dangerous to jump on, a key, and a door (see Figure(a)).

Original: The human sprite could be moved with  help of arrow keys and the agent obtained a reward of +1  when it reached the door after taking the key thereby terminating the game.
Revised: The agent sprite can be moved with the help of arrow keys. A terminal reward of +1 is provided when the agent reaches the door after having to taken the key, thereby terminating the game.

Original: The game was reset whenever the agent touched the enemy, jumped on the spike, or fell below the lowermost platform.
Revised: The game is reset whenever the agent touches the enemy, jumps on the spike, or falls below the lowest platform.

Original: We made this game to resemble the exploration challenges faced in the classic ATARI game of Montezuma’s revenge that has proven to be very challenging forstate of deep reinforcement learning techniques (Bellemare et al., 2016; Mnih et al., 2015).
Revised: We made this game to resemble the exploration problems faced in the classic ATARI game of Montezuma’s Revenge that has proven to be very challenging for deep reinforcement learning techniques (Bellemare et al., 2016; Mnih et al., 2015).

Revised: Unlike the motivating example, this game is too large-scale to be solved by RL agents, but provides the complexity we need to run a wide range of human experiments.

Original: Wesystematically created different versions of this game by re-rendering various entities such as ladders, enemies, keys, platforms etc.
Revised: We created different versions of the video game by re-rendering various entities such as ladders, enemies, keys, platforms etc.

Original: using alternate textures (see Figure 2).
Revised: using alternate textures (Figure 2).

Original: These textures were chosen to mask various forms of prior knowledge that are described in the experiments section.
Revised: These textures were chosen to mask various forms of prior knowledge that are described in the experiments section.

Original: Our experiment style draws inspiration from the neuroscience literature wherein researchers study aspects about the human brain by performing lesion studies (M ¨ uller & Knight (2006), Shi & Davis (1999)). For the purposes of our experiment, since it was not possible to go directly inside participant’s brain in order to study the importance ofvarious priors, we did thenext best thing possiblemask those priors.
Revised: We also changed various physical properties of the game, such as the effect of gravity, and the way the agent interacts with its environment. Note that all the games were exactly the same in their underlying structure and reward, as well as the shortest path to reach the goal, thereby ensuring that the change in human performance (if any) is only due to masking of the priors.

Original: For each version of the gamecreated, we quantiﬁed human performance by recruiting 120 participants from Amazon Mechanical Turk.
Revised: We quantiﬁed human performance on each version of the game by recruiting 120 participants from Amazon Mechanical Turk.

Original: Each participant was instructed touse the arrow keys to move and ﬁnish the game as soon as possible. No information about the goals or the reward structure of the game was communicated to the participants.
Revised: Each participant was instructed to ﬁnish the game as quickly as possible using the arrow keys as controls, but no information about the goals or the reward structure of the game was communicated.

Original: Each participant was paid $1 for successfully ﬁnishing the game.
Revised: Each participant was paid $1 for successfully completing the game.

Original: The maximum timefor allowed for playing the game was set to 30 minutes.
Revised: The maximum time allowed for playing the game was set to 30 minutes.

Original: For each participant we recorded the ( x, y ) position of the player at every step of the game, the total time taken by the participant to ﬁnish the game(i e. achieve the reward of +1) and the total number of deaths prior to ﬁnishing the game.
Revised: For each participant, we recorded the ( x, y ) position of the player at every step of the game, the total time taken by the participant to ﬁnish the game and the total number of deaths before ﬁnishing the game.

Original: We used this data to quantify participant’s performance.
Revised: We used this data to quantify the performance of each participant.

Original: Note that, we didn’t repeat participants and thus no participant  played more than one instance of the game.
Revised: Note that each participant was only allowed to complete a game once, and could not participate again (i.e. different 120 participants played each version of the game).

Original:  Q UANTIFYING THE IMPORTANCE OF OBJECT PRIORS
Revised:  Q UANTIFYING THE IMPORTANCE OF OBJECT PRIORS

Original: The ﬁrst version of the game  is shown in Figure 2(a),game link.
Revised: The original game (available to play at this link) is shown in Figure 2(a).

Original: From a single glance at the game, human playerscan employ their prior knowledge to interpret that thegame agentcan climb on ladders, it is supported by platforms, the pink slimy sprite is dangerous, spikes are to be avoided and probably the goal of the game is to take the key to open the door.
Revised: A single glance at this game is enough to inform human players that the agent sprite has to reach the key to open the door while avoiding the dangerous objects like spikes and angry pink slime.

Original: As expected such interpretations enable humansto quickly solve the game.
Revised: Unsurprisingly, humans quickly solve this game.

Original: Figure 3(a) shows that the average time taken to complete the game is 1 . 8 minutes (blue bar) and the average number of deaths (orange bar) and  game states visited by humans (yellow bar) are  quite small.
Revised: Figure 3(a) shows that the average time taken to complete the game is 1 . 8 minutes (blue bar) and the average number of deaths ( 3 . 3 , orange bar) and unique game states visited ( 3011 , yellow bar) are all quite small.

Original: 3. S EMANTICS
Revised: 3. S EMANTICS

Original: To study  importance of prior knowledge about object semantics, we rendered objects and ladders with blocks of uniform color as shown in Figure 2(b),game link.
Revised: To study the importance of prior knowledge about object semantics, we rendered objects and ladders with blocks of uniform color as shown in Figure 2(b).

Revised: This game can be played at this link.

Original: Thus, in this game manipulation, the  appearance of objects conveys no information about their semantics.
Revised: In this version, the visual appearance of objects conveys no information about their semantics.

Original: Results in Figure 3(b), show that human players take more than twice the time, have higher number of deaths, and explore signiﬁcantly larger number of states (p-value < 0.01 for all measures) as compared to the originalversion of the game  clearly demonstrating that lesioning semantics hurts human performance.
Revised: Results in Figure 3(b) show that human players take more than twice the time ( 4 . 3 minutes), have higher number of deaths ( 11 . 1 ), and explore signiﬁcantly larger number of states ( 7205 ) as compared to the original game (p-value: p < 0 . 01 ). This clearly demonstrates that masking semantics hurts human performance.

Original: A naturalnext question is how do humans make use of semantic information?
Revised: A natural question is how do humans make use of semantic information?

Original: One hypothesis is that knowledge of semantics enables humans to infer the latent reward structure of the game.
Revised: One hypothesis is that knowledge of semantics enables humans to infer the latent reward structure of the game.

Original: If this indeed is the case, then in the original game players should ﬁrst visit the key and then go to the door, while in the version of the game without semantics, players should not exhibitany such bias.
Revised: If this indeed is the case, then in the original game, where the key and the door are both visible, players should ﬁrst visit the key and then go to the door, while in the version of the game without semantics, players should not exhibit such bias.

Original: We found that in the original game, where key and door were both visible, almost all120 participants reached the key ﬁrst, while in the version with masked semantics only 42 out of 120 participants reached the key before the door (see Figure 4(a)).
Revised: We found that in the original game, nearly all participants reached the key ﬁrst, while in the version with masked semantics only 42 out of 120 participants reached the key before the door (see Figure 4(a)).

Original: Further investigation into time taken by human players  to reach the door after taking the key revealed that they take signiﬁcantly larger amount of time when the semantics are masked (see Figure 4(b)).
Revised: Moreover, human players took signiﬁcantly longer to reach the door after taking the key as compared to the original game (see Figure 4(b)).

Original: This  provides further evidence that  humans are unable to infer the reward structure and consequently signiﬁcantly increase their exploration when semantics are masked.
Revised: This result provides further evidence that in the absence of semantics, humans are unable to infer the reward structure and consequently signiﬁcantly increase their exploration.

Original: Note that, to rule out the possibility that increase in time is simply due to the fact players take more time to ﬁnish the game without semantics, the time to reach the door after taking the key was normalized by the total amount ofthe timethat was spent by the player to complete the game.
Revised: To rule out the possibility that increase in time is simply due to the fact players take longer to ﬁnish the game without semantics, the time to reach the door after taking the key was normalized by the total amount of time spent by the player to complete the game.

Original: To further quantify  importance of semantics, instead of simply masking, we reversed the semantics.
Revised: To further quantify the importance of semantics, instead of simply masking, we manipulated the semantic prior by swapping the semantics between different entities.

Revised: As seen on Figure 4(c), we replaced the pink enemy and spikes by coins and ice-cream objects respectively which have a positive connotation; the ladder by ﬁre, the key and the door by spikes and enemies which have negative connotations (see game link).

Original: This condition further deteriorated human performance and the results are detailed in the appendix.
Revised: As shown in Figure 3(c), the participants took longer to solve this game ( 6 . 1 minutes, p < 0 . 01 ). The average number of deaths ( 13 . 7 ) was also signiﬁcantly more and the participants explored more states ( 9400 ) compared to the original version ( p < 0 . 01 for both). Interestingly, the participants also took longer compared to the masked semantics version ( p < 0 . 05 ) implying that when we reverse semantic information, humans ﬁnd the game even tougher.

Original: 3. O BJECTS AS S UBGOALS FOR E XPLORATION
Revised: 3. O BJECTS AS S UB - GOALS FOR E XPLORATION

Original: While blocks of uniform color in  game shown in Figure 2(b) convey no semantics, they are distinct from the background and seem to attract human attention.
Revised: While blocks of uniform color in the game shown in Figure 2(b) convey no semantics, they are distinct from the background and seem to attract human attention.

Original: It is possible that humans infer these distinct entities (or objects) as subgoals, which results in more efﬁcient exploration than random search.
Revised: It is possible that humans infer these distinct entities (or objects) as sub-goals, which results in more efﬁcient exploration than random search.

Original: This leads to the hypothesis that humans have a prior to treat visually distinct entities as subgoals to guide exploration.
Revised: That is, there is something special about objects that draws human attention compared to any random piece of texture.

Original: In order to test this, we modiﬁed the game to cover each space on the platform with a block of different color to hide where the objects are (see Figure 2(c), game link).
Revised: To test this, we modiﬁed the game to cover each space on the platform with a block of different color to hide where the objects are (see Figure 2(d), game link).

Original: Note that most colored blocks are placebos and do not correspond to any object and the actual objects have the same color and form as in the previous version of the game without semantics (i.e. Figure 2(b)).
Revised: Most colored blocks are placebos and do not correspond to any object and the actual objects have the same color and form as in the previous version of the game with masked semantics (i.e., Figure 2(b)).

Original: If the prior knowledgeabout entities thatare visibly distinct  are interesting to explore is critical, thismanipulation in the game structure should lead to a signiﬁcant change in human performance.
Revised: If the prior knowledge that visibly distinct entities are interesting to explore is critical, this game manipulation should lead to a signiﬁcant drop in human performance.

Original: Results in Figure 3(c) show that masking where objectsare leads to drastic deterioration in performance.
Revised: Results in Figure 3(d) show that masking the concept of objects leads to drastic deterioration in performance.

Original: The average time taken by human players to solve the game is nearly four times, number of deaths is nearly six times  and humans explore four times as many game states  as compared to the original game (Figure 3(c)).
Revised: The average time taken by human players to solve the game is nearly four times longer ( 7 . 7 minutes), the number of deaths is nearly six times greater ( 20 . 2 ), and humans explore four times as many game states ( 12 , 232 ) as compared to the original game.

Original: When compared to  game version in which only semantic information was removed, the time taken, number of deaths, and number of states are all signiﬁcantly greater (p-value < 0.01).
Revised: When compared to the game version in which only semantic information was removed (Figure 3(b)), the time taken, number of deaths and number of states are all signiﬁcantly greater ( p < 0 . 01 ).

Original: When only semantics are masked, after encountering one object the human player is aware of what possible locations might be interesting to explore next.
Revised: When only semantics are removed, after encountering one object, human players become aware of what possible locations might be interesting to explore next.

Original: However, when  objects are also masked it is unclear what to explore next.
Revised: However, when concept of objects is also masked, it is unclear what to explore next.

Original: This effect can be seen by the increase in normalized time taken to reach the door from the key as compared to the game where only semantics are masked (Figure 4(b)).
Revised: This effect can be seen by the increase in normalized time taken to reach the door from the key as compared to the game where only semantics are masked (Figure 4(b)).

Original: All these results suggest that  knowing that visibly distinct entities are interesting and can be used as subgoals for exploration is a  more importantprior than knowledge of semantics.
Revised: All these results suggest that concept of objects i.e. knowing that visibly distinct entities are interesting and can be used as sub-goals for exploration, is a critical prior and perhaps more important than knowledge of semantics.

Original: 3. A FFORDANCE
Revised: 3. A FFORDANCES

Original: Uptil now, we manipulated objects in ways that made inferring the underlying reward structure of the game non-trivial.
Revised: Until now, we manipulated objects in ways that made inferring the underlying reward structure of the game non-trivial.

Original: However, in these games it was obvious for humans that platforms can support agent sprites, ladders could be climbed to reach different platforms (even when the ladders were colored in uniform red in games shown in Figure 2(b,c), the connectivity pattern revealed what ladders were) and black parts of the game constitutes free space .
Revised: However, in these games it was obvious for humans that platforms can support agent sprites, ladders could be climbed to reach different platforms (even when the ladders were colored in uniform red in games shown in Figure 2(b,c), the connectivity pattern revealed where the ladders were) and black parts of the game constitute free space .

Original: Such knowledge about the use of an entity is referred to as affordance of the entity( Gibson (2014)).
Revised: Here, the platforms and ladders afford the actions of walking and climbing (Gibson, 2014), irrespective of their appearance.

Original: Note that we have purposefully constructed a difference between entities such as key, door, enemy, spike which cannot directly be used by the agent but convey the task structure and entities such as platforms, ladders and free space which do not necessarily convey the reward structure but are used to explore the environment.

Original: In the next set of experiments we manipulated the game to mask the affordance prior.
Revised: In the next set of experiments, we manipulated the game to mask the affordance prior.

Original: One way to mask affordances is to render the free space with random textures, which are visually similar to textures used for rendering ladders andplatforms. Such rendering makes it difﬁcult for humans to infer what parts of the game screen belong to platformsor ladders (see Figure 2(d), game link).
Revised: One way to mask affordances is to ﬁll free space with random textures, which are visually similar to textures used for rendering ladders and platforms (see Figure 2(e), game link).

Original: Note that in this game manipulation, objects and their semantics are clearly observable.
Revised: Note that in this game manipulation, objects and their semantics are clearly observable.

Original: When tasked to play this game,  humans require signiﬁcantly more time, visit larger number of states, and die more often (p-value < 0.01) as compared to the original game.
Revised: When tasked to play this game, as shown in Figure 3(e), humans require signiﬁcantly more time ( 4 . 7 minutes), die more often ( 10 . 7 ), and visit more states ( 7031 ) compared to the original game ( p < 0 . 01 ).

Original: On the other hand, there is no signiﬁcant different in performanceof humans in this game when compared toperformance of humans in the game without semantics i.e. Figure 2(b), implying that the affordance prior is as important as the semantics prior in our setup.
Revised: On the other hand, there is no signiﬁcant difference in performance compared to the game without semantics, i.e., Figure 2(b), implying that the affordance prior is as important as the semantics prior in our setup.

Original: 3. T HINGS THAT LOOK SIMILAR BEHAVE SIMILAR
Revised: 3. T HINGS THAT LOOK SIMILARLY , BEHAVE SIMILARLY

Original: In the previous game, although we masked affordance information, once the player realizes that it is possible to stand on a particular texture and climb a speciﬁc texture, it is easy to use color/texture similarity to identify other platforms and ladders in the game.
Revised: In the previous game, although we masked affordance information, once the player realizes that it is possible to stand on a particular texture and climb a speciﬁc texture, it is easy to use color/texture similarity to identify other platforms and ladders in the game.

Original: Similarly, in the game with masked semantics (Figure 2(b)), visual similarity can be used to identify other enemies and spikes.
Revised: Similarly, in the game with masked semantics (Figure 2(b)), visual similarity can be used to identify other enemies and spikes.

Original: These considerations suggest that a general prior of the form that things that look the same act the same might help humans efﬁciently explore environments where semantics or affordances are hidden.
Revised: These considerations suggest that a general prior of the form that things that look the same act the same might help humans efﬁciently explore environments where semantics or affordances are hidden.

Original: We tested this hypothesis by modifying the masked affordance game in a way that none of the platforms and ladders had the same visual signature (Figure 2(e), game link).
Revised: We tested this hypothesis by modifying the masked affordance game in a way that none of the platforms and ladders had the same visual signature (Figure 2(f), game link).

Original: Such rendering prevented human players from using the similarity prior.
Revised: Such rendering prevented human players from using the similarity prior.

Original: Figure 3(e)) shows that performance of humans was signiﬁcantly worse in comparison to the original game (Figure 2(a)), the game with masked semantics (Figure 2(b)) and the game with masked affordances (Figure 2(d))
Revised: Figure 3(f)) shows that performance of humans was signiﬁcantly worse in comparison to the original game (Figure 2(a)), the game with masked semantics (Figure 2(b)) and the game with masked affordances (Figure 2(e))

Original: (p-value < 0.01).
Revised: ( p < 0 . 01 ).

Original: When compared to the game with no object information (Figure 2(c)), the time to complete the game  and the number of states explored by players were similar, but the number of deaths  was signiﬁcantly lower (p-value < 0.01).
Revised: When compared to the game with no object information (Figure 2(d)), the time to complete the game ( 7 . 6 minutes) and the number of states explored by players were similar ( 11 , 715 ), but the number of deaths ( 14 . 8 ) was signiﬁcantly lower ( p < 0 . 01 ).

Original: These results suggests that visual similarity is the second most important prior used by humans in game play after the knowledge of directing exploration towards objects.
Revised: These results suggest that visual similarity is the second most important prior used by humans in gameplay after the knowledge of directing exploration towards objects.

Original: In order to gain insight into how this prior knowledge effects humans, we investigated the exploration pattern of human players.
Revised: In order to gain insight into how this prior knowledge affects humans, we investigated the exploration pattern of human players.

Original: In the game when all information is visible we expect that  progress of humans would be uniform in time.
Revised: In the game when all information is visible we expected that the progress of humans would be uniform in time.

Original: In the case when affordances are removed, the human players would initially take sometime to ﬁgure out what visual pattern corresponds to what entity and then quickly make progress in the game.
Revised: In the case when affordances are removed, the human players would initially take some time to ﬁgure out what visual pattern corresponds to what entity and then quickly make progress in the game.

Original: Finally, in the case when the similarity prior is removed, we would expect human players to be unable to generalize any knowledge across the game and  take large amounts of time exploring the environment even towards the end.
Revised: Finally, in the case when the similarity prior is removed, we would expect human players to be unable to generalize any knowledge across the game and to take large amounts of time exploring the environment even towards the end.

Original: We investigated if this indeed was true by computing the time taken by each player to reach different vertical distances in the game for the ﬁrst time.
Revised: We investigated if this indeed was true by computing the time taken by each player to reach different vertical distances in the game for the ﬁrst time.

Original: Note that the door is on the top of the game, so the moving up corresponds to getting closer to solving the game.
Revised: Note that the door is on the top of the game, so the moving up corresponds to getting closer to solving the game.

Original: The results of this analysis are shown in Figure 4(c).
Revised: The results of this analysis are shown in Figure 4(c).

Original: The x-axis shows the height reached by the player and the y-axis show the average time taken by the players.
Revised: The horizontal-axis shows the height reached by the player and the vertical-axis show the average time taken by the players.

Original: As the ﬁgure shows, the results conﬁrm our hypothesis.
Revised: As the ﬁgure shows, the results conﬁrm our hypothesis.

Original: 3. H OW TO INTERACT WITH OBJECTS
Revised: 3. H OW TO INTERACT WITH OBJECTS

Original: Until now we have analyzed  prior knowledge used by humans to interpret the visual structure in the game.
Revised: Until now we have analyzed the prior knowledge used by humans to interpret the visual structure in the game.

Original: However, interpretation of visual structure is only useful if the player understands what to do with the interpretation.
Revised: However, interpretation of visual structure is only useful if the player understands what to do with the interpretation.

Original: Humans seem to possess prior knowledge about how to interact with different objects.
Revised: Humans seem to possess prior knowledge about how to interact with different objects.

Original: For e.g., monsters can be avoided by jumping over them, ladders can be climbed by pressing the up key repeatedly etc.
Revised: For example, monsters can be avoided by jumping over them, ladders can be climbed by pressing the up key repeatedly etc.

Original: Deep reinforcement learning agents on the other hand do not possess such priors and must learn how to interact with objects by mere hit and trial.
Revised: Deep reinforcement learning agents, on the other hand, do not possess such priors and must learn how to interact with objects by mere trial and error.

Original: To test how criticalis such prior knowledge, we created a version of the game in which the ladders couldn’t be climbed by simply pressing the up key.
Revised: To test how critical such prior knowledge is, we created a version of the game in which the ladders couldn’t be climbed by simply pressing the up key.

Original: Instead, the ladders were zigzag in nature and in order to climb the ladder players had to press the up key, followed by alternating presses between the right and left key.
Revised: Instead, the ladders were zigzag in nature and in order to climb the ladder players had to press the up key, followed by alternating presses between the right and left key.

Original: Note that the ladders in this version looked like normal ladders, so players couldn’t infer the properties of the ladder by simply looking at them (see Figure 2(f), game link).
Revised: Note that the ladders in this version looked like normal ladders, so players couldn’t infer the properties of the ladder by simply looking at them (see Figure 2(g), game link).

Original: As shown in Figure 3(f), changing the property of the ladder increases the time taken, number of deaths, and states explored  when compared to the original game(p-value < 0.01). The time spent by the agent in different parts of the game visualized in the original game (top row of Figure 4(d)) and this game reveals (bottom row of Figure 4(d)) that humans spend signiﬁcantly more amount of time in the ﬁrst ladder in the modiﬁed version of the game. However, once they learn about how to use the ladder they are able to quickly climb the second ladder. When compared to game versions without semantics (Figure 2(b)) and without affordance (Figure 2(d)), we note that the number of deaths and states explored are signiﬁcantly lesser ( p < 0 . 01 ).
Revised: As shown in Figure 3(g), changing the property of the ladder increases the time taken ( 3 . 6 minutes), number of deaths ( 6 ), and states explored ( 5942 ) when compared to the original game ( p < 0 . 01 ).

Original: This ﬁnding suggests that while prior knowledge about object properties plays a critical role in human gameplay, knowledge about semantics and affordances may be more important than this prior.

Original:  T AXONOMY OF OBJECT PRIORS
Revised:  T AXONOMY OF OBJECT PRIORS

Original: In previous sections, we studied how different priors about objects affect human performance one at a time.
Revised: In previous sections, we studied how different priors about objects affect human performance one at a time.

Original: We next sought to quantify human performance when all object priors investigated so far are simultaneously masked. This led to creation of the game shown in Figure 5(a) that hid all information about objects, semantics, affordance, and similarity(game link).
Revised: To quantify human performance when all object priors investigated so far are simultaneously masked, we created the game shown in Figure 5(b) that hid all information about objects, semantics, affordance, and similarity(game link).

Original: As shown in Figure 5(b), human performance was extremely poor in thisversion of the game.
Revised: Results in Figure 5(c) show that humans found it extremely hard to play this game.

Original: The average time taken to solve the game increased to 20 minutes and the average number of deaths rose sharply to 40 .
Revised: The average time taken to solve the game increased to 20 minutes and the average number of deaths rose sharply to 40 .

Original: Remarkably, the exploration trajectory of humans is now almost completely random (refer to Figure 5(c)) with the number of unique states visited by the human players increasing by a factor of 9.
Revised: Remarkably, the exploration trajectory of humans is now almost completely random as shown in Figure 5(e) with the number of unique states visited by the human players increasing by a factor of 9 as compared to the original game.

Original: Due to difﬁcult in completing the game, we noticed a high dropout of human participants before they ﬁnished the game.
Revised: Due to difﬁculty in completing this game, we noticed a high dropout of human participants before they ﬁnished the game.

Original: We had to increase the pay to $2.25 to incentivize participantsto not  quit.
Revised: We had to increase the pay to $2.25 to encourage participants not to quit.

Original: Many participants noted that they couldonly solve the game  by memorizing it.
Revised: Many participants noted that they could solve the game only by memorizing it.

Original: Even though we preserved priors related to physics (e.g. objects fall down) and motor control (e.g. pressing left key moves the agent sprite to the left), just by rendering the game in a way that makes it impossible to use prior knowledge about how to visually interpret the game screen, makes the game extremely hard to play.
Revised: Even though we preserved priors related to physics (e.g., objects fall down) and motor control (e.g., pressing left key moves the agent sprite to the left), just by rendering the game in a way that makes it impossible to use prior knowledge about how to visually interpret the game screen makes the game extremely hard to play.

Original: To further test the limits of human ability, we designed a harder game where we also reversed gravity and randomly re-mapped the key presses to how it affect’s the motion of agent’s sprite.
Revised: To further test the limits of human ability, we designed a harder game where we also reversed gravity and randomly re-mapped the key presses to how it affect’s the motion of agent’s sprite.

Original: We, the creators of the game, having played  previous version of the game hundred of times had an extremely hard time trying to complete this version of the game.
Revised: We, the creators of the game, having played a previous version of the game hundreds of times had an extremely hard time trying to complete this version of the game.

Original: This game placed us in the shoes of reinforcement learning (RL) agents that start off without the immense prior knowledge that humans possess.
Revised: This game placed us in the shoes of reinforcement learning (RL) agents that start off without the immense prior knowledge that humans possess.

Original: While improvements in  performance of RL agents with better algorithms and better computational resources is inevitable, our results make a strong case for developing algorithms to incorporate prior knowledge as a way for improving the performance of artiﬁcial agents.
Revised: While improvements in the performance of RL agents with better algorithms and better computational resources is inevitable, our results make a strong case for developing algorithms that incorporate prior knowledge as a way to improve the performance of artiﬁcial agents.

Original: While there are many possible directions on how to incorporate priors in RL and more generally AI agents, it is informative to study how humans acquire such priors.
Revised: While there are many possible directions on how to incorporate priors in RL and more generally AI agents, it is informative to study how humans acquire such priors.

Original: Studies in developmental psychology suggest that human infants as young as two months old possess  primitive notion of objects and expect them to move as connected and bounded wholes that allows them to perceive object boundaries and therefore possibly distinguish them from  background ( Spelke (1990); Spelke & Kinzler (2007)).
Revised: Studies in developmental psychology suggest that human infants as young as 2 months old possess a primitive notion of objects and expect them to move as connected and bounded wholes that allows them to perceive object boundaries and therefore possibly distinguish them from the background (Spelke, 1990; Spelke & Kinzler, 2007).

Original: At this stage, infants do not reason about object categories.
Revised: At this stage, infants do not reason about object categories.

Original: By the age of 3months, infants start exhibiting categorization behavior based on similarity and familiarity. (Mandler (1998), Mareschal & Quinn (2001)).
Revised: By the age of 3months, infants start exhibiting categorization behavior based on similarity and familiarity (Mandler, 1998; Mareschal & Quinn, 2001).

Original: The ability to recognize individual objects rapidly and accurately emerges comparatively late in development (usually by the time babies are 18-24 months old, Pereira & Smith (2009)).
Revised: The ability to recognize individual objects rapidly and accurately emerges comparatively late in development (usually by the time babies are 18-24 months old (Pereira & Smith, 2009)).

Original: Similarly, while young infants exhibit some knowledge about affordances early during development, the ability to distinguish a walkable step from a cliff emerges only by the time they are 18 months old (Kretch & Adolph (2013)).
Revised: Similarly, while young infants exhibit some knowledge about affordances early during development, the ability to distinguish a walkable step from a cliff emerges only by the time they are 18 months old (Kretch & Adolph, 2013).

Original: These results in infant development suggest that starting with a primitive notion of objects, infants gradually learn about visual similarity and eventually about object semantics and affordances.
Revised: These results in infant development suggest that starting with a primitive notion of objects, infants gradually learn about visual similarity and eventually about object semantics and affordances.

Original: It is quite interesting to note that the order in which infants increase their knowledge matches the importance of different object priors such as  existence of objects as sub-goals for exploration, visual similarity, object semantics, and affordances.
Revised: It is quite interesting to note that the order in which infants increase their knowledge matches the importance of different object priors such as the existence of objects as sub-goals for exploration, visual similarity, object semantics, and affordances.

Original: Based on these results, we suggest a possible taxonomy and ranking of object priors in Figure 6.
Revised: Based on these results, we suggest a possible taxonomy and ranking of object priors in Figure 6.

Original: Wehere put object properties at the bottom as in the context of our problem, knowledge about how to interact with speciﬁc objects can be only learnt once recognition is performed.
Revised: We put ‘object interaction’ at the bottom as in the context of our problem, knowledge about how to interact with speciﬁc objects can be only learned once recognition is performed.

Original: P RIOR KNOWLEDGE IS NOT ALWAYS DESIRABLE

Original: For many interesting real world tasks, for pragmatic reasons it is often only possible to provide agents with a terminal reward when they succeed and they receive no external rewards otherwise.

Original: Success in such scenarios critically depends on the agent’s ability to explore its environment and then quickly learn from its success (i.e. exploitation).

Original: While understanding what enables an agent to efﬁciently exploit is an interesting question, without a good exploration strategy no exploitation is possible.

Original: It therefore naturally follows that agents that can efﬁciently explore their environment will be good at completing tasks with sparse rewards.

Original: In this vein, our results demonstrate the importance of prior knowledge in helping humans explore efﬁciently in these sparse reward environments.

Original: However, that being said, being equipped with strong prior knowledge may not be beneﬁcial with regards to reward optimization in all kinds of environments.

Original: To illustrate this, we again recruited participants from Mechanical Turk ( n = 30 ) to play a short game that simply consisted of a player and a princess at a short distance away from the player (Figure 7.a).

Original: Unknown to the participants, the game consisted of 10 hidden rewards (shown in yellow for illustration purposes) and the participants were given a bonus upon discovering them.

Original: As shown in (Figure 7.b), human players do not explore this environment and end up with suboptimal rewards.

Original: Upon entering the game, the players saw the princess and mostly inferred that as the goal and immediately reached her, thereby terminating the game.

Original: In contrast, a random agent (30 seeds of episode count=1 to simulate human experiments) ends up obtaining almost 4 times the rewards than human players.

Original: Research in developmental psychology has also demonstrated such instances wherein children have been shown to be better learners than adults in some cases (Lucas et al.

Original: Thus, while incorporating prior knowledge in RL agents has many potential beneﬁts, it is also important to consider if that could lead to inﬂexibility in an algorithm leading to inefﬁcient exploration.

Original: C ONCLUSION

Original: While there is no doubt that the performance of recent deep RL algorithms is impressive, there is much to be learned from human cognition if our goal is to enable RL agents to solve sparse reward tasks with human-like efﬁciency.

Original: Humans have the amazing ability to bring to bear their past knowledge (i e., priors) to solve new tasks quickly.

Original: Our work takes one of the ﬁrst steps to quantify the importance of various priors that humans employ in solving sparse reward tasks and in understanding how prior knowledge makes humans good at reinforcement learning tasks.

Original: We believe that our results will inspire researchers to think about different mechanisms of incorporating prior knowledge in the design of RL agents instead of starting from scratch.

Original: We also hope that our experimental platform of video games, available in open-source, will fuel more detailed studies investigating human priors and a benchmark for quantifying the efﬁcacy of different mechanisms of incorporating prior knowledge into RL agents

Original: A F URTHER EXPERIMENTS ON SEMANTICS A. R EVERSING SEMANTIC INFORMATION

Original: In Section 3.1, we masked semantic information by recoloring objects with plain colors.

Original: An alternate mechanism to manipulate the semantic prior is by reversing the semantic of different entities (i.e. objects that people associate as good are bad, and vice versa)).

Original: We created this version by replacing the pink enemy and spikes by coins and ice-cream sprite respectively which have a positive connotation, the ladder by ﬁre, the key and the door by spikes and slimes which have negative connotations (Figure 8(a)).

Original: As shown in Figure 8(b), participants took longer to solve this game compared to the original version with average time taken equal to 6 minutes (p-value < 0.05).

Original: The average number of deaths was also signiﬁcantly greater and participants explored more compared to the original version (p-value < 0.01 for both).

Original: Interestingly, participants also took longer to solve this game when compared to the masked semantics version (p-value < 0.05) implying that when we reverse semantic information, humans ﬁnd the game even tougher to solve.

Original: This experiment further demonstrates that in absence of semantics (or reversal of semantics as in this case), human players performance in video games drops signiﬁcantly.

Original: B  P HYSICS AND MOTOR CONTROL PRIORS
Revised:  P HYSICS AND MOTOR CONTROL PRIORS

Original: In addition to prior knowledge about objects, humans also bring in rich prior knowledge about intuitive physics as well as bring strong motor control priors when they approach a new task (Hespos et al. (2009), Baillargeon (2004), Wolpert & Ghahramani (2000), Baillargeon (1994)).
Revised: In addition to prior knowledge about objects, humans also bring in rich prior knowledge about intuitive physics and strong motor control priors when they approach a new task (Hespos et al., 2009; Baillargeon, 2004; 1994; Wolpert & Ghahramani, 2000).

Original: Here, we have taken some initial steps to explore the importance of such priors in context of human gameplay.
Revised: Here, we have taken some initial steps to explore the importance of such priors in context of human gameplay.

Original: B. G RAVITY
Revised: 5. G RAVITY

Original: One of the most obvious  knowledge that we have about the physical world is with regards to gravity i.e. things fall from up to down.
Revised: One of the most obvious forms of knowledge that we have about the physical world is with regards to gravity, i.e., things fall from up to down.

Original: To mask this prior, we created a version of the game in which the whole game window was rotated 90 ◦ .
Revised: To mask this prior, we created a version of the game in which the whole game window was rotated 90 ◦ (refer to Figure 2(h)).

Original: In this way, the gravity was reversed from left to right (as opposed to up to down).
Revised: In this way, the gravity was reversed from left to right (as opposed to up to down).

Original: As shown in Figure 9, participants spent more time to solve this game compared to the original version with average time taken close to 3 minutes (p-value < 0.01).
Revised: As shown in Figure 7, participants spent more time to solve this game compared to the original version with average time taken close to 3 minutes ( p < 0 . 01 ).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).
Revised: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version ( p < 0 . 01 ).

Original: B. N ON - UNIFORM GRAVITY

Original: In the previous game, although we manipulated the gravity prior by reversing the gravity, participants still had access to more general notions about gravity such as gravity in the game is uniform and constant.

Original: We hypothesized that such a general prior about gravity might guide human exploration in an environment even when the gravity is reversed.

Original: To test this, we modiﬁed the original game such that different platforms in the game had different gravity.

Original: This meant that some platforms had a very strong gravity so that the agent sprite couldn’t jump on these platforms, some platforms had a very weak gravity so that the agent sprite could jump signiﬁcantly higher, and some platforms had moderate gravity.

Original: Thus, in this version, participants had to learn about the dynamics of the game (related to gravity and jumping) from scratch.

Original: As shown in Figure 9, participants took a signiﬁcantly longer time to solve this game compared to the version with reversed gravity with average time taken close to 5 minutes (p-value < 0.01).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the version with reverse gravity (p-value < 0.01).

Original: This suggests that similar to our results on object priors, general priors related to physics (such as uniform gravity) serve a prominent role in guiding efﬁcient human gameplay.

Original: B. M USCLE MEMORY
Revised: 5. M USCLE MEMORY

Original: Human players also come with knowledge of the form such as pressing arrow keys moves the agent sprite in the corresponding directions (i.e. pressing up makes the agent sprite jump, pressing left makes the agent sprite go left and so forth).
Revised: Human players also come with knowledge about the consequences of actions such as pressing arrow keys moves the agent sprite in the corresponding directions (i.e., pressing up makes the agent sprite jump, pressing left makes the agent sprite go left and so forth).

Original: We created a version of the game in which we reversed the arrow key controls.
Revised: We created a version of the game in which we reversed the arrow key controls.

Original: Thus, pressing  left arrow key made the agent sprite go right, pressing  right key moved the sprite left, pressing  down key made the player jump (or go up the stairs), and pressing  up key made the player go down the stairs.
Revised: Thus, pressing the left arrow key made the agent sprite go right, pressing the right key moved the sprite left, pressing the down key made the player jump (or go up the stairs), and pressing the up key made the player go down the stairs.

Original: Participants again took longer to solve this game compared to the original version with average time taken close to 3 minutes (refer to Figure 9).
Revised: Participants again took longer to solve this game compared to the original version with average time taken close to 3 minutes (refer to Figure 7).

Original: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version (p-value < 0.01).
Revised: The average number of deaths and number of states explored was also signiﬁcantly larger than the original version ( p < 0 . 01 ).

Original: Interestingly, the performance of players when the gravity was reversed and key controls were reversed is similar with no signiﬁcant difference between the two conditions.
Revised: Interestingly, the performance of players when the gravity was reversed, and key controls were reversed is similar, with no signiﬁcant difference between the two conditions.

Original:  C P ERFORMANCE OF RL AGENT ON VARIOUS GAME MANIPULATIONS
Revised:  C ONTROLLING FOR CHANGE IN COMPLEXITY

Original: In this section, we investigated how the RL agent (ICM-A3C; Pathak et al.
Revised: So far in this paper, we have manipulated various visual priors while keeping the underlying game and reward structure exactly the same.

Original: performed in each of the lesioned settings we investigated with humans.
Revised: We have assumed that this will inﬂuence human performance while keeping RL agent performance unchanged, since RL does not have any priors to begin with. However, one possible confound is that the visual complexity of the modiﬁed games might have changed from the original game version, because masking out priors without changing visual complexity is extremely difﬁcult. To control for this confound, we investigated the performance of an RL agent on the various game manipulations.

Original: While deep RL agents don’t come with any prior knowledge, they can at least ﬁnd and exploit regularities in data. Thus the experiments in  this section can help shed light on how statistical regularities in the data inﬂuence deep RL agents.
Revised: If RL agents are not affected by the game manipulations, then it would suggest that prior knowledge and not visual complexity is the main reason behind the change in human performance. Note that this confound is not present in the physics and motor control experiments as the visual input stays the same as the original game.

Original: To do this, we systematically created different versions of the game in Figure 1(a) to mask semantics,  concept of object, affordance as well as similarity (refer to Figure 10) and ran 10 random seeds of the RL agent on each of the game versions. Note that, we modiﬁed the game  in Figure 1(a) and not
Revised: To this end, we systematically created different versions of the game in Figure 1(a) to ablate semantics, the concept of object, affordance, and similarity as shown in Figure 8. Note that the game used for human experiments shown in Figure 2 is more complex than the game used for RL experiments in Figure 8.

Original: Figure 2(a) to run the RL experiments as the  gamein Figure 2(a) was  too hard for the RL agent to solve.
Revised: This is because the larger game was simply too hard for state-of-the-art RL agents to solve.

Original: As shown inFigure 11, the RL agent is unaffected by the removal of semantics, concept of objects as well as affordance information – there is no signiﬁcant difference between the group means of the RL agent on these games and  the original version.
Revised: Apart from the difference in the game size, we tried to make the games as similar as possible. Even though this version of the game is simpler (regarding size, number of objects etc.), we note that this game is still non-trivial for an RL agent. For instance, due to the sparse reward structure of the game, both A3C (Mnih et al., 2016) and breadth-ﬁrst search didn’t come close to solving the game even after 10 million steps.

Original: The performance of  the RL agent on game without object information (Figure 10(c)) is especially interesting because this prior information is extremely critical to human gameplay.
Revised: Hence, for our purpose, we used an RL algorithm augmented with a curiosity based exploration strategy (Pathak et al., 2017). For each game version, we report the mean performance of ﬁve random seeds that succeeded. As shown in Figure 8(e), the RL agent was unaffected by the removal of semantics, the concept of objects, as well as affordances – there is no signiﬁcant difference between the mean score of the RL agent on these games when compared to the performance on the original game ( p > 0 . 05 ). This suggests that the drop in human performance in these game manipulations is not due to the change in visual complexity, but it is rather due to the masking of the various priors.

Original: Interestingly, the RL agent is affected by removal of similarity information as it takes nearly two times to solve that version of  the game implying that 
Revised: On the other hand, the performance of the RL agent does worsen when visual similarity is masked as it takes nearly twice as many interactions to complete the game compared to the original version. We believe this is due to to the use of convolutional neural networks that implicitly impose the prior of visual similarity rather than simply due to the change in visual complexity.  D ISCUSSION While there is no doubt that the performance of deep RL algorithms is impressive, there is much to be learned from human cognition if our goal is to enable RL agents to solve sparse reward tasks with human-like efﬁciency.

Original: RL agents do exploit visual similarity in  the data.
Revised: Humans have the amazing ability to use their past knowledge (i.e., priors) to solve new tasks quickly. Success in such scenarios critically depends on the agent’s ability to explore its environment and then promptly learn from its successes (Daw et al., 2006; Cohen et al., 2007). In this vein, our results demonstrate the importance of prior knowledge in helping humans explore efﬁciently in these sparse reward environments (Knox et al., 2012; Gershman & Niv, 2015). However, being equipped with strong prior knowledge can sometimes lead to constrained exploration that might not be optimal in all environments (Lucas et al., 2014; Bonawitz et al., 2011).

Revised: For instance, consider the game shown in Figure 9 consisting of a robot and a princess object.

Revised: The game environment also includes rewards in hidden locations (shown as dashed yellow boxes only for illustration).

Revised: When tasked to play this game, human participants (n= 30 ) immediately assume that princess is the goal and do not explore the free space containing hidden rewards.

Revised: They directly reach the princess and thereby terminate the game with sub-optimal rewards.

Revised: In contrast, a random agent ( 30 seeds) ends up obtaining almost four times more reward than human players as shown in

Original: Future work aims to investigate how this visual similarity is automatically learned.
Revised: Figure 9. Thus, while incorporating prior knowledge in RL agents has many potential beneﬁts, future work should also consider challenges regarding under-constrained exploration in certain kinds of settings. While our paper primarily investigated object priors (and physics priors to some extent), humans also possess rich prior knowledge about the world in the form of intuitive psychology and also bring in various priors about general video game playing such as that moving up and to the right in games is generally correlated with progress, games have goals, etc. Studying the importance of such priors will be an interesting future direction of research.

Original: The RL agent performs similarly on all games except for  the one
Revised: Building RL algorithms that require fewer interactions to reach the goal (i.e., sample efﬁcient algorithms) is an active area of research, and further progress is inevitable. In addition to developing better optimization methods, we believe that instead of always initializing learning from scratch, either incorporating prior knowledge directly or constructing mechanisms for condensing experience into reusable knowledge (i.e., learning priors through continual learning) might be critical for building RL agents with human-like efﬁciency. Our work takes ﬁrst steps toward quantifying the importance of various priors that humans employ in solving video games and in understanding how prior knowledge makes humans good at such complex tasks. We believe that our results will inspire researchers to think about different mechanisms of incorporating prior knowledge in the design of

Revised: RL agents.

Revised: We also hope that our experimental platform of video games, available in open-source, will fuel more detailed studies investigating human priors and a benchmark for quantifying the efﬁcacy of different mechanisms of incorporating prior knowledge into RL agents.

Revised: A CKNOWLEDGEMENT

Revised: We thank Jordan Suchow, Michael Chang, Shubham Tulsiani, Alison Gopnik and other members of the BAIR community for helpful discussions and comments.

Revised: This work has been supported, in part, by Google, ONR MURI N00014-14-1-0671, Berkeley DeepDrive, NVIDIA Graduate Fellowship to

Revised: DP and the Valrhona Reinforcement Learning Fellowship