Original: R ELEVANT A CTIONS M ATTER : M OTIVATING A GENTS WITH A CTION U SEFULNESS
Revised: R ELEVANT A CTIONS M ATTER : M OTIVATING A GENTS WITH A CTION U SEFULNESS

Original: inria.fr
Revised: mathieu.seurin@inria.fr

Original: A BSTRACT
Revised: A BSTRACT

Original: Sparse rewards are double-edged training signals in reinforcement learning: easy to design but hard to optimize.
Revised: Sparse rewards are double-edged training signals in reinforcement learning: easy to design but hard to optimize.

Original: Intrinsic motivation guidances have thus been developed toward alleviating the resulting exploration problem.
Revised: Intrinsic motivation guidances have thus been developed toward alleviating the resulting exploration problem.

Original: They usually incentivize agents to look for new states through novelty signals.
Revised: They usually incentivize agents to look for new states through novelty signals.

Original: Yet, such methods encourage exhaustive exploration of the state space rather than focusing on the environment’s salient interaction opportunities.
Revised: Yet, such methods encourage exhaustive exploration of the state space rather than focusing on the environment’s salient interaction opportunities.

Original: We propose a new exploration method, called Relevant Actions Matter (RAM), shifting the emphasis from state novelty to state with relevant actions.
Revised: We propose a new exploration method, called Relevant Actions Matter (RAM), shifting the emphasis from state novelty to state with relevant actions.

Original: While most actions consistently change the state when used e g. moving the agent, some actions are only effective in speciﬁc states, e g. , opening a door, grabbing an object.
Revised: While most actions consistently change the state when used, e.g. moving the agent, some actions are only effective in speciﬁc states, e.g. , opening a door, grabbing an object.

Original: RAM detects and rewards actions that seldom affect the environment.
Revised: RAM detects and rewards actions that seldom affect the environment.

Original: We evaluate RAM on the procedurallygenerated environment MiniGrid, against state-of-the-art methods and show that
Revised: We evaluate RAM on the procedurallygenerated environment MiniGrid, against state-of-the-art methods and show that

Original: RAM greatly reduces sample complexity
Revised: RAM greatly reduces sample complexity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the reinforcement learning (RL) problem in which an agent learns to interact with its environment optimally w.r.t.
Revised: We consider the reinforcement learning (RL) problem in which an agent learns to interact with its environment optimally w.r.t.

Original: a cumulative function of reward signals collected along its trajectories (Sutton and Barto, 2018; Kaelbling et al. , 1996).
Revised: a cumulative function of reward signals collected along its trajectories (Sutton and Barto, 2018; Kaelbling et al. , 1996).

Original: To do so, an RL agent explores its surrounding, aiming at retrieving the most prominent course of actions, and updates its behavior accordingly.
Revised: To do so, an RL agent explores its surrounding, aiming at retrieving the most prominent course of actions, and updates its behavior accordingly.

Original: When the environment provides abundant rewards, the agent may successfully collect enough training signals by performing random actions (Mnih et al. , 2016; Lillicrap et al. , 2015).
Revised: When the environment provides abundant rewards, the agent may successfully collect enough training signals by performing random actions (Mnih et al. , 2016; Lillicrap et al. , 2015).

Original: But as soon as the environment provides scarce rewards, the agent is reduced to inefﬁciently waver around without being able to update its policy.
Revised: But as soon as the environment provides scarce rewards, the agent is reduced to inefﬁciently waver around without being able to update its policy.

Original: To palliate this lack of training signals, one common method consists in intrinsically motivating the agent to explore its environment using a self-rewarding mechanism (Schmidhuber, 1991; Oudeyer et al. , 2007; 2008; Schmidhuber, 2010).
Revised: To palliate this lack of training signals, one common method consists in intrinsically motivating the agent to explore its environment using a self-rewarding mechanism (Schmidhuber, 1991; Oudeyer et al. , 2007; 2008; Schmidhuber, 2010).

Original: In the online RL literature, a widespread strategy is to augment the sparse extrinsic reward from the environment with a generated dense intrinsic reward that steers exploration (Chentanez et al. , 2005;
Revised: In the online RL literature, a widespread strategy is to augment the sparse extrinsic reward from the environment with a generated dense intrinsic reward that steers exploration (Chentanez et al. , 2005;

Original: S¸ims¸ek and Barto, 2006).
Revised: S¸ims¸ek and Barto, 2006).

Original: Hence, the intrinsic reward should encode a degree of “novelty,” “surprise,” or “curiosity” (Berlyne, 1965) which is often encoded as an estimate of the agent’s visitation frequency of state-action pairs.
Revised: Hence, the intrinsic reward should encode a degree of “novelty,” “surprise,” or “curiosity” (Berlyne, 1965) which is often encoded as an estimate of the agent’s visitation frequency of state-action pairs.

Original: The agent is incentivized to diversely interact with its environment to collect intrinsic rewards, which may ultimately trigger extrinsic rewards.
Revised: The agent is incentivized to diversely interact with its environment to collect intrinsic rewards, which may ultimately trigger extrinsic rewards.

Original: Nonetheless, establishing intrinsic motivation signals remains double-edged as it introduces human-priors, may lead to sub-optimal policies or foster reward hacking behavior.
Revised: Nonetheless, establishing intrinsic motivation signals remains double-edged as it introduces human-priors, may lead to sub-optimal policies or foster reward hacking behavior.

Original: All in all, different novelty measures have been studied, where each of them entails different exploration behaviors.
Revised: All in all, different novelty measures have been studied, where each of them entails different exploration behaviors.

Original: For instance, count-based methods keep counts of previous observations to bait the agent to explore unseen states (Lopes et al. , 2012; Ostrovski et al. , 2017; Bellemare et al. , 2016; Ecoffet et al. , 2019).
Revised: For instance, count-based methods keep counts of previous observations to bait the agent to explore unseen states (Lopes et al. , 2012; Ostrovski et al. , 2017; Bellemare et al. , 2016; Ecoffet et al. , 2019).

Original: Yet, these approaches implicitly encourage an exhaustive search of the state space.
Revised: Yet, these approaches implicitly encourage an exhaustive search of the state space.

Original: Differently, curiosity-based methods train a model that encapsulates the environment dynamics, before nudging the agent to visit state-transitions with high prediction errors (Pathak et al. , 2017; Burda et al. , 2018; Haber et al. , 2018; Houthooft et al. , 2016) or large change in the value of state features (Raileanu and Rockt¨aschel, 2019).
Revised: Differently, curiosity-based methods train a model that encapsulates the environment dynamics, before nudging the agent to visit state-transitions with high prediction errors (Pathak et al. , 2017; Burda et al. , 2018; Haber et al. , 2018; Houthooft et al. , 2016) or large change in the value of state features (Raileanu and Rockt¨aschel, 2019).

Original: However, the ﬁrst category suffers from reward decay across episodes and poor generalization within procedurally-generated environments.
Revised: However, the ﬁrst category suffers from reward decay across episodes and poor generalization within procedurally-generated environments.

Original: We here observe that the second category insufﬁciently favors exploration towards novel and useful actions.
Revised: We here observe that the second category insufﬁciently favors exploration towards novel and useful actions.

Original: In this paper, we therefore aim to shift the emphasis from state novelty distributions towards novel action distributions to develop new intrinsic motivation signals, and consequently, change the exploration behavior.
Revised: In this paper, we therefore aim to shift the emphasis from state novelty distributions towards novel action distributions to develop new intrinsic motivation signals, and consequently, change the exploration behavior.

Original: More precisely, we aim at encouraging the agent to visit states that allow rare and relevant actions, i.e. actions that can only be performed in rare occasions.
Revised: More precisely, we aim at encouraging the agent to visit states that allow rare and relevant actions, i.e. actions that can only be performed in rare occasions.

Original: Imagine that an infant discovers that pushing a button triggers a light; s/he is likely to push everywhere to switch on new lights.
Revised: Imagine that an infant discovers that pushing a button triggers a light; s/he is likely to push everywhere to switch on new lights.

Original: By repeating his/her action, the infant may eventually uncover new buttons, and start associating the action push to the relevant state features of buttons .
Revised: By repeating his/her action, the infant may eventually uncover new buttons, and start associating the action push to the relevant state features of buttons .

Original: A similar observation can be made within virtual environments and embodied agents.
Revised: A similar observation can be made within virtual environments and embodied agents.

Original: We expect the agent to ﬁrst detect rare actions to learn while being nudged towards the states that allow performing such actions.
Revised: We expect the agent to ﬁrst detect rare actions to learn while being nudged towards the states that allow performing such actions.

Original: In this spirit, we propose a new approach we name Relevant Actions Matter (RAM).
Revised: In this spirit, we propose a new approach we name Relevant Actions Matter (RAM).

Original: Instead of uniformly seeking for novel states, RAM encourages exploring states allowing actions that are rarely useful; those rarely relevant actions are generally hard to retrieve by random exploration.
Revised: Instead of uniformly seeking for novel states, RAM encourages exploring states allowing actions that are rarely useful; those rarely relevant actions are generally hard to retrieve by random exploration.

Original: In other words, the agent is intrinsically rewarded when successfully performing an action that is usually ineffectual.
Revised: In other words, the agent is intrinsically rewarded when successfully performing an action that is usually ineffectual.

Original: We observe that this simple mechanism induces a remarkably different exploration behavior differing from the common state-count and curiosity-based patterns.
Revised: We observe that this simple mechanism induces a remarkably different exploration behavior differing from the common state-count and curiosity-based patterns.

Original: Formally, RAM keeps track of two quantities for each action: the number of times the action has been used and the number of times the action led to a state change.
Revised: Formally, RAM keeps track of two quantities for each action: the number of times the action has been used and the number of times the action led to a state change.

Original: The resulting intrinsic reward is inversely proportional to the number of times the action has led to a state change.
Revised: The resulting intrinsic reward is inversely proportional to the number of times the action has led to a state change.

Original: Noticeably, RAM primarily keeps count of actions, and can thus be deﬁned as an action count-based method.
Revised: Noticeably, RAM primarily keeps count of actions, and can thus be deﬁned as an action count-based method.

Original: Besides, tracking actions (as opposed to states) naturally scales in RL: in the discrete case, there is generally less than a few thousand actions, allowing for an exact count.
Revised: Besides, tracking actions (as opposed to states) naturally scales in RL: in the discrete case, there is generally less than a few thousand actions, allowing for an exact count.

Original: In the continuous case, actions may easily be discretized without using complex density models (Tang and Agrawal, 2020).
Revised: In the continuous case, actions may easily be discretized without using complex density models (Tang and Agrawal, 2020).

Original: This paper ﬁrst provides an overview of recent exploration methods before introducing RAM as an action-driven intrinsic motivation method.
Revised: This paper ﬁrst provides an overview of recent exploration methods before introducing RAM as an action-driven intrinsic motivation method.

Original: We then study this approach in the MiniGrid procedurally generated environment (Chevalier-Boisvert et al. , 2018).
Revised: We then study this approach in the MiniGrid procedurally generated environment (Chevalier-Boisvert et al. , 2018).

Original: Despite their apparent simplicity, these tasks contain intermediate decisive actions, e.g. picking keys, which have kept in check advanced exploration methods (Raileanu and Rockt¨aschel, 2019; Campero et al. , 2020).
Revised: Despite their apparent simplicity, these tasks contain intermediate decisive actions, e.g. picking keys, which have kept in check advanced exploration methods (Raileanu and Rockt¨aschel, 2019; Campero et al. , 2020).

Original: We empirically show that RAM reduces the sample complexity by a factor of 2 to 10 in a diverse set of environments while resolving the hardest tasks.
Revised: We empirically show that RAM reduces the sample complexity by a factor of 2 to 10 in a diverse set of environments while resolving the hardest tasks.

Original: We then study how RAM amends the agent’s behavior and compare it to other methods.
Revised: We then study how RAM amends the agent’s behavior and compare it to other methods.

Original: Finally, we also analyze whether RAM may lead to unwanted agent behavior when facing environment with multiple interactions, which we refer as the BallPit-problem .
Revised: Finally, we also analyze whether RAM may lead to unwanted agent behavior when facing environment with multiple interactions, which we refer as the BallPit-problem .

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: RL algorithms require the agents to acquire knowledge about their environment to update their policy; exploration has thus been one of the longest running problems of RL (Sutton and Barto, 2018;
Revised: RL algorithms require the agents to acquire knowledge about their environment to update their policy; exploration has thus been one of the longest running problems of RL (Sutton and Barto, 2018;

Original: Mozer and Bachrach, 1990; Sato et al. , 1988; Schmidhuber, 1991; Barto and Singh, 1991).
Revised: Mozer and Bachrach, 1990; Sato et al. , 1988; Schmidhuber, 1991; Barto and Singh, 1991).

Original: Exploration methods have quickly been categorized into two broad categories: directed and undirected exploration (Thrun, 1992).
Revised: Exploration methods have quickly been categorized into two broad categories: directed and undirected exploration (Thrun, 1992).

Original: On the one hand, undirected exploration does not use any domain knowledge and ensures exploration by introducing stochasticity in the agent’s policy.
Revised: On the one hand, undirected exploration does not use any domain knowledge and ensures exploration by introducing stochasticity in the agent’s policy.

Original: This approach includes methods such as random walk, (cid:15) -greedy, or Boltzmann exploration.
Revised: This approach includes methods such as random walk, (cid:15) -greedy, or Boltzmann exploration.

Original: Although they enable learning the optimal policy in the tabular setting, they require a number of steps that grows exponentially with the state space (Whitehead, 1991; Kakade and others, 2003; Strehl et al. , 2006).
Revised: Although they enable learning the optimal policy in the tabular setting, they require a number of steps that grows exponentially with the state space (Whitehead, 1991; Kakade and others, 2003; Strehl et al. , 2006).

Original: Despite this inherent lack of sample efﬁciency, they remain valuable task-agnostic exploration strategies in large-scale problems with dense rewards (Mnih et al. , 2016).
Revised: Despite this inherent lack of sample efﬁciency, they remain valuable task-agnostic exploration strategies in large-scale problems with dense rewards (Mnih et al. , 2016).

Original: Besides, recent undirected exploration methods have been developed to ﬁt deep neural architectures such as injecting random noise in the network parameter space (Fortunato et al. , 2018;
Revised: Besides, recent undirected exploration methods have been developed to ﬁt deep neural architectures such as injecting random noise in the network parameter space (Fortunato et al. , 2018;

Original: Plappert et al. , 2018).
Revised: Plappert et al. , 2018).

Original: Yet, undirected methods still struggle with sparse reward signals (Plappert et al. , 2018), or any task requiring deep exploration (Osband et al. , 2016).
Revised: Yet, undirected methods still struggle with sparse reward signals (Plappert et al. , 2018), or any task requiring deep exploration (Osband et al. , 2016).

Original: On the other hand, directed methods incorporate external priors to orient the exploration strategy through diverse heuristics or measures.
Revised: On the other hand, directed methods incorporate external priors to orient the exploration strategy through diverse heuristics or measures.

Original: Among others, uncertainty has been used to guide exploration towards ill-estimated state-action pairs by relying on the Bellman equation (Geist and
Revised: Among others, uncertainty has been used to guide exploration towards ill-estimated state-action pairs by relying on the Bellman equation (Geist and

Original: Pietquin, 2010; O’Donoghue et al. , 2018) or by bootstrapping multiple Q-functions (Osband et al. , 2016).
Revised: Pietquin, 2010; O’Donoghue et al. , 2018) or by bootstrapping multiple Q-functions (Osband et al. , 2016).

Original: Despite being theoretically sound, these methods face scaling difﬁculties.
Revised: Despite being theoretically sound, these methods face scaling difﬁculties.

Original: In this paper, we study another directed exploration approach based on reward bonuses to densify the reward signal.
Revised: In this paper, we study another directed exploration approach based on reward bonuses to densify the reward signal.

Original: In this setting, the environment reward, namely extrinsic rewards, is augmented with an exploration guidance reward signal, namely intrinsic rewards (Chentanez et al. , 2005; S¸ims¸ek and Barto, 2006).
Revised: In this setting, the environment reward, namely extrinsic rewards, is augmented with an exploration guidance reward signal, namely intrinsic rewards (Chentanez et al. , 2005; S¸ims¸ek and Barto, 2006).

Original: This intrinsic reward spurs exploration by tipping the agent to take a speciﬁc course of actions.
Revised: This intrinsic reward spurs exploration by tipping the agent to take a speciﬁc course of actions.

Original: Furthermore, it makes undirected exploration mechanism applicable again by spreading milestone rewards during training.
Revised: Furthermore, it makes undirected exploration mechanism applicable again by spreading milestone rewards during training.

Original: Inspired by cognitive science, this intrinsic reward often encodes a degree of “novelty,” “surprise,” ,“curiosity” (Oudeyer et al. , 2007; Berlyne, 1965; Schmidhuber, 1991) , “learning progress” (Lopes et al. , 2012) or “boredom” (Schmidhuber, 1991; Oudeyer et al. , 2008).
Revised: Inspired by cognitive science, this intrinsic reward often encodes a degree of “novelty,” “surprise,” ,“curiosity” (Oudeyer et al. , 2007; Berlyne, 1965; Schmidhuber, 1991) , “learning progress” (Lopes et al. , 2012) or “boredom” (Schmidhuber, 1991; Oudeyer et al. , 2008).

Original: These common intrinsic motivation mechanisms are broadly categorized into three families: countbased, curiosity-based, and goal-based methods.
Revised: These common intrinsic motivation mechanisms are broadly categorized into three families: countbased, curiosity-based, and goal-based methods.

Original: Count-based exploration aims to catalog visited states (or action-states pairs) along episodes to detect unseen states, and drive the agent towards them.
Revised: Count-based exploration aims to catalog visited states (or action-states pairs) along episodes to detect unseen states, and drive the agent towards them.

Original: It has ﬁrst been proposed as an exploration heuristic in the early days of RL (Thrun, 1992; Sato et al. , 1988; Barto and Singh, 1991) before being framed as an intrinsic exploration reward mechanisms in the tabular case (Strehl and Littman, 2008;
Revised: It has ﬁrst been proposed as an exploration heuristic in the early days of RL (Thrun, 1992; Sato et al. , 1988; Barto and Singh, 1991) before being framed as an intrinsic exploration reward mechanisms in the tabular case (Strehl and Littman, 2008;

Original: Kolter and Ng, 2009).
Revised: Kolter and Ng, 2009).

Original: Pseudo-counts were then introduced to approximate the state counts (Lopes et al. , 2012), where pseudo-counts were estimated through different density models to produce intrinsic rewards.
Revised: Pseudo-counts were then introduced to approximate the state counts (Lopes et al. , 2012), where pseudo-counts were estimated through different density models to produce intrinsic rewards.

Original: Density models range from raw image downscaling with or without handcrafted state features (Ecoffet et al. , 2019; Zhang et al. , 2020), contextual trees (Bellemare et al. , 2016), generative neural models, e.g. PixelCNN (Ostrovski et al. , 2017), or autoencoders combined with a local hashing function (Tang et al. , 2017).
Revised: Density models range from raw image downscaling with or without handcrafted state features (Ecoffet et al. , 2019; Zhang et al. , 2020), contextual trees (Bellemare et al. , 2016), generative neural models, e.g. PixelCNN (Ostrovski et al. , 2017), or autoencoders combined with a local hashing function (Tang et al. , 2017).

Original: Differently, Burda et al.
Revised: Differently, Burda et al.

Original: (2018) use the prediction error between a randomly initialized network and a trained network as a state-count proxy.
Revised: (2018) use the prediction error between a randomly initialized network and a trained network as a state-count proxy.

Original: Yet, count-based methods may explore the immediate surrounding and heavily depend on the state representation quality.
Revised: Yet, count-based methods may explore the immediate surrounding and heavily depend on the state representation quality.

Original: By shifting the emphasis on counting action, we thus address these representation constraints and push for distant interactions.
Revised: By shifting the emphasis on counting action, we thus address these representation constraints and push for distant interactions.

Original: Curiosity-based exploration aims to encourage the agent to uncover the environment dynamics rather than cataloging states.
Revised: Curiosity-based exploration aims to encourage the agent to uncover the environment dynamics rather than cataloging states.

Original: Inspired by cognitive science, such agents learn a world model predicting the consequences of their actions; and they take an interest in challenging and reﬁning it (Haber et al. , 2018; Oudeyer et al. , 2007).
Revised: Inspired by cognitive science, such agents learn a world model predicting the consequences of their actions; and they take an interest in challenging and reﬁning it (Haber et al. , 2018; Oudeyer et al. , 2007).

Original: In RL, this intuition is transposed by taking the current state and action to predict the next state representation; the resulting prediction error is then turned into the intrinsic reward signal.
Revised: In RL, this intuition is transposed by taking the current state and action to predict the next state representation; the resulting prediction error is then turned into the intrinsic reward signal.

Original: Approaches mostly differ in learning the state representation: Stadie et al.
Revised: Approaches mostly differ in learning the state representation: Stadie et al.

Original: (2015) compress raw observation with autoencoders, Burda et al.
Revised: (2015) compress raw observation with autoencoders, Burda et al.

Original: (2019) use random projections, Houthooft et al.
Revised: (2019) use random projections, Houthooft et al.

Original: (2016) capture the environment stochasticity by maximizing mutual information with Bayesian
Revised: (2016) capture the environment stochasticity by maximizing mutual information with Bayesian

Original: Networks.
Revised: Networks.

Original: In parallel, Pathak et al.
Revised: In parallel, Pathak et al.

Original: (2017) argue that the state representation should mainly encode features altered by the agent.
Revised: (2017) argue that the state representation should mainly encode features altered by the agent.

Original: They thus introduce an inverse model that predicts the action given two consequent states as a training signal.
Revised: They thus introduce an inverse model that predicts the action given two consequent states as a training signal.

Original: Yet, those intrinsic rewards based on prediction errors may attract the agent into irrelevant yet unpredictable transitions.
Revised: Yet, those intrinsic rewards based on prediction errors may attract the agent into irrelevant yet unpredictable transitions.

Original: Another drawback is reward evanescence: the intrinsic reward slowly vanishes as the model is getting better.
Revised: Another drawback is reward evanescence: the intrinsic reward slowly vanishes as the model is getting better.

Original: Schmidhuber (1991) originally proposed to measure the mean error evolution rather than immediate errors to account for the agent progress.
Revised: Schmidhuber (1991) originally proposed to measure the mean error evolution rather than immediate errors to account for the agent progress.

Original: Differently, Raileanu and Rockt¨aschel (2019) replace the error prediction by the difference between consecutive representation states, removing the need to compute a vanishing prediction error.
Revised: Differently, Raileanu and Rockt¨aschel (2019) replace the error prediction by the difference between consecutive representation states, removing the need to compute a vanishing prediction error.

Original: In this paper, we also compare successive states in a similar spirit, but we use it to catalog actions and bias state visitation through a different exploration scheme.
Revised: In this paper, we also compare successive states in a similar spirit, but we use it to catalog actions and bias state visitation through a different exploration scheme.

Original: Goal-based methods (Colas et al. , 2020) provide identiﬁable and intermediate goals to reward the agent upon completion.
Revised: Goal-based methods (Colas et al. , 2020) provide identiﬁable and intermediate goals to reward the agent upon completion.

Original: Such approaches perform an explicit curriculum by slowly increasing the exploration depth through goal difﬁculties.
Revised: Such approaches perform an explicit curriculum by slowly increasing the exploration depth through goal difﬁculties.

Original: They often build on top of the UVFA framework to condition the agent policy (Schaul et al. , 2015).
Revised: They often build on top of the UVFA framework to condition the agent policy (Schaul et al. , 2015).

Original: Goal-based methods may take several forms ranging from hindsight experience replay (Andrychowicz et al. , 2017), adversarial goal-generation (Forestier et al. , 2017; Campero et al. , 2020) and hand-crafted goals.
Revised: Goal-based methods may take several forms ranging from hindsight experience replay (Andrychowicz et al. , 2017), adversarial goal-generation (Forestier et al. , 2017; Campero et al. , 2020) and hand-crafted goals.

Original: Yet, they may face to unstable training, complex goal deﬁnition, or require fully observable environment (Campero et al. , 2020).
Revised: Yet, they may face to unstable training, complex goal deﬁnition, or require fully observable environment (Campero et al. , 2020).

Original: Other forms of intrinsic reward have been explored with empowerment (Mohamed and Jimenez Rezende, 2015) or trajectory diversities (Savinov et al. , 2018), but they are facing scalability issues.
Revised: Other forms of intrinsic reward have been explored with empowerment (Mohamed and Jimenez Rezende, 2015) or trajectory diversities (Savinov et al. , 2018), but they are facing scalability issues.

Original: Hussenot et al.
Revised: Hussenot et al.

Original: (2020) also tried to retrieve intrinsic motivation signals from human trajectories through inverse reinforcement learning.
Revised: (2020) also tried to retrieve intrinsic motivation signals from human trajectories through inverse reinforcement learning.

Original: Finally, intrinsic motivation have been explored in hierarchical reinforcement learning (Barto et al. , 2004; Kulkarni et al. , 2016), but it goes beyond the scope of this paper.
Revised: Finally, intrinsic motivation have been explored in hierarchical reinforcement learning (Barto et al. , 2004; Kulkarni et al. , 2016), but it goes beyond the scope of this paper.

Original:  R EINFORCEMENT L EARNING B ACKGROUND
Revised:  R EINFORCEMENT L EARNING B ACKGROUND

Original: Notation The environment is modeled as a Markov Decision Problem (MDP), where the MDP is deﬁned as a tuple {S , A , P , R , γ } .
Revised: Notation The environment is modeled as a Markov Decision Problem (MDP), where the MDP is deﬁned as a tuple {S , A , P , R , γ } .

Original: At each time step t , the agent is in a state s t ∈ S , where it selects an action a t ∈ A according to its policy π : S → A .
Revised: At each time step t , the agent is in a state s t ∈ S , where it selects an action a t ∈ A according to its policy π : S → A .

Original: It then receives a reward r t from the environment’s reward function r : S ×
Revised: It then receives a reward r t from the environment’s reward function r : S ×

Original: A × S → R and moves to the next state s t + with probability p ( s t +1 | s t , a t ) according to the transition kernel P .
Revised: A × S → R and moves to the next state s t + with probability p ( s t +1 | s t , a t ) according to the transition kernel P .

Original: Hence, the agent generates a trajectory τ =
Revised: Hence, the agent generates a trajectory τ =

Original: [ s 0 , a 0 , r 0 , s 1 , r 1 , a 1 , . .
Revised: [ s 0 , a 0 , r 0 , s 1 , r 1 , a 1 , . .

Original: , s T , a T , r T ] of length T .
Revised: , s T , a T , r T ] of length T .

Original: In practice, the policy is often parameterized by a weight vector θ ∈ Θ .
Revised: In practice, the policy is often parameterized by a weight vector θ ∈ Θ .

Original: The goal is then to search for the optimal policy π θ ∗ that maximizes the expected return J ( θ ) =
Revised: The goal is then to search for the optimal policy π θ ∗ that maximizes the expected return J ( θ ) =

Original: E π θ (cid:2) (cid:80) t =

Revised: E π θ (cid:2) (cid:80) t =

Original: γ t r ( s t , a t , s t +1 ) (cid:3) by directly optimizing the policy parameters θ .
Revised: γ t r ( s t , a t , s t +1 ) (cid:3) by directly optimizing the policy parameters θ .

Original: Intrinsic Motivation In this setting, the reward function is decomposed into an extrinsic reward returned by the environment r e ( s t , a t ) and a new intrinsic reward r
Revised: Intrinsic Motivation In this setting, the reward function is decomposed into an extrinsic reward returned by the environment r e ( s t , a t ) and a new intrinsic reward r

Original: i ( s t , a t , s t +1 ) .
Revised: i ( s t , a t , s t +1 ) .

Original: Therefore, the new reward function is deﬁned as r ( s t , a t , s t +1 ) =
Revised: Therefore, the new reward function is deﬁned as r ( s t , a t , s t +1 ) =

Original: r e ( s t , a t , s t +1 )
Revised: r e ( s t , a t , s t +1 )

Original: + βr i ( s t , a t , s t +1 ) where β is an hyperparameter to balance the two return signals.
Revised: + βr i ( s t , a t , s t +1 ) where β is an hyperparameter to balance the two return signals.

Original: In practice, the extrinsic reward is often a sparse task-speciﬁc signal while the intrinsic reward is usually a dense training signal that fosters exploration.
Revised: In practice, the extrinsic reward is often a sparse task-speciﬁc signal while the intrinsic reward is usually a dense training signal that fosters exploration.

Original:  R ELEVANT A CTIONS M ATTER
Revised:  R ELEVANT A CTIONS M ATTER

Original: Intuition While most actions consistently move the agent to a new state, some actions do not affect speciﬁc states, i.e., the agent remains in the same state after performing it.
Revised: Intuition While most actions consistently move the agent to a new state, some actions do not affect speciﬁc states, i.e., the agent remains in the same state after performing it.

Original: We hence deﬁne an effective action if the new state of the environment is different from what it would have been if no action were to be taken.
Revised: We hence deﬁne an effective action if the new state of the environment is different from what it would have been if no action were to be taken.

Original: For instance, in tasks involving embodied interaction, such state-action pairs include moving forward while facing a wall or grabbing non-existent objects.
Revised: For instance, in tasks involving embodied interaction, such state-action pairs include moving forward while facing a wall or grabbing non-existent objects.

Original: Although one may update the MDP only to keep effective actions, such an operation may not always be feasible or desirable in practice.
Revised: Although one may update the MDP only to keep effective actions, such an operation may not always be feasible or desirable in practice.

Original: It is thus up to the agent to learn the correct actionable states through exploration.
Revised: It is thus up to the agent to learn the correct actionable states through exploration.

Original: Noticeably, those rare state-actions are often landmarks in the environment dynamics, e.g., triggering buttons or opening doors.
Revised: Noticeably, those rare state-actions are often landmarks in the environment dynamics, e.g., triggering buttons or opening doors.

Original: One idea is thus to bias the agent to visit states that effectively allow rare actions.
Revised: One idea is thus to bias the agent to visit states that effectively allow rare actions.

Original: RAM encapsulates this exploration pattern by (1) detecting rare but effective actions, (2) rewarding the agent when effectively performing these rare actions.
Revised: RAM encapsulates this exploration pattern by (1) detecting rare but effective actions, (2) rewarding the agent when effectively performing these rare actions.

Original: In short, rare and effective actions are the relevant actions that matter.
Revised: In short, rare and effective actions are the relevant actions that matter.

Original: Method For every action a i , the agent tracks two quantities.
Revised: Method For every action a i , the agent tracks two quantities.

Original: The number U of times an action has been u sed during past trajectories, and the number E of times the action was e ffective, i.e. change the state s t (cid:54) =
Revised: The number U of times an action has been u sed during past trajectories, and the number E of times the action was e ffective, i.e. change the state s t (cid:54) =

Original: s t +1 .
Revised: s t +1 .

Original: Formally, given the whole history of transitions (across all episodes)
Revised: Formally, given the whole history of transitions (across all episodes)

Revised: H =

Revised: ( s

Original: [Equation]
Revised: [Equation]

Original:  s h
Revised: a h , s h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: =0 :

Original: [Equation]
Revised: [Equation]

Original: where 1 is the indicator function and × the product operator.
Revised: where 1 is the indicator function and × the product operator.

Original: Intuitively, the ratio E H ( a ) /U H ( a ) indicates how often the action a has been effective along the history H .
Revised: Intuitively, the ratio E H ( a ) /U H ( a ) indicates how often the action a has been effective along the history H .

Original: For instance, actions that move an agent would update the state most of the time, therefore
Revised: For instance, actions that move an agent would update the state most of the time, therefore

Original: U ( a i ) ≈ E ( a i ) .
Revised: U ( a i ) ≈ E ( a i ) .

Original: On the other hand, grabbing objects only changes the state in rare occurrences, and
Revised: On the other hand, grabbing objects only changes the state in rare occurrences, and

Original: U ( a i ) ≥ E ( a i ) .
Revised: U ( a i ) ≥ E ( a i ) .

Original: We then deﬁne the bonus as:
Revised: We then deﬁne the bonus as:

Original: [Equation]
Revised: [Equation]

Original: where η is a hyperparameter.
Revised: where η is a hyperparameter.

Original: This function is a continuous approximation of an exponential decay exp − ηE H ( a t ) /U
Revised: This function is a continuous approximation of an exponential decay exp − ηE H ( a t ) /U

Original: H ( a t ) .
Revised: H ( a t ) .

Original: It ranges from 1 when E H = 0 and goes to 0 when E H =
Revised: It ranges from 1 when E H = 0 and goes to 0 when E H =

Original: U H .
Revised: U H .

Original: Small η leads to a uniform bonus on all actions whereas, large values favor rare and efﬁcient actions
Revised: Small η leads to a uniform bonus on all actions whereas, large values favor rare and efﬁcient actions

Original: An intrinsic reward mechanism often requires to discount the intrinsic bonus within an episode.
Revised: An intrinsic reward mechanism often requires to discount the intrinsic bonus within an episode.

Original: Hence, it prevents the agent from overexploiting, and being stuck in local exploration minima.
Revised: Hence, it prevents the agent from overexploiting, and being stuck in local exploration minima.

Original: Inspired by theoretically sound count-based methods (Strehl and Littman, 2008), we thus divide the previous ratio by an episodic state-count.
Revised: Inspired by theoretically sound count-based methods (Strehl and Littman, 2008), we thus divide the previous ratio by an episodic state-count.

Original: Finally, we want to reward action only in context where they are effective, thus the agent is rewarded only when st (cid:54) =
Revised: Finally, we want to reward action only in context where they are effective, thus the agent is rewarded only when st (cid:54) =

Original: s t +1 , deﬁning the ﬁnal RAM intrinsic reward:
Revised: s t +1 , deﬁning the ﬁnal RAM intrinsic reward:

Original: [Equation]
Revised: [Equation]

Original: where N τ ( s ) = (cid:80) th =0 1 { s = s h } is an episodic state count which is reset at the beginning of each episode.
Revised: where N τ ( s ) = (cid:80) th =0 1 { s = s h } is an episodic state count which is reset at the beginning of each episode.

Original: In high-dimensional state space, the episodic state count can be replaced by a pseudocount (Bellemare et al. , 2016) or an episodic novelty mechanism (Badia et al. , 2020).
Revised: In high-dimensional state space, the episodic state count can be replaced by a pseudocount (Bellemare et al. , 2016) or an episodic novelty mechanism (Badia et al. , 2020).

Original: Action-based Counter
Revised: Action-based Counter

Original: As counting methods may sound anachronistic, we emphasize again that actions are ascertainable in RL, i.e. they can be easily enumerated.
Revised: As counting methods may sound anachronistic, we emphasize again that actions are ascertainable in RL, i.e. they can be easily enumerated.

Original: As opposed to state-counting which requires complex density models (Ostrovski et al. , 2017), discrete action suffers less from the curse of dimensionality, and can easily be binned together in the case of a large action set (DulacArnold et al. , 2015).
Revised: As opposed to state-counting which requires complex density models (Ostrovski et al. , 2017), discrete action suffers less from the curse of dimensionality, and can easily be binned together in the case of a large action set (DulacArnold et al. , 2015).

Original: Besides, although RAM relies on an episodic state count, a raw approximation is sufﬁcient as it encodes a reward decay.
Revised: Besides, although RAM relies on an episodic state count, a raw approximation is sufﬁcient as it encodes a reward decay.

Original:  E XPERIMENTAL SETTINGS
Revised:  E XPERIMENTAL SETTINGS

Original: We evaluate RAM in the procedurally-generated environments MiniGrid (Chevalier-Boisvert et al. , 2018).
Revised: We evaluate RAM in the procedurally-generated environments MiniGrid (Chevalier-Boisvert et al. , 2018).

Original: MiniGrid is a partially observable 2D gridworld with a diverse set of tasks.
Revised: MiniGrid is a partially observable 2D gridworld with a diverse set of tasks.

Original: The RL agent needs to collect items and open locked doors before reaching a ﬁnal destination.
Revised: The RL agent needs to collect items and open locked doors before reaching a ﬁnal destination.

Original: Despite its apparent simplicity, several MiniGrid environments require the agent to perform exploration with few speciﬁc interactions, and have kept in check state-of-the-art exploration procedures (Raileanu and
Revised: Despite its apparent simplicity, several MiniGrid environments require the agent to perform exploration with few speciﬁc interactions, and have kept in check state-of-the-art exploration procedures (Raileanu and

Original: Rockt¨aschel, 2019).
Revised: Rockt¨aschel, 2019).

Original: For each experiment, we report the rolling mean (over 40k timesteps) and standard deviation over5 seeds.
Revised: For each experiment, we report the rolling mean (over 40k timesteps) and standard deviation over seeds.

Original: [Figure]
Revised: [Figure]

Original: 5. M INI G RID E NVIRONMENT
Revised: 5. M INI G RID E NVIRONMENT

Original: Each new MiniGrid world contains a combination of rooms that are populated with objects (balls, boxes or keys)
Revised: Each new MiniGrid world contains a combination of rooms that are populated with objects (balls, boxes or keys)

Original: [Figure]
Revised: [Figure]

Original: and are linked together through (locked/unlocked) doors.
Revised: and are linked together through (locked/unlocked) doors.

Original: Balls and keys can be picked up or dropped and the agent may only carry one of them at a time.
Revised: Balls and keys can be picked up or dropped and the agent may only carry one of them at a time.

Original: Boxes can be opened to discover a hidden colored key.
Revised: Boxes can be opened to discover a hidden colored key.

Original: Doors can be unlocked with keys matching their color.
Revised: Doors can be unlocked with keys matching their color.

Original: The agent is rewarded after reaching the goal tile.
Revised: The agent is rewarded after reaching the goal tile.

Original: At each step, the agent observes a 7x7 representation of its ﬁeld of view and the item it carries if any.
Revised: At each step, the agent observes a 7x7 representation of its ﬁeld of view and the item it carries if any.

Original: The agent may perform one out of seven actions: move forward, turn right, turn left, pick-up object, drop object, toggle.
Revised: The agent may perform one out of seven actions: move forward, turn right, turn left, pick-up object, drop object, toggle.

Original: Noticeably, some actions are ineffective in speciﬁc states, e.g. moving forward in front of a wall, picking-up/dropping/toggling objects when none is available.
Revised: Noticeably, some actions are ineffective in speciﬁc states, e.g. moving forward in front of a wall, picking-up/dropping/toggling objects when none is available.

Original: Following (Raileanu and Rockt ¨ aschel, 2019; Campero et al. , 2020), we focus on three hard exploration tasks, which are illustrated in Figure 2.
Revised: Following (Raileanu and Rockt ¨ aschel, 2019; Campero et al. , 2020), we focus on three hard exploration tasks, which are illustrated in Figure 2.

Original: MultiRoom( N - S ):
Revised: MultiRoom( N - S ):

Original: The agent must navigate through a sequence of empty rooms connected by doors of different colors.
Revised: The agent must navigate through a sequence of empty rooms connected by doors of different colors.

Original: A map contains N rooms, whose indoor width and height are sampled within 2 and S − 2 tiles.
Revised: A map contains N rooms, whose indoor width and height are sampled within 2 and S − 2 tiles.

Original: MultiRoom maps entail limited interaction as the agent only has to toggle doors and no object manipulation is required.
Revised: MultiRoom maps entail limited interaction as the agent only has to toggle doors and no object manipulation is required.

Original: Yet, this bare-bone environment constitutes a good preliminary trial.
Revised: Yet, this bare-bone environment constitutes a good preliminary trial.

Original: KeyCorridor( S - R ): The agent must explore multiple adjacent unlocked rooms to retrieve a key, open the remaining locked room, and collect the green ball.
Revised: KeyCorridor( S - R ): The agent must explore multiple adjacent unlocked rooms to retrieve a key, open the remaining locked room, and collect the green ball.

Original: A map contains a large main corridor connected to 2 × R square rooms of ﬁxed indoor dimension S − 2 .
Revised: A map contains a large main corridor connected to 2 × R square rooms of ﬁxed indoor dimension S − 2 .

Original: Solving a KeyCorridor map requires the agent to perform a speciﬁc sequence of interactions, which makes the task more difﬁcult than MultiRoom.
Revised: Solving a KeyCorridor map requires the agent to perform a speciﬁc sequence of interactions, which makes the task more difﬁcult than MultiRoom.

Original: ObstructedMaze:
Revised: ObstructedMaze:

Original: The agent must explore a grid of rooms that are randomly connected to each others in order to collect a blue ball.
Revised: The agent must explore a grid of rooms that are randomly connected to each others in order to collect a blue ball.

Original: Some of the doors are locked and the agent has to either directly collect the keys or toggle boxes to reveal them.
Revised: Some of the doors are locked and the agent has to either directly collect the keys or toggle boxes to reveal them.

Original: Besides, distractor balls are added to block door access.
Revised: Besides, distractor balls are added to block door access.

Original: ObstructedMaze can quickly become a hard maze with false leads and complex interactions.
Revised: ObstructedMaze can quickly become a hard maze with false leads and complex interactions.

Original: 5. E XPERIMENTAL S ETTING
Revised: 5. E XPERIMENTAL S ETTING

Original: Training We follow the training protocol deﬁned in (Raileanu and Rockt¨aschel, 2019; Campero et al. , 2020).
Revised: Training We follow the training protocol deﬁned in (Raileanu and Rockt¨aschel, 2019; Campero et al. , 2020).

Original: We use 3 convolution layers with a kernel size of 3, followed by 2 fully-connected layers of size 1024, and an LSTM of hidden size 1024.
Revised: We use 3 convolution layers with a kernel size of 3, followed by 2 fully-connected layers of size 1024, and an LSTM of hidden size 1024.

Original: Finally, we use two separate fully-connected layers of size 1024 for the actor’s and critic’s head.
Revised: Finally, we use two separate fully-connected layers of size 1024 for the actor’s and critic’s head.

Original: We train our model with the distributed actor-critic
Revised: We train our model with the distributed actor-critic

Original: [Figure]
Revised: [Figure]

Original: algorithm IMPALA (Espeholt et al. , 2018)
Revised: algorithm IMPALA (Espeholt et al. , 2018)

Original: TorchBeast implementation (K¨uttler et al. , 2019).
Revised: TorchBeast implementation (K¨uttler et al. , 2019).

Original: (All parameters in subsection A.3).
Revised: (All parameters in subsection A.3).

Original: Baselines We here cover the three common families of intrinsically motivated reward mechanisms.
Revised: Baselines We here cover the three common families of intrinsically motivated reward mechanisms.

Original: COUNT (Strehl and Littman, 2008) is a counting method that baits the agent to explore less visited states.
Revised: COUNT (Strehl and Littman, 2008) is a counting method that baits the agent to explore less visited states.

Original: In this setting, we use a tabular-count to catalog the state-action pairs.
Revised: In this setting, we use a tabular-count to catalog the state-action pairs.

Original: RND (Burda et al. , 2018) acts as a states’ pseudo-count method.
Revised: RND (Burda et al. , 2018) acts as a states’ pseudo-count method.

Original: A network is trained to predict randomly projected states and the normalized predicton error is used as intrinsic reward.
Revised: A network is trained to predict randomly projected states and the normalized predicton error is used as intrinsic reward.

Original: RIDE (Raileanu and
Revised: RIDE (Raileanu and

Original: Rockt ¨ aschel, 2019) is a curiosity-based model that builds upon (Pathak et al. , 2017).
Revised: Rockt ¨ aschel, 2019) is a curiosity-based model that builds upon (Pathak et al. , 2017).

Original: It computes the difference between two consecutive states, encouraging the agent to perform actions that lead it to a maximally different states.
Revised: It computes the difference between two consecutive states, encouraging the agent to perform actions that lead it to a maximally different states.

Original: AMIGO (Campero et al. , 2020) is a hierarchical goal-based method, splitting the agent into two components: an adversarial goal-setter and a goal-condition learner that adversarially creates goals.
Revised: AMIGO (Campero et al. , 2020) is a hierarchical goal-based method, splitting the agent into two components: an adversarial goal-setter and a goal-condition learner that adversarially creates goals.

Original:  E XPERIMENTAL R ESULTS 6. B ASE ENVIRONMENT
Revised:  E XPERIMENTAL R ESULTS 6. B ASE ENVIRONMENT

Original: Figure 2 displays the results on 8 MiniGrid tasks.
Revised: Figure 2 displays the results on 8 MiniGrid tasks.

Original: Noticeably, RAM outperforms all the baselines in sample complexity, and even solves among the most complex worlds.
Revised: Noticeably, RAM outperforms all the baselines in sample complexity, and even solves among the most complex worlds.

Original: In MultiRoom, we observe that RAM outperforms RIDE, RND, and COUNT in the simple setup (N7S4), and matches RIDE’s sample complexity performance on the challenging setup (N12S10).
Revised: In MultiRoom, we observe that RAM outperforms RIDE, RND, and COUNT in the simple setup (N7S4), and matches RIDE’s sample complexity performance on the challenging setup (N12S10).

Original: Note RAM does not seem to be penalized by the small amount of possible interactions.
Revised: Note RAM does not seem to be penalized by the small amount of possible interactions.

Original: In KeyCorridor and ObstructedMaze, RIDE,
Revised: In KeyCorridor and ObstructedMaze, RIDE,

Original: RND, and AMIGO learn in the easiest instances but they struggle as the difﬁculty, i.e. exploration depth, increases as ﬁrst observed in (Campero et al. , 2020).
Revised: RND, and AMIGO learn in the easiest instances but they struggle as the difﬁculty, i.e. exploration depth, increases as ﬁrst observed in (Campero et al. , 2020).

Original: On the other hand, RAM consistently solves all the environments, even the challenging ObstructedMaze-Full.
Revised: On the other hand, RAM consistently solves all the environments, even the challenging ObstructedMaze-Full.

Original: We derive two hypotheses from those results: (1) State-count rewards exhaustively explore the state space, reducing the overall exploration coverage (2) Curiosity-based rewards do not emphasize enough salient interactions and then explore new but irrelevant state-action pairs.
Revised: We derive two hypotheses from those results: (1) State-count rewards exhaustively explore the state space, reducing the overall exploration coverage (2) Curiosity-based rewards do not emphasize enough salient interactions and then explore new but irrelevant state-action pairs.

Original: Although such approaches were successful in many environments, those exploration behaviors may fail as soon as speciﬁc interactions must be regularly performed in the exploration process.
Revised: Although such approaches were successful in many environments, those exploration behaviors may fail as soon as speciﬁc interactions must be regularly performed in the exploration process.

Original: In the following, we thus try to assess those hypotheses.
Revised: In the following, we thus try to assess those hypotheses.

Original: 6. I NTRINSIC EXPLORATION BEHAVIOR
Revised: 6. I NTRINSIC EXPLORATION BEHAVIOR

Original: We ﬁrst conduct a series of experiments without external reward to study what type of exploration each bonus creates.
Revised: We ﬁrst conduct a series of experiments without external reward to study what type of exploration each bonus creates.

Original: In other words, what are the inductive exploration biases that arise from the different intrinsic reward mechanisms.
Revised: In other words, what are the inductive exploration biases that arise from the different intrinsic reward mechanisms.

Original: To do so, we rely on two metrics: the state visit (plotted as heatmaps) and the action distribution (plotted as bar plots).
Revised: To do so, we rely on two metrics: the state visit (plotted as heatmaps) and the action distribution (plotted as bar plots).

Original: Rewardless Playground
Revised: Rewardless Playground

Original: In this spirit, we design a sandbox environment without any speciﬁc goal to observe the agent behavior visually, akin to a kindergarten.
Revised: In this spirit, we design a sandbox environment without any speciﬁc goal to observe the agent behavior visually, akin to a kindergarten.

Original: This environment contains multiple keys, balls, and boxes located in the corners and spawns the agent facing a random direction.
Revised: This environment contains multiple keys, balls, and boxes located in the corners and spawns the agent facing a random direction.

Original: Figure 3 shows the agent state visits for during 10 6 training timesteps when only using the intrinsic reward signal.
Revised: Figure 3 shows the agent state visits for during 10 6 training timesteps when only using the intrinsic reward signal.

Original: We observe that RND and RAM are both attracted by the objects and explore the space thoroughly, whereas RIDE and COUNT remain close to the center and seldomly reach the objects.
Revised: We observe that RND and RAM are both attracted by the objects and explore the space thoroughly, whereas RIDE and COUNT remain close to the center and seldomly reach the objects.

Original: This observation backs our results in ObstructedMaze2Dlh, where RND and RAM are the only methods exploring thoughtfully the environment.
Revised: This observation backs our results in ObstructedMaze2Dlh, where RND and RAM are the only methods exploring thoughtfully the environment.

Original: It also conﬁrms our hypothesis that standard state-based approaches, e.g., COUNT, may not be pushed enough to perform in-depth exploration.
Revised: It also conﬁrms our hypothesis that standard state-based approaches, e.g., COUNT, may not be pushed enough to perform in-depth exploration.

Original: Surprisingly, the curiosity-based method RIDE has not been strongly incentivized to interact with remote objects, suggesting that it may suffer from its dependency on the state representation.
Revised: Surprisingly, the curiosity-based method RIDE has not been strongly incentivized to interact with remote objects, suggesting that it may suffer from its dependency on the state representation.

Original: However, these experiments do not explain the performance difference between RND and RAM on the most challenging setups.
Revised: However, these experiments do not explain the performance difference between RND and RAM on the most challenging setups.

Original: Thus, looking at the action distribution is necessary
Revised: Thus, looking at the action distribution is necessary.

Original: [Figure]
Revised: [Figure]

Original: Rewardless KeyCorridorS4R

Revised: Rewardless KeyCorridorS4R

Original: We then study the behavior that is solely intrinsically motivated in the KeyCorridor environment to better grasp the RAM performance in this setting.
Revised: We then study the behavior that is solely intrinsically motivated in the KeyCorridor environment to better grasp the RAM performance in this setting.

Original: Similarly, we trained the agents on KeyCorridorS4R3 for 10 7 timesteps with only the intrinsic reward signal, and results are displayed in Figure 4.
Revised: Similarly, we trained the agents on KeyCorridorS4R3 for 10 7 timesteps with only the intrinsic reward signal, and results are displayed in Figure 4.

Original: All the baselines – RIDE, RND, and COUNT – remain mostly stuck in the central corridor, where
Revised: All the baselines – RIDE, RND, and COUNT – remain mostly stuck in the central corridor, where

Original: RAM explores rooms more uniformly.
Revised: RAM explores rooms more uniformly.

Original: More impressively, the RAM agent naturally picks the key, enters the locked room, and grabs the ball 7% of the times without any extrinsic reward.
Revised: More impressively, the RAM agent naturally picks the key, enters the locked room, and grabs the ball 7% of the times without any extrinsic reward.

Original: COUNT,
Revised: COUNT,

Original: RIDE, and RND all have a success ratio below 0.6%, which may explain why RAM manages to solve this task.
Revised: RIDE, and RND all have a success ratio below 0.6%, which may explain why RAM manages to solve this task.

Original: Further details can be found in Figure 9 and Table 1.
Revised: Further details can be found in Figure 9 and Table 1.

Original: We also observe a large discrepancy in the action distribution between the different methods.
Revised: We also observe a large discrepancy in the action distribution between the different methods.

Original: First, we observe that RND and RAM action distributions remain approximately uniform while RIDE and
Revised: First, we observe that RND and RAM action distributions remain approximately uniform while RIDE and

Original: COUNT favor moving actions, reducing the opportunity for interactions.
Revised: COUNT favor moving actions, reducing the opportunity for interactions.

Original: Second, and crucially, the impact distributions E H ( a ) differs drastically between RAM and other methods.
Revised: Second, and crucially, the impact distributions E H ( a ) differs drastically between RAM and other methods.

Original: All agents are trying actions such as pick, toggle or drop , but those actions are rarely changing the agent’s state.
Revised: All agents are trying actions such as pick, toggle or drop , but those actions are rarely changing the agent’s state.

Original: These actions are not used in the appropriate context, i.e., in front of an object.
Revised: These actions are not used in the appropriate context, i.e., in front of an object.

Original: It means that rewarding state novelty might not be enough to discover effective actions, thus wasting samples.
Revised: It means that rewarding state novelty might not be enough to discover effective actions, thus wasting samples.

Original: Although RAM and RND had similar state-visitation and action distribution patterns, only RAM correctly apprehend rarely effective actions, and correctly use them to explore its environment.
Revised: Although RAM and RND had similar state-visitation and action distribution patterns, only RAM correctly apprehend rarely effective actions, and correctly use them to explore its environment.

Original: 6. I NTRINSIC M OTIVATION P ITFALLS
Revised: 6. I NTRINSIC M OTIVATION P ITFALLS

Original: The Ball Pit Problem As RAM biases the state visit distribution towards performing rare actions, it may introduce a poor exploration pattern when facing too many of such states.
Revised: The Ball Pit Problem As RAM biases the state visit distribution towards performing rare actions, it may introduce a poor exploration pattern when facing too many of such states.

Original: We refer to this potential issue as the Ballpit problem : the agent remains in rooms with plenty of balls to interact with.
Revised: We refer to this potential issue as the Ballpit problem : the agent remains in rooms with plenty of balls to interact with.

Original: We created four versions of Multiroom (normal, small, more, max), and randomly spawned objects to assess the agent behavior (more details in subsection A.1).
Revised: We created four versions of Multiroom (normal, small, more, max), and randomly spawned objects to assess the agent behavior (more details in subsection A.1).

Original: [Figure]
Revised: [Figure]

Original: As the number of objects grows, the performance of all algorithms deteriorates.
Revised: As the number of objects grows, the performance of all algorithms deteriorates.

Original: RND, COUNT are mostly affected by this problem, as the number of states is growing exponentially; thus, counting state occurrence is challenging.
Revised: RND, COUNT are mostly affected by this problem, as the number of states is growing exponentially; thus, counting state occurrence is challenging.

Original: RIDE is less affected by the BallPit problem, but most surprisingly,
Revised: RIDE is less affected by the BallPit problem, but most surprisingly,

Original: RAM is the only one to reach the exit consistently in the most challenging setup.
Revised: RAM is the only one to reach the exit consistently in the most challenging setup.

Original: The E H ( a ) /U H ( a ) ratio correctly balances the exploration bonus, and does not take over the ﬁnal extrinsic reward.
Revised: The E H ( a ) /U H ( a ) ratio correctly balances the exploration bonus, and does not take over the ﬁnal extrinsic reward.

Original: The Noisy-TV problem State-count based agent are attracted to state-action pairs with random noise.
Revised: The Noisy-TV problem State-count based agent are attracted to state-action pairs with random noise.

Original: In its current deﬁnition, RAM is also affected while computing E H .
Revised: In its current deﬁnition, RAM is also affected while computing E H .

Original: Similar to (Burda et al. , 2018), this effect can be circumvented by using an inverse model, and we leave it for future work.
Revised: Similar to (Burda et al. , 2018), this effect can be circumvented by using an inverse model, and we leave it for future work.

Original: ColorMaze
Revised: ColorMaze

Original: In Figure 6, we design a map with a sequence of open rooms, colored ﬂoor changing every episode, two boxes with one hidden key, and a locked door leading to the reward.
Revised: In Figure 6, we design a map with a sequence of open rooms, colored ﬂoor changing every episode, two boxes with one hidden key, and a locked door leading to the reward.

Original: All baselines remain in the ﬁrst part of the maze while RAM quickly reaches the objects and solves the task.
Revised: All baselines remain in the ﬁrst part of the maze while RAM quickly reaches the objects and solves the task.

Original: This experiment highlights again how shifting the emphasis from exhaustive state-visit to relevant statevisit can be beneﬁcial, and change the exploration pattern. 
Revised: This experiment highlights again how shifting the emphasis from exhaustive state-visit to relevant statevisit can be beneﬁcial, and change the exploration pattern. ColorMaze

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce Relevant Actions Matter (RAM), a new action-based intrinsic exploration algorithm.
Revised: We introduce Relevant Actions Matter (RAM), a new action-based intrinsic exploration algorithm.

Original: As opposed to count-based and curiosity-driven methods, RAM shifts the emphasis from novel state to state with relevant actions, rewarding actions that are rarely effective in the environment.
Revised: As opposed to count-based and curiosity-driven methods, RAM shifts the emphasis from novel state to state with relevant actions, rewarding actions that are rarely effective in the environment.

Original: Combined with a simple episodic count, RAM outperforms recent exploration methods on a variety of hard exploratory tasks in a Minigrid environment.
Revised: Combined with a simple episodic count, RAM outperforms recent exploration methods on a variety of hard exploratory tasks in a Minigrid environment.

Original: This proof of concept illustrates that actionbased exploration is a promising approach as it induces surprisingly different exploration patterns.
Revised: This proof of concept illustrates that actionbased exploration is a promising approach as it induces surprisingly different exploration patterns.

Original: We also pointed out a new category of problems called BallPit , which deteriorate performance of many intrinsically motivated reward approaches.
Revised: We also pointed out a new category of problems called BallPit , which deteriorate performance of many intrinsically motivated reward approaches.

Original:  A CKNOWLEDGMENTS
Revised:  A CKNOWLEDGMENTS

Original: We would like the thank the anonymous reviewers for useful feedbacks, Bilal Piot and Olivier Teboul for proofreading the manuscript and the stimulating environment of Scool.
Revised: We would like the thank the anonymous reviewers for useful feedbacks, Bilal Piot and Olivier Teboul for proofreading the manuscript and the stimulating environment of Scool.

Original: Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientiﬁc interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).
Revised: Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientiﬁc interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

Original: A A PPENDIX A. E NVIRONMENTS INTRODUCED IN THE PAPER
Revised: A A PPENDIX A. E NVIRONMENTS INTRODUCED IN THE PAPER

Original: Playground is designed to visually assess how exploration strategies behave when lots of object are present.
Revised: Playground is designed to visually assess how exploration strategies behave when lots of object are present.

Original: Playground is a 14x14 grid, objects are always spawned at the same location but the color changes from episode to episode.
Revised: Playground is a 14x14 grid, objects are always spawned at the same location but the color changes from episode to episode.

Original: An episode lasts 200 steps, no external reward is given and the agent is always spawned in the center, facing a random direction.
Revised: An episode lasts 200 steps, no external reward is given and the agent is always spawned in the center, facing a random direction.

Original: Ballpit environment The Ballpit environment was designed to assess how each algorithm is affected by the Ballpit problem .
Revised: Ballpit environment The Ballpit environment was designed to assess how each algorithm is affected by the Ballpit problem .

Original: We build it on top of MultiRoomN4S6, a simple environment solved by all baselines.
Revised: We build it on top of MultiRoomN4S6, a simple environment solved by all baselines.

Original: No Ball is the baseline environment.
Revised: No Ball is the baseline environment.

Original: In Small instances, 1 random object is spawned, More contains 3 random objects and Max rooms are completely ﬁlled without blocking the agent path.
Revised: In Small instances, 1 random object is spawned, More contains 3 random objects and Max rooms are completely ﬁlled without blocking the agent path.

Original: We emphasize that the agent does not need to interact with anything (with the exception of door) to solve the environment and objects are never in the way of the agent.
Revised: We emphasize that the agent does not need to interact with anything (with the exception of door) to solve the environment and objects are never in the way of the agent.

Original: ColorMaze ColorMaze is a variation of ObstructedMaze2Dlh.
Revised: ColorMaze ColorMaze is a variation of ObstructedMaze2Dlh.

Original: Before accessing the room containing keys within boxes, the agent must cross 4 colored rooms.
Revised: Before accessing the room containing keys within boxes, the agent must cross 4 colored rooms.

Original: A distractor room is placed at the beginning, leading to nothing.
Revised: A distractor room is placed at the beginning, leading to nothing.

Original: The colors are picked randomly at the beginning of an episode and remains the same until the episode’s end.
Revised: The colors are picked randomly at the beginning of an episode and remains the same until the episode’s end.

Original: The agent must catch the blue ball within a 576 steps limit to receive the reward.
Revised: The agent must catch the blue ball within a 576 steps limit to receive the reward.

Original: One main difference between MultiRoom and ColorMaze is the absence of doors between colored room, which make this environment harder for RAM as no interaction is possible until the ﬁrst door in room 4.
Revised: One main difference between MultiRoom and ColorMaze is the absence of doors between colored room, which make this environment harder for RAM as no interaction is possible until the ﬁrst door in room 4.

Original: A. D ECAY ILLUSTRATION
Revised: A. D ECAY ILLUSTRATION

Original: RAM reward is a function of the action ratio : E H ( a ) the number of times an action impacted the state over U H ( a ) the number of usage.
Revised: RAM reward is a function of the action ratio : E H ( a ) the number of times an action impacted the state over U H ( a ) the number of usage.

Original: The function acts as an exponential decay starting fromwhen the ratio is 0 (the action never impacted the environment so we want to reward it highly when it is actually modifying the state) and giving 0 reward when the ratio is 1 (The action a modiﬁes the state all the time; thus, a is of low interest).
Revised: The function acts as an exponential decay starting fromwhen the ratio is 0 (the action never impacted the environment so we want to reward it highly when it is actually modifying the state) and giving 0 reward when the ratio is 1 (The action a modiﬁes the state all the time; thus, a is of low interest).

Original: In Figure 8, we illustrate how the parameter η shape the intrinsic reward.
Revised: In Figure 8, we illustrate how the parameter η shape the intrinsic reward.

Original: [Equation]
Revised: [Equation]

Original: [Table] [Figure] [Figure]
Revised: [Table] [Table]

Original: action ratio
Revised: action ratio

Original:  [Figure]n t r i n s i
Revised: [Table] [Figure]

Original: c r e w a r d
Revised: [Table] [Figure]

Original: Ratio decay =decay =decay =
Revised: [Table] [Figure] n [Figure]

Original: Parameters Value
Revised: i n s

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: c

Original: [Figure]
Revised: [Figure]

Original:  Ratio decay η Learning rate 1eOptimizer RMSprop
Revised: e w a r d Ratio decay =decay =decay =

Original: Entropy cost 0. batch size LSTM unroll length Number of actors
Revised: [Table]

Original: Max norm gradient Baseline cost 0. Discount factor 0. All parameters are kept the same across all environments, except for MultiRoomN12-S10 where the entropy cost is 0.0001.
Revised: All parameters are kept the same across all environments, except for MultiRoomN12-S10 where the entropy cost is 0.0001.