Original: N OISY A GENTS : S ELF - SUPERVISED E XPLORATION BY P REDICTING A UDITORY E VENTS
Revised: N OISY A GENTS : S ELF - SUPERVISED E XPLORATION BY P REDICTING A UDITORY E VENTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Humans integrate multiple sensory modalities ( e g. , visual and audio) to build a causal understanding of the physical world.
Revised: Humans integrate multiple sensory modalities ( e g. , visual and audio) to build a causal understanding of the physical world.

Original: In this work, we propose a novel type of intrinsic motivation for Reinforcement Learning (RL) that encourages the agent to understand the causal effect of its actions through auditory event prediction.
Revised: In this work, we propose a novel type of intrinsic motivation for Reinforcement Learning (RL) that encourages the agent to understand the causal effect of its actions through auditory event prediction.

Original: First, we allow the agent to collect a small amount of acoustic data and use K-means to discover underlying auditory event clusters.
Revised: First, we allow the agent to collect a small amount of acoustic data and use K-means to discover underlying auditory event clusters.

Original: We then train a neural network to predict the auditory events and use the prediction errors as intrinsic rewards to guide RL exploration.
Revised: We then train a neural network to predict the auditory events and use the prediction errors as intrinsic rewards to guide RL exploration.

Original: We ﬁrst conduct an in-depth analysis of our module using a set of Atari games.
Revised: We ﬁrst conduct an in-depth analysis of our module using a set of Atari games.

Original: We then apply our model to audio-visual exploration using the Habitat simulator and active learning using the ThreeDWorld (TDW) simulator.
Revised: We then apply our model to audio-visual exploration using the Habitat simulator and active learning using the ThreeDWorld (TDW) simulator.

Original: Experimental results demonstrate the advantages of using audio signals over vision-based models as intrinsic rewards to guide RL explorations
Revised: Experimental results demonstrate the advantages of using audio signals over vision-based models as intrinsic rewards to guide RL explorations

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep Reinforcement Learning algorithms aim to learn a policy of an agent to maximize its cumulative rewards by interacting with environments and have demonstrated substantial success in a wide range of application domains, such as video game (Mnih et al., 2015), board games (Silver et al., 2016), and visual navigation (Zhu et al., 2017).
Revised: Deep Reinforcement Learning algorithms aim to learn a policy of an agent to maximize its cumulative rewards by interacting with environments and have demonstrated substantial success in a wide range of application domains, such as video game (Mnih et al., 2015), board games (Silver et al., 2016), and visual navigation (Zhu et al., 2017).

Original: While these results are remarkable, one of the critical constraints is the prerequisite of carefully engineered dense reward signals, which are not always accessible.
Revised: While these results are remarkable, one of the critical constraints is the prerequisite of carefully engineered dense reward signals, which are not always accessible.

Original: To overcome these constraints, researchers have proposed a range of intrinsic reward functions.
Revised: To overcome these constraints, researchers have proposed a range of intrinsic reward functions.

Original: For example, curiosity-driven intrinsic reward based on prediction error of current (Burda et al., 2018b) or future state (Pathak et al., 2017) on the latent feature spaces have shown promising results.
Revised: For example, curiosity-driven intrinsic reward based on prediction error of current (Burda et al., 2018b) or future state (Pathak et al., 2017) on the latent feature spaces have shown promising results.

Original: Nevertheless, visual state prediction is a non-trivial problem as visual state is high-dimensional and tends to be highly stochastic in real-world environments.
Revised: Nevertheless, visual state prediction is a non-trivial problem as visual state is high-dimensional and tends to be highly stochastic in real-world environments.

Original: The occurrence of physical events ( e.g. objects coming into contact with each other, or changing state) often correlates with both visual and auditory signals.
Revised: The occurrence of physical events ( e.g. objects coming into contact with each other, or changing state) often correlates with both visual and auditory signals.

Original: Both sensory modalities should thus offer useful cues to agents learning how to act in the world.
Revised: Both sensory modalities should thus offer useful cues to agents learning how to act in the world.

Original: Indeed, classic experiments in cognitive and developmental psychology show that humans naturally attend to both visual and auditory cues, and their temporal coincidence, to arrive at a rich understanding of physical events and human activity such as speech (Spelke, 1976; McGurk & MacDonald, 1976).
Revised: Indeed, classic experiments in cognitive and developmental psychology show that humans naturally attend to both visual and auditory cues, and their temporal coincidence, to arrive at a rich understanding of physical events and human activity such as speech (Spelke, 1976; McGurk & MacDonald, 1976).

Original: In artiﬁcial intelligence, however, much more attention has been paid to the ways visual signals (e.g., patterns in pixels) can drive learning.
Revised: In artiﬁcial intelligence, however, much more attention has been paid to the ways visual signals (e.g., patterns in pixels) can drive learning.

Original: We believe this misses important structure learners could exploit.
Revised: We believe this misses important structure learners could exploit.

Original: As compared to visual cues, sounds are often more directly or easily observable causal effects of actions and interactions.
Revised: As compared to visual cues, sounds are often more directly or easily observable causal effects of actions and interactions.

Original: This is clearly true when agents interact: most communication uses speech or other nonverbal but audible signals.
Revised: This is clearly true when agents interact: most communication uses speech or other nonverbal but audible signals.

Original: However, it is just as much in physics.
Revised: However, it is just as much in physics.

Original: Almost any time two objects collide, rub or slide against each other, or touch in any way, they make a sound.
Revised: Almost any time two objects collide, rub or slide against each other, or touch in any way, they make a sound.

Original: That sound is often clearly distinct from background auditory textures, localized in both time and spectral properties, hence relatively easy to detect and identify; in contrast, speciﬁc visual events can be much harder to separate from all the ways high-dimensional pixel inputs are changing over the course of a scene.
Revised: That sound is often clearly distinct from background auditory textures, localized in both time and spectral properties, hence relatively easy to detect and identify; in contrast, speciﬁc visual events can be much harder to separate from all the ways high-dimensional pixel inputs are changing over the course of a scene.

Original: The sounds that result from object interactions also allow us to estimate underlying causally relevant variables, such as material properties ( e.g. , whether objects are hard or soft, solid or hollow, smooth, or rough), which can be critical for planning actions.
Revised: The sounds that result from object interactions also allow us to estimate underlying causally relevant variables, such as material properties ( e.g. , whether objects are hard or soft, solid or hollow, smooth, or rough), which can be critical for planning actions.

Original: These facts bring a natural question of how to use audio signals to beneﬁt policy learning in RL.
Revised: These facts bring a natural question of how to use audio signals to beneﬁt policy learning in RL.

Original: In this paper, our main idea is to use sound prediction as an intrinsic reward to guide RL exploration.
Revised: In this paper, our main idea is to use sound prediction as an intrinsic reward to guide RL exploration.

Original: Intuitively, we want to exploit the fact that sounds are frequently made when objects interact, or other causally signiﬁcant events occur, like cues to causal structure or candidate subgoals an agent could discover and aim for.
Revised: Intuitively, we want to exploit the fact that sounds are frequently made when objects interact, or other causally signiﬁcant events occur, like cues to causal structure or candidate subgoals an agent could discover and aim for.

Original: A na¨ıve strategy would be to directly regress feature embeddings of audio clips and use feature prediction errors as intrinsic rewards.
Revised: A na¨ıve strategy would be to directly regress feature embeddings of audio clips and use feature prediction errors as intrinsic rewards.

Original: However, prediction errors on feature space do not accurately reﬂect how well an agent understands the underlying causal structure of events and goals.
Revised: However, prediction errors on feature space do not accurately reﬂect how well an agent understands the underlying causal structure of events and goals.

Original: It also remains an open problem on how to perform appropriate normalizations to solve intrinsic reward diminishing issues.
Revised: It also remains an open problem on how to perform appropriate normalizations to solve intrinsic reward diminishing issues.

Original: To bypass these limitations, we formulate the sound-prediction task as a classiﬁcation problem, in which we train a neural network to predict auditory events that occurred after applying action to a visual scene.
Revised: To bypass these limitations, we formulate the sound-prediction task as a classiﬁcation problem, in which we train a neural network to predict auditory events that occurred after applying action to a visual scene.

Original: We use classiﬁcation errors as an exploration bonus for deep reinforcement learning.
Revised: We use classiﬁcation errors as an exploration bonus for deep reinforcement learning.

Original: Concretely, our pipeline consists of two exploration phases.
Revised: Concretely, our pipeline consists of two exploration phases.

Original: In the beginning, the agent receives an incentive to actively collect a small amount of auditory data by interacting with the environment.
Revised: In the beginning, the agent receives an incentive to actively collect a small amount of auditory data by interacting with the environment.

Original: Then we cluster the sound data into auditory events using K-means.
Revised: Then we cluster the sound data into auditory events using K-means.

Original: In the second phase, we train a neural network to predict the auditory events conditioned on the embedding of visual observations and actions.
Revised: In the second phase, we train a neural network to predict the auditory events conditioned on the embedding of visual observations and actions.

Original: The state that has the wrong prediction is rewarded and encouraged to be visited more.
Revised: The state that has the wrong prediction is rewarded and encouraged to be visited more.

Original: We demonstrate the effectiveness of our intrinsic module on game playing with in Atari (Bellemare et al., 2013), audio-visual exploration in Habitat (Savva et al., 2019), active learning using a rolling robot in ThreeDWorld (TDW) (Gan et al., 2020a).
Revised: We demonstrate the effectiveness of our intrinsic module on game playing with in Atari (Bellemare et al., 2013), audio-visual exploration in Habitat (Savva et al., 2019), active learning using a rolling robot in ThreeDWorld (TDW) (Gan et al., 2020a).

Original: In summary, our work makes the following contributions
Revised: In summary, our work makes the following contributions

Original: : • We introduce a novel and effective auditory event prediction (AEP) framework to make use of the auditory signals as intrinsic rewards for RL exploration.
Revised: : • We introduce a novel and effective auditory event prediction (AEP) framework to make use of the auditory signals as intrinsic rewards for RL exploration.

Original: • We perform an in-depth study on Atari games to understand our audio-driven exploration works well under what circumstances.
Revised: • We perform an in-depth study on Atari games to understand our audio-driven exploration works well under what circumstances.

Original: • We demonstrate our new model can enable a more efﬁcient exploration strategy for audio-visual embodied navigation on the Habitat environment.
Revised: • We demonstrate our new model can enable a more efﬁcient exploration strategy for audio-visual embodied navigation on the Habitat environment.

Original: • We show that our new intrinsic module is more stable in the 3D multi-modal physical world environment and can encourage interest actions that involved physical interactions.
Revised: • We show that our new intrinsic module is more stable in the 3D multi-modal physical world environment and can encourage interest actions that involved physical interactions.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Audio-Visual Learning.
Revised: Audio-Visual Learning.

Original: In recent years, audio-visual learning has been studied extensively.
Revised: In recent years, audio-visual learning has been studied extensively.

Original: By leveraging audio-visual correspondences in videos, it can help to learn powerful audio and visual representations through self-supervised learning (Owens et al., 2016b; Aytar et al., 2016; Arandjelovic & Zisserman, 2017;
Revised: By leveraging audio-visual correspondences in videos, it can help to learn powerful audio and visual representations through self-supervised learning (Owens et al., 2016b; Aytar et al., 2016; Arandjelovic & Zisserman, 2017;

Original: Korbar et al., 2018; Owens & Efros, 2018).
Revised: Korbar et al., 2018; Owens & Efros, 2018).

Original: In contrast to the widely used correspondences between these two modalities, we take a step further by considering sound as causal effects of actions.
Revised: In contrast to the widely used correspondences between these two modalities, we take a step further by considering sound as causal effects of actions.

Original: RL Explorations.
Revised: RL Explorations.

Original: The problem of exploration in Reinforcement Learning (RL) has been an active research topic for decades.
Revised: The problem of exploration in Reinforcement Learning (RL) has been an active research topic for decades.

Original: There are various solutions that have been investigated for encouraging the agent to explore novel states, including rewarding information gain (Little & Sommer, 2013), surprise (Schmidhuber, 1991; 2010), state visitation counts (Tang et al., 2017; Bellemare et al., 2016), empowerment (Klyubin et al., 2005), curiosity (Pathak et al., 2017; Burda et al., 2018a) disagreement (Pathak et al., 2019) and so on.
Revised: There are various solutions that have been investigated for encouraging the agent to explore novel states, including rewarding information gain (Little & Sommer, 2013), surprise (Schmidhuber, 1991; 2010), state visitation counts (Tang et al., 2017; Bellemare et al., 2016), empowerment (Klyubin et al., 2005), curiosity (Pathak et al., 2017; Burda et al., 2018a) disagreement (Pathak et al., 2019) and so on.

Original: A separate line of work (Osband et al., 2019; 2016) adopts parameter noises and Thompson sampling heuristics for exploration.
Revised: A separate line of work (Osband et al., 2019; 2016) adopts parameter noises and Thompson sampling heuristics for exploration.

Original: For example, Osband et al.
Revised: For example, Osband et al.

Original: (2019) trains multiple value functions and makes use of the bootstraps for deep exploration.
Revised: (2019) trains multiple value functions and makes use of the bootstraps for deep exploration.

Original: Here, we mainly focus on the problem of using intrinsic rewards to drive explorations.
Revised: Here, we mainly focus on the problem of using intrinsic rewards to drive explorations.

Original: The most widely used intrinsic motivation could be roughly divided into two families.
Revised: The most widely used intrinsic motivation could be roughly divided into two families.

Original: The ﬁrst one is countbased approaches (Strehl & Littman, 2008; Bellemare et al., 2016;
Revised: The ﬁrst one is countbased approaches (Strehl & Littman, 2008; Bellemare et al., 2016;

Original: Tang et al., 2017; Ostrovski et al., 2017; Martin et al., 2017; Burda et al., 2018b), which encourage the agent to visit novel states.
Revised: Tang et al., 2017; Ostrovski et al., 2017; Martin et al., 2017; Burda et al., 2018b), which encourage the agent to visit novel states.

Original: For example, Burda et al.
Revised: For example, Burda et al.

Original: (2018b) employs the prediction errors of a self-state feature extracted from a ﬁxed and random initialized network as exploration bonuses and encourage the agent to visit more previous unseen states.
Revised: (2018b) employs the prediction errors of a self-state feature extracted from a ﬁxed and random initialized network as exploration bonuses and encourage the agent to visit more previous unseen states.

Original: Another one is the curiosity-based approach (Stadie et al., 2015; Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018a), which is formulated as the uncertainty in predicting the consequences of the agent’s actions.
Revised: Another one is the curiosity-based approach (Stadie et al., 2015; Pathak et al., 2017; Haber et al., 2018; Burda et al., 2018a), which is formulated as the uncertainty in predicting the consequences of the agent’s actions.

Original: For instance, Pathak et al.
Revised: For instance, Pathak et al.

Original: (2017) and Burda et al.
Revised: (2017) and Burda et al.

Original: (2018a) use the errors of predicting the next state in the latent feature space as rewards.
Revised: (2018a) use the errors of predicting the next state in the latent feature space as rewards.

Original: The agent is then encouraged to improve its knowledge about the environment dynamics.
Revised: The agent is then encouraged to improve its knowledge about the environment dynamics.

Original: In contrast to previous work purely using vision, we make use of the sound signals as rewards for RL explorations.
Revised: In contrast to previous work purely using vision, we make use of the sound signals as rewards for RL explorations.

Original: 𝜙(𝑠𝑜𝑢𝑛𝑑 ! ) 𝑐𝑙𝑎𝑠𝑠, !"
Revised: 𝜙(𝑠𝑜𝑢𝑛𝑑 ! ) 𝑐𝑙𝑎𝑠𝑠, !"

Original: # s𝑜𝑢𝑛𝑑 !
Revised: # s𝑜𝑢𝑛𝑑 !

Original: s𝑜𝑢𝑛𝑑 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑠 𝑜𝑛𝑙𝑖𝑛𝑒 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑖𝑛𝑔 𝑖𝑚𝑎𝑔𝑒 !
Revised: s𝑜𝑢𝑛𝑑 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑠 𝑜𝑛𝑙𝑖𝑛𝑒 𝑐𝑙𝑢𝑠𝑡𝑒𝑟𝑖𝑛𝑔 𝑖𝑚𝑎𝑔𝑒 !

Original: 𝜙(𝑠𝑜𝑢𝑛𝑑 !"
Revised: 𝜙(𝑠𝑜𝑢𝑛𝑑 !"

Original: # ) c𝑙𝑎𝑠𝑠 !"
Revised: # ) c𝑙𝑎𝑠𝑠 !"

Original: # 𝛹(𝑖𝑚𝑎𝑔𝑒 ! )
Revised: # 𝛹(𝑖𝑚𝑎𝑔𝑒 ! )

Original: s𝑜𝑢𝑛𝑑 !"
Revised: s𝑜𝑢𝑛𝑑 !"

Original: # 𝑳𝒐𝒔𝒔 𝑎𝑐𝑡𝑖𝑜𝑛 !
Revised: # 𝑳𝒐𝒔𝒔 𝑎𝑐𝑡𝑖𝑜𝑛 !

Original: 𝐼𝑛𝑡𝑟𝑖𝑛𝑠𝑖𝑐 𝑅𝑒𝑤𝑎𝑟𝑑
Revised: 𝐼𝑛𝑡𝑟𝑖𝑛𝑠𝑖𝑐 𝑅𝑒𝑤𝑎𝑟𝑑

Original: [Figure]
Revised: [Figure]

Original: Sounds and Actions.
Revised: Sounds and Actions.

Original: There are numerous works to explore the associations between sounds and actions.
Revised: There are numerous works to explore the associations between sounds and actions.

Original: For example, Owens et al.
Revised: For example, Owens et al.

Original: (2016a) made the ﬁrst attempt to collect an audio-video dataset through physical interaction with objects and train an RNN model to generate sounds for silent videos.
Revised: (2016a) made the ﬁrst attempt to collect an audio-video dataset through physical interaction with objects and train an RNN model to generate sounds for silent videos.

Original: Shlizerman et al.
Revised: Shlizerman et al.

Original: ; Ginosar et al.
Revised: ; Ginosar et al.

Original: (2019) explore the problem of predicting body dynamics from music and body gesture from speech.
Revised: (2019) explore the problem of predicting body dynamics from music and body gesture from speech.

Original: Gan et al.
Revised: Gan et al.

Original: (2020b)
Revised: (2020b)

Original: and
Revised: and

Original: Chen et al.
Revised: Chen et al.

Original: (2020) introduce an interesting audio-visual embodied task in 3D simulation environments.
Revised: (2020) introduce an interesting audio-visual embodied task in 3D simulation environments.

Original: More recently, Dhiraj et al.
Revised: More recently, Dhiraj et al.

Original: (2020) collected a large sound-action-vision dataset using Tilt-bolt and demonstrates sound signals could provide valuable information for ﬁnd-grained object recognition, inverse model learning, and forward dynamics model prediction.
Revised: (2020) collected a large sound-action-vision dataset using Tilt-bolt and demonstrates sound signals could provide valuable information for ﬁnd-grained object recognition, inverse model learning, and forward dynamics model prediction.

Original: More related to us are the papers from Aytar et al.
Revised: More related to us are the papers from Aytar et al.

Original: and
Revised: and

Original: Omidshaﬁei et al.
Revised: Omidshaﬁei et al.

Original: (2018), which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games.
Revised: (2018), which have shown that the sound signals could provide useful supervisions for imitation learning and reinforcement learning in Atari games.

Original: Concurrent to our work, Dean et al.
Revised: Concurrent to our work, Dean et al.

Original: (2020) uses novel associations of audio and visual signals as intrinsic rewards to guide RL exploration.
Revised: (2020) uses novel associations of audio and visual signals as intrinsic rewards to guide RL exploration.

Original: Different from them, we mainly studied if the sound signals alone could be utilized as intrinsic rewards for RL explorations.
Revised: Different from them, we use auditory event predictions as intrinsic rewards to drive RL explorations.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we ﬁrst introduce some background knowledge of reinforcement learning and intrinsic rewards.
Revised: In this section, we ﬁrst introduce some background knowledge of reinforcement learning and intrinsic rewards.

Original: And then we elaborate on the pipeline of self-supervised exploration through auditory event predictions.
Revised: And then we elaborate on the pipeline of self-supervised exploration through auditory event predictions.

Original: The pipeline of our system is outlined in Figure 1.
Revised: The pipeline of our system is outlined in Figure 1.

Original: 3. B ACKGROUND
Revised: 3. B ACKGROUND

Original: MDPs We formalize the decision procedure in our context as a standard Markov Decision Process (MDP), deﬁned as ( S , A , r, T , µ, γ ) .
Revised: MDPs We formalize the decision procedure in our context as a standard Markov Decision Process (MDP), deﬁned as ( S , A , r, T , µ, γ ) .

Original: S , A and µ ( s ) : S → [0 , 1] denote the sets of state, action and the distribution of initial state respectively.
Revised: S , A and µ ( s ) : S → [0 , 1] denote the sets of state, action and the distribution of initial state respectively.

Original: The transition function T ( s (cid:48) | s, a ) : S ×
Revised: The transition function T ( s (cid:48) | s, a ) : S ×

Original: A × S → [0 , 1] deﬁnes the transition probability to next-step state s (cid:48) if the agent takes action a at current state s .
Revised: A × S → [0 , 1] deﬁnes the transition probability to next-step state s (cid:48) if the agent takes action a at current state s .

Original: The agent will receive a reward r after taking an action a according to the reward function R ( s, a ) , discounted by γ ∈ (0 , 1) .
Revised: The agent will receive a reward r after taking an action a according to the reward function R ( s, a ) , discounted by γ ∈ (0 , 1) .

Original: The goal of training reinforcement learning is to learn an optimal policy π ∗ that can maximize the expected rewards under the discount factor γ as
Revised: The goal of training reinforcement learning is to learn an optimal policy π ∗ that can maximize the expected rewards under the discount factor γ as

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where ζ represents the agent’s trajectory, namely { ( s 0 , a 0 ) , ( s 1 , a 1 ) , · · · } .
Revised: where ζ represents the agent’s trajectory, namely { ( s 0 , a 0 ) , ( s 1 , a 1 ) , · · · } .

Original: The agent chooses an action a from a policy π ( a | s ) : S ×
Revised: The agent chooses an action a from a policy π ( a | s ) : S ×

Original: A → [0 , 1] that speciﬁes the probability of taking action a ∈ A under state s .
Revised: A → [0 , 1] that speciﬁes the probability of taking action a ∈ A under state s .

Original: In this paper, we concentrate on the MDPs whose states are raw image-based observations as well as audio clips, actions are discrete, and T is provided by the game engine.
Revised: In this paper, we concentrate on the MDPs whose states are raw image-based observations as well as audio clips, actions are discrete, and T is provided by the game engine.

Original: Intrinsic Rewards for Exploration.
Revised: Intrinsic Rewards for Exploration.

Original: Designing intrinsic rewards for exploration has been widely used to resolve the sparse reward issues in the deep RL communities.
Revised: Designing intrinsic rewards for exploration has been widely used to resolve the sparse reward issues in the deep RL communities.

Original: One effective approach is to use the errors of a predictive model as exploration bonuses (Pathak et al., 2017; Haber et al., 2018;
Revised: One effective approach is to use the errors of a predictive model as exploration bonuses (Pathak et al., 2017; Haber et al., 2018;

Original: Burda et al., 2018a).
Revised: Burda et al., 2018a).

Original: The intrinsic rewards will encourage the agent to explore those states with less familiarity.
Revised: The intrinsic rewards will encourage the agent to explore those states with less familiarity.

Original: We aim to train a policy that can maximize the errors of auditory event predictions.
Revised: We aim to train a policy that can maximize the errors of auditory event predictions.

Original: 3. R EPRESENTATIONS OF A UDITORY E VENTS
Revised: 3. R EPRESENTATIONS OF A UDITORY E VENTS

Original: Consider an agent that sees a visual observation s v,t , takes an action a t and transits to the next state with visual observation s v,t +1 and sound effect s s,t +1 .
Revised: Consider an agent that sees a visual observation s v,t , takes an action a t and transits to the next state with visual observation s v,t +1 and sound effect s s,t +1 .

Original: The main objective of our intrinsic module is to predict auditory events of the next state, given feature representations of the current visual observation s v,t and the agent’s action a t .
Revised: The main objective of our intrinsic module is to predict auditory events of the next state, given feature representations of the current visual observation s v,t and the agent’s action a t .

Original: We hypothesize that the agents, through this process, could learn the underlying causal structure of the physical world and use that to make predictions about what will happen next, and as well as plan actions to achieve their goals.
Revised: We hypothesize that the agents, through this process, could learn the underlying causal structure of the physical world and use that to make predictions about what will happen next, and as well as plan actions to achieve their goals.

Original: To better capture the statistic of the raw auditory data, we extract sound textures (McDermott &
Revised: To better capture the statistic of the raw auditory data, we extract sound textures (McDermott &

Original: Simoncelli, 2011) Φ( s s,t ) to represent each audio clip s s,t .
Revised: Simoncelli, 2011) Φ( s s,t ) to represent each audio clip s s,t .

Original: There are many possible ways to compute audio features.
Revised: There are many possible ways to compute audio features.

Original: We choose sound texture mainly for its computation efﬁciency and effectiveness in audio-visual learning (Owens et al., 2016a;b).
Revised: We choose sound texture mainly for its computation efﬁciency and effectiveness in audio-visual learning (Owens et al., 2016a;b).

Original: For the task of auditory event predictions, perhaps the most straightforward option is to directly regress the sound features Φ( s s,t +1 ) given the feature embeddings of the image observation s v,t and agent’s actions a t .
Revised: For the task of auditory event predictions, perhaps the most straightforward option is to directly regress the sound features Φ( s s,t +1 ) given the feature embeddings of the image observation s v,t and agent’s actions a t .

Original: Nevertheless, we ﬁnd that not very effective.
Revised: Nevertheless, we ﬁnd that not very effective.

Original: We hypothesize that the main reasons are: 1) the mean squared error (MSE) loss used for regression is satisﬁed with “blurry” predictions.
Revised: We hypothesize that the main reasons are: 1) the mean squared error (MSE) loss used for regression is satisﬁed with “blurry” predictions.

Original: This might not capture the full distribution over possible sounds and a categorical distribution over clusters; 2) the MSE loss does not accurately reﬂect how well an agent understands these auditory events.
Revised: This might not capture the full distribution over possible sounds and a categorical distribution over clusters; 2) the MSE loss does not accurately reﬂect how well an agent understands these auditory events.

Original: Therefore, we choose instead to deﬁne explicit auditory events categories and formulate this auditory event prediction problem as a classiﬁcation task, similar to (Owens et al., 2016b).
Revised: Therefore, we choose instead to deﬁne explicit auditory events categories and formulate this auditory event prediction problem as a classiﬁcation task, similar to (Owens et al., 2016b).

Original: 3. A UDITORY E VENTS P REDICTION B ASED I NTRINSIC R EWARD
Revised: 3. A UDITORY E VENTS P REDICTION B ASED I NTRINSIC R EWARD

Original: Our AEP framework consists of two stages: sound clustering and auditory event prediction.
Revised: Our AEP framework consists of two stages: sound clustering and auditory event prediction.

Original: We need to collect a small set of diverse auditory data in the ﬁrst stage and use them to deﬁne the underlying auditory event classes.
Revised: We need to collect a small set of diverse auditory data in the ﬁrst stage and use them to deﬁne the underlying auditory event classes.

Original: To achieve this goal, we ﬁrst train an RL policy that rewards the agents based on sound novelty.
Revised: To achieve this goal, we ﬁrst train an RL policy that rewards the agents based on sound novelty.

Original: And then, we run a K-means algorithm to group these data into a ﬁxed number of auditory events.
Revised: And then, we run a K-means algorithm to group these data into a ﬁxed number of auditory events.

Original: In the second phase, we train a forward dynamics network that takes input as the embedding of visual observation and action and predicts which auditory event will happen next.
Revised: In the second phase, we train a forward dynamics network that takes input as the embedding of visual observation and action and predicts which auditory event will happen next.

Original: The prediction error is then utilized as an intrinsic reward to encourage the agent to explore those auditory events with more uncertainty, thus improving its ability to understand the consequences of its actions.
Revised: The prediction error is then utilized as an intrinsic reward to encourage the agent to explore those auditory events with more uncertainty, thus improving its ability to understand the consequences of its actions.

Original: We will elaborate on the details of these two phases below.
Revised: We will elaborate on the details of these two phases below.

Original: Sound clustering.
Revised: Sound clustering.

Original: The agents start to collect audio data by interacting with the environment.
Revised: The agents start to collect audio data by interacting with the environment.

Original: The goal of this phase is to gather diverse data that could be used to deﬁne auditory events.
Revised: The goal of this phase is to gather diverse data that could be used to deﬁne auditory events.

Original: For this purpose, we train an RL policy by maximizing the occurrences of novel sound effects.
Revised: For this purpose, we train an RL policy by maximizing the occurrences of novel sound effects.

Original: In particular, we design an online clustering-based intrinsic motivation module to guide explorations.
Revised: In particular, we design an online clustering-based intrinsic motivation module to guide explorations.

Original: Assuming we have a series of sound embeddings Φ( s s,t ) , t ∈ 1 , 2 , ..., T and temporarily grouped into K clusters.
Revised: Assuming we have a series of sound embeddings Φ( s s,t ) , t ∈ 1 , 2 , ..., T and temporarily grouped into K clusters.

Original: Given a new coming sound embedding Φ( s s,t +1 ) , we compute its distance to the closest cluster centers and use that as an exploration bonus.
Revised: Given a new coming sound embedding Φ( s s,t +1 ) , we compute its distance to the closest cluster centers and use that as an exploration bonus.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: where r t denotes an intrinsic reward at time t , and c i , i ∈ { 1 , 2 , ..., K } represents a cluster center.
Revised: where r t denotes an intrinsic reward at time t , and c i , i ∈ { 1 , 2 , ..., K } represents a cluster center.

Original: During this exploration, the number of clusters will grow, and each cluster’s center will also be updated.
Revised: During this exploration, the number of clusters will grow, and each cluster’s center will also be updated.

Original: Through this process, the agent is encouraged to collect novel auditory data that could enrich cluster diversity.
Revised: Through this process, the agent is encouraged to collect novel auditory data that could enrich cluster diversity.

Original: After the number of the clusters is saturated, we then perform the K-Means clustering algorithm (Lloyd, 1982) on the collected data to deﬁne the auditory event classes and use the center of each cluster for the subsequent auditory event prediction task.
Revised: After the number of the clusters is saturated, we then perform the K-Means clustering algorithm (Lloyd, 1982) on the collected data to deﬁne the auditory event classes and use the center of each cluster for the subsequent auditory event prediction task.

Original: To be noted, the number of cluster K is determined automatically in our experiments.
Revised: To be noted, the number of cluster K is determined automatically in our experiments.

Original: In practice, we deﬁne K ∈ [5 , 30] for clustering at each time step and use the silhouette score to automatically decide the best K .
Revised: In practice, we deﬁne K ∈ [5 , 30] for clustering at each time step and use the silhouette score to automatically decide the best K .

Original: When the best K keeps unchanged for 3 updates, we believe it is “saturated”.
Revised: When the best K keeps unchanged for 3 updates, we believe it is “saturated”.

Original: We visualize the corresponding visual states in two games ( Frostbite and Assault ) that belong to the same sound clusters, and it can be observed that each cluster always contains identical or similar auditory events (see Figure 2).
Revised: We visualize the corresponding visual states in two games ( Frostbite and Assault ) that belong to the same sound clusters, and it can be observed that each cluster always contains identical or similar auditory events (see Figure 2).

Original: Auditory event predictions.
Revised: Auditory event predictions.

Original: Since we have already explicitly deﬁned the auditory event categories, the prediction problem can then be easily formulated as a classiﬁcation task.
Revised: Since we have already explicitly deﬁned the auditory event categories, the prediction problem can then be easily formulated as a classiﬁcation task.

Original: We label each sound texture with the index of the closest center, and then train a forward dynamics network f (Ψ( s v,t ) , a t ; θ p ) that takes the embeddings of visual observation Ψ( s v,t ) and action a t as input to predict which auditory event cluster the incurred sound Φ( s s,t +1 ) belongs to.
Revised: We label each sound texture with the index of the closest center, and then train a forward dynamics network f (Ψ( s v,t ) , a t ; θ p ) that takes the embeddings of visual observation Ψ( s v,t ) and action a t as input to predict which auditory event cluster the incurred sound Φ( s s,t +1 ) belongs to.

Original: The forward dynamics model is trained on collected data using gradient descent to minimize the cross-entropy loss L between the estimated class probabilities with the ground truth distributions y t +1 as:
Revised: The forward dynamics model is trained on collected data using gradient descent to minimize the cross-entropy loss L between the estimated class probabilities with the ground truth distributions y t +1 as:

Original: [Equation]
Revised: [Equation]

Original: The prediction is expected to fail for novel associations of visual and audio data.
Revised: The prediction is expected to fail for novel associations of visual and audio data.

Original: We use the loss L as intrinsic reward.
Revised: We use the loss L as intrinsic reward.

Original: It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario.
Revised: It will reward the agent at that stage and encourage it to visit more such states since it is uncertain about this scenario.

Original: In practice, we do ﬁnd that the agent can learn to avoid dying scenarios in the games since that gives a similar sound effect it has already encountered many times and can predict very well.
Revised: In practice, we do ﬁnd that the agent can learn to avoid dying scenarios in the games since that gives a similar sound effect it has already encountered many times and can predict very well.

Original: By avoiding potential dangers and seeking novel events, agents can learn causal knowledge of the world for planing their actions to achieve the goal.
Revised: By avoiding potential dangers and seeking novel events, agents can learn causal knowledge of the world for planing their actions to achieve the goal.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: As a proof of concept, we ﬁrst conduct experiments on Atari game environments (Bellemare et al., 2013) to understand our model.
Revised: As a proof of concept, we ﬁrst conduct experiments on Atari game environments (Bellemare et al., 2013) to understand our model.

Original: And then we apply our model to more realistic 3D environments, such as Habitat (Savva et al., 2019) and TDW (Gan et al., 2020a).
Revised: And then we apply our model to more realistic 3D environments, such as Habitat (Savva et al., 2019) and TDW (Gan et al., 2020a).

Original: 4. S ETUP
Revised: 4. S ETUP

Original: Atari Game Environment Our primary goal is to investigate whether we could use auditory event prediction as intrinsic rewards to help RL exploration.
Revised: Atari Game Environment Our primary goal is to investigate whether we could use auditory event prediction as intrinsic rewards to help RL exploration.

Original: For this purpose, we use the Gym Retro (Nichol et al., 2018) as a testbed to measure agents’ competency quantitatively.
Revised: For this purpose, we use the Gym Retro (Nichol et al., 2018) as a testbed to measure agents’ competency quantitatively.

Original: Gym Retro consists of a diverse set of Atari games, and also supports an audio API to provide the sound effects of each state.
Revised: Gym Retro consists of a diverse set of Atari games, and also supports an audio API to provide the sound effects of each state.

Original: We use 20 familiar video games that contain sound effects to compare the intrinsic reward only exploration against several previous state-of-the-art intrinsic modules.
Revised: We use 20 familiar video games that contain sound effects to compare the intrinsic reward only exploration against several previous state-of-the-art intrinsic modules.

Original: We follow the standard setting in (Pathak et al., 2017; Burda et al., 2018b), where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy.
Revised: We follow the standard setting in (Pathak et al., 2017; Burda et al., 2018b), where an agent can use external rewards as an evaluation metric to quantify the performance of the exploration strategy.

Original: This is because doing well on extrinsic rewards usually requires having explored thoroughly.
Revised: This is because doing well on extrinsic rewards usually requires having explored thoroughly.

Original: Audio-Visual Explorations on Habitat
Revised: Audio-Visual Explorations on Habitat

Original: Following (Dean et al., 2020), we use the the apartmentin Replica scene (Straub et al., 2019) with the Habitat simulator for experiments.
Revised: Following (Dean et al., 2020), we use the the apartmentin Replica scene (Straub et al., 2019) with the Habitat simulator for experiments.

Original: This scene has 211 unique locations as the state space, and the action space contains three discrete actions, namely
Revised: This scene has 211 unique locations as the state space, and the action space contains three discrete actions, namely

Original: [Table]
Revised: [Table]

Original: Active learning with a rolling robot on TDW We ﬁnally test our module on a 3D multi-modal physic simulation platform, built on top of TDW (Gan et al., 2020a).
Revised: Active learning with a rolling robot on TDW We ﬁnally test our module on a 3D multi-modal physic simulation platform, built on top of TDW (Gan et al., 2020a).

Original: As shown in Figure 7 , we place a rolling robot agent ( i.e. , red sphere) on a billiard table.
Revised: As shown in Figure 7 , we place a rolling robot agent ( i.e. , red sphere) on a billiard table.

Original: The agent is required to execute actions to interact with objects of different materials and shapes.
Revised: The agent is required to execute actions to interact with objects of different materials and shapes.

Original: When two objects collide, the environment could generate collision sound based on the physical properties of objects.
Revised: When two objects collide, the environment could generate collision sound based on the physical properties of objects.

Original: We would like to compare the agent’s behaviors in this 3D world physical environment using different intrinsic rewards.
Revised: We would like to compare the agent’s behaviors in this 3D world physical environment using different intrinsic rewards.

Original: Baselines.
Revised: Baselines.

Original: We consider four state-of-the-art intrinsic motivation modules for comparisons, including
Revised: We consider four state-of-the-art intrinsic motivation modules for comparisons, including

Original: Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), Random Feature Networks (RFN) (Burda et al., 2018a), Random Network Distillation (RND) (Burda et al., 2018a), and Model Disagreement (DIS) (Pathak et al., 2019).
Revised: Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), Random Feature Networks (RFN) (Burda et al., 2018a), Random Network Distillation (RND) (Burda et al., 2018a), and Model Disagreement (DIS) (Pathak et al., 2019).

Original: We use their open-source code to reproduce the results.
Revised: We use their open-source code to reproduce the results.

Original: Implementation details.
Revised: Implementation details.

Original: For all the experiments, we choose PPO algorithm (Schulman et al., 2017) based on the PyTorch implementation to train our RL agent since it is robust and requires little hyper-parameter tuning.
Revised: For all the experiments, we choose PPO algorithm (Schulman et al., 2017) based on the PyTorch implementation to train our RL agent since it is robust and requires little hyper-parameter tuning.

Original: We use the open-source toolbox 1 to extract sound texture features.
Revised: We use the open-source toolbox 1 to extract sound texture features.

Original: For experiments on Atari games, we use gray-scale image observations of size 84 × 84 and 60ms audio clip.
Revised: For experiments on Atari games, we use gray-scale image observations of size 84 × 84 and 60ms audio clip.

Original: We set the skip frame S = 4 for all the experiments.
Revised: We set the skip frame S = 4 for all the experiments.

Original: We use a 4-layer CNN as the encoder of the policy network.
Revised: We use a 4-layer CNN as the encoder of the policy network.

Original: As for the auditory prediction network, we choose a 3-layer CNN with output channel 32, 64, 64 to encode the image observation and use 2-layer MLP with 512 hidden units to predict the auditory events.
Revised: As for the auditory prediction network, we choose a 3-layer CNN with output channel 32, 64, 64 to encode the image observation and use 2-layer MLP with 512 hidden units to predict the auditory events.

Original: For all experiments, our model use 10K interactions for stage 1 exploration, which is also included in the beginning of each reported curve.
Revised: For all experiments, our model use 10K interactions for stage 1 exploration, which is also included in the beginning of each reported curve.

Original: The intrinsic reward in both phases are clipped to [-1, 1] after normalization.
Revised: The intrinsic reward in both phases are clipped to [-1, 1] after normalization.

Original: We use 8, 1, and 8 parallel environments for Atari, Habitat, and
Revised: We use 8, 1, and 8 parallel environments for Atari, Habitat, and

Original: TDW experiments, respectively.
Revised: TDW experiments, respectively.

Original: 4. E XPERIMENTS ON A TARI
Revised: 4. E XPERIMENTS ON A TARI

Original: Overall Results.
Revised: Overall Results.

Original: Figure 3 summarizes the evaluation curves of mean extrinsic reward in 20 Atari games.
Revised: Figure 3 summarizes the evaluation curves of mean extrinsic reward in 20 Atari games.

Original: For each method, we experiment with three different random seeds and report the standard deviation in the shaded region.
Revised: For each method, we experiment with three different random seeds and report the standard deviation in the shaded region.

Original: As the ﬁgure shows, our module achieves signiﬁcantly better results than previous vision-only intrinsic motivation modules in 15 out of 20 games.
Revised: As the ﬁgure shows, our module achieves signiﬁcantly better results than previous vision-only intrinsic motivation modules in 15 out of 20 games.

Original: Another interesting observation is that the earned score keeps going up, even only using intrinsic rewards.
Revised: Another interesting observation is that the earned score keeps going up, even only using intrinsic rewards.

Original: Result Analysis.
Revised: Result Analysis.

Original: We would like to provide an in-depth understanding of under what circumstances our algorithm works well.
Revised: We would like to provide an in-depth understanding of under what circumstances our algorithm works well.

Original: The sound effects in Atari games fall into three different categories: 1) event-driven sounds which emitted when agents achieve a speciﬁc condition ( e.g. , picking up a coin, the explosion of an aircraft, etc.); 2) action-driven sounds which emitted when agents implement a speciﬁc action ( e.g. , shooting, jumping, etc.) and 3) background noise/music.
Revised: The sound effects in Atari games fall into three different categories: 1) event-driven sounds which emitted when agents achieve a speciﬁc condition ( e.g. , picking up a coin, the explosion of an aircraft, etc.); 2) action-driven sounds which emitted when agents implement a speciﬁc action ( e.g. , shooting, jumping, etc.) and 3) background noise/music.

Original: According to the dominant sound effects in each game, we summarize the 20 Atari games in Table 1.
Revised: According to the dominant sound effects in each game, we summarize the 20 Atari games in Table 1.

Original: It is also worth noticing that, the sounds in Atari games include positive ( e.g. earning a bonus), negative ( e.g. dying) and meaningless sound ( e.g. background music).
Revised: It is also worth noticing that, the sounds in Atari games include positive ( e.g. earning a bonus), negative ( e.g. dying) and meaningless sound ( e.g. background music).

Original: None of the category accounts for the majority.
Revised: None of the category accounts for the majority.

Original: Based on the category deﬁned in Table 1 and the performance shown in Figure 3, we can draw three conclusions.
Revised: Based on the category deﬁned in Table 1 and the performance shown in Figure 3, we can draw three conclusions.

Original: First, both event-driven and action-driven sounds boost the performance of our
Revised: First, both event-driven and action-driven sounds boost the performance of our

Original: [Figure]
Revised: [Figure]

Original: v
Revised: v

Original: [Figure]
Revised: [Figure]

Original: r a g e E x t r i n
Revised: r a g e E x t r i n

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: r a g
Revised: r a g

Original: [Figure]
Revised: [Figure]

Original: t r i n s i
Revised: t r i n s i

Original: c R
Revised: c R

Original: [Figure]
Revised: [Figure]

Original: w a r d s
Revised: w a r d s

Original: FishingDerby
Revised: FishingDerby

Original: [Figure]
Revised: [Figure]

Original: v e r a g e E x t r i n s i
Revised: v e r a g e E x t r i n s i

Original: c
Revised: c

Original: R e w a r d s
Revised: R e w a r d s

Original: [Figure]
Revised: [Figure]

Original: v e r a g e E x t r i n s i
Revised: v e r a g e E x t r i n s i

Original: c
Revised: c

Original: R e w a r d s
Revised: R e w a r d s

Original: [Figure]
Revised: [Figure]

Original: algorithm.
Revised: algorithm.

Original: Since the sound is more observable effects of action and events, understanding these casual effects is essential to learn a better exploration policy.
Revised: Since the sound is more observable effects of action and events, understanding these casual effects is essential to learn a better exploration policy.

Original: Second, our algorithm performs better on the games dominant with event-driven sounds compared to those with action-driven sounds.
Revised: Second, our algorithm performs better on the games dominant with event-driven sounds compared to those with action-driven sounds.

Original: We believe that event-driven sounds contain higher-level information, such as the explosion of an aircraft or collecting coins, which can be more beneﬁcial for the agent to understand the physical world.
Revised: We believe that event-driven sounds contain higher-level information, such as the explosion of an aircraft or collecting coins, which can be more beneﬁcial for the agent to understand the physical world.

Original: Third, our algorithm falls short in comparison with baselines when the sound effects mainly consist of meaningless background noise or background music ( i.e. CrazyClimber, MsPacman, Gopher,
Revised: Third, our algorithm falls short in comparison with baselines when the sound effects mainly consist of meaningless background noise or background music ( i.e. CrazyClimber, MsPacman, Gopher,

Original: Tutankham, WizardOfWor, and BattleZone).
Revised: Tutankham, WizardOfWor, and BattleZone).

Original: These are also reasonable since our noisy agents are incentivized to make interventions that lead to making surprising sounds.
Revised: These are also reasonable since our noisy agents are incentivized to make interventions that lead to making surprising sounds.

Original: Sometimes sound events will occur independently of the agent’s decisions and do not differentiate between different policies, and therefore do not guide learning.
Revised: Sometimes sound events will occur independently of the agent’s decisions and do not differentiate between different policies, and therefore do not guide learning.

Original: The events that guide learning are those that are a consequence, directly or indirectly, of an agent’s actions.
Revised: The events that guide learning are those that are a consequence, directly or indirectly, of an agent’s actions.

Original: The agent is encouraged to understand the causal effect of its actions and thereby take actions that lead to better exploration.
Revised: The agent is encouraged to understand the causal effect of its actions and thereby take actions that lead to better exploration.

Original: Ablated Study.
Revised: Ablated Study.

Original: We further carry out additional experiments to study the following questions of model design.
Revised: We further carry out additional experiments to study the following questions of model design.

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e E x t r i n s i
Revised: A v e r a g e E x t r i n s i

Original: c
Revised: c

Original: R e w a r d s
Revised: R e w a r d s

Original: Amidar
Revised: Amidar

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Predict auditory events or sound features?
Revised: Predict auditory events or sound features?

Original: One main contribution of our paper is to use auditory event prediction as an intrinsic reward.
Revised: One main contribution of our paper is to use auditory event prediction as an intrinsic reward.

Original: We conduct an ablated study by replacing this module with sound feature prediction module.
Revised: We conduct an ablated study by replacing this module with sound feature prediction module.

Original: We train a neural network that takes the embedding of visual state and action as input and predicts the sound textures.
Revised: We train a neural network that takes the embedding of visual state and action as input and predicts the sound textures.

Original: The comparison curves are plotted in Figure 4.
Revised: The comparison curves are plotted in Figure 4.

Original: We observe that the auditory event prediction module indeed earned more rewards.
Revised: We observe that the auditory event prediction module indeed earned more rewards.

Original: We speculate that auditory events provide more structured knowledge of the world, thus lead to better RL explorations.
Revised: We speculate that auditory events provide more structured knowledge of the world, thus lead to better RL explorations.

Original: Sound clustering or auditory event prediction?
Revised: Sound clustering or auditory event prediction?

Original: We adopt a two-stage exploration strategy.
Revised: We adopt a two-stage exploration strategy.

Original: A natural question is if this is necessary.
Revised: A natural question is if this is necessary.

Original: We show the curve of using phase-1 only ( i.e. sound clustering) in Figure 4.
Revised: We show the curve of using phase-1 only ( i.e. sound clustering) in Figure 4.

Original: This is a novelty driven exploration strategy.
Revised: This is a novelty driven exploration strategy.

Original: The reward function is deﬁned as the distance to the closest clusters.
Revised: The reward function is deﬁned as the distance to the closest clusters.

Original: We notice that the returned extrinsic reward is similar to sound feature prediction, but worse than auditory event prediction.
Revised: We notice that the returned extrinsic reward is similar to sound feature prediction, but worse than auditory event prediction.

Original: Is active exploration necessary?
Revised: Is active exploration necessary?

Original: We propose an online clustering-based intrinsic module for active audio data collections.
Revised: We propose an online clustering-based intrinsic module for active audio data collections.

Original: To verify its efﬁcacy, we replace this module with random explorations and
Revised: To verify its efﬁcacy, we replace this module with random explorations and

Original: RND (Burda et al., 2018a).
Revised: RND (Burda et al., 2018a).

Original: For fair comparisons, we allow both models to use 10K interaction data to deﬁne the event classes with the same K-mean clustering algorithm.
Revised: For fair comparisons, we allow both models to use 10K interaction data to deﬁne the event classes with the same K-mean clustering algorithm.

Original: The comparison results are shown in Figure 5.
Revised: The comparison results are shown in Figure 5.

Original: We can ﬁnd that the proposed active explorations indeed achieve better results.
Revised: We can ﬁnd that the proposed active explorations indeed achieve better results.

Original: We also compute the cluster distances of these models and ﬁnd that the sound clusters discovered by active exploration are more diverse, thus facilitating the agents to perform in-depth explorations.
Revised: We also compute the cluster distances of these models and ﬁnd that the sound clusters discovered by active exploration are more diverse, thus facilitating the agents to perform in-depth explorations.

Original: 4. E XPERIMENTS ON H ABITAT
Revised: 4. E XPERIMENTS ON H ABITAT

Original: [Figure]
Revised: [Figure]

Original: To evaluate the exploration ability of agents trained with our approach, we report the unique state coverage, given a limited exploration budget.
Revised: To evaluate the exploration ability of agents trained with our approach, we report the unique state coverage, given a limited exploration budget.

Original: In Figure 6, using only 16K exploration steps, our agent has already explored all unique states (211 states), while other methods have visited less than 195 unique states (190, 157, 170, andfor ICM, RND, RFN, and DIS, respectively) on average.
Revised: In Figure 6, using only 16K exploration steps, our agent has already explored all unique states (211 states), while other methods have visited less than 195 unique states (190, 157, 170, andfor ICM, RND, RFN, and DIS, respectively) on average.

Original: Even exploring for 20k steps, all other methods have not yet fully explored the 211 states.
Revised: Even exploring for 20k steps, all other methods have not yet fully explored the 211 states.

Original: These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.
Revised: These results demonstrate that our agent explores environments more quickly and fully, showing the potential ability of exploring the real world.

Original: We believe the reason is that our auditory event prediction encourages agents to explore the novel areas with uncertain sounds.
Revised: We believe the reason is that our auditory event prediction encourages agents to explore the novel areas with uncertain sounds.

Original: 4. E XPERIMENTS ON TDW
Revised: 4. E XPERIMENTS ON TDW

Original: [Figure]
Revised: [Figure]

Original: Setup.
Revised: Setup.

Original: We take an image observation of 84 × 84 size and 50ms audio clip as input.
Revised: We take an image observation of 84 × 84 size and 50ms audio clip as input.

Original: We use a threelayer convolutional network to encode the image and extract sound textures from the audio clip.
Revised: We use a threelayer convolutional network to encode the image and extract sound textures from the audio clip.

Original: Same as the previous experiment, we train the policy using the PPO algorithm.
Revised: Same as the previous experiment, we train the policy using the PPO algorithm.

Original: The action space consists of moving to eight directions and stop.
Revised: The action space consists of moving to eight directions and stop.

Original: An action is repeated 4 times on each frame.
Revised: An action is repeated 4 times on each frame.

Original: We run all the experiments for 200K steps with 8 parallel environments.
Revised: We run all the experiments for 200K steps with 8 parallel environments.

Original: Result Analysis.
Revised: Result Analysis.

Original: To understand and quantify the agent’s behaviors in this 3D physical world, we show the number of collision events and intrinsic rewards in Figure 7.
Revised: To understand and quantify the agent’s behaviors in this 3D physical world, we show the number of collision events and intrinsic rewards in Figure 7.

Original: We noticed two major issues with the previous vision-based curiosity models.
Revised: We noticed two major issues with the previous vision-based curiosity models.

Original: First, the prediction errors on latent feature space could not accurately reﬂect subtle object state changes in the 3D photo-realistic world, in which a physical event happens.
Revised: First, the prediction errors on latent feature space could not accurately reﬂect subtle object state changes in the 3D photo-realistic world, in which a physical event happens.

Original: Second, the intrinsic reward can diminish quickly during training, since the learned predictive model usually converges to a stable state representation of the environment.
Revised: Second, the intrinsic reward can diminish quickly during training, since the learned predictive model usually converges to a stable state representation of the environment.

Original: Instead, our auditory event prediction driven exploration will lead agents to interact more with objects in the physical world, which is critical to learning the dynamics of the environments.
Revised: Instead, our auditory event prediction driven exploration will lead agents to interact more with objects in the physical world, which is critical to learning the dynamics of the environments.

Original: 4. D ISCUSSIONS
Revised: 4. D ISCUSSIONS

Original: This section aims to examine if our module leverages the knowledge of different auditory events for
Revised: This section aims to examine if our module leverages the knowledge of different auditory events for

Original: RL explorations.
Revised: RL explorations.

Original: Do we need to perform event classiﬁcations?
Revised: Do we need to perform event classiﬁcations?

Original: We aim to understand the role that the auditory event prediction module plays in our learning framework.
Revised: We aim to understand the role that the auditory event prediction module plays in our learning framework.

Original: There are two alternatives to leverage multisensory signals.
Revised: There are two alternatives to leverage multisensory signals.

Original: One option is to design a simple binary classiﬁcation task by predicting making a sound or not(See Bin Cls in Figure 8).
Revised: One option is to design a simple binary classiﬁcation task by predicting making a sound or not(See Bin Cls in Figure 8).

Original: The other one is to predict the associations between visual frames and audio clips (Dean et al., 2020)
Revised: The other one is to predict the associations between visual frames and audio clips (Dean et al., 2020)

Original: (See SHE in Figure 8).
Revised: (See SHE in Figure 8).

Original: We will reward an agent if its prediction is incorrect.
Revised: We will reward an agent if its prediction is incorrect.

Original: Otherwise, it will be no reward.
Revised: Otherwise, it will be no reward.

Original: We conduct these ablated studies in 3 Atari games, Habitat, and TDW environments.
Revised: We conduct these ablated studies in 3 Atari games, Habitat, and TDW environments.

Original: From Figure 8, we observe that predicting auditory events can indeed lead to a better exploration strategy.
Revised: From Figure 8, we observe that predicting auditory events can indeed lead to a better exploration strategy.

Original: These results further demonstrate that understanding the causal structures of different events is essential for multi-sensory RL explorations.
Revised: These results further demonstrate that understanding the causal structures of different events is essential for multi-sensory RL explorations.

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: Do we need audio to deﬁne events?
Revised: Do we need audio to deﬁne events?

Original: We also want to understand if it is necessary to use audios to deﬁne high-level events.
Revised: We also want to understand if it is necessary to use audios to deﬁne high-level events.

Original: Perhaps, the results will also be good if we replace the audio features with visual features to cluster the event types.
Revised: Perhaps, the results will also be good if we replace the audio features with visual features to cluster the event types.

Original: We plot the curves of clustering visual frame to get the event type (see Img Cls in Figure 8).
Revised: We plot the curves of clustering visual frame to get the event type (see Img Cls in Figure 8).

Original: The results are signiﬁcantly worse than using audio features.
Revised: The results are signiﬁcantly worse than using audio features.

Original: We hypothesize that it might be hard to capture the patterns of pixel changes when these events happen, but sounds are more observable to discover these events.
Revised: We hypothesize that it might be hard to capture the patterns of pixel changes when these events happen, but sounds are more observable to discover these events.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we introduce an intrinsic reward function of predicting sound for RL exploration.
Revised: In this work, we introduce an intrinsic reward function of predicting sound for RL exploration.

Original: Our model employs the errors of auditory event prediction as an exploration bonus, which allows RL agents to explore novel physical interactions of objects.
Revised: Our model employs the errors of auditory event prediction as an exploration bonus, which allows RL agents to explore novel physical interactions of objects.

Original: We demonstrate our proposed methodology and compared it against a number of baselines on Atari games.
Revised: We demonstrate our proposed methodology and compared it against a number of baselines on Atari games.

Original: Based on the experimental result above, we conclude that sound conveys rich information and is powerful for agents to build a causal model of the physical world for efﬁcient RL explorations.
Revised: Based on the experimental result above, we conclude that sound conveys rich information and is powerful for agents to build a causal model of the physical world for efﬁcient RL explorations.

Original: We hope our work could inspire more works on using multi-modality cues for planing and control.
Revised: We hope our work could inspire more works on using multi-modality cues for planing and control.

Original: A T RAINING D ETAILS
Revised: A T RAINING D ETAILS

Original: Table 2 shows the hyper-parameters used in our algorithm.
Revised: Table 2 shows the hyper-parameters used in our algorithm.

Original: Hyperparameter
Revised: Hyperparameter

Original: Value
Revised: Value

Original: Rollout length
Revised: Rollout length

Original: Number of minibatches
Revised: Number of minibatches

Original: [Figure]
Revised: [Figure]

Original: Learning rate
Revised: Learning rate

Original: [Figure]
Revised: [Figure]

Original: Clip parameter
Revised: Clip parameter

Original: [Figure]
Revised: [Figure]

Original: Entropy coefﬁcient
Revised: Entropy coefﬁcient

Original: [Figure]
Revised: [Figure]

Original: B A N A ¨ VE BASELINE FOR MULTI - MODAL RL
Revised: B A N A ¨ VE BASELINE FOR MULTI - MODAL RL

Original: [Figure]
Revised: [Figure]

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we carry out ablated experiments to demonstrate that the gains in our method are caused by the audio-event prediction, rather than the use of multi-modality information.
Revised: we carry out ablated experiments to demonstrate that the gains in our method are caused by the audio-event prediction, rather than the use of multi-modality information.

Original: For four baselines ( i
Revised: For four baselines ( i

Original: [Figure]
Revised: [Figure]

Original: e. RND, RFN, ICM
Revised: e. RND, RFN, ICM

Original: [Figure]
Revised: [Figure]

Original: and DIS), instead of predicting audio-event
Revised: and DIS), instead of predicting audio-event

Original: [Figure]
Revised: [Figure]

Original: they consider sound information by concatenating both visual and sound features to predict the image embedding in the next time step.
Revised: they consider sound information by concatenating both visual and sound features to predict the image embedding in the next time step.

Original: As shown in Figure 9, our algorithm signiﬁcantly outperforms other baselines in ﬁve Atari games.
Revised: As shown in Figure 9, our algorithm signiﬁcantly outperforms other baselines in ﬁve Atari games.

Original: This indicates that it is non-trivial to exploit sound information for RL, and our algorithm beneﬁts from the carefully designed audio-event prediction as an intrinsic reward.
Revised: This indicates that it is non-trivial to exploit sound information for RL, and our algorithm beneﬁts from the carefully designed audio-event prediction as an intrinsic reward.

Original: C H ARD E XPLORATIONS G AMES
Revised: C H ARD E XPLORATIONS G AMES

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we investigate whether the audio-driven intrinsic rewards could be further utilized to improve policy learning in the hard exploration scenarios, where extrinsic rewards exist but very sparse.
Revised: we investigate whether the audio-driven intrinsic rewards could be further utilized to improve policy learning in the hard exploration scenarios, where extrinsic rewards exist but very sparse.

Original: We use six hard exploration environments in Atari games, including Venture, Solaris, Private
Revised: We use six hard exploration environments in Atari games, including Venture, Solaris, Private

Original: Eye, Pitfall!, Gravitar and Montezuma’s Revenge for experiments.
Revised: Eye, Pitfall!, Gravitar and Montezuma’s Revenge for experiments.

Original: Following the strategy proposed
Revised: Following the strategy proposed

Original: [Figure]
Revised: [Figure]

Original: by RND (Burda et al., 2018a), we use two value heads separately for the intrinsic and extrinsic reward module and then combine their returns.
Revised: by RND (Burda et al., 2018a), we use two value heads separately for the intrinsic and extrinsic reward module and then combine their returns.

Original: We also normalize intrinsic rewards to make up the variances among different environments.
Revised: We also normalize intrinsic rewards to make up the variances among different environments.

Original: We compare our model against those vision-only baselines including
Revised: We compare our model against those vision-only baselines including

Original: ICM, RND, RFN, DIS mentioned in the main paper.
Revised: ICM, RND, RFN, DIS mentioned in the main paper.

Original: All experiments are run for 4 million steps with 32 parallel environments.
Revised: All experiments are run for 4 million steps with 32 parallel environments.

Original: We use the max episodic returns to measure the ability of explorations.
Revised: We use the max episodic returns to measure the ability of explorations.

Original: The comparison results are shown in Figure 10.
Revised: The comparison results are shown in Figure 10.

Original: Note the X-axis here refers to the frames rather than parameter updates like RND.
Revised: Note the X-axis here refers to the frames rather than parameter updates like RND.

Original: All the baselines are reproduced using their open-source code.
Revised: All the baselines are reproduced using their open-source code.

Original: Our approach achieves best/comparable results in 4 out of 6 Atari games.
Revised: Our approach achieves best/comparable results in 4 out of 6 Atari games.