Original: N EURAL E PISODIC C ONTROL WITH S TATE A BSTRACTION
Revised: N EURAL E PISODIC C ONTROL WITH S TATE A BSTRACTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency.
Revised: Existing Deep Reinforcement Learning (DRL) algorithms suffer from sample inefficiency.

Original: Generally, episodic control-based approaches arethose solutions that leverage highly-rewarded past experiences to improvethe DRL algorithms’ sample efficiency.
Revised: Generally, episodic control-based approaches are solutions that leverage highly-rewarded past experiences to improve sample efficiency of DRL algorithms.

Original: However, previous episodic control-based approaches have failed to utilize the latent information from the historical behaviors ( e.g. , state transitions, topological similarities, etc. ) and lack scalability during DRL training.
Revised: However, previous episodic control-based approaches fail to utilize the latent information from the historical behaviors ( e.g. , state transitions, topological similarities, etc. ) and lack scalability during DRL training.

Original: This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic control containing a more comprehensive episodic memory, a novel state evaluation, and a multi-step state analysis.
Revised: This work introduces Neural Episodic Control with State Abstraction (NECSA), a simple but effective state abstraction-based episodic control containing a more comprehensive episodic memory, a novel state evaluation, and a multi-step state analysis.

Original: We evaluate our approach to the MuJoCo environments in OpenAI gym domains.
Revised: We evaluate our approach to the MuJoCo and Atari tasks in OpenAI gym domains.

Original: The experimental results indicate that NECSA achieve higher sample efficiency than the state-of-the-art episodic control-based approaches.
Revised: The experimental results indicate that NECSA achieves higher sample efficiency than the state-of-the-art episodic control-based approaches.

Original: Our data and code are available at the website: https://sites.google.com/view/drl-necsa.
Revised: Our data and code are available at the project website 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (DRL) has garnered much attention in both research and industry, with applications in various fields related to artificial intelligence (AI) such as games (Mnih et al., 2013; Silver et al., 2018), autonomous driving (Xu et al., 2020),  and robotics (Thomaz & Breazeal, 2008).
Revised: Deep reinforcement learning (DRL) has garnered much attention in both research and industry, with applications in various fields related to artificial intelligence (AI) such as games (Mnih et al., 2013; Silver et al., 2018; Shen et al., 2020), autonomous driving (Xu et al., 2020), software testing (Zheng et al., 2019; 2021c) and robotics (Thomaz & Breazeal, 2008).

Original: DRL usually achieves excellent performance for many tasks and sometimes outperforms human beings.
Revised: DRL usually achieves excellent performance for many tasks and sometimes outperforms human beings.

Original: However, human-level DRL policies usually require a tremendous amount of data and millions of training steps, which are demonstrated to be sample inefficient (Arulkumaran et al., 2017; Tsividis et al., 2017).
Revised: However, human-level DRL policies usually require a tremendous amount of data and millions of training steps, which are demonstrated to be sample inefficient (Arulkumaran et al., 2017; Tsividis et al., 2017).

Original: To mitigate this problem, many approaches have been proposed, such as improving the exploration (Yu, 2018; Burda et al., 2018), modeling the environment (Moerland et al., 2020), state abstraction (Vezhnevets et al., 2017) and knowledge transfer (Lazaric et al., 2008).
Revised: To mitigate this problem, many approaches have been proposed, such as improving the exploration (Yu, 2018; Burda et al., 2018), modeling the environment (Moerland et al., 2020), state abstraction (Vezhnevets et al., 2017) and knowledge transfer (Lazaric et al., 2008; Zhang et al., 2020; Cao et al., 2022).

Original: However, this paper focuses on resolving the problem of sample inefficiency through episodic control.
Revised: However, this paper focuses on resolving the problem of sample inefficiency through episodic control.

Original: Episodic control is designed to assist DRL agents in making the appropriate decisions in unseen environments using past experiences.
Revised: Episodic control is designed to assist DRL agents in making the appropriate decisions in unseen environments using past experiences.

Original: The idea is inspired by a biological mechanism, hippocampus (Lengyel & Dayan, 2007).
Revised: The idea is inspired by a biological mechanism, hippocampus (Lengyel & Dayan, 2007).

Original: Moreover, episodic control has been adopted to tackle the sample inefficiency in DRL (Blundell et al., 2016; Pritzel et al., 2017).
Revised: Moreover, episodic control has been adopted to tackle the sample inefficiency in DRL (Blundell et al., 2016; Pritzel et al., 2017).

Original: Previous neural episodic control-based approacheshave usually stored past experiences in a tabular memory.
Revised: Previous neural episodic control-based approaches usually store past experiences in a tabular memory.

Original: Therefore, the agent could retrieve the past highly-rewarded experiences by looking up similar cached states from the episodic memory.
Revised: Therefore, the agent could retrieve historical highly-rewarded experiences by looking up similar cached states from the episodic memory.

Original: Then the state (action) values could be estimated based on the similar states retrieved.
Revised: Then the state (action) values could be estimated based on the similar states retrieved.

Original: In this way, the policy can quickly reduce the bias between episodic and model estimated state values and generalize the past highly-rewarded cases.
Revised: In this way, the policy can efficiently reduce the bias between episodic and model estimated state values and generalize the past highly-rewarded cases.

Original: Although many episodic control-based approaches were proposed to improve the sample efficiency of DRL policy, all of them suffer from obvious limitations (Hu et al., 2021; Pinto, 2020; Kuznetsov &
Revised: Although many episodic control-based approaches were proposed to improve the sample efficiency of DRL policy, all of them suffer from obvious limitations (Hu et al., 2021; Pinto, 2020; Kuznetsov &

Original: Filchenkov, 2021).
Revised: Filchenkov, 2021).

Original: In general, they only store the concrete states, actions, and state values (Blundell et al., 2016).
Revised: In general, they only store the concrete states, actions, and state values (Blundell et al., 2016).

Original: On the other hand, their episodic memory does not record information such as time steps and transitions in the traces.
Revised: On the other hand, their episodic memory does not record information such as time steps and transitions in the traces.

Original: As a result, some latent semantics, such as state transitions and topological similarities, cannot be explored and exploited.
Revised: As a result, some latent semantics, such as state transitions and topological similarities, cannot be explored and exploited.

Original: However, many previous works demonstrate that such latent information can be used to improve sample efficiency (Kuznetsov & Filchenkov, 2021; Zhu et al., 2020).
Revised: However, many previous works demonstrate that such latent information can be used to improve sample efficiency (Kuznetsov & Filchenkov, 2021; Zhu et al., 2020).

Original: For instance, a DRL model is trained to make decisions continuously, and the influence ( e.g. , approximation errors) of a state-action pair might be accumulated and affect the forward scenes (Dynkin, 1965).
Revised: For instance, a DRL model is trained to make decisions continuously, and the influence ( e.g. , approximation errors) of a state-action pair might be accumulated and affect the forward scenes (Dynkin, 1965).

Original: In other words, the root cause of a bad decision in the current state can either come from the latest state or the much earlier states.
Revised: In other words, the root cause of a bad decision in the current state can either come from the latest state or the much earlier states.

Original: The topological state transitions can help trace the root cause of the bad decisions.
Revised: The topological state transitions can help trace the root cause of the bad decisions.

Original: Moreover, the data structure of existing episodic memories does not efficiently support storing and exploring latent semantics.
Revised: Moreover, the data structure of existing episodic memories does not efficiently support storing and exploring latent semantics.

Original: The reason is that the concrete state representations usually consist of float numbers.
Revised: The reason is that the concrete state representations usually consist of float numbers.

Original: Thus almost none of the states are the same.
Revised: Thus almost none of the states are the same.

Original: Consequently, we cannot directly count and retrieve them from episodic memory, and it is impossible to identify the critical state transitions.
Revised: Consequently, we cannot directly count and retrieve them from episodic memory, and it is impossible to identify the critical state transitions.

Original: In addition, existing episodic control-based approaches, which utilize distance-based measurement to retrieve the k -st most similar concrete states ( e.g. , k-nearest neighbors, i.e. , kNN search) and the weighted sum of the state (action) values for the estimation (Pritzel et al., 2017; Lin et al., 2018), are inevitably resource-consuming.
Revised: In addition, existing episodic control-based approaches, which utilize distance-based measurement to retrieve the k -st most similar concrete states ( e.g. , k-nearest neighbors, i.e. , kNN search) and the weighted sum of the state (action) values for the estimation (Pritzel et al., 2017; Lin et al., 2018), are inevitably resource-consuming.

Original: Overall, existing episodic control-based approaches lack (1) a more comprehensive episodic memory analysis ( i.e. , multi-step state transitions) and (2) a more scalable storage and retrieval strategy for episodic data.
Revised: Overall, existing episodic control-based approaches lack (1) a more comprehensive episodic memory analysis ( i.e. , multi-step state transitions) and (2) a more scalable storage and retrieval strategy for episodic data.

Original: We propose NECSA, a state abstraction-based neural episodic control approach that enables a more comprehensive analysis of episodic data and better sample efficiency to address the above issues.
Revised: We propose NECSA, a state abstraction-based neural episodic control approach that enables a more comprehensive analysis of episodic data and better sample efficiency to address the above issues.

Original: Inspired by multi-grid and model-based reinforcement learning (Grze´s & Kudenko, 2008; Kaiser et al., 2019), we discretize the continuous state space into finite grids on each dimension, and the states located in the same grid will be labeled with a unique ID .
Revised: Inspired by multi-grid and model-based reinforcement learning (Grze ´ s & Kudenko, 2008; Kaiser et al., 2019), we discretize the continuous state space into finite grids on each dimension, and the states located in the same grid will be labeled with a unique ID .

Original: Naturally, we conduct a multi-step analysis of the state transitions by treating the consecutive state transitions as a fixed pattern.
Revised: Naturally, we conduct a multi-step analysis of the state transitions by treating the consecutive state transitions as a fixed pattern.

Original: Finally, we make the policy generalize different patterns, as we infer that analyzing and generalizing such multi-step patterns might result in better performance than just focusing on a single state (Sutton & Barto, 2018).
Revised: Finally, we make the policy generalize different patterns, as we infer that analyzing and generalizing such multi-step patterns might result in better performance than just focusing on a single state (Sutton & Barto, 2018).

Original: Such abstraction enables the following strengths: (1) based on the abstracted state space, more advanced semantic characteristics, such as state transitions and topological similarities, can be analyzed for improving performance (Grze ´ s & Kudenko, 2008); (2) the complexity of storing and retrieving the episodic data is reduced O ( N ) to O (1) since we can retrieve the episodic memory using an exact match.
Revised: Such abstraction enables the following strengths: (1) based on the abstracted state space, more advanced semantic characteristics, such as state transitions and topological similarities, can be analyzed for improving performance (Grze´s & Kudenko, 2008); (2) the complexity of storing and retrieving the episodic data is reduced O ( N ) to O (1) since we can retrieve the episodic memory using an exact match.

Original: Previous episodic controls used average state values as the state measurement to correct the DRL policy estimation (Lin et al., 2018; Kuznetsov & Filchenkov, 2021).
Revised: Previous episodic controls used average state values as the state measurement to correct the DRL policy estimation (Lin et al., 2018; Kuznetsov & Filchenkov, 2021).

Original: Nevertheless, such a state measurement cannot be computed directly for a multi-step pattern.
Revised: Nevertheless, such a state measurement cannot be computed directly for a multi-step pattern.

Original: Instead, we propose an intrinsic state measurement based on state abstraction instead of past state values.
Revised: Instead, we propose an intrinsic state measurement based on state abstraction instead of past state values.

Original: Specifically, we record the returns of those episodes where they occur in an abstract pattern.
Revised: Specifically, we record the returns of those episodes where they occur in an abstract pattern.

Original: Then we compute the average returns to measure the abstract pattern.
Revised: Then we compute the average returns to measure the abstract pattern.

Original: This measurement can efficiently identify those patterns which can result in higher rewards.
Revised: This measurement can efficiently identify those patterns which can result in higher rewards.

Original: By utilizing such intrinsic rewards (Burda et al., 2018), we revise the policy and accelerate the learning by encouraging those states with higher measurements but punishing those with relatively low measurements.
Revised: By utilizing such intrinsic rewards (Burda et al., 2018), we revise the policy and accelerate the learning by encouraging those states with higher measurements but punishing those with relatively low measurements.

Original: Finally, we evaluate NECSA on MuJoCo (Todorov et al., 2012)  in OpenAI gym (Brockman et al., 2016) domains.
Revised: Finally, we evaluate NECSA on MuJoCo (Todorov et al., 2012) and Atari tasks in OpenAI gym (Brockman et al., 2016) domains.

Original: The evaluation shows that our approach can significantly improve the sample efficiency and outperform state-of-the-art episodic control.
Revised: The evaluation shows that our approach can significantly improve the sample efficiency and outperform state-of-the-art episodic control.

Original: In summary, we make the following contributions: ) we propose a multi-step analysis of state transitions to achieve better policies; (2) we propose a comprehensive episodic memory that enables a more advanced analysis of past experiences; (3) we propose an intrinsic reward-based episodic control method to optimize the policy.
Revised: In summary, we make the following contributions: (1) we propose a multi-step analysis of state transitions to achieve better policies; (2) we propose a comprehensive episodic memory that enables a more advanced analysis of past experiences; (3) we propose an intrinsic reward-based episodic control method to optimize the policy.

Original:  R ELATED W ORK2. N EURAL E PISODIC C ONTROL 
Revised:  R ELATED W ORK

Revised: N EURAL E PISODIC C ONTROL

Original: Episodic control (Lengyel & Dayan, 2007) was creatively applied on model-free DRL tasks to retrieve episodic memory-based state values (Blundell et al., 2016) for resolving sample inefficiency.
Revised: Episodic control (Lengyel & Dayan, 2007) was creatively applied on model-free DRL tasks to retrieve episodic memory-based state values (Blundell et al., 2016) for resolving sample inefficiency.

Original: The distance-based measurements were applied for looking up similar episodic data (Pritzel et al., 2017).
Revised: The distance-based measurements were applied for looking up similar episodic data (Pritzel et al., 2017).

Original: The episodic memory buffer can be smaller by applying Gaussian Random Projection to reduce the dimension of concrete states (Lin et al., 2018).
Revised: The episodic memory buffer can be smaller by applying Gaussian Random Projection to reduce the dimension of concrete states (Lin et al., 2018).

Original: The retrieved state values from episodic memory can be combined with the predictions of the critic network (Hansen et al., 2018).
Revised: The retrieved state values from episodic memory can be combined with the predictions of the critic network (Hansen et al., 2018).

Original: Novel episodic memory structure such as associative memory and generalized memory efficiently propagates state values to stored memory items (Zhu et al., 2020;
Revised: Novel episodic memory structure such as associative memory and generalized memory efficiently propagates state values to stored memory items (Zhu et al., 2020;

Original: Hu et al., 2021).
Revised: Hu et al., 2021).

Original: Episodic control was also combined with multi-agent tasks (Zheng et al., 2021) with curiosity-based exploration and modelbased reinforcement learning (Le et al., 2021).
Revised: Episodic control was also combined with multi-agent tasks (Zheng et al., 2021a) with curiosity-based exploration and model-based reinforcement learning (Le et al., 2021).

Original: Recently, the episodic control was also applied to the continuous control (Zhang et al., 2019; Kuznetsov & Filchenkov, 2021), which outperformed the state-of-the-art DRL algorithms.
Revised: Recently, the episodic control was also applied to the continuous control (Zhang et al., 2019; Kuznetsov & Filchenkov, 2021), which outperformed the state-of-the-art DRL algorithms.

Original: The universal episodic control (Hu et al., 2021; Pinto, 2020) was proved effective on both discrete and concrete action space by adopting a generalized episodic memory ( e.g. , neural network) to fit the past state values.
Revised: The universal episodic control (Hu et al., 2021; Pinto, 2020) was proved effective on both discrete and continuous action space by adopting a generalized episodic memory ( e.g. , neural network) to fit the past state values.

Original: Offline tasks can be solved by the state value-based episodic buffer and avoid the over-generalization of actions in the dataset (Ma et al., 2021).
Revised: Offline tasks can be solved by the state value-based episodic buffer and avoid the over-generalization of actions in the dataset (Ma et al., 2021).

Original: Episodic memory was also used to search the optimal hyperparameter for policy gradient methods (Le et al., 2022).
Revised: Episodic memory was also used to search the optimal hyperparameter for policy gradient methods (Le et al., 2022).

Original: This paper proposes  a novel and simple episodic control, NECSA, which adopts a grid-based state abstraction instead of distance-based measurement to store and retrieve episodic memory more efficiently.
Revised: This paper proposes NECSA, a novel and simple episodic control-based approach. NECSA adopts a grid-based state abstraction instead of distance-based measurement to operate episodic memory more efficiently.

Original: Moreover, unlike previous works, our episodic memory is more comprehensive since the state abstraction strategy enables us to explore abstract state transitions and topological information, which can be used to improve performance (Yin & Li, 2020).
Revised: Moreover, unlike previous works, our episodic memory is more comprehensive since the state abstraction strategy enables us to explore abstract state transitions and topological information. Such information can be used to improve the sample efficiency (Yin & Li, 2020).

Original: 2. S TATE A BSTRACTION
Revised: 2. S TATE A BSTRACTION

Original: The purpose of state abstraction is to groupthose states which share the same characteristics into a single cluster.
Revised: The purpose of state abstraction is to group states which share the same characteristics into a single cluster.

Original: One solution is discretizing the continuous domains (Dougherty et al., 1995).
Revised: One solution is discretizing the continuous domains (Dougherty et al., 1995).

Original: In model-based RL tasks (Kaiser et al., 2019), the state abstraction (Jiang et al., 2015; Burda et al., 2018) plays an essential role in reducing the data scales, although such abstraction requires strong domain knowledge of the environment or assumptions.
Revised: In modelbased RL tasks (Kaiser et al., 2019), the state abstraction (Jiang et al., 2015; Burda et al., 2018) plays an essential role in reducing the data scales, although such abstraction requires strong domain knowledge of the environment or assumptions.

Original: In model-free reinforcement learning, the grid-based state abstraction has proved effective (Anderson & Crawford-Hines, 1994; Grze ´ s & Kudenko, 2008).
Revised: In model-free reinforcement learning, the grid-based state abstraction has proved effective (Anderson & Crawford-Hines, 1994; Grze´s & Kudenko, 2008).

Original: Particularly in continuous control, the continuous action space can be divided into equal intervals, then those concrete actions in the same value scales can be labeled by a common abstraction (Pazis & Lagoudakis, 2009; Tang & Agrawal, 2020).
Revised: Particularly in continuous control, the continuous action space can be divided into equal intervals, then those concrete actions in the same value scales can be labeled by a common abstraction (Pazis & Lagoudakis, 2009; Tang & Agrawal, 2020; Du et al., 2019; Xie et al., 2019; Zhu et al., 2021).

Original: Another work of state abstraction in reinforcement learning is DreamerV2 (Hafner et al., 2020) which encodes the observations directly as discrete state variables.
Revised: Another work of state abstraction in reinforcement learning is DreamerV2 (Hafner et al., 2020) which encodes the observations directly as discrete state variables.

Original: The idea of smoothing the state space (Gangwani et al., 2020) shares similarities with our work exploring topological information on state transitions.
Revised: The idea of smoothing the state space (Gangwani et al., 2020) shares similarities with our work exploring topological information on state transitions.

Original: Dynamic state clustering (Mannor et al., 2004) is proved to be effective, although it is hard to be applied to the multi-step analysis.
Revised: Dynamic state clustering (Mannor et al., 2004) is proved to be effective, although it is hard to be applied to the multi-step analysis.

Original: Moreover, previous works partially focus on state or action space, and few works have been done on abstracting state-action pairs.
Revised: Moreover, previous works partially focus on state or action space, and few works have been done on abstracting state-action pairs.

Original: Inspired by the above works, we use the grid-based abstraction to group the state-action pairs simultaneously.
Revised: Inspired by the above works, we use the grid-based abstraction to group the state-action pairs simultaneously.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Generally, the process of reinforcement learning (RL) can be defined as a Markov Decision Process (MDP) (Sutton & Barto, 2018).
Revised: Generally, the process of reinforcement learning (RL) can be defined as a Markov Decision Process (MDP) (Sutton & Barto, 2018).

Original: MDP is a four-tuple ⟨ S, A, R, P ⟩ , where S and A represent the sets of states and actions respectively.
Revised: MDP is a four-tuple ⟨ S, A, R, P ⟩ , where S and A represent the sets of states and actions respectively.

Original: The agent interacts with the environment at each time step t by observing the current state s t ∈ S , choosing an action a t ∈
Revised: The agent interacts with the environment at each time step t by observing the current state s t ∈ S , choosing an action a t ∈

Original: A to execute, receiving an immediate  reward r t =
Revised: A to execute, receiving an immediate [Figure] reward r t =

Original: R ( s t , a t ) after doing a t , and transferring to a new state s t +1 ∼ P ( s t , a t ) .
Revised: R ( s t , a t ) after doing a t , and transferring to a new state s t +1 ∼ P ( s t , a t ) .

Original: The R ( · ) and
Revised: The R ( · ) and

Original: P ( · ) are the reward function and state transition function respectively.
Revised: P ( · ) are the reward function and state transition function respectively.

Original: The agent selects an action a ∼ π ( s ) to execute according to a policy π ( · ) and interacts with the environment to generate a trace, [( s 0 , a 0 , r 0 ) , ..., ( s t , a t , r t ) , ... ] , where the subscripts denote different[Figure] time steps.
Revised: The agent selects an action a ∼ π ( s ) to execute according to a policy π ( · ) and interacts with the environment to generate a trace, [( s 0 , a 0 , r 0 ) , ..., ( s t , a t , r t ) , ... ] , where the subscripts denote different time steps.

Original: The return of each trace is defined by (cid:80)
Revised: The return of each trace is defined by (cid:80)

Original: Tt =

Revised: Tt =

Original: γ t r t , where rewards are discounted by a factor γ ∈ [0 , 1) .
Revised: γ t r t , where rewards are discounted by a factor γ ∈ [0 , 1) .

Original: RL generally aims to find an optimal policy to maximize the returns.
Revised: RL generally aims to find an optimal policy to maximize the returns.

Original: The state-action value function: Q π ( s t , a t ) =
Revised: The state-action value function: Q π ( s t , a t ) =

Original: E π (cid:2) (cid:80) T − 1 t =

Revised: E π (cid:2) (cid:80) T − 1 t =

Original: γ t r t | s t , a t (cid:3) is widely used in many RL algorithms.
Revised: γ t r t | s t , a t (cid:3) is widely used in many RL algorithms.

Original: In value-based approaches (e.g., DQN (Mnih et al., 2013)), action with maximal
Revised: In value-based approaches (e.g., DQN (Mnih et al., 2013)), action with maximal

Original: Q π ( s t , a t ) value will be selected, and these approaches have achieved great success in discrete action space environments.
Revised: Q π ( s t , a t ) value will be selected, and these approaches have achieved great success in discrete action space environments.

Original: The policy-based approaches such as A3C (Mnih et al., 2016) are more efficient and suitable for continuous action spaces than value-based approaches, which generate a policy distribution and sample actions from it instead of iterating in the infinite action space.
Revised: The policy-based approaches such as A3C (Mnih et al., 2016) are more efficient and suitable for continuous action spaces than value-based approaches, which generate a policy distribution and sample actions from it instead of iterating in the infinite action space.

Original: DDPG (Lillicrap et al., 2016) is used in a continuous action setting to improve the sample efficiency over the vanilla actor-critic, in which the deterministic means the Actor can directly output the actions instead of computing a probability distribution over actions.
Revised: DDPG (Lillicrap et al., 2016) is used in a continuous action setting to improve the sample efficiency over the vanilla actor-critic, in which the deterministic means the Actor can directly output the actions instead of computing a probability distribution over actions.

Original: Twin Delayed DDPG (TD3) (Fujimoto et al., 2018) tackles the over-estimation of state values.
Revised: Twin Delayed DDPG (TD3) (Fujimoto et al., 2018) tackles the over-estimation of state values.

Original: In this work, we build our episodic control in continuous action space based on TD3 (Fujimoto et al., 2018).
Revised: In this work, we build our episodic control in continuous action space based on TD3 (Fujimoto et al., 2018).

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: This paper proposes an effective and efficient episodic control framework, NECSA.
Revised: This paper proposes an effective and efficient episodic control framework, NECSA.

Original: Figure 1 shows the main procedure of our framework.
Revised: Figure 1 shows the main procedure of our framework.

Original: We have two additional modules: (1) a grid-based abstraction to convert the concrete states to abstract ID ( i.e. , the ˆ s i as noted later); (2) a key-value-manner episodic memory module to store the abstract states with the state measurement ( i.e. , the reward confidence scores c i as mentioned later).
Revised: We have two additional modules: (1) a grid-based abstraction to convert the concrete states to abstract ID ( i.e. , the ˆ s i as noted later); (2) a key-value-manner episodic memory module to store the abstract states with the state measurement ( i.e. , the reward confidence scores c i as mentioned later).

Original: The traces are revised based on the state measurement and further flow to the replay buffer for policy optimization.
Revised: The traces are revised based on the state measurement and further flow to the replay buffer for policy optimization.

Original: NECSA is highly supplementary, which can be applied to the general reinforcement learning paradigm.
Revised: NECSA is highly supplementary, which can be applied to the general reinforcement learning paradigm.

Original: 4. G RID - BASED S TATE A BSTRACTION
Revised: 4. G RID - BASED S TATE A BSTRACTION

Original: The K -dimensional state space R K contains infinite concrete states, each state s i is represented as ( s 0 i , ..., s Ki ) .
Revised: The K -dimensional state space R K contains infinite concrete states, each state s i is represented as ( s 0 i , ..., s Ki ) .

Original: Each component s mi in the m -th dimension satisfies s mi ∈ [ l m , u m ] , where the l m and u m are the lower and upper boundaries, respectively.
Revised: Each component s mi in the m -th dimension satisfies s mi ∈ [ l m , u m ] , where the l m and u m are the lower and upper boundaries, respectively.

Original: We split each dimension [ l m , u m ] into N equal intervals.
Revised: We split each dimension [ l m , u m ] into N equal intervals.

Original: Thus the concrete state space R K will be divided into N K grids as follows:
Revised: Thus the concrete state space R K will be divided into N K grids as follows:

Original: [Equation]
Revised: [Equation]

Original: where e mn represents the n -th interval on the m -th dimension in the state space.
Revised: where e mn represents the n -th interval on the m -th dimension in the state space.

Original: In this way, (1) each concrete will fall into a grid; (2) all the concrete states s i which fall into the same grid can be identified as in the same cluster; (3) each cluster ( i.e. each grid) can be represented by a unique abstract state ˆ s :
Revised: In this way, (1) each concrete will fall into a grid; (2) all the concrete states s i which fall into the same grid can be identified as in the same cluster; (3) each cluster ( i.e. each grid) can be represented by a unique abstract state ˆ s :

Original: [Equation]
Revised: [Equation]

Original: Figure 2 is a schematic diagram of grid-based clustering, in which the infinite concrete state space is converted into the finite discrete state space.
Revised: Figure 2 is a schematic diagram of grid-based clustering, in which the infinite concrete state space is converted into the finite discrete state space.

Original: The environment determines the l m and u m .
Revised: The environment determines the l m and u m .

Original: It is worth mentioning that we can either perform the abstraction on concrete states or state-action pairs.
Revised: It is worth mentioning that we can either perform the abstraction on concrete states or state-action pairs.

Original: As the experimental results suggest, abstracting state-action pairs can perform better than focusing on concrete states.
Revised: As the experimental results suggest, abstracting state-action pairs can perform better than focusing on concrete states.

Revised: Take Walker2d-v3 as an example; the state vector’s shape is (1,17), and the action vector is (1,6).

Revised: Then, we combine the state and action vectors into a state-action pair with a shape of (1,23).

Revised: The boundaries of the state and action vectors are [-10,10] and [-1,1], respectively.

Revised: For instance, we obtain the fixed lower bounds as (-10, -10, ..., -10, -1, -1, ..., -1).

Original: We use a  random projection  to reduce the dimension of the concrete state-action pairs and make our method more scalable for high-dimensional tasks.
Revised: finally, we abstract the state-action pairs by grid-based clustering. Each dimension of a state-action pair is split into equal intervals. In general, the high-dimensional concrete state (-action pairs) would increase the difficulty and complexity of state abstraction. Inspired by Gaussian random projection (Dasgupta, 2013), we apply a random matrix to reduce the dimensions of state (-action) vectors.

Revised: When creating the matrix, we ensure that the values of the matrix elements follow the Gaussian distribution and that the value scope is [-1,1].

Revised: By multiplying the state (-action) vector ( e.g. , (1,376) in Humanoid-v3) to a Gaussian random matrix with a shape of (376,24), we can obtain a smaller state (-action) representation vector as (1,24).

Revised: The Gaussian random matrix is initialized at the beginning of the training.

Original: 4. E PISODIC M EMORY WITH M ULTI - STEP A NALYSIS AND S TATE M EASUREMENT
Revised: Then, it would remain the same during the training. Finally, all the state (-action) vectors are projected to a smaller one by a common Gaussian random matrix. Note that abstracting images directly can be inefficient in Atari games. Pixel-based image vectors consist of multiple channels, which can be unsuitable for abstraction using grid-based methods.

Revised: Therefore, we use the hidden outputs for abstraction, considering that (1) the hidden outputs are natural features of images but relatively smaller than the raw images, and (2) hidden outputs reflect the states of actions.

Revised: Moreover, incorporating hidden outputs with episodic control is also effective in previous episodic control approaches such as EVA (Hansen et al., 2018).

Revised: 4. E PISODIC M EMORY WITH M ULTI - STEP A NALYSIS AND S TATE M EASUREMENT

Original: Figure 3 shows the logic of extending the state abstraction to multi-step patterns.
Revised: Figure 3 shows the logic of extending the state abstraction to multi-step patterns.

Original: The abstract patterns are extracted by sliding a N -step window on each trace.
Revised: The abstract patterns are extracted by sliding a N -step window on each trace.

Original: We treat N -step continuous abstract states as a whole pattern.
Revised: We treat N -step continuous abstract states as a whole pattern.

Original: Moreover, we count the patterns iteratively and orderly, where { ˆ s i , ˆ
Revised: Moreover, we count the patterns iteratively and orderly, where { ˆ s i , ˆ

Original: s i +1 ,
Revised: s i +1 ,

Original: ˆ s i +2 } and { ˆ s i +1 , ˆ s i +2 , ˆ s i +3 } are treated as different patterns.
Revised: ˆ s i +2 } and { ˆ s i +1 , ˆ s i +2 , ˆ s i +3 } are treated as different patterns.

Original: To revise the state (action) values ( i.e. Q ) estimation, previous episodic control-based approaches usually leverage a shift of episodic returns to correct the critic Q-value.
Revised: To revise the state (action) values ( i.e. Q ) estimation, previous episodic control-based approaches usually leverage a shift of episodic returns to correct the critic Q-value.

Original: Such approaches can achieve better performance while requiring domain-specific knowledge of corresponding DRL algorithms, model structure reform, and engineering work efforts.
Revised: Such approaches can achieve better performance while requiring domain-specific knowledge of corresponding DRL algorithms, model structure reform, and engineering work efforts.

Original: Furthermore, measuring states by the statistics of Q-values is only applicable in one-step episodic memory.
Revised: Furthermore, measuring states by the statistics of Q-values is only applicable in one-step episodic memory.

Original: With multi-step patterns, we propose a state measurement named reward confidence scores.
Revised: With multi-step patterns, we propose a state measurement named reward confidence scores.

Original: The score c i represents the expectation of the episode return when ˆ s i occurs.
Revised: The score c i represents the expectation of the episode return when ˆ s i occurs.

Original: Although the scores are similar to state (-action) values in terms of the definitions, they stand for different semantics.
Revised: Although the scores are similar to state (-action) values in terms of the definitions, they stand for different semantics.

Original: State (action) values represent the forwarding rewards after the state ˆ s i , but the scores measure the episode reward over the whole trace.
Revised: State (action) values represent the forwarding rewards after the state ˆ s i , but the scores measure the episode reward over the whole trace.

Original: We design a simple key-value-manner episodic memory C based on state abstraction and measurement.
Revised: We design a simple key-value-manner episodic memory C based on state abstraction and measurement.

Original: Each item in the episodic memory is indexed by the ˆ
Revised: Each item in the episodic memory is indexed by the ˆ

Original: s i with three values: the current score
Revised: s i with three values: the current score

Original: c i , the historical total episodic rewards E ithat earned by the traces which contain the ˆ s i , and the total number of occurrences η i of ˆ s i .
Revised: c i , the historical total episodic rewards E i earned by the traces which contain the ˆ s i , and the total number of occurrences η i of ˆ s i .

Original: The following equation is the implementation of computing scores:
Revised: The following equation is the implementation of computing scores:

Original: [Equation]
Revised: [Equation]

Original: Where the (cid:80) r is the final episodic return of the trace, which contains ˆ s i , in this way, our memory buffer can efficiently record past experiences and update the scores.
Revised: Where the (cid:80) r is the final episodic return of the trace, which contains ˆ s i . In this way, our memory buffer can efficiently record past experiences and update the scores.

Original: Our episodic memory enables three operations: ADD , LOOKUP  and UPDATE .
Revised: Our episodic memory enables three operations: ADD , LOOKUP , and UPDATE .

Original: ADD is the operation to store the ˆ s i into C .
Revised: ADD is the operation to store the ˆ s i into C .

Original: LOOKUP returns score c i for ˆ s i by taking O (1) .
Revised: LOOKUP returns score c i for ˆ s i by taking O (1) .

Original: UPDATE is to revise the c i of ˆ s i .
Revised: UPDATE is to revise the c i of ˆ s i .

Original: Moreover, the episodic memory enables the normalizing of the scores to [0,1], and the average of the score c i is adopted as the intrinsic reward for abstract pattern ˆ s i . 
Revised: Moreover, the episodic memory enables the normalizing of the scores to [0,1], and the average of the score c i is adopted as the intrinsic reward for abstract pattern ˆ s i . Algorithm 1 NECSA. : ˆ s i ← State _

Revised: Abstract ( s i , a i ) 6: ˆ S.append (ˆ s i ) 7: c i ← Inquire _ Score (ˆ s i , C ) 8: ˆ r i ← Reward _ Revision ( r i , c i ) 9:

Revised: R.append ( r i ) 10:

Original: 4. E PISODIC C ONTROL -B ASED L EARNING
Revised: B .add ( s i , a i , ˆ r i ) 11: if d i is true then 12: C ← Update _ Score ( ˆ S, R, C ) 13: S, ˆ S, A, R, D ← ∅ 14: end if 15: i ← i +16:

Original:  As inspired by previous episodic control-based approaches, we infer that a statistic-based measurement of state-action pairs can be a helpful reference to revise the estimation of state (-action) values by the critic network (Lin et al., 2018; Kuznetsov & Filchenkov, 2021).Therefore, we use an intrinsic
Revised: M ← Train ( M , B .sample ()) 17: end while 4. E PISODIC C ONTROL -B ASED L EARNING As inspired by previous episodic control-based approaches, we infer that a statistic-based measurement of state-action pairs can be a helpful reference to revise the estimation of state (-action) values by the critic network (Lin et al., 2018; Kuznetsov & Filchenkov, 2021).

Original: Algorithm 1 NECSA.
Revised: The core of our method is to measure the abstract patterns by scores and make the agent generalize the high-score experiences. Previous work like reward shaping (Harutyunyan et al., 2015; Laud & DeJong, 2003; Devlin &

Original: reward (Burda et al., 2018) to revise the policy.
Revised: Kudenko, 2012; Burda et al., 2018) helps the policy incorporate domain knowledge into policy optimization.

Revised: Therefore, it is natural to reshape the reward with a score-based intrinsic reward to revise the policy.

Original: Formally, the revised reward
Revised: Formally, the revised reward

Original: ˆ r i =
Revised: ˆ r i =

Original: r t + ∆ , where r t is the return of the original reward function, ∆ is the intrinsic reward (computed based on the reward confidence scores) which represents the semantics for improving the rewarding mechanism.
Revised: r t + ∆ , where r t is the return of the original reward function, ∆ is the intrinsic reward (computed based on the reward confidence scores) which represents the semantics for improving the rewarding mechanism.

Original: We perform reward revision for each pattern as follows:
Revised: We perform reward revision for each pattern as follows:

Original: [Equation]
Revised: [Equation]

Original: where the ˆ r i is the revised reward based on r i .
Revised: where the ˆ r i is the revised reward based on r i .

Original: M is the total number of abstract patterns.
Revised: M is the total number of abstract patterns.

Original: (cid:80)
Revised: (cid:80)

Original: Mm {+=
+}
Revised: Mm {+=
+}

Original: [Equation]

Original: c m M is the average of all the scores.
Revised: c m M is the average of all the scores.

Original: The term ∆ will be less than 

Revised: The term ∆ will be less than

Original: if the ˆ s i score is less than the average.
Revised: if the ˆ s i score is less than the average.

Original: Thus the corresponding concrete state will be punished.
Revised: Thus the corresponding concrete state will be punished.

Original: The state will be encouraged if ∆ is greater than 0.
Revised: The state will be encouraged if ∆ is greater than 0.

Original: The agent will earn more rewards if it passes through ˆ s i .
Revised: The agent will earn more rewards if it passes through ˆ s i .

Original: In a word, the lower the score c i is, the more the pattern is punished.
Revised: In a word, the lower the score c i is, the more the pattern is punished.

Original: ϵ is a hyperparameter that helps control the magnitude of punishment and encouragement.
Revised: ϵ is a hyperparameter that helps control the magnitude of punishment and encouragement.

Original: We implement NECSA as Algorithm 1.
Revised: We implement NECSA as Algorithm 1.

Original: We use a module named State_Abstract in line 5 which can covert the concrete states to abstract patterns in run-time.
Revised: We use a module named State_Abstract in line 5, which can covert the concrete states to abstract patterns in run-time.

Original: While tuning the model, scores can be inquired from the episodic memory C in line 7, and the reward will be revised in line 8.
Revised: While tuning the model, scores can be inquired from the episodic memory C in line 7, and the reward will be revised in line 8.

Original: We also added a module named Update_Score in line 12, which helps calculate the scores for the abstract patterns in ˆ S and update the scores in C as each termination of an episode.
Revised: We also added a module named Update_Score in line 12, which helps calculate the scores for the abstract patterns in ˆ S and update the scores in C as each termination of an episode.

Original: The revised rewards will be stored in the replay buffer B .
Revised: The revised rewards will be stored in the replay buffer B .

Original: Finally, the policy will be updated with the sampled data from the B in line 16.
Revised: Finally, the policy will be updated with the sampled data from the B in line 16.

Original:  E XPERIMENT5. E XPERIMENT S ETUP
Revised:  E XPERIMENT

Revised: 5. E XPERIMENT S ETUP

Original: We conduct the experiments on nine MuJoCo environments in OpenAI gym
Revised: We conduct the experiments on nine MuJoCo tasks and six Atari games in OpenAI gym

Original: (Brockman et al., 2016) domains, including Walker2d-v3, Hopper-v3, Humanoid-v3, Ant-v3, HalfCheetah-v3, Swimmer-v3, Reacher-v2, InvertedPendulum-v2, and InvertedDoublePendulum-v2.
Revised: (Brockman et al., 2016) domains.

Original: The selected environments are typical continuous control tasks.
Revised: For continuous control tasks in MuJoCo, we select four baselines including

Original: We select four baselines include EMAC (Kuznetsov & Filchenkov, 2021), GEM (Hu et al., 2021) 
Revised: EMAC (Kuznetsov & Filchenkov, 2021), GEM (Hu et al., 2021) DDPG (Lillicrap et al., 2016) and

Original: DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) forthe comparison. 
Revised: TD3 (Fujimoto et al., 2018) for comparison. We also compare NECSA to four baselines, including

Revised: DQN (Mnih et al., 2013), Rainbow (Hessel et al., 2018), EVA (Hansen et al., 2018) and GEM (Hu et al., 2021) on Atari games, which use image-based states and discrete action spaces.

Original: EMAC and GEM are state-of-the-art episodic control approaches and proved effective on continuous control tasks.
Revised: EMAC is a state-of-the-art episodic control approach that proved effective on continuous control tasks.

Original: EMAC retrieves experiences from a non-parametric episodic memory to minimize the gap between the estimation and statistics of state (-action) values via a novel critic loss.
Revised: EVA is effective on discrete action spaces by performing ephemeral adjustments of Q-values to impact the parametric value functions.

Original: GEM integrates a neural network to generalize episodic experiences, enabling fast episodic memory retrieval.
Revised: GEM integrates a neural network to generalize episodic experiences, enabling fast episodic memory retrieval.

Original: We adopt the benchmark of DDPG and TD3 in Tianshou (Weng et al., 2021).

Original: The implementation of NECSA is based on TD3.

Original: We evaluate our approach on InvertedPendulum-v2 and InvertedDoublePendulum-v2 for 50,000 time steps, Reacher-v2 for 100,000 steps, Swimmer-v3 for 200,000 steps, Walker2d-v3 and Hopper-v for 400,000 steps, Humanoid-v3, Ant-v3 and HalfCheetah-v3 for 1,000,000 steps, respectively.

Original: We run each experiment with five different random seeds to counteract the randomness and compare the average performance over respective training steps. It takes 4 to 8 GPU hours to finish five seeds on each task. All the experiments were run on the powerful servers with CPU (Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz), GPU(NVIDIA Corporation GA102GL [RTX A6000]) with 128GB RAM.
Revised: GEM can be applied to both continuous and discrete action spaces.

Original: 5. E VALUATION R ESULTS
Revised: 5. E VALUATION R ESULTS

Original: Figure 4 shows the main evaluation  of our approaches.
Revised: Figure 4 shows the main evaluation results of each approach on MuJoCo and Atari tasks.

Revised: Note that more evaluation results are in the Appendix A.2.

Original: In our experiments, we set the length of the multi-step abstract pattern m =3.
Revised: In our experiments, we set the length of the multi-step abstract pattern m =3.

Original: As shown in Figure 4, our approach NECSA significantly outperforms the baselines on six tasks, including Walker2d-v3, Hopper-v3, Humanoid-v3 and
Revised: As shown in Figure 4, our approach NECSA significantly outperforms the baselines on all tasks.

Original: Ant-v3, HalfCheetah-v3, and Swimmer-v3, in terms of sample efficiency.

Original: Furthermore, NECSA reaches (slightly outperforms) state-of-the-art performance in Reacher-v2, InvertedPendulum-v and InvertedDoublePendulum-v2.

Original: Table 1 is the average return of the learned policies over the respective training steps. The bold entries in Table 1 mark the best average returns of each method.
Revised: Table 3 and Table 4 in the Appendix are the average returns and standard deviations of the learned policies over multiple random seeds.

Original: NECSA achieves the best results on all tasks in the limited training steps. Overall, we found that (1) NECSA is the most sample efficient algorithm on all tasks while other algorithms only establish good performance on the part of tasks; (2) NECSA performs better on relatively complex tasks ( e.g. , high-dimensional tasks).
Revised: Overall, we found that (1) NECSA is the most sample-efficient algorithm on various tasks, while other algorithms only establish good performance on the part of tasks; (2) NECSA performs better on relatively complex tasks ( e.g. , high-dimensional states and image-based states).

Original: Such results prove that NECSA is a scalable episodic control on continuous  control tasks.
Revised: Such results prove that NECSA is a scalable episodic control on continuous and discrete control tasks.

Original: Moreover, EMAC and GEM struggle on Humanoid-v3 (a relatively complex task in MuJoCo since the observation space  is large), while NECSA can perform better than all the baselines on this task.
Revised: Take Humanoid-v3 as an example, a 376-dimensions state space with five grids on each dimension generates a 5 376 abstract state space, which is impossible to efficiently store and retrive in the episodic memory.

Original: Note
Revised: On Atari games, we take the hidden outputs after the convolutional neural network layers of the policy network to represent the image states (-action pairs).

Original: [Table] InvertedDoublePendulum- [Table] that in Humanoid-v3, a 376-dimensions state space with five grids on each dimension will generate a 5 376 abstract state space, which is almost impossible for efficient storage and retrieving in the episodic memory.
Revised: However, the dimension of hidden outputs is also unsuitable for grid-based abstraction.

Original: Gaussian random projection can be a promising solution to make the concrete states  smaller (Pritzel et al., 2017; Kuznetsov & Filchenkov, 2021).
Revised: Gaussian random projection can be a promising solution to make the concrete states (or hidden outputs) smaller (Pritzel et al., 2017; Kuznetsov & Filchenkov, 2021).

Original: Therefore, we perform a dimension reduction of the concrete state vectors to 1 × 25 by a random projection and divide each dimension into eight intervals.
Revised: Therefore, we perform a dimension reduction of the concrete state vectors to 1 × 24 by a random projection and divide each dimension into fixed intervals.

Original: The results show that the random projection reduces dimension without significant information loss, thus making NECSA a scalable episodic control on high-dimensional  tasks. 
Revised: The results show that the random projection reduces dimension without significant side effects, thus making NECSA a scalable episodic control on high-dimensional continuous control and image-states tasks. The experimental results on MuJoCo tasks and Atari games prove that NECSA is effective on various DRL tasks. We performed an in-depth ablation studies on the critical parts of NECSA in the following section.

Revised: Besides, we have conducted detailed hyperparameter analyses.

Revised: Note that the hyperparameter analyses are in Appendix A.3.

Original:  5. A BLATION S TUDY
Revised: We also proved that NECSA is effective on continuous control tasks with image-based inputs. The related experiments are conducted on DMControl (Tunyasuvunakool et al., 2020) tasks. Please refer to Appendix A.4 for details. 5. A BLATION S TUDY

Original: One of the most critical parts of this paper is to adopt a multi-step analysis of past experiences.
Revised: One of the most critical parts of this paper is to adopt a multi-step analysis of past experiences.

Original: To better demonstrate the effectiveness of the multi-step episodic control, we compare the performance with one-step and two-step settings.
Revised: To better demonstrate the effectiveness of the multi-step episodic control, we compare the performance between different multi-step settings.

Original: The results in Figure 5 show that the three-step NECSA outperforms one-step and two-step settings.
Revised: The results in Figure 5 show that the three-step NECSA outperforms one-step and two-step settings.

Original: Another observation is that the two-step NECSA usually establishes better sample efficiency than the one-step settings.

Original: Such results prove that multi-step analysis of past traces can significantly improve performance on various tasks.

Original: However, longer steps are not always better.

Original: In our evaluation, the default setting of NECSA multi-step ( m ) analysis is set to 3.
Revised: In our evaluation, the default setting of NECSA multistep ( m ) analysis is set to 3.

Original: The reason is that longer patterns cause the exact matches to become exceedingly rare, which makes episodic control less helpful for improving performance.
Revised: The reason is that longer patterns cause the exact matches to become exceedingly rare, which makes episodic control less helpful for improving performance.

Original: We also study the effectiveness of reward confidence scores.
Revised: We also study the effectiveness of reward confidence scores.

Original: We use scores to measure each stateaction pair, although previous works mainly focus on the state-action values ( i.e. , the Q-values).
Revised: We use scores to measure each stateaction pair, although previous works mainly focus on the state-action values ( i.e. , the Q-values).

Original: Specifically, we use the average of scores and Q-values (Lin et al., 2018; Kuznetsov & Filchenkov, 2021) for each abstract state (pattern) to compute the intrinsic reward.
Revised: Specifically, we use the average of scores and Q-values (Lin et al., 2018; Kuznetsov & Filchenkov, 2021) for each abstract state (pattern) to compute the intrinsic reward.

Original: Note that we cannot directly attain the Q-value of a multi-step abstract pattern.
Revised: Note that we cannot directly attain the Q-value of a multi-step abstract pattern.

Original: We instead use the average Q value of the corresponding concrete states (Lin et al., 2018).
Revised: We instead use the average Q value of the corresponding concrete states (Lin et al., 2018).

Original: The results in Figure 6 show that measuring the state-action pairs through scores (NECSA_Scores) can lead to better sample efficiency than using
Revised: The results in Figure 6 show that measuring the state-action pairs through scores (NECSA_Scores) can lead to better sample efficiency than using

Original: Q-values (NECS_Q-values).
Revised: Q-values (NECSA_Q-values).

Original: We claim that such state measurement is practical and necessary since the scores contain the overall impact of an abstract on a whole trace.
Revised: We claim that such state measurement is effective and necessary since the scores contain the overall impact of an abstract on a whole trace.

Original: At the same time, Q-values are the expectations of forwarding rewards.

Original: Furthermore, such a difference makes the scores more comprehensive since we reference the historical impact on the abstract states.
Revised: Furthermore, such a difference makes the scores more comprehensive since we reference the historical impact on the abstract states.

Original: We conclude that NECSA can perform better than the state-of-the-art episodic control-based approaches based on the experiments.
Revised: We conclude that NECSA performs better than the state-of-the-art episodic control-based approaches based on the experiments.

Original: State abstraction-based episodic memory enables more advanced analysis of past experiences.
Revised: State abstraction-based episodic memory enables more advanced analysis of past experiences.

Original: The scores of state abstractions are valuable, and the rationale for state measurement and the multi-step analysis of state transitions canhelp significantly improve the performance.
Revised: The scores of state abstractions are valuable, and the rationale for state measurement and the multi-step analysis of state transitions can significantly improve the performance.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We adopt grid-based abstraction instead of kNN .
Revised: We adopt grid-based abstraction instead of kNN .

Original: Although kNN can group concrete states into unique
Revised: Although kNN can group concrete states into unique

Original: IDs, it cannotsatisfactorily update episodic memory in real time.
Revised: IDs, it cannot update episodic memory in real-time.

Original: For instance, if we perform the clustering by kNN in 10,000 steps, then in the 11,000 steps, the clustering in 10,000 steps may lose generality and accuracy.

Original: However, performing new clustering costs extra computation time.

Original: Gridbased state abstraction can avoid this since it can handle any newcomer concrete states without altering the previous clustering.

Original: Moreover, kNN does not support inquiries about the state measurement of multi-step patterns.
Revised: Moreover, kNN does not support inquiries about the state measurement of multi-step patterns.

Original: We extend the episodic control to multi-step analysis, which is one of the most important contributions of NECSA.
Revised: We extend the episodic control to multi-step analysis, which is one of the most important contributions of NECSA.

Original: Overall, we select a grid-based abstraction (clustering) instead of kNN for (1) more efficient state abstraction and (2) multi-step abstract pattern analysis.
Revised: Overall, we select a grid-based abstraction (clustering) instead of kNN for (1) more efficient state abstraction and (2) multi-step abstract pattern analysis.

Original: Although NECSA establishes promising sample efficiency, it still has much room for improvement.
Revised: Although NECSA is sample efficient, there still exists room for improvement.

Original: First of all, the state abstraction strategy might cause variances or errors.
Revised: First of all, the state abstraction strategy might cause variances or errors.

Original: For example, we divide each dimension by the same number of intervals, though we can not ensure that the value scale of each dimension is  the same.
Revised: For example, we divide each dimension by the same number of intervals, though sometimes the value scale of each dimension is not the same.

Original: Therefore, someexisting concrete states might not be labeled by abstract ID .
Revised: Therefore, some concrete states might not be labeled by abstract ID .

Original: Based on prior knowledge of the states’ space, we infer that a possible solution may be to  divide each dimension into appropriate but not fixed intervals automatically.
Revised: Based on prior knowledge of the state space, we infer that a possible solution may be to automatically divide each dimension into appropriate but not fixed intervals.

Original: The motivation behind NECSA is to construct a more comprehensive episodic memory.
Revised: The motivation behind NECSA is to construct a more comprehensive episodic memory.

Original: We consider that past experiences need to be further explored, and more valuable information will be found and exploited.
Revised: We consider that past experiences need to be further explored, and more valuable information could be obtained and exploited.

Original: Furthermore, exploring latent semantics based on the abstraction-based episodic memory promises to improve the sample efficiency.
Revised: Furthermore, exploring latent semantics based on the abstraction-based episodic memory promises to improve the sample efficiency.

Original: In addition, NECSA is highly supplementary since it does not require reforming the models or loss functions. Therefore, we believe that NECSA can help improve the performance of more tasks.
Revised: In addition, NECSA is highly scalable and can be applied to various DRL algorithms and tasks.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We propose NECSA, a novel state abstraction-based neural episodic control approach to improve
Revised: We propose NECSA, a novel state abstraction-based neural episodic control approach to improve

Original: DRL’s sample efficiency.
Revised: DRL’s sample efficiency.

Original: We transform dense concrete state-action pairs into abstract patterns, which enables efficient storage and retrieval of episodic data.
Revised: We transform dense concrete state-action pairs into abstract patterns, which enables efficient storage and retrieval of episodic data.

Original: We also perform a multi-step analysis of the pattern transitions to enhance their performance.
Revised: We also perform a multi-step analysis of the pattern transitions to enhance their performance.

Original: Furthermore, we propose a novel state measurement metric, reward confidence scores, which can be incorporated with extrinsic rewards to accelerate policy optimization.
Revised: Furthermore, we propose a novel state measurement metric, reward confidence scores, which can be incorporated with extrinsic rewards to accelerate policy optimization.

Original: The evaluation results prove that our approach is more sample efficient than state-of-the-art episodic control approaches.
Revised: The evaluation results prove that our approach is more sample efficient than state-of-the-art episodic control approaches.

Original: We believe that more advanced research can be conducted based on the state abstraction, state measurement, and multi-step analysis of state transitions, including (1) abstracting the behaviors of the policy via state abstraction since we can build an automaton base on the abstract states; (2) analyzing the state transitions and topological relationships of multi-agent tasks; (3) applying our approach to model-based reinforcement learning with on-the-fly planning.
Revised: We believe that more advanced research can be conducted based on the state abstraction, state measurement, and multi-step analysis of state transitions, including (1) abstracting the behaviors of the policy via state abstraction since we can build an automaton base on the abstraction; (2) analyzing the state transitions and topological relationships of multi-agent tasks (Zheng et al., 2018; 2021b); (3) applying our approach to model-based reinforcement learning with dynamic planning.

Original: We leave these topics for future works.
Revised: We leave these topics for future work.

Original: A A PPENDIX A. C OMMON H YPERPARAMETER AND S ETTINGS
Revised: A CKNOWLEDGMENTS

Revised: This work was supported in part by JSPS KAKENHI Grant No.JP19H04086 and No.JP20H04168, JST-Mirai Program Grant No. JPMJMI20B8, JST SPRING Grant

Revised: No.

Revised: JPMJSP2136, as well as Canada

Revised: CIFAR AI Chairs Program, the Natural Sciences and Engineering Research Council of Canada (NSERC No.

Revised: RGPIN-2021-02549, No.RGPAS-2021-00034,

Revised: No.DGECR-2021-00019)

Revised: , the National

Revised: Natural Science Foundation of China (Grant No.62106172), the “New Generation of Artificial Intelli- gence” Major Project of Science & Technology 2030 (Grant No.2022ZD0116402), and the Science and Technology on Information Systems Engineering Laboratory (Grant No.

Revised: WDZC20235250409,

Revised: No.WDZC20205250407).

Revised: A A PPENDIX A. C OMMON H YPERPARAMETER AND S ETTINGS

Original: Our hyperparameter settings are available in Table  2.
Revised: Our hyperparameter settings are listed in Table 1 and Table 2.

Original: In addition, we directly evaluate the program released by the authors of EMAC (Kuznetsov & Filchenkov, 2021) and
Revised: In addition, we directly evaluate the program released by the authors of EMAC (Kuznetsov & Filchenkov, 2021), EVA (Hansen et al., 2018)

Revised: [Table]

Original:  GEM (Hu et al., 2021).
Revised: and GEM (Hu et al., 2021).

Original: We also adopt the implementation of DDPG and TD3 in (Weng et al., 2021).
Revised: We also adopt the implementation of DDPG, TD3, DQN, and Rainbow in (Weng et al., 2021).

Original: We build NECSA based on TD3. The neural network consists of two hidden layers.
Revised: For MuJoCo tasks, the neural network consists of two hidden layers.

Original: The size of each layer is 256.
Revised: The size of each layer is 256.

Original: The activation unit is ReLU.
Revised: The activation unit is ReLU.

Original: Both the Actor and Critic networks share the same structure.
Revised: Both the Actor and Critic networks share the same structure.

Revised: For Atari tasks, we use a three-layer convolution neural network head and a fully connected layer to output the Q-values of each action.

Revised: The specific parameter values are listed in

Original: [Table]
Revised: Table 2.

Revised: More details are available in the code repository

Original:  A. A NALYSIS OF S TATE V ISITING D ENSITY
Revised: [Table] [Table] A. T HE RESULTS

Original:  We plot the density of state visiting times in Figure 7.
Revised: [Table] We adopt the implementations of DDPG, TD3, DQN, and Rainbow in Tianshou (Weng et al., 2021).

Original: Take Walker2d-v3 as an example.
Revised: The implementation of NECSA is based on TD3. We use the public replicate package of the respective baselines for experiments. We run each experiment with five different random seeds to counteract the randomness and compare the average performance over respective training steps. All the experiments were run on powerful servers with CPU (Intel(R) Core(TM) i9-10940X CPU @ 3.30GHz)

Original: The 1-order
Revised: [Table] and

Original:  NECSA generated 198,613 abstract states. 163,273 of them were visited one time, and 35,340 were visited more than one time.
Revised: GPU(NVIDIA Corporation GA102GL [RTX A6000]) with 128GB RAM. We have evaluated NECSA on nine MuJoCo tasks and six Atari games in total.

Original: All the 35,340 multi-times visited states were visited 236,727 times in 400,000 training steps, which is more than 1-time visit.
Revised: Table 3 and Tableare the performance and standard deviation of all the learned policies of the respective approaches. Figure 7 is the performance trends of all the approaches on different tasks ( i.e. , except the results we have reported in Figure 4). Based on the above results, we found that the learned policies of

Original: The m-order NECSA generated 377, abstract patterns.
Revised: NECSA achieve the highest performance.

Original: 372,253 of them were visited one time, and 5,376 of them were visited more than one time. All the 5,376 multi-times visited patterns were visited 27,747 times, which is more than the one -time visited pattern.
Revised: Most of the improvements against other approaches are significant. On the tasks such as InvertedPendulum-v2, InvertedDoublePendulum-v2, and Reacher-v2, NECSA reaches the same performance or slightly outperforms the baselines.

Original: The density plot shows that 1-time visits are more in multi-step analysis.
Revised: The reason is that such tasks are relatively easy than others.

Original: However, the performance in Figure 4 shows that multi-step analysis canstill achieve the best sample efficiency in our settings.
Revised: Therefore NECSA and other baseline approaches can achieve state-of-the-art performance in fewer training steps.

Original: Therefore, we assume that the multi-times visited patterns played an essential role in enhancing the sample efficiency.
Revised: Overall, the evaluation results demonstrate that NECSA is effective and sample-efficient on MuJoCo and Atari tasks. [Table]

Original: A. S PECIAL H YPERPARAMETERS AND S ETTINGS IN NECSA
Revised: A

Revised: S PECIAL H YPERPARAMETERS AND S ETTINGS IN NECSA

Original: During the experiments, we evaluated different settings of several hyperparameters, which can help us better understand the effectiveness of NECSA.
Revised: During the experiments, we evaluated different settings of several hyperparameters, which can help us better understand the effectiveness of NECSA.

Original: We discuss four settings: (1) Should we abstract the state or state-action pairs?
Revised: We discuss four settings and answer the following questions through detailed experiments: Should we abstract the state or state-action pairs (or the hidden outputs)?

Original: (2) What is the effect of discrete state space grid numbers on performance?
Revised: What is the effect of discrete state space grid numbers on performance?

Original: (3) How to control the magnitude of revising environment rewards via intrinsic reward ( i.e. , reward confidence scores)?
Revised: How to control the magnitude of revising environment rewards via intrinsic reward ( i.e. , reward confidence scores)?

Original: (4) How do the backbone algorithms affect the performance of NECSA?
Revised: How do the backbone algorithms affect the performance of NECSA?

Original: We answer the above questions through the following experiments.

Original: Figure 8 shows the performance of abstracting state and state-action pairs.
Revised: Figure 8 shows the performance of abstracting state and state-action pairs.

Original: NECSA_state is the curve that we only abstract the concrete states.
Revised: NECSA_state is the curve that we only abstract the concrete states (or the static features of the image states).

Original: On the other hand, NECSA_state-action means that we abstract state-action pairs.
Revised: On the other hand, NECSA_state-action means that we abstract state-action pairs.

Original: It reveals that abstracting the state-action pairs can significantly improve performance.
Revised: It reveals that abstracting the state-action pairs can significantly improve performance on MuJoCo tasks.

Original: The reason is that abstracting state-action pairs can involve more information for modeling the policy than just focusing on states.
Revised: The reason is that abstracting state-action pairs can involve more information for modeling the policy than just focusing on states.

Original: For example, state and state-action values in DRL can be directly converted to each other, and intuitively, the scores between states and state-action pairs should also be the same. However, we found that although a set of concrete states can be located in the same grid, their corresponding state-action pairs might be divided into different grids after abstracting the state-action space.
Revised: NECSA_hidden means that we use the hidden outputs of the policy network for abstracting. Using the hidden outputs on Atari games can also outperform the performance of just abstracting the states. The hidden outputs are natural features of images but relatively smaller than the raw images.

Original: Therefore variances of thescores between states and state-action pairs are amplified. Due to such variances, abstraction with state-action pairs can improve evaluation accuracy.
Revised: Moreover, the hidden outputs reflect the states of actions. Therefore, we infer that using hidden outputs for abstraction shares the same benefits as using state-action pairs.

Original: Figure 9 shows the performance under different abstraction granularities.
Revised: Figure 9 shows the performance under different abstraction granularities.

Original: Grid-N means we divide each dimension of state-action pairs into N equal intervals.
Revised: Grid-N means we divide each dimension of state-action pairs into N equal intervals.

Original: The results show that grid numbersand 10 are the best abstraction choices in Walker2d-v3 and Hopper-v3.
Revised: The results show that grid numbers 8 and 10 are the best abstraction choices in Walker2d-v3, Hopper-v3 and Alien.

Original: Our evaluation is limited to 10 grids since a greater grid number causes the exponential growth of the abstract number, which is very memory-consuming.
Revised: Our evaluation is limited to 10 grids since a greater grid number causes the exponential growth of the abstract number, which is very memory-consuming.

Original: We found that more grids can not always achieve better performance.
Revised: We found that more grids cannot always achieve better performance.

Original: For example, in Walker2d-v3, 8-grids abstraction outperforms 10-grids.
Revised: For example, in Walker2d-v3, 8-grids abstraction outperforms 10-grids.

Original: The reason is that although more grids can cluster similar concrete states more accurately, as the grid number of the abstraction becomes smaller, the scores of abstract states and patterns are less representative.
Revised: The reason is that although more grids can cluster similar concrete states more accurately, as the grid number of the abstraction becomes smaller, the scores of abstract states and patterns are less representative.

Original: We set the grid numbers to 10, 5, and 5 in Swimmer-v3, InvertedPendulum-v2, and InvertedDoublePendulum-v2, respectively. 
Revised: For MuJoCo tasks, we set the grid numbers to 10 for Swimmer-v3 and Hopper-v3, and 5 for other tasks, respectively. For Atari tasks, we uniformly set the grid numbers to 5.

Original: Figure 10 shows how the parameter ϵ in the equation Eq. equation 4 influences the performance.
Revised: Figure 10 shows how the parameter ϵ in Equation 4 influences the performance.

Original: The results suggest two major conclusions: (1) ϵ 
Revised: The results suggest two major conclusions: (1) ϵ in [0.1, 0.2] is the best choice for controlling reward shaping in

Original: 2 and ϵ = 0 . 1 are the best for controlling reward shaping in Walker2d-v3 and Hopper-v3, respectively; (2) a value for ϵ that is greater or smaller will reduce performance.
Revised: Walker2d-v3 and Hopper-v3, respectively; (2) a value for ϵ that is greater or smaller will reduce performance.

Original: The reason is that a smaller ϵ reduces the impact of scores, and a greater ϵ will increase the weight of state evaluation and completely reform the rewarding mechanism.
Revised: The reason is that a smaller ϵ reduces the impact of scores, and a greater ϵ will increase the weight of state evaluation and completely reform the rewarding mechanism.

Original: In addition,  we set ϵ = 0 .
Revised: In addition, for MuJoCo tasks, we set ϵ = 0 .

Original: 15 for Swimmer-v3 and ϵ = 0 .
Revised: 15 for Swimmer-v3 and ϵ = 0 .

Original: 2 for InvertedPendulum-v2 and InvertedDoublePendulum-v2.
Revised: 2 for other tasks. We set ϵ = 0 . 1 for all the Atari games.

Original: Figure 11 shows how the backbone algorithms affect the performance of NECSA.
Revised: Figure 11 shows how the backbone algorithms affect the performance of NECSA.

Original: In this evaluation, we build NECSA on TD3 and DDPG and compare the performance with the primary algorithms.
Revised: In this evaluation, we build NECSA on TD3 and DDPG and compare the performance with the primary algorithms.

Original: The results show that (1) algorithms can achieve better sample efficiency after incorporating with NECSA and (2) the backbone algorithms affect the performance of NECSA.
Revised: The results show that (1) algorithms can achieve better sample efficiency by incorporating with NECSA and (2) the backbone algorithms affect the performance of NECSA.

Original: For example, in Walker2d-v and Hopper-v3, TD3 is more sample efficient than DDPG. 
Revised: For example, in Walker2d-v and Hopper-v3, TD3 is more sample efficient than DDPG. In Alien, Rainbow outperforms DQN.

Original:  Such a conclusion can also be drawn after implementing NECSA, demonstrating that NECSA is scalable and effective on different backbone algorithms.
Revised: After applying NECSA to these algorithms, the comparison results remain the same trend. Such a conclusion demonstrates that NECSA is scalable and effective on different backbone algorithms.

Revised: A. P ERFORMANCE OF APPLYING NECSA TO DMC ONTROL TASKS

Revised: Figure 12 shows the performance of NECSA on DMControl (Tunyasuvunakool et al., 2020) tasks.

Revised: In this evaluation, we build NECSA on DrQ-v2 (Yarats et al., 2021), an approach to solve continuous tasks with image-based states.

Revised: NECSA outperforms DrQ-v2 on Walker-Walk and Hopper-Stand, and achieves the same performance on Cartpole-Balance.

Revised: The results show that (1) NECSA improves the sample efficiency of DrQ-v2 and (2) NECSA is general and effective for DRL tasks with image-based states and continuous action spaces.

Revised: A. A NALYSIS OF S TATE V ISITING D ENSITY

Revised: We plot the density of state visiting times in Figure 13.

Revised: Take Walker2d-v3 as an example.

Revised: The 1-step NECSA generated 198,613 abstract states, the 2-step NECSA generated 282,717 abstract patterns, and the 3-step NECSA generated 377,629 abstract patterns.

Revised: In 1-step setting, 163, abstract patterns were visited by only one time, and 35,340 were visited more than one time.

Revised: In 3-step setting, 372,253 abstract patterns were visited one time, and 5,376 of them were visited more than one

Revised: All 5, patterns visited pattern.

Revised: The density plot shows that the number of 1-time visits increases in multi-step statr abstraction.

Revised: However, the performance in Figure 5 shows that multi-step analysis can still achieve the best sample efficiency in our settings.

Revised: Therefore, we assume that the multi-times visited patterns played an essential role in enhancing the sample efficiency.

Revised: Neverthleass, we assume that there exists better choice of the abstraction steps ( e.g. , 5-step, 10-step).

Revised: Due to the abstraction state aprsity issue, we might need a novel state encoder instead of grid-based abstraction.

Revised: We leave the above assumption as future work.