Original: G ENETIC S OFT U PDATES FOR P OLICY E VOLUTION IN D EEP R EINFORCEMENT L EARNING
Revised: G ENETIC S OFT U PDATES FOR P OLICY E VOLUTION IN D EEP R EINFORCEMENT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The combination of Evolutionary Algorithms (EA) and Deep Reinforcement
Revised: The combination of Evolutionary Algorithms (EAs) and Deep Reinforcement

Original: Learning (DRL) has been recently proposed to merge the beneﬁts of both solutions.
Revised: Learning (DRL) has been recently proposed to merge the beneﬁts of both solutions.

Original: Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present signiﬁcant overhead.
Revised: Existing mixed approaches, however, have been successfully applied only to actor-critic methods and present signiﬁcant overhead.

Original: We address these issues by introducing a novel mixed framework that exploits a periodical genetic evaluation to soft update the weights of a DRL agent.
Revised: We address these issues by introducing a novel mixed framework that exploits a periodical genetic evaluation to soft update the weights of a DRL agent.

Original: The resulting approach is applicable with any DRL method and, in a worst-case scenario, it does not exhibit detrimental behaviours.
Revised: The resulting approach is applicable with any DRL method and, in a worst-case scenario, it does not exhibit detrimental behaviours.

Original: Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that signiﬁcantly outperforms prior DRL, EA, and mixed approaches.
Revised: Experiments in robotic applications and continuous control benchmarks demonstrate the versatility of our approach that signiﬁcantly outperforms prior DRL, EAs, and mixed approaches.

Original: Finally, we employ formal veriﬁcation to conﬁrm the policy improvement, mitigating the inefﬁcient exploration and hyper-parameter sensitivity of DRL.
Revised: Finally, we employ formal veriﬁcation to conﬁrm the policy improvement, mitigating the inefﬁcient exploration and hyper-parameter sensitivity of DRL

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The key to a wider and successful application of DRL techniques in real scenarios is the ability to adapt to the surrounding environment by generalizing from training experiences.
Revised: The key to a wider and successful application of DRL techniques in real scenarios is the ability to adapt to the surrounding environment by generalizing from training experiences.

Original: These solutions have to cope with the uncertainties of the operational environment, requiring a huge number of trials to achieve good performance.
Revised: These solutions have to cope with the uncertainties of the operational environment, requiring a huge number of trials to achieve good performance.

Original: Hence, devising robust learning approaches while improving sample efﬁciency is one of the challenges for wider utilization of DRL.
Revised: Hence, devising robust learning approaches while improving sample efﬁciency is one of the challenges for wider utilization of DRL.

Original: Despite the promising results (Tai et al., 2017; Zhang et al., 2017; Marchesini et al., 2019), DRL also suffer from convergence to local optima, which is mainly caused by the lack of diverse exploration when operating in high-dimensional spaces.
Revised: Despite the promising results (Tai et al., 2017; Zhang et al., 2017; Marchesini et al., 2019), DRL also suffer from convergence to local optima, which is mainly caused by the lack of diverse exploration when operating in high-dimensional spaces.

Original: Several studies address the exploration problem (e.g., curiosity-driven exploration (Pathak et al., 2017), count-based exploration (Ostrovski et al., 2017)), but they typically rely on sensitive task-speciﬁc hyper-parameters.
Revised: Several studies address the exploration problem (e.g., curiosity-driven exploration (Pathak et al., 2017), count-based exploration (Ostrovski et al., 2017)), but they typically rely on sensitive task-speciﬁc hyper-parameters.

Original: The sensitivity to such hyper-parameters is another signiﬁcant issue in DRL as it typically results in brittle convergence properties and poor performance in practical tasks (Haarnoja et al., 2018).
Revised: The sensitivity to such hyper-parameters is another signiﬁcant issue in DRL as it typically results in brittle convergence properties and poor performance in practical tasks (Haarnoja et al., 2018).

Original: Evolutionary Algorithms (Fogel, 2006) have been recently employed as a promising gradient-free optimization alternative to DRL.
Revised: Evolutionary Algorithms (Fogel, 2006) have been recently employed as a promising gradient-free optimization alternative to DRL.

Original: The redundancy of these population-based approaches has the advantages of enabling diverse exploration and improve robustness, leading to a more stable convergence.
Revised: The redundancy of these population-based approaches has the advantages of enabling diverse exploration and improve robustness, leading to a more stable convergence.

Original: In particular, Genetic Algorithms (GA) (Montana & Davis, 1989) show competitive results compared to gradient-based DRL (Such et al., 2017) and are characterized by low computational cost.
Revised: In particular, Genetic Algorithms (GA) (Montana & Davis, 1989) show competitive results compared to gradient-based DRL (Such et al., 2017) and are characterized by low computational cost.

Original: These gradient-free approaches, however, struggle to solve high-dimensional problems having poor generalization skills and are signiﬁcantly less sample efﬁcient than gradient-based methods.
Revised: These gradient-free approaches, however, struggle to solve high-dimensional problems having poor generalization skills and are signiﬁcantly less sample efﬁcient than gradient-based methods.

Original: An emergent research direction proposes the combination of gradient-free and gradient-based methods following the physical world, where evolution and learning cooperate to assimilate the best of both solutions (Simpson, 1953).
Revised: An emergent research direction proposes the combination of gradient-free and gradient-based methods following the physical world, where evolution and learning cooperate to assimilate the best of both solutions (Simpson, 1953).

Original: The ﬁrst mixed approach, Evolutionary Reinforcement Learning (ERL) (Khadka & Tumer, 2018), relies on actor-critic architecture to inject information in an evolutionary population while both the gradient-free and gradient-based training phases proceed in parallel.
Revised: The ﬁrst mixed approach, Evolutionary Reinforcement Learning (ERL) (Khadka & Tumer, 2018), relies on actor-critic architecture to inject information in an evolutionary population while both the gradient-free and gradient-based training phases proceed in parallel.

Original: Similarly, Proximal Distilled ERL (PDERL) (Bodnar, 2020) extends ERL with different evolutionary methods.
Revised: Similarly, Proximal Distilled ERL (PDERL) (Bodnar, 2020) extends ERL with different evolutionary methods.

Original: CEM-RL (Pourchot, 2019) brings this research direction into the family of distributed approaches, combining a portfolio of
Revised: CEM-RL (Pourchot, 2019) brings this research direction into the family of distributed approaches, combining a portfolio of

Original: TD3 (Fujimoto et al., 2018) learners with the Cross-Entropy Method (Yan Duan, 2016).
Revised: TD3 (Fujimoto et al., 2018) learners with the Cross-Entropy Method (Yan Duan, 2016).

Original: These mixed approaches however also present several limitations, which we address through our work: (i) the parallel training phases of the DRL and EA components (Khadka & Tumer, 2018; Bodnar, 2020), or the multitude of learners (Pourchot, 2019) result in signiﬁcant overhead (detailed in Section 4).
Revised: These mixed approaches however also present several limitations, which we address through our work: (i) the parallel training phases of the DRL and EA components (Khadka & Tumer, 2018; Bodnar, 2020), or the multitude of learners (Pourchot, 2019) result in signiﬁcant overhead (detailed in Section 4).

Original: (ii)
Revised: (ii)

Original: The actor-critic formalization of previous mixed approaches, allows them to be easily evaluated in continuous locomotion benchmarks (Brockman et al., 2016;
Revised: The actor-critic formalization of previous mixed approaches, allows them to be easily evaluated in continuous locomotion benchmarks (Brockman et al., 2016;

Original: Todorov et al., 2012).
Revised: Todorov et al., 2012).

Original: However, this also hinders their combination with value-based DRL.
Revised: However, this also hinders their combination with value-based DRL (Marchesini & Farinelli, 2020a).

Original: This is important as recent work (Matheron et al., 2019) shows the limitation of actor-critic in deterministic tasks, that in contrast can be effectively addressed with value-based methods.
Revised: This is important as recent work (Matheron et al., 2019) shows the limitation of actor-critic in deterministic tasks, that in contrast can be effectively addressed with value-based DRL.

Original: In particular,we show in Section 4  that a value-based implementation of Khadka & Tumer (2018) does not converge in our discrete robotic task.
Revised: In particular, Section 4 shows that a value-based implementation of Khadka & Tumer (2018) does not converge in our discrete robotic task.

Original: (iii)
Revised: (iii)

Original: The combination strategy does not ensure better performance compared to the DRL agent as it does not prevent detrimental behaviours (e.g., drop in performance).
Revised: The combination strategy does not ensure better performance compared to the DRL agent as it does not prevent detrimental behaviours (e.g., drop in performance).

Original: This is shown in the poor performance of a value-based implementation of ERL  (Section 4).
Revised: This is shown in the poor performance of a value-based implementation of ERL and PDERL (Section 4).

Original: We propose a novel mixed framework, called Soft Updates for Policy Evolution (Supe-RL), that enables us to combine the characteristics of GAs with any DRL algorithm, addressing the limitations of previous approaches.
Revised: We propose a novel mixed framework, called Soft Updates for Policy Evolution (Supe-RL), that enables us to combine the characteristics of GAs with any DRL algorithm, addressing the limitations of previous approaches.

Original: Supe-RL (Figure 1) beneﬁts from the high sampling efﬁciency of gradientbased DRL while incorporating gradient-free GA to generate diverse experiences and ﬁnd better policies.
Revised: Supe-RL (Figure 1) beneﬁts from the high sampling efﬁciency of gradientbased DRL while incorporating gradient-free GA to generate diverse experiences and ﬁnd better policies.

Original: Summarizing, Supe-RL based algorithms perform a periodical genetic evaluation applying
Revised: Summarizing, Supe-RL based algorithms perform a periodical genetic evaluation applying

Original: GAs to the agent network.
Revised: GAs to the agent network.

Original: A selection operator uses a ﬁtness metric to evaluate the population, choosing the best performing genome (i.e., the weights of the network) that is used to update the weights of the DRL agent.
Revised: A selection operator uses a ﬁtness metric to evaluate the population, choosing the best performing genome (i.e., the weights of the network) that is used to update the weights of the DRL agent.

Original: In contrast to previous work, our genetic evaluation is only performed periodically, drastically reducing the overhead.
Revised: In contrast to previous work, our genetic evaluation is only performed periodically, drastically reducing the overhead.

Original: Furthermore, our soft update (Section 3) allows a direct integration of GAs to any DRL algorithm as it is similar to perform a gradient step towards a better policy, avoiding detrimental behaviours.
Revised: Furthermore, our soft update (Section 3) allows a direct integration of GAs to any DRL algorithm as it is similar to perform a gradient step towards a better policy, avoiding detrimental behaviours.

Original: As detailed in Section 3.1, this allows using valuebased DRL, exploiting the variety of optimizations developed for the well-known DQN (van Hasselt et al., 2016;
Revised: As detailed in Section 3.1, this allows using valuebased DRL, exploiting the variety of optimizations developed for the well-known DQN (van Hasselt et al., 2016;

Original: Schaul et al., 2016; Wang et al., 2016; Fortunato et al., 2017; Bellemare et al., 2017).
Revised: Schaul et al., 2016; Wang et al., 2016; Fortunato et al., 2017; Bellemare et al., 2017).

Original: Crucially, our genetic component inﬂuences the DRL agent policy only if one of its mutated version performs better in a subset of evaluation episodes.
Revised: Crucially, our genetic component inﬂuences the DRL agent policy only if one of its mutated version performs better in a subset of evaluation episodes.

Original: Hence, as detailed in Section 3, with a sufﬁcient number of episodes we obtain a good estimation of the overall performance of the population.
Revised: Hence, as detailed in Section 3, with a sufﬁcient number of episodes we obtain a good estimation of the overall performance of the population.

Original: Our evaluation focuses on mapless navigation, a well-known problem in robotics and recent DRL (Zhang et al., 2017; Wahid et al., 2019; Marchesini & Farinelli, 2020).
Revised: Our evaluation focuses on mapless navigation, a well-known problem in robotics and recent DRL (Zhang et al., 2017; Wahid et al., 2019; Marchesini & Farinelli, 2020b).

Original: In particular, we consider two tasks developed with Unity (Juliani et al., 2018): (i) a discrete action space indoor scenario with obstacles for a mobile robot and (ii) a continuous task for aquatic drones, with dynamic waves and physically realistic water.
Revised: In particular, we consider two tasks developed with Unity (Juliani et al., 2018): (i) a discrete action space indoor scenario with obstacles for a mobile robot and (ii) a continuous task for aquatic drones, with dynamic waves and physically realistic water.

Original: Besides considering standard metrics related to performance (success rate and reward), we also consider safety properties that are particularly important in these domains (e.g., the agent does not collide with obstacles).
Revised: Besides considering standard metrics related to performance (success rate and reward), we also consider safety properties that are particularly important in these domains (e.g., the agent does not collide with obstacles).

Original: In more detail, we employ formal veriﬁcation (Neurify (Wang et al., 2018b)) to compute the percentage of input cases that cause violations of these properties.
Revised: In more detail, we employ formal veriﬁcation ((Corsi et al., 2020)) to compute the percentage of input cases that cause violations of these properties.

Original: This is important to conﬁrm our claim that Supe-RL based approaches correctly bias the exploration process in the direction of more robust policy regions with higher returns.
Revised: This is important to conﬁrm our claim that Supe-RL based approaches correctly bias the exploration process in the direction of more robust policy regions with higher returns.

Original: Results show that Supe-RL algorithms improve performance (i.e., training time, success rate, average reward), stability, and safety over value-based and policy-gradient DRL (Rainbow (Hessel et al., 2018), PPO (Schulman et al., 2017)) and ERL.
Revised: Results show that Supe-RL algorithms improve performance (i.e., training time, success rate, average reward), stability, and safety over value-based and policy-gradient DRL (Rainbow (Hessel et al., 2018), PPO (Schulman et al., 2017)) and ERL.

Original: Finally, we performed additional comparisons of Supe-RL with: (i) PDERL (Bodnar, 2020) to evidence the poor performance of previous mixed approaches when combined with value-based DRL; (ii) CEM-RL (Pourchot, 2019) in the aquatic scenario, to show the differences with a multi learner approach;
Revised: Finally, we performed additional comparisons of Supe-RL with: (i) PDERL (Bodnar, 2020) to evidence the poor performance of previous mixed approaches when combined with value-based DRL; (ii) CEM-RL (Pourchot, 2019) in the aquatic scenario, to show the differences with a multi learner approach;

Original: (iii) ERL in standard continuous benchmarks (i.e., MuJoCo locomotion (Brockman et al., 2016; Todorov et al., 2012)), where results conﬁrm the superior performance of Supe-RL.
Revised: (iii) ERL in standard continuous benchmarks (i.e., MuJoCo locomotion (Brockman et al., 2016; Todorov et al., 2012)), where results conﬁrm the superior performance of Supe-RL.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: We formalize robotic navigation as a RL problem, deﬁned over a Markov Decision Process, as described in recent DRL literature (Zhang et al., 2017; Wahid et al., 2019).
Revised: We formalize robotic navigation as a RL problem, deﬁned over a Markov Decision Process, as described in recent DRL literature (Tai et al., 2017; Zhang et al., 2017; Wahid et al., 2019).

Original: DRL for robotic navigation(Tai et al., 2017; Zhang et al., 2017; Wahid et al., 2019) focus exclusively on continuous action algorithms such as actor-critic DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018) and PPO (Schulman et al., 2017).
Revised: DRL for robotic navigation focus exclusively on continuous action algorithms such as actor-critic DDPG (Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018) and PPO (Schulman et al., 2017).

Original: Such methods have been adopted following the idea that value-based DQN (Mnih et al., 2013) can not deal with high-dimensional action spaces.
Revised: Such methods have been adopted following the idea that value-based DQN (Mnih et al., 2013) can not deal with high-dimensional action spaces.

Original: However, discrete value-based solutions typically result in shorter training time being more sample efﬁcient, and show better performance even in continuous settings.
Revised: However, discrete value-based solutions typically result in shorter training time being more sample efﬁcient, and show better performance even in continuous settings.

Original: In detail, Marchesini & Farinelli (2020) shows that discrete DRL is a more efﬁcient alternative to continuous DRL in robotic navigation.
Revised: In detail, Marchesini & Farinelli (2020b) shows that discrete DRL is a more efﬁcient alternative to continuous DRL in robotic navigation.

Original: Moreover, Tavakoli et al.
Revised: Moreover, Tavakoli et al.

Original: (2018) proposes an adaptation of Dueling DQN (Wang et al., 2016) with Double DQN (van Hasselt et al., 2016) that achieves competitive results in locomotion benchmarks (Brockman et al., 2016; Todorov et al., 2012).
Revised: (2018) proposes an adaptation of Dueling DQN (Wang et al., 2016) with Double DQN (van Hasselt et al., 2016) that achieves competitive results in locomotion benchmarks (Brockman et al., 2016; Todorov et al., 2012).

Original: More recently, de Wiele et al.
Revised: More recently, de Wiele et al.

Original: (2020) designed a DQN-based algorithm to handle enormous discrete and continuous action spaces.
Revised: (2020) designed a DQN-based algorithm to handle enormous discrete and continuous action spaces.

Original: These researches further motivate our contribution in the design of a mixed approach that also works with value-based DRL.
Revised: These researches further motivate our contribution in the design of a mixed approach that also works with value-based DRL.

Original: Evolutionary Algorithms EA are an alternative black-box optimization characterized by three main operators (Fogel, 2006): generation, alteration, and selection.
Revised: Evolutionary Algorithms EA are an alternative black-box optimization characterized by three main operators (Fogel, 2006): generation, alteration, and selection.

Original: In detail, Montana & Davis (1989) evolves a population of N individuals, each one represented by the network vector parameter θ (genome).
Revised: In detail, Montana & Davis (1989) evolves a population of N individuals, each one represented by the network vector parameter θ (genome).

Original: Each θ i ( i ∈ [0 , .., N − 1] ) is evaluated to produce a ﬁtness F ( θ i ) , used by the selection operator to choose the best genome.
Revised: Each θ i ( i ∈ [0 , .., N − 1] ) is evaluated to produce a ﬁtness F ( θ i ) , used by the selection operator to choose the best genome.

Original: For the EA component of Supe-RL, we consider a mutationbased GA that shown competitive performance over gradient-based DRL (Such et al., 2017).
Revised: For the EA component of Supe-RL, we consider a mutationbased GA that shown competitive performance over gradient-based DRL (Such et al., 2017).

Original: Combining EA and DRL Following the trend of using EA as an alternative for DRL (Salimans et al., 2017), an emergent research ﬁeld focuses on combining gradient-free and gradient-based solutions.
Revised: Combining EA and DRL Following the trend of using EA as an alternative for DRL (Salimans et al., 2017), an emergent research ﬁeld focuses on combining gradient-free and gradient-based solutions.

Original: In particular, ERL (Khadka & Tumer, 2018) considers an actor-critic DDPG agent (Lillicrap et al., 2015) and a concurrent EA training that generates a population of individuals, which are mutated and selected based on their ﬁtness.
Revised: In particular, ERL (Khadka & Tumer, 2018) considers an actor-critic DDPG agent (Lillicrap et al., 2015) and a concurrent EA training that generates a population of individuals, which are mutated and selected based on their ﬁtness.

Original: The DRL agent is trained in parallel from the samples generated by both the training phases and it is periodically injected in the running population which is used to collect the training performance.
Revised: The DRL agent is trained in parallel from the samples generated by both the training phases and it is periodically injected in the running population which is used to collect the training performance.

Original: The mutation function of ERL ensures that, in a certain number of episodes, the gradient-based policy outperforms its evolutionary siblings, introducing the gradient-based beneﬁts into the population.
Revised: The mutation function of ERL ensures that, in a certain number of episodes, the gradient-based policy outperforms its evolutionary siblings, introducing the gradient-based beneﬁts into the population.

Original: Hence, biasing the selection process of the next generation and its performance.
Revised: Hence, biasing the selection process of the next generation and its performance.

Original: In their experiments, the authors highlight an efﬁcient transfer of information between the two families of algorithms, outperforming DDPG in well-known locomotion benchmarks (Brockman et al., 2016; Todorov et al., 2012).
Revised: In their experiments, the authors highlight an efﬁcient transfer of information between the two families of algorithms, outperforming DDPG in well-known locomotion benchmarks (Brockman et al., 2016; Todorov et al., 2012).

Original: However, both the introduction of all the experiences in the buffer and forcing the DRL agent to perform better than the EA population, bias the training and can cause detrimental behaviours.
Revised: However, both the introduction of all the experiences in the buffer and forcing the DRL agent to perform better than the EA population, bias the training and can cause detrimental behaviours.

Original: Inspired by ERL, several combinations have been proposed (Bodnar, 2020; Colas et al., 2018; Pourchot, 2019; Khadka et al., 2019).
Revised: Inspired by ERL, several combinations have been proposed (Bodnar, 2020; Colas et al., 2018; Pourchot, 2019; Khadka et al., 2019).

Original: While GEP-PG (Bodnar, 2020) can be considered as a simpliﬁed version of a mixed approach, where a curiosity-driven approach is used to ﬁll the buffer of the agent, Proximal Distilled ERL (PDERL) (Bodnar, 2020) addresses the EA component of ERL, introducing novel operators to compensate for the simplicity of the genetic representation (as investigated by Lehman et al.
Revised: While GEP-PG (Bodnar, 2020) can be considered as a simpliﬁed version of a mixed approach, where a curiositydriven approach is used to ﬁll the buffer of the agent, Proximal Distilled ERL (PDERL) (Bodnar, 2020) addresses the EA component of ERL, introducing novel operators to compensate for the simplicity of the genetic representation (as investigated by Lehman et al.

Original: (2018), where authors address destructive behaviors of biologically-inspired variation operators applied to neural networks, which causes catastrophic forgetting).
Revised: (2018), where authors address destructive behaviors of biologically-inspired variation operators applied to neural networks, which causes catastrophic forgetting).

Original: However, as detailed in Section 3.1, our genetic evaluation is used to soft update the DRL agent only in the case of performance improvement, hence it does not show such catastrophic forgetting.
Revised: However, as detailed in Section 3.1, our genetic evaluation is used to soft update the DRL agent only in the case of performance improvement, hence it does not show such catastrophic forgetting.

Original: We also mention CERL (Khadka et al., 2019) and CEM-RL (Wang et al., 2016) as they are extensions of ERL for distributed training (which we do not consider here) with multiple active learners, which leads to non-negligible overhead (nonetheless, Section 4. reports an additional experiment in our continuous task with CEM-RL, to provide a more heterogeneous overview of the superior performance of Supe-RL).
Revised: We also mention CERL (Khadka et al., 2019) and CEM-RL (Pourchot, 2019) as they are extensions of ERL for distributed training (which we do not consider here) with multiple active learners, which leads to non-negligible overhead (nonetheless, Section 4.2 reports an additional experiment in our continuous task with CEM-RL, to provide a more heterogeneous overview of the superior performance of Supe-RL).

Original: These works share a common baseline as they all rely on actor-critic DRL and are built on the insights of ERL, which is the most closely related to Supe-RL.
Revised: These works share a common baseline as they all rely on actor-critic DRL and are built on the insights of ERL, which is the most closely related to Supe-RL.

Original: Hence, we choose ERL for complete performance comparison.
Revised: Hence, we choose ERL for complete performance comparison.

Original: Section 4.2 also shows a comparison with PDERL, to further highlights the poor performance of previous approaches when combined with value-based DRL.
Revised: Section 4.2 also shows a comparison with PDERL, to further highlights the poor performance of previous approaches when combined with value-based DRL.

Original: Finally, we use formal veriﬁcation to support our claims on the beneﬁcial effects of our genetic component into the policy.
Revised: Finally, we use formal veriﬁcation to support our claims on the beneﬁcial effects of our genetic component into the policy.

Original: We report in Appendix A a brief description of the considered methodology.
Revised: We report in Appendix A a brief description of the considered methodology.

Original:  S UPE -RL
Revised:  S UPE -RL

Original: The main insight of Supe-RL is to soft update a DRL agent towards better policy regions, enabling the combination with any DRL algorithm.
Revised: The main insight of Supe-RL is to soft update a DRL agent towards better policy regions, enabling the combination with any DRL algorithm.

Original: In detail, we combine a mutation-based GA (Such et al., 2017) with two DRL algorithms 1 : (i) Rainbow (Hessel et al., 2018) as a value-based algorithm for the discrete task and (ii) PPO (Schulman et al., 2017) as policy-gradient one for the continuous scenario, giving rise to two algorithms named SGRainbow and GPPO.
Revised: In detail, we combine a mutation-based GA (Such et al., 2017) with two DRL algorithms 1 : (i) Rainbow (Hessel et al., 2018) as a value-based algorithm for the discrete task and (ii) PPO (Schulman et al., 2017) as policy-gradient one for the continuous scenario, giving rise to two algorithms named SGRainbow and GPPO.

Original: A typical Supe-RL based algorithm (Appendix B provides a general pseudocode) proceeds as follows: the weights of a DRL agent drl a (referred as genome or θ a interchangeably)
Revised: A typical Supe-RL based algorithm (Appendix B provides a general pseudocode) proceeds as follows: the weights of a DRL agent drl a (referred as genome or θ a interchangeably)

Original: , are initialized with random values and, as in a standard training setup, drl a starts to collect experiences interacting with its environment.
Revised: , are initialized with random values and, as in a standard training setup, drl a starts to collect experiences interacting with its environment.

Original: Such experiences are stored in a buffer R to train the network.
Revised: Such experiences are stored in a buffer R to train the network.

Original: Periodically (every GE episodes), Supe-RL makes a genetic evaluation by generating a population of children, each one characterized by a different genome.
Revised: Periodically (every GE episodes), Supe-RL makes a genetic evaluation by generating a population of children, each one characterized by a different genome.

Original: Weights θ a are used to create the N individuals applying noise to the parameter vector:
Revised: Weights θ a are used to create the N individuals applying noise to the parameter vector:

Original: θ a + mut p n , where n ∼ N (0 , mut v ) and mut p is the mutation probability.
Revised: θ a + mut p n , where n ∼ N (0 , mut v ) and mut p is the mutation probability.

Original: In contrast, the mutation function of ERL multiplies the randomly chosen weights by
Revised: In contrast, the mutation function of ERL multiplies the randomly chosen weights by

Original: N (0 , mut v ) .
Revised: N (0 , mut v ) .

Original: Such mutations act in a similar fashion of a dropout layer, biasing the DRL agent to perform better than the evolutionary population in the long term.
Revised: Such mutations act in a similar fashion of a dropout layer, biasing the DRL agent to perform better than the evolutionary population in the long term.

Original: In more detail, ERL authors mutate 10% of the network weights in each episode (or epoch), multiplying them by N (0 , 0 . 1) (plus a mutation with standard deviation 10 or a reset mutation in a small percentage of cases).
Revised: In more detail, ERL authors mutate 10% of the network weights in each episode (or epoch), multiplying them by N (0 , 0 . 1) (plus a mutation with standard deviation 10 or a reset mutation in a small percentage of cases).

Original: Given that their evolutionary component is running in parallel with the gradient-based agent, we noticed that the weights in the population tend to 0, hence causing a detrimental behavior.
Revised: Given that their evolutionary component is running in parallel with the gradient-based agent, we noticed that the weights in the population tend to 0, hence causing a detrimental behavior.

Original: The population and a copy of θ a are then independently tested over a set of evaluation episodes which shares the same goals, to ﬁnd the overall best performing individual θ best based on the ﬁtness (the ﬁtness deﬁnition is domain-speciﬁc; in the case of navigation, it is computed as the number of targets the agent reaches over the evaluation episodes).
Revised: The population and a copy of θ a are then independently tested over a set of evaluation episodes which shares the same goals, to ﬁnd the overall best performing individual θ best based on the ﬁtness (the ﬁtness deﬁnition is domain-speciﬁc; in the case of navigation, it is computed as the number of targets the agent reaches over the evaluation episodes).

Original: Ideally, designing a sufﬁcient number of evaluation episodes (and targets), we can assume that θ best will not return detrimental behaviours to the DRL agent (as shown in Section 4).

Original: In this phase, we can also store a portion of diverse experiences in the buffer R used by drl a , to further exploit the population-based component.
Revised: In this phase, we can also store a portion of diverse experiences in the buffer R used by drl a , to further exploit the population-based component.

Original: Finally, if the selected genome belongs to one of the children, drl a weights are updated towards the mutated version and the training phase continues with the new weights.
Revised: Finally, if the selected genome belongs to one of the children, drl a weights are updated towards the mutated version and the training phase continues with the new weights.

Original: In contrast, if the best score belongs to drl a , the training phase, which was running in parallel, continues.
Revised: In contrast, if the best score belongs to drl a , the training phase, which was running in parallel, continues.

Original: Since the evaluation does not require any interaction among the population, we instantiate an independent copy of the environment for each
Revised: Since the evaluation does not require any interaction among the population, we instantiate an independent copy of the environment for each

Original: N + 1 population component in a separate thread and test them in parallel, drastically reducing the overhead for the drl a .
Revised: N + 1 population component in a separate thread and test them in parallel, drastically reducing the overhead for the drl a .

Original: The multi-thread nature of the Unity game engine makes this parallel testing phase straightforward and particularly efﬁcient.
Revised: The multi-thread nature of the Unity game engine makes this parallel testing phase straightforward and particularly efﬁcient.

Original: This approach has both the advantage of search a better-performing policy exploiting GA mutations (similar to noisy exploration (Fortunato et al., 2017)) and enrich the replay buffer with new diversiﬁed experiences.
Revised: This approach has both the advantage of search a better-performing policy exploiting GA mutations (similar to noisy exploration (Fortunato et al., 2017)) and enrich the replay buffer with new diversiﬁed experiences.

Original: As detailed in our empirical evaluation and the experiments with formal veriﬁcation tools (Section 3.1), our genetic evaluation leads to safer policies and a signiﬁcant reduction in training time, with Supe-RL resulting approximately two times faster than ERL in the same scenario.
Revised: As detailed in our empirical evaluation and the experiments with formal veriﬁcation tools (Section 3.1), our genetic evaluation leads to safer policies and a signiﬁcant reduction in training time, with Supe-RL resulting approximately two times faster than ERL in the same scenario.

Original: It is important to mention that both Supe-RL and previous mixed approaches are especially designed for scenarios in which it is possible to parallelize the learning process.
Revised: It is important to mention that both Supe-RL and previous mixed approaches are especially designed for scenarios in which it is possible to parallelize the learning process.

Original: Hence, when the training phase is performed on real physical systems, it is not possible in general to evaluate the population in parallel.
Revised: Hence, when the training phase is performed on real physical systems, it is not possible in general to evaluate the population in parallel.

Original: This could signiﬁcantly increase the convergence time.
Revised: This could signiﬁcantly increase the convergence time.

Original: Crucially, in contrast to previous mixed approaches, Supe-RL based algorithms are designed to improve the performance of drl a as our combination schema does not bias the choice of the betterperforming children in the long term.
Revised: Crucially, in contrast to previous mixed approaches, Supe-RL based algorithms are designed to improve the performance of drl a as our combination schema does not bias the choice of the betterperforming children in the long term.

Original: In the worst-case scenario, the main agent is always the best genome in the population and does not improve the current policy, hence a Supe-RL based training will match the performance of the chosen DRL algorithm.
Revised: In the worst-case scenario, the main agent is always the best genome in the population and does not improve the current policy, hence a Supe-RL based training will match the performance of the chosen DRL algorithm.

Original: 3. V ALUE -B ASED AND P OLICY -G RADIENT I MPLEMENTATIONS
Revised: 3. V ALUE -B ASED AND P OLICY -G RADIENT I MPLEMENTATIONS

Original: Robotic navigation allows evaluating Supe-RL in a variety of scenarios (e.g., discrete and continuous action spaces).
Revised: Robotic navigation allows evaluating Supe-RL in a variety of scenarios (e.g., discrete and continuous action spaces).

Original: Hence, our two tasks present different characteristics (e.g., static and dynamic environments, sparse and dense rewards).
Revised: Hence, our two tasks present different characteristics (e.g., static and dynamic environments, sparse and dense rewards).

Original: In this section, we present the value-based and policy-gradient implementation of Supe-RL, combined with a mutation-based GA (Such et al., 2017).
Revised: In this section, we present the value-based and policy-gradient implementation of Supe-RL, combined with a mutation-based GA (Such et al., 2017).

Original: 3.1. SGR AINBOW IN D ISCRETE I NDOOR N AVIGATION
Revised: 3.1. SGR AINBOW IN D ISCRETE I NDOOR N AVIGATION

Original: We consider a Turtlebot3 2 indoor environment with obstacles, using a discrete action space and a dense reward function.
Revised: We consider a Turtlebot3 2 indoor environment with obstacles, using a discrete action space and a dense reward function.

Original: Environment description Target goals randomly spawn in the scenario and are guaranteed to be obstacle-free.
Revised: Environment description Target goals randomly spawn in the scenario and are guaranteed to be obstacle-free.

Original: The reward r t is structured as two sparse value in case of reaching the target R reach = 1 within error µ = 5 cm from the target goal, or crashing R fail = − 1 which terminates an episode (resetting the robot to its starting position).
Revised: The reward r t is structured as two sparse value in case of reaching the target R reach = 1 within error µ = 5 cm from the target goal, or crashing R fail = − 1 which terminates an episode (resetting the robot to its starting position).

Original: A dense part is used during the travel: ω ( d t − 1 − d t ) , where d t − 1 , d t indicate the euclidean distance between the robot and the goal at two consecutive time steps and ω = 10 is a multiplicative factor.
Revised: A dense part is used during the travel: ω ( d t − 1 − d t ) , where d t − 1 , d t indicate the euclidean distance between the robot and the goal at two consecutive time steps and ω = 10 is a multiplicative factor.

Original: Network architecture: the input layer contains 19 sparse laser scans, sampled in [ − 90 , 90] degrees in a ﬁxed angle distribution and the target position (expressed in polar coordinates); a similar setting is used in Tai et al.
Revised: Network architecture: the input layer contains 19 sparse laser scans, sampled in [ − 90 , 90] degrees in a ﬁxed angle distribution and the target position (expressed in polar coordinates); a similar setting is used in Tai et al.

Original: Long et al.
Revised: Long et al.

Original: We did explore other encodings for the problem (e.g., adding linear velocities as output) but a higher complexity of the problem causes longer training times with negligible improvements.
Revised: We did explore other encodings for the problem (e.g., adding linear velocities as output) but a higher complexity of the problem causes longer training times with negligible improvements.

Original: We performed multiple trials on different network sizes and seeds (Chen & Chang, 1996) and the outcome led us to use two relu hidden layers with 64 neurons each and 5 linear nodes to encode the output angular velocities [ − 90 , − 45 , 0 , 45 , 90] deg/s .
Revised: We performed multiple trials on different network sizes and seeds (Chen & Chang, 1996) and the outcome led us to use two relu hidden layers with 64 neurons each and 5 linear nodes to encode the output angular velocities [ − 90 , − 45 , 0 , 45 , 90] deg/s .

Original: Methodology for SGRainbow: the genetic evaluation of the value-based agent presents challenges due to the instability of the training algorithm (which is a known drawback of DQN-based algorithms) and the poor scalability on high-dimensional action spaces.
Revised: Methodology for SGRainbow: the genetic evaluation of the value-based agent presents challenges due to the instability of the training algorithm (which is a known drawback of DQN-based algorithms) and the poor scalability on high-dimensional action spaces.

Original: Nonetheless, recent work shows the beneﬁts of such solutions (Marchesini & Farinelli, 2020; Tavakoli et al., 2018; de Wiele et al., 2020) in these challenging settings, further motivating the requirement for a mixed approach that works also with value-based DRL.
Revised: Nonetheless, recent work shows the beneﬁts of such solutions (Marchesini & Farinelli, 2020b; Tavakoli et al., 2018; de Wiele et al., 2020) in these challenging settings, further motivating the requirement for a mixed approach that works also with value-based DRL.

Original: Here we will discuss only the elements of the algorithm which are relevant to SGRainbow, referring the interested reader to (Hessel et al., 2018) for further details on Rainbow.
Revised: Here we will discuss only the elements of the algorithm which are relevant to SGRainbow, referring the interested reader to (Hessel et al., 2018) for further details on Rainbow.

Original: We developed different approaches to update the drl a with the genome of the better performing child.
Revised: We developed different approaches to update the drl a with the genome of the better performing child.

Original: We ﬁrstly switch drl a with the child, soft updating only the target network (originally developed for DDQN van Hasselt et al.
Revised: We ﬁrstly switch drl a with the child, soft updating only the target network (originally developed for DDQN van Hasselt et al.

Original: to approach the new weights.
Revised: to approach the new weights.

Original: We tried different settings for the target network, but a soft update with τ (cid:48)
Revised: We tried different settings for the target network, but a soft update with τ (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 1 showed us better performance.
Revised: 1 showed us better performance.

Original: This method however leads to an unusual optimizer choice, which is crucial considering the high-variance of a value-based algorithm.
Revised: This method however leads to an unusual optimizer choice, which is crucial considering the high-variance of a value-based algorithm.

Original: In particular, the widely adopted Adam optimizer (Kingma & Ba, 2014) with its self-adaptable learning rate typically requires minimal hyper-parameter tuning but, after a switch of the drl a model with a better genome, a drop of performance occurs.
Revised: In particular, the widely adopted Adam optimizer (Kingma & Ba, 2014) with its self-adaptable learning rate typically requires minimal hyper-parameter tuning but, after a switch of the drl a model with a better genome, a drop of performance occurs.

Original: We motivate this as the current learning rate of Adam is ”balanced” for the old drl a weights and requires time to adjust its value for the new unexpected and better-performing model.
Revised: We motivate this as the current learning rate of Adam is ”balanced” for the old drl a weights and requires time to adjust its value for the new unexpected and better-performing model.

Original: For this reason, this ﬁrst application, which we refer to as GRainbow, required to tune an SGD optimizer for this scenario, decaying its initial learning rate from 0 .
Revised: For this reason, this ﬁrst application, which we refer to as GRainbow, required to tune an SGD optimizer for this scenario, decaying its initial learning rate from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: 001 based on the current success rate of the model.
Revised: 001 based on the current success rate of the model.

Original: As shown in
Revised: As shown in

Original: Section 4, this approach already outperformed both the Rainbow algorithm and the GA.
Revised: Section 4, this approach already outperformed both the Rainbow algorithm and the GA.

Original: The tuning for SGD requires several trials and is one of the main limitations of GRainbow.
Revised: The tuning for SGD requires several trials and is one of the main limitations of GRainbow.

Original: To address this, we considered Tessler et al.
Revised: To address this, we considered Tessler et al.

Original: (2019), where authors improved DDQN stability by copying the agent model to the target when the former performs better.
Revised: (2019), where authors improved DDQN stability by copying the agent model to the target when the former performs better.

Original: Hence, we ﬂipped our approach by soft updating drl a towards the best genome (using τ (cid:48)
Revised: Hence, we ﬂipped our approach by soft updating drl a towards the best genome (using τ (cid:48)

Original: = 0 . 1 )
Revised: = 0 . 1 )

Original: and switching the target network with such genome.
Revised: and switching the target network with such genome.

Original: This enabled us to use the Adam optimizer, improving the performance of GRainbow.
Revised: This enabled us to use the Adam optimizer, improving the performance of GRainbow.

Original: We refer the interested reader to Appendix C of the Supplementary Material for a performance comparison between GRainbow with SGD and Adam.
Revised: We refer the interested reader to Appendix C of the Supplementary Material for a performance comparison between GRainbow with SGD and Adam.

Original: Finally, we further improved both training performance and stability, exploiting the soft update (Lillicrap et al., 2015) technique for both networks.
Revised: Finally, we further improved both training performance and stability, exploiting the soft update (Lillicrap et al., 2015) technique for both networks.

Original: We refer to the resultant algorithm as Soft
Revised: We refer to the resultant algorithm as Soft

Original: GRainbow (SGRainbow).
Revised: GRainbow (SGRainbow).

Original: In particular, we soft update both the drl a agent and target models towards the weights of the best performing child to smooth the transition of the networks towards the bettermutated policy.
Revised: In particular, we soft update both the drl a agent and target models towards the weights of the best performing child to smooth the transition of the networks towards the bettermutated policy.

Original: The update rule for the drl a networks is then: θ a = τ (cid:48) θ a + (1 − τ (cid:48) )
Revised: The update rule for the drl a networks is then: θ a = τ (cid:48) θ a + (1 − τ (cid:48) )

Original: θ best .
Revised: θ best .

Original: We tried different values for τ (cid:48) (results in Appendix C), obtaining the best performance with τ (cid:48)
Revised: We tried different values for τ (cid:48) (results in Appendix C), obtaining the best performance with τ (cid:48)

Original: = 0 .
Revised: = 0 .

Original: We believe that these periodical slight changes in the drl a policy, that simulate a gradient step towards better network weights, are the core mechanism that allowed Supe-RL to work and improve the performance of value-based Supe-RL, while also reducing the variance across different runs.
Revised: We believe that these periodical slight changes in the drl a policy, that simulate a gradient step towards better network weights, are the core mechanism that allowed Supe-RL to work and improve the performance of value-based Supe-RL, while also reducing the variance across different runs.

Original: A useful side message is that using SGD seems better than Adam when drl a ”jumps” due to target network instabilities, as Adam works better if the transition of the target network is more stable.
Revised: A useful side message is that using SGD seems better than Adam when drl a ”jumps” due to target network instabilities, as Adam works better if the transition of the target network is more stable.

Original: Finally, results in Section 4 introduce part of the experiences of the genetic evaluation into the same prioritized buffer of the drl a (Appendix C contains an ablation experiment on the inﬂuence of these diversiﬁed experiences with SGRainbow).
Revised: Finally, results in Section 4 introduce part of the experiences of the genetic evaluation into the same prioritized buffer of the drl a (Appendix C contains an ablation experiment on the inﬂuence of these diversiﬁed experiences with SGRainbow).

Original: 3.1. GPPO IN C ONTINUOUS A QUATIC N AVIGATION
Revised: 3.1. GPPO IN C ONTINUOUS A QUATIC N AVIGATION

Original: We consider an aquatic drone navigation task characterized by dynamic waves, with a continuous action space and a sparse reward function.
Revised: We consider an aquatic drone navigation task characterized by dynamic waves, with a continuous action space and a sparse reward function.

Original: Sources of the aquatic simulator as a novel DRL task and a video with an overview of the environment are available at tinyurl.com/y22xh43c .
Revised: Sources of the aquatic simulator as a novel DRL task and a video with an overview of the environment are available at tinyurl.com/y22xh43c .

Original: Environment description: The aquatic drone is a differential drive platform, where a continuous action is mapped to the motor power.
Revised: Environment description: The aquatic drone is a differential drive platform, where a continuous action is mapped to the motor power.

Original: The drone receives a sparse reward r t structured as two values in case of reaching the target R reach = 1 within error µ = 5 cm from the target goal, or reaching timeout R fail = − 1 which terminates an episode (resetting the robot to its starting position).
Revised: The drone receives a sparse reward r t structured as two values in case of reaching the target R reach = 1 within error µ = 5 cm from the target goal, or reaching timeout R fail = − 1 which terminates an episode (resetting the robot to its starting position).

Original: Network architecture: the input layer contains the target position expressed in polar coordinates with respect to the drone and the pitch of the boat.
Revised: Network architecture: the input layer contains the target position expressed in polar coordinates with respect to the drone and the pitch of the boat.

Original: It is possible to retrieve these values using the
Revised: It is possible to retrieve these values using the

Original: GPS and the compass of the boat.
Revised: GPS and the compass of the boat.

Original: Our initial evaluation for the network size led us to use two tanh hidden layers with 32 neurons each.
Revised: Our initial evaluation for the network size led us to use two tanh hidden layers with 32 neurons each.

Original: Finally, two tanh output nodes encode each motor velocity (multiplied by a constant value to obtain our velocity limit).
Revised: Finally, two tanh output nodes encode each motor velocity (multiplied by a constant value to obtain our velocity limit).

Original: Methodology for GPPO: in contrast to the value-based implementation, the stability of on-policy
Revised: Methodology for GPPO: in contrast to the value-based implementation, the stability of on-policy

Original: PPO does not present issues related to the Adam optimizer.
Revised: PPO does not present issues related to the Adam optimizer.

Original: Hence, we considered the same soft update strategy adopted with SGRainbow.
Revised: Hence, we considered the same soft update strategy adopted with SGRainbow.

Original: Trivially we did not use experiences of the best child due to the on-policy nature of the algorithm.
Revised: Trivially we did not use experiences of the best child due to the on-policy nature of the algorithm.

Original: For further details on the implementation of the PPO algorithm, we refer to (Schulman et al., 2017).
Revised: For further details on the implementation of the PPO algorithm, we refer to (Schulman et al., 2017).

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: The goal of our empirical evaluation is to investigate whether Supe-RL approaches combine the beneﬁts of GA with both value-based and policy-gradient DRL while maintaining minimal overhead for the training.
Revised: The goal of our empirical evaluation is to investigate whether Supe-RL approaches combine the beneﬁts of GA with both value-based and policy-gradient DRL while maintaining minimal overhead for the training.

Original: Data are collected on an i7-9700k, using the implementation of Section 3.1.
Revised: Data are collected on an i7-9700k, using the implementation of Section 3.1.

Original: We considered the same set of hyper-parameters (reported in Appendix D) for the baselines and Supe-RL based approaches.
Revised: We considered the same set of hyper-parameters (reported in Appendix D) for the baselines and Supe-RL based approaches.

Original: Our ERL implementations refer to (Khadka & Tumer, 2018) for the evolutionary component, while we use Rainbow and PPO for the gradient-based part.
Revised: Our ERL implementations refer to (Khadka & Tumer, 2018) for the evolutionary component, while we use Rainbow and PPO for the gradient-based part.

Original: In order to get reproducible and consistent results when comparing different algorithms, the random seed is ﬁxed across a single run of every algorithm (because there may exist a sequence of targets that favor a run or a better network initialization), while it varies in different runs.
Revised: In order to get reproducible and consistent results when comparing different algorithms, the random seed is ﬁxed across a single run of every algorithm (because there may exist a sequence of targets that favor a run or a better network initialization), while it varies in different runs.

Original: As a consequence, a certain run of every algorithm executes the same sequence of targets and initializes the networks with the same weights.
Revised: As a consequence, a certain run of every algorithm executes the same sequence of targets and initializes the networks with the same weights.

Original: All the trained models, except for the value-based ERL (ERL-R) and the GA, are able to navigate generalizing: starting and target position, and velocity.
Revised: All the trained models, except for the value-based ERL (ERL-R) and the GA, are able to navigate generalizing: starting and target position, and velocity.

Original: The Turtlebot3 lidar allows navigating in unknown environments with different obstacles, while the boat maintains similar performance in different wave conditions.
Revised: The Turtlebot3 lidar allows navigating in unknown environments with different obstacles, while the boat maintains similar performance in different wave conditions.

Original: For each graph, we report the mean and standard deviation of ﬁve statistically independent runs, considering (i) success rate: how many successful trajectories are performed; (ii) total reward.
Revised: For each graph, we report the mean and standard deviation of ten statistically independent runs, considering (i) success rate: how many successful trajectories are performed; (ii) total reward.

Original: Results are smoothed over one hundred episodes.
Revised: Results are smoothed over one hundred episodes.

Original: [Table]
Revised: [Table]

Original: To collect signiﬁcant data in the evaluation of navigation performance, we chose a set of targets reachable by every model.
Revised: To collect signiﬁcant data in the evaluation of navigation performance, we chose a set of targets reachable by every model.

Original: Table 1 resumes the results for the discrete (Rainbow, SGRainbow) and the continuous (PPO, ERL, GPPO) tasks, considering only models with acceptable performance (i.e., we did not include GA and ERL-R).
Revised: Table 1 resumes the results for the discrete (Rainbow, SGRainbow) and the continuous (PPO, ERL, GPPO) tasks, considering only models with acceptable performance (i.e., we did not include GA and ERL-R).

Original: For clarity, in the aquatic task we collected the dense reward used in the discrete one.
Revised: For clarity, in the aquatic task we collected the dense reward used in the discrete one.

Original: Hence, rewards will be similar (agents navigate towards the target, collecting positive rewards), but time and number of steps (i.e., trajectory length) differ signiﬁcantly ( ≈ 31% and ≈ 37 .
Revised: Hence, rewards will be similar (agents navigate towards the target, collecting positive rewards), but time and number of steps (i.e., trajectory length) differ signiﬁcantly ( ≈ 31% and ≈ 37 .

Original: 5% for SGRainbow over Rainbow; ≈ 18% and ≈ 8% for GPPO over PPO and ERL, respectively).
Revised: 5% for SGRainbow over Rainbow; ≈ 18% and ≈ 8% for GPPO over PPO and ERL, respectively).

Original: In both tasks, Supe-RL outperforms DRL algorithms and ERL in every considered metric.
Revised: In both tasks, Supe-RL outperforms DRL algorithms and ERL in every considered metric.

Original: 4. V ALUE -B ASED E VALUATION
Revised: 4. V ALUE -B ASED E VALUATION

Original: Here we compare the GA, Rainbow, ERL-R, GRainbow, and SGRainbow.
Revised: Here we compare the GA, Rainbow, ERL-R, GRainbow, and SGRainbow.

Original: As previously discussed,
Revised: As previously discussed,

Original: Figures 2A, B show that a direct combination of ERL with the value-based algorithm can not cope the issues of such DRL approaches, resulting in very poor performance.
Revised: Figures 2A, B show that a direct combination of ERL with the value-based algorithm can not cope the issues of such DRL approaches, resulting in very poor performance.

Original: Given these results, we decided to perform an additional experiment with the improved version of ERL, PDERL (Bodnar, 2020), which introduces novel genetic operators to improve ERL robustness.
Revised: Given these results, we decided to perform an additional experiment with the improved version of ERL, PDERL (Bodnar, 2020), which introduces novel genetic operators to improve ERL robustness.

Original: Nonetheless, we conjecture that the detrimental behaviour of previous mixed approaches is related to their DRL injection pattern, rather than the simplicity of the genetic approach (the goodness of a simple genetic representation is evident in our results and in Such et al.
Revised: Nonetheless, we conjecture that the detrimental behaviour of previous mixed approaches is related to their DRL injection pattern, rather than the simplicity of the genetic approach (the goodness of a simple genetic representation is evident in our results and in Such et al.

Original: In contrast, our periodical genetic evaluation with a soft update strategy simulates a gradient step towards a better policy while, in a worst-case scenario, we do not update the DRL agent, avoiding detrimental behaviours.
Revised: In contrast, our periodical genetic evaluation with a soft update strategy simulates a gradient step towards a better policy while, in a worst-case scenario, we do not update the DRL agent, avoiding detrimental behaviours.

Original: Our results (detailed in Section 4.2) conﬁrm the superior performance of PDERL over ERL in both our environments.
Revised: Our results (detailed in Section 4.2) conﬁrm the superior performance of PDERL over ERL in both our environments.

Original: However, PDERL still provides inferior performance compared to Supe-RL based approaches and shares the poor performance of ERL when combined with the value-based algorithm.
Revised: However, PDERL still provides inferior performance compared to Supe-RL based approaches and shares the poor performance of ERL when combined with the value-based algorithm.

Original: In further detail, Figure 2A shows that even GRainbow with the SGD optimizer, where the best genome fully substitutes the main DRL agent, outperforms Rainbow (i.e., 80% of success rate over 60% ).
Revised: In further detail, Figure 2A shows that even GRainbow with the SGD optimizer, where the best genome fully substitutes the main DRL agent, outperforms Rainbow (i.e., 80% of success rate over 60% ).

Original: Moreover, the soft genetic update further improves the performance while reducing the variance and SGRainbow reaches a performance of 90% success rate in about 2000 epochs that correspond to 60 minutes of training (in contrast, Rainbow reached 60% success rate in similar training time).
Revised: Moreover, the soft genetic update further improves the performance while reducing the variance and SGRainbow reaches a performance of 90% success rate in about 2000 epochs that correspond to 60 minutes of training (in contrast, Rainbow reached 60% success rate in similar training time).

Original: Furthermore, the standalone GA was not able to cope with the complexity of the task, where the algorithm needs to generalize the navigation while exploiting the laser values to avoid obstacles.
Revised: Furthermore, the standalone GA was not able to cope with the complexity of the task, where the algorithm needs to generalize the navigation while exploiting the laser values to avoid obstacles.

Original: 4. P OLICY -G RADIENT E VALUATION
Revised: 4. P OLICY -G RADIENT E VALUATION

Original: Here we compare PPO, an adapted version of ERL with PPO, and GPPO.
Revised: Here we compare PPO, an adapted version of ERL with PPO, and GPPO.

Original: Figures 2C, D show that also in the continuous domain, Supe-RL based method offers better performances considering our evaluation metrics.
Revised: Figures 2C, D show that also in the continuous domain, Supe-RL based method offers better performances considering our evaluation metrics.

Original: In detail, GPPO reaches over 98% of average success rate in aboutepochs that correspond to 110 minutes of training, while PPO, similarly to ERL, was able to reach ≈ 82% of average success rate in ≈ 1700 epochs ( 160 and 210 minutes of training, respectively).
Revised: In detail, GPPO reaches over 98% of average success rate in aboutepochs that correspond to 110 minutes of training, while PPO, similarly to ERL, was able to reach ≈ 82% of average success rate in ≈ 1700 epochs ( 160 and 210 minutes of training, respectively).

Original: Furthermore, as reported in Table 1, GPPO uses a lower number of actions compared to PPO and
Revised: Furthermore, as reported in Table 1, GPPO uses a lower number of actions compared to PPO and

Original: ERL, which translates into shorter paths and travel time for the drone towards the same target.
Revised: ERL, which translates into shorter paths and travel time for the drone towards the same target.

Original: In this task, we also compared the efﬁciency of our genetic evaluation in ﬁnding better policies with respect to ERL.
Revised: In this task, we also compared the efﬁciency of our genetic evaluation in ﬁnding better policies with respect to ERL.

Original: Figure 3 shows the cumulative number of injections (ERL) and evaluations (SupeRL) (x-axis), over the successful ones (y-axis) through the training, to reach similar performance.
Revised: Figure 3 shows the cumulative number of injections (ERL) and evaluations (SupeRL) (x-axis), over the successful ones (y-axis) through the training, to reach similar performance.

Original: We compare injections and evaluations as they represent the approaches of ERL and Supe-RL to improve the policy, respectively.
Revised: We compare injections and evaluations as they represent the approaches of ERL and Supe-RL to improve the policy, respectively.

Original: Results show that our mutation schema ﬁnds more often betterperforming policies, in contrast to ERL where successful network injection occurs more rarely (ERL requires 400% more injection trials, i.e., 250 over 25, to match Supe-RL performance, i.e., 98% success).
Revised: Results show that our mutation schema ﬁnds more often betterperforming policies, in contrast to ERL where successful network injection occurs more rarely (ERL requires 400% more injection trials, i.e., 250 over 25, to match Supe-RL performance, i.e., 98% success).

Original: Evaluation in Standard Benchmarks We performed additional experiments on Reacher-v2,
Revised: Evaluation in Standard Benchmarks We performed additional experiments on Reacher-v2,

Original: HalfCheetah-v2, Hopper-v2, and Ant-v2 MuJoCo tasks (Brockman et al., 2016; Todorov et al., 2012) with GPPO and ERL.
Revised: HalfCheetah-v2, Hopper-v2, and Ant-v2 MuJoCo tasks (Brockman et al., 2016; Todorov et al., 2012) with GPPO and ERL.

Original: In particular, we considered the same speciﬁcs for data collection detailed in Khadka & Tumer (2018), hence we report the same performance metrics instead of a success rate.
Revised: In particular, we considered the same speciﬁcs for data collection detailed in Khadka & Tumer (2018), hence we report the same performance metrics instead of a success rate.

Original: Our ERL implementation with PPO returned comparable results to the original ones presented in Khadka & Tumer (2018); Pourchot (2019);
Revised: Our ERL implementation with PPO returned comparable results to the original ones presented in Khadka & Tumer (2018); Pourchot (2019);

Original: Khadka et al.
Revised: Khadka et al.

Original: (data were collected using the same hardware, GPPO parameters, and averaged over 5 runs).
Revised: (data were collected using the same hardware, GPPO parameters, and averaged over 5 runs).

Original: Crucially, Figure 4A, B,
Revised: Crucially, Figure 4A, B,

Original: C, D shows that our Supe-RL based algorithm has comparable or better performance across the considered tasks.
Revised: C, D shows that our Supe-RL based algorithm has comparable or better performance across the considered tasks.

Original: Furthermore, 4C highlights the detrimental behaviour of ERL.
Revised: Furthermore, 4C highlights the detrimental behaviour of ERL.

Original: Comparison with CEM-RL and PDERL To conﬁrm that previous mixed approaches can not be directly combined with value-based approaches, due to their injection pattern rather than their genetic representation, we performed an additional experiment with PDERL (Bodnar, 2020).
Revised: Comparison with CEM-RL and PDERL To conﬁrm that previous mixed approaches can not be directly combined with value-based approaches, due to their injection pattern rather than their genetic representation, we performed an additional experiment with PDERL (Bodnar, 2020).

Original: Such approach addresses the poor genetic representation of ERL, improving its performance and robustness.
Revised: Such approach addresses the poor genetic representation of ERL, improving its performance and robustness.

Original: Figure 5C, D shows the collected data in the discrete scenario, where we combined PDERL with our value-based baseline, and in the continuous task, where we combined it with the PPO baseline.
Revised: Figure 5C, D shows the collected data in the discrete scenario, where we combined PDERL with our value-based baseline, and in the continuous task, where we combined it with the PPO baseline.

Original: We considered PDERL GitHub for the genetic component, and the same hyper-parameters and random seeds used in our previous experiments.
Revised: We considered PDERL GitHub for the genetic component, and the same hyper-parameters and random seeds used in our previous experiments.

Original: Results conﬁrm the superior performance of PDERL over
Revised: Results conﬁrm the superior performance of PDERL over

Original: ERL in both environments (i.e., improved success rate and reduced variance).
Revised: ERL in both environments (i.e., improved success rate and reduced variance).

Original: Furthermore, the two parallel training phases do not provide a robust evaluation of the gradient-free population in the same set of tasks, hence, the resultant best agent does not represent an overall best policy for the task.
Revised: Furthermore, the two parallel training phases do not provide a robust evaluation of the gradient-free population in the same set of tasks, hence, the resultant best agent does not represent an overall best policy for the task.

Original: Follows that PDERL shares the detrimental performance of previous mixed approaches when combined with value-based algorithms.
Revised: Follows that PDERL shares the detrimental performance of previous mixed approaches when combined with value-based algorithms.

Original: Finally, as detailed in Section 2, the recent ﬁeld of mixed approaches has been extended with distributed solutions that use a portfolio of active DRL learners, such as CERL (Khadka et al., 2019) and CEM-RL (Pourchot, 2019).
Revised: Finally, as detailed in Section 2, the recent ﬁeld of mixed approaches has been extended with distributed solutions that use a portfolio of active DRL learners, such as CERL (Khadka et al., 2019) and CEM-RL (Pourchot, 2019).

Original: This intuitively results in signiﬁcant overhead in the training process.
Revised: This intuitively results in signiﬁcant overhead in the training process.

Original: We decided to compare GPPO with the TD3 version of CEM-RL, as the authors claim it outperforms previous mixed approaches (to further conﬁrm the overhead, Sec.
Revised: We decided to compare GPPO with the TD3 version of CEM-RL, as the authors claim it outperforms previous mixed approaches (to further conﬁrm the overhead, Sec.

Original: 5.2.2 of CEM-RL (Pourchot, 2019) state that tests are on limited timesteps, due to computational demands).
Revised: 5.2.2 of CEM-RL (Pourchot, 2019) state that tests are on limited timesteps, due to computational demands).

Original: In contrast, Supe-RL uses one DRL learner and the population is only used for policy evaluation.
Revised: In contrast, Supe-RL uses one DRL learner and the population is only used for policy evaluation.

Original: We used the GitHub provided by the authors to test CEM-TD3 in the continuous scenario (aquatic drone) and Figures 5A, B show that, as expected, CEM-RL required signiﬁcantly more time (250 minutes over 110).
Revised: We used the GitHub provided by the authors to test CEM-TD3 in the continuous scenario (aquatic drone) and Figures 5A, B show that, as expected, CEM-RL required signiﬁcantly more time (250 minutes over 110).

Original: In particular, CEM-TD3 reaches 98% success rate (the performance of Supe-RL at epoch 1200) in approximately 600 episodes.
Revised: In particular, CEM-TD3 reaches 98% success rate (the performance of Supe-RL at epoch 1200) in approximately 600 episodes.

Original: However, it required 125% more wall-clock time with respect to the time required by Supe-RL to reach similar performance (data were collected using the same hardware and averaged over 5 runs).
Revised: However, it required 125% more wall-clock time with respect to the time required by Supe-RL to reach similar performance (data were collected using the same hardware and averaged over 5 runs).

Original: [Table]
Revised: [Table]

Original: 4. R OBUSTNESS OF S UPE -RL USING F ORMAL V ERIFICATION
Revised: 4. R OBUSTNESS OF S UPE -RL USING F ORMAL V ERIFICATION

Original: An important result in both tasks is the limited variance shown by Supe-RL approaches, across runs with different network initialization seeds.
Revised: An important result in both tasks is the limited variance shown by Supe-RL approaches, across runs with different network initialization seeds.

Original: Appendix E shows a more detailed analysis of our results
Revised: Appendix E shows a more detailed analysis of our results

Original: [Table]
Revised: [Table]

Original: where Supe-RL based approaches seem to not suffer from detrimental network initialization seeds
Revised: where Supe-RL based approaches seem to not suffer from detrimental network initialization seeds

Original: [Table]
Revised: [Table]

Original: To further conﬁrm our claims on the beneﬁcial transfer of information of mixed approaches
Revised: To further conﬁrm our claims on the beneﬁcial transfer of information of mixed approaches

Original: [Table]
Revised: [Table]

Original: we employ a formal veriﬁcation tool, Neurify (Wang et al., 2018b)
Revised: we employ a formal veriﬁcation tool Corsi et al.

Original: [Table] to verify the behavior of our trained models with respect to a series of safety properties.
Revised: (2020), to verify the behavior of our trained models with respect to a series of safety properties.

Original: The idea behind this evaluation is to conﬁrm that models trained with mixed approaches lead to more robust policies.
Revised: The idea behind this evaluation is to conﬁrm that models trained with mixed approaches lead to more robust policies.

Original: Given the inferior performance of ERL, we also expect that Supe-RL based models will present fewer conﬁgurations that cause undesirable behaviors (i [Table] e., violates the safety property).
Revised: Given the inferior performance of ERL, we also expect that Supe-RL based models will present fewer conﬁgurations that cause undesirable behaviors (i.e., violates the safety property).

Original: According to the navigation scenarios, we selected the following safety properties for the I indoor and A aquatic navigation tasks:
Revised: According to the navigation scenarios, we selected the following safety properties for the I indoor and A aquatic navigation tasks:

Original: Θ I, 0 : If TurtleBot3 has obstacles too close on the right and on the front, it must turn left.
Revised: Θ I, 0 : If TurtleBot3 has obstacles too close on the right and on the front, it must turn left.

Original: Θ I, 1 : If TurtleBot3 has obstacles too close on the left and on the front, it must turn right.
Revised: Θ I, 1 : If TurtleBot3 has obstacles too close on the left and on the front, it must turn right.

Original: Θ I, 2 : If TurtleBot3 has obstacles too close on the front, it must turn in any of the other directions.
Revised: Θ I, 2 : If TurtleBot3 has obstacles too close on the front, it must turn in any of the other directions.

Original: Θ A, 0 : If the aquatic drone has a target on the right, it must turn right.
Revised: Θ A, 0 : If the aquatic drone has a target on the right, it must turn right.

Original: Θ A, 1 : If the aquatic drone has a target on the left, it must turn left.
Revised: Θ A, 1 : If the aquatic drone has a target on the left, it must turn left.

Original: Table 2 shows violation percentage, computation time, and memory returned by the veriﬁcation tool, to test our safety properties.
Revised: Table 2 shows violation percentage, computation time, and memory returned by the veriﬁcation tool, to test our safety properties.

Original: In detail, models trained with mixed approaches (i.e., SGRainbow, GPPO, and ERL) present fewer violations in every considered property.
Revised: In detail, models trained with mixed approaches (i.e., SGRainbow, GPPO, and ERL) present fewer violations in every considered property.

Original: Furthermore, there is also a signiﬁcant improvement over the computation time and memory required by the veriﬁer.
Revised: Furthermore, there is also a signiﬁcant improvement over the computation time and memory required by the veriﬁer.

Original: This conﬁrms our claims on the policy improvement of mixed approaches as they evaluate with a signiﬁcant difference the output values, which translates into fewer bounds re-computations for the veriﬁer.
Revised: This conﬁrms our claims on the policy improvement of mixed approaches as they evaluate with a signiﬁcant difference the output values, which translates into fewer bounds re-computations for the veriﬁer.

Original: Crucially, the superior safety performance of Supe-RL based approaches compared to ERL, further motivate the introduction of our framework.
Revised: Crucially, the superior safety performance of Supe-RL based approaches compared to ERL, further motivate the introduction of our framework.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We presented Supe-RL, a novel mixed framework that exploits the robustness of population-based
Revised: We presented Supe-RL, a novel mixed framework that exploits the robustness of population-based

Original: GA to improve value-based and policy-gradient DRL agents.
Revised: GA to improve value-based and policy-gradient DRL agents.

Original: We evaluate Supe-RL in two mapless navigation scenarios: an indoor navigation for a TurtleBot3 platform, an aquatic navigation task, and in locomotion benchmarks.
Revised: We evaluate Supe-RL in two mapless navigation scenarios: an indoor navigation for a TurtleBot3 platform, an aquatic navigation task, and in locomotion benchmarks.

Original: Our empirical evaluation shows that Supe-RL signiﬁcantly outperforms
Revised: Our empirical evaluation shows that Supe-RL signiﬁcantly outperforms

Original: DRL baselines (Rainbow and PPO), the GA, and the recent ERL and PDERL, which also show poor performance when combined with value-based DRL.
Revised: DRL baselines (Rainbow and PPO), the GA, and the recent ERL and PDERL, which also show poor performance when combined with value-based DRL.

Original: Crucially, Supe-RL is the ﬁrst framework that combines GA and DRL in the ﬁeld of value-based discrete methods.
Revised: Crucially, Supe-RL is the ﬁrst framework that combines GA and DRL in the ﬁeld of value-based discrete methods.

Original: Furthermore, we exploited a formal veriﬁcation tool to conﬁrm the beneﬁcial effects of mixed approaches in policy improvement.
Revised: Furthermore, we exploited a formal veriﬁcation tool to conﬁrm the beneﬁcial effects of mixed approaches in policy improvement.

Original: This evaluation conﬁrms the superior performance of Supe-RL based approaches, that returned safer models.
Revised: This evaluation conﬁrms the superior performance of Supe-RL based approaches, that returned safer models.

Original: This work paves the way for several interesting research directions which include exploiting complex evolutionary mechanisms and different crossover techniques to further improve our framework as well as the possibility of extending Supe-RL to concurrently optimize the total reward and the desired safety properties.
Revised: This work paves the way for several interesting research directions which include exploiting complex evolutionary mechanisms and different crossover techniques to further improve our framework as well as the possibility of extending Supe-RL to concurrently optimize the total reward and the desired safety properties.

Original: A I NTERVAL A NALYSIS FOR F ORMAL V ERIFICATION
Revised: A I NTERVAL A NALYSIS FOR F ORMAL V ERIFICATION

Original: We consider formal veriﬁcation to support our claims on the beneﬁcial effects of our genetic component on the policy.
Revised: We consider formal veriﬁcation to support our claims on the beneﬁcial effects of our genetic component on the policy.

Original: We brieﬂy describe the main veriﬁcation concept and refer the interested reader to the references provided.
Revised: We brieﬂy describe the main veriﬁcation concept and refer the interested reader to the references provided.

Original: Recently, Wang et al.
Revised: Recently, Wang et al.

Original: (2018a)

Original: ; Singh et al.
Revised: ; Singh et al.

Original: (2019) propose an extension of interval analysis for the veriﬁcation of neural networks.
Revised: (2019) propose an extension of interval analysis for the veriﬁcation of neural networks.

Original: Given a set of input bounds (area) and a safety property to verify, these methods propagate the input, layer by layer, to compute the bounds for each output node.
Revised: Given a set of input bounds (area) and a safety property to verify, these methods propagate the input, layer by layer, to compute the bounds for each output node.

Original: Afterwards, they exploit a standard formalization from interval algebra (Moore, 1963): y 

Revised: Afterwards, they exploit a standard formalization from interval algebra (Moore, 1963): y

Original: =
Revised: =

Original: [ a, b ] < y 1 =
Revised: [ a, b ] < y 1 =

Original: [ c, d ] ⇒ b < c to prove or deny properties.
Revised: [ c, d ] ⇒ b < c to prove or deny properties.

Original: In decision-making scenarios, such as our navigation, we can verify a property by analyzing the relationships among the computed output bounds.
Revised: In decision-making scenarios, such as our navigation, we can verify a property by analyzing the relationships among the computed output bounds.

Original: To simplify the understanding of this process, we visualize it as a 2d graph in Figure 6 which shows the possible scenarios after computing the bounds for a decision-making task: (i) there is an overlap between output bounds; (ii) an output bound that satisfy the property is always higher than the others; (iii) an output bound corresponding to an action that violates the property is greater than the others.
Revised: To simplify the understanding of this process, we visualize it as a 2d graph in Figure 6 which shows the possible scenarios after computing the bounds for a decision-making task: (i) there is an overlap between output bounds; (ii) an output bound that satisfy the property is always higher than the others; (iii) an output bound corresponding to an action that violates the property is greater than the others.

Original: In the last two cases, we can conclude that the property is either veriﬁed or denied in the input set.
Revised: In the last two cases, we can conclude that the property is either veriﬁed or denied in the input set.

Original: However, the ﬁrst case requires to re-compute the output bounds in smaller input subsets (Figure 6 shows the bounds on three smaller subsets on the x-axis) to obtain a more precise estimation that falls in one of the other veriﬁable cases.
Revised: However, the ﬁrst case requires to re-compute the output bounds in smaller input subsets (Figure 6 shows the bounds on three smaller subsets on the x-axis) to obtain a more precise estimation that falls in one of the other veriﬁable cases.

Original: In summary, by exploiting formal veriﬁcation (and speciﬁcally Wang et al.(2018b)) we can measure the number of input conﬁgurations that would cause a violation of our safety properties.
Revised: In summary, by exploiting formal veriﬁcation (and speciﬁcally Corsi et al. we can measure the number of input conﬁgurations that would cause a violation of our safety properties.

Original: We measure this as the percentage of area that may cause a violation with respect to the entire input area (i.e., the violation percentage).
Revised: We measure this as the percentage of area that may cause a violation with respect to the entire input area (i.e., the violation percentage).

Original: The lower this number, the more robust is the policy (i.e., properties may be violated by a lower amount of input conﬁgurations).
Revised: The lower this number, the more robust is the policy (i.e., properties may be violated by a lower amount of input conﬁgurations).

Original: This translates, de facto, in an overall better policy as the trained network correctly evaluate with a signiﬁcant difference the output values (output bounds have very different values), i.e., the action to choose.
Revised: This translates, de facto, in an overall better policy as the trained network correctly evaluate with a signiﬁcant difference the output values (output bounds have very different values), i.e., the action to choose.

Original: In detail, fewer bounds re-computation (i.e., fewer overlaps) means faster computation and less required memory.
Revised: In detail, fewer bounds re-computation (i.e., fewer overlaps) means faster computation and less required memory.

Original: B S UPE -RL P SEUDOCODE
Revised: B S UPE -RL P SEUDOCODE

Original: We introduce a general pseudocode of Supe-RL based algorithms.
Revised: We introduce a general pseudocode of Supe-RL based algorithms.

Original: Algorithm 1 Supe-RL 1: Input: size N of the population P , periodicity GE for the genetic evaluation, seed s , timeout
Revised: Algorithm 1 Supe-RL 1: Input: size N of the population P , periodicity GE for the genetic evaluation, seed s , timeout

Original: T .
Revised: T .

Original: Initialize network weights θ a , replay buffer R and env with seed s .
Revised: Initialize network weights θ a , replay buffer R and env with seed s .

Original: 2: for episode = 1 to ∞ do 3: for t = 1 to T do 4:
Revised: 2: for episode = 1 to ∞ do 3: for t = 1 to T do 4:

Original: Following the algorithm policy, select action a t 5:
Revised: Following the algorithm policy, select action a t 5:

Original: Execute action a t , observe reward r t and state s t +1 , and Store ( s t , a t , r t , s t +1 ) in R 6: end for 7: env ← ResetEnvironment() 8:
Revised: Execute action a t , observe reward r t and state s t +1 , and Store ( s t , a t , r t , s t +1 ) in R 6: end for 7: env ← ResetEnvironment() 8:

Original: Sample a random mini-batch of transitions from R 9:
Revised: Sample a random mini-batch of transitions from R 9:

Original: Perform a gradient descent step, updating θ a 10: if episode % GE =
Revised: Perform a gradient descent step, updating θ a 10: if episode % GE =

Original: = 

Revised: =

Original: then 11:
Revised: then 11:

Original: Start periodical genetic evaluation, in parallel 12:
Revised: Start periodical genetic evaluation, in parallel 12:

Original: P, Genvs ← GenerateChildren( N , θ a , env ) 13:
Revised: P, Genvs ← GenerateChildren( N , θ a , env ) 13:

Original: Score ← GeneticEvaluation( P, Genvs ) 14: θ best ← Select best θ ∈ P based on Score 15: θ a ← GeneticUpdate
Revised: Score ← GeneticEvaluation( P, Genvs ) 14: θ best ← Select best θ ∈ P based on Score 15: θ a ← GeneticUpdate

Original: ( θ a , θ best ) 16:
Revised: ( θ a , θ best ) 16:

Original: End periodical genetic evaluation 17: end if 18: end for
Revised: End periodical genetic evaluation 17: end if 18: end for

Original: 1: procedure GenerateChildren( N , θ a , env ) 2: Children ← Initialize  N +  copies  θ a 3: Cenvs ← Initialize  N + 1 copies  env 4: for  to N do 5: for 
Revised: 1: procedure GenerateChildren( N , θ a , env ) 2: Children ← Initialize with N + 1 copies of θ a 3: Cenvs ← Initialize with N + 1 copies of env 4: for i = 1 to N do 5: for iteration

Original:  1 to mut p * 
Revised: = 1 to mut p * | Children

Original: Children i ]  do 6:
Revised: [ i ] | do 6:

Original: Randomly sample indices r, c for Children [ i ] 7:
Revised: Randomly sample indices r, c for Children [ i ] 7:

Original: Children [ i ][ r, c ]+ = N (0 , mut v ) 8: end for 9: end for 10: Return Children , Cenvs
Revised: Children [ i ][ r, c ]+ = N (0 , mut v ) 8: end for 9: end for 10: Return Children , Cenvs

Original: C A DDITIONAL SGR AINBOW E XPERIMENTS
Revised: C A DDITIONAL SGR AINBOW E XPERIMENTS

Original: In this section, we show the performance of GRainbow with SGD and Adam and the results of the tuning of τ (cid:48) parameter for SGRainbow.
Revised: In this section, we show the performance of GRainbow with SGD and Adam and the results of the tuning of τ (cid:48) parameter for SGRainbow.

Original: We also present an ablation experiment on the inﬂuence of the population experiences in the buffer of the DRL agent.
Revised: We also present an ablation experiment on the inﬂuence of the population experiences in the buffer of the DRL agent.

Original: Adam and SGD GRainbow Tessler et al.
Revised: Adam and SGD GRainbow Tessler et al.

Original: (2019) improved the original DDQN evaluating the performance of the agent model over its target model.
Revised: (2019) improved the original DDQN evaluating the performance of the agent model over its target model.

Original: In particular, they show that a more stable training phase can be obtained replacing the target model with the agent weights when the latter statistically perform better than the former.
Revised: In particular, they show that a more stable training phase can be obtained replacing the target model with the agent weights when the latter statistically perform better than the former.

Original: Inspired by this approach, when our genetic evaluation ﬁnds a better child, the main agent is soft updated towards the best genome, and the latter substitutes the target model.
Revised: Inspired by this approach, when our genetic evaluation ﬁnds a better child, the main agent is soft updated towards the best genome, and the latter substitutes the target model.

Original: As Figure 7A shows, this precaution allows us to use GRainbow with Adam and outperform the manually tuned GRainbow with SGD.
Revised: As Figure 7A shows, this precaution allows us to use GRainbow with Adam and outperform the manually tuned GRainbow with SGD.

Original: Tuning τ (cid:48) for SGRainbow We further improved GRainbow performance, exploiting the soft update (Lillicrap et al., 2015) developed to improve the naive approach in DDQN which periodically substitutes the target with the agent model.
Revised: Tuning τ (cid:48) for SGRainbow We further improved GRainbow performance, exploiting the soft update (Lillicrap et al., 2015) developed to improve the naive approach in DDQN which periodically substitutes the target with the agent model.

Original: We performed multiple trials with different seeds to ﬁnd the best value of τ (cid:48) for the soft genetic update function that characterizes SGRainbow.
Revised: We performed multiple trials with different seeds to ﬁnd the best value of τ (cid:48) for the soft genetic update function that characterizes SGRainbow.

Original: In particular,
Revised: In particular,

Original: Figure 7B clearly shows that SGRainbow outperforms GRainbow and reports the average success rate of our training phases with τ (cid:48) ∈ [0 . 15 , 0 . 3 , 0 . 6] .
Revised: Figure 7B clearly shows that SGRainbow outperforms GRainbow and reports the average success rate of our training phases with τ (cid:48) ∈ [0 . 15 , 0 . 3 , 0 . 6] .

Original: Population Experiences in SGRainbow Finally, given our best performing algorithm (i.e.,
Revised: Population Experiences in SGRainbow Finally, given our best performing algorithm (i.e.,

Original: SGRainbow with τ (cid:48)
Revised: SGRainbow with τ (cid:48)

Original: = 0 . 3 ), we performed an ablation experiment to test the inﬂuence of the experiences collected in the genetic evaluation phase in the main agent buffer.
Revised: = 0 . 3 ), we performed an ablation experiment to test the inﬂuence of the experiences collected in the genetic evaluation phase in the main agent buffer.

Original: Figure 7C shows performance improvement with the introduction of the best child experiences.
Revised: Figure 7C shows performance improvement with the introduction of the best child experiences.

Original: D H YPER - PARAMETERS
Revised: D H YPER - PARAMETERS

Original: Genetic Evaluation We use the same genetic hyper-parameters for both SGRainbow and GPPO for the genetic evaluation.
Revised: Genetic Evaluation We use the same genetic hyper-parameters for both SGRainbow and GPPO for the genetic evaluation.

Original: In particular, the size N of the population P is set to 10 and the number of trials conducted in the evaluation phase to compute the ﬁtness score ranges from 10 to 20 across tasks.
Revised: In particular, the size N of the population P is set to 10 and the number of trials conducted in the evaluation phase to compute the ﬁtness score ranges from 10 to 20 across tasks.

Original: The mutation probability of a network weight is mut p = { 0 .
Revised: The mutation probability of a network weight is mut p = { 0 .

Original: based on the current success rate of the drl a and a constant mutation value mut v = 0 . 1 .
Revised: based on the current success rate of the drl a and a constant mutation value mut v = 0 . 1 .

Original: Rainbow, GRainbow, and SGRainbow Here we discuss only the hyper-parameters of the algorithmic implementations which are relevant to GRainbow and SGRainbow, referring to the original papers for further details (notice that the three approaches share the same hyper-parameters).
Revised: Rainbow, GRainbow, and SGRainbow Here we discuss only the hyper-parameters of the algorithmic implementations which are relevant to GRainbow and SGRainbow, referring to the original papers for further details (notice that the three approaches share the same hyper-parameters).

Original: Based on our experiments, we decided to use a Priority Experience Replay (Schaul et al., 2016) buffer of size 30000 with a batch size of 64 , α = 0 .
Revised: Based on our experiments, we decided to use a Priority Experience Replay (Schaul et al., 2016) buffer of size 30000 with a batch size of 64 , α = 0 .

Original: 6 , β start = 0 .
Revised: 6 , β start = 0 .

Original: 4 and β increment = 0 .
Revised: 4 and β increment = 0 .

Original: The soft update of the target network (Lillicrap et al., 2015) for Rainbow is performed with τ = 0 .
Revised: The soft update of the target network (Lillicrap et al., 2015) for Rainbow is performed with τ = 0 .

Original: Furthermore, given the well-structured reward function, we notice that a simple (cid:15) − greedy exploration strategy with decay = 0 .
Revised: Furthermore, given the well-structured reward function, we notice that a simple (cid:15) − greedy exploration strategy with decay = 0 .

Original: 99 and (cid:15) min = 0 .
Revised: 99 and (cid:15) min = 0 .

Original: 02 led to a faster training phase compared to the introduction of noisy exploration in the network (Fortunato et al., 2017).
Revised: 02 led to a faster training phase compared to the introduction of noisy exploration in the network (Fortunato et al., 2017).

Original: Finally, we noticed that a genetic evaluation collects on average 30000 total transitions, and adding a random 10% of these transitions into the priority buffer (i.e., trans p = 0 . 1 ), showed the best results.
Revised: Finally, we noticed that a genetic evaluation collects on average 30000 total transitions, and adding a random 10% of these transitions into the priority buffer (i.e., trans p = 0 . 1 ), showed the best results.

Original: PPO, GPPO, and ERL First, PPO presents two versions and we choose to implement the clipped one, which was recommended in the original work (Schulman et al., 2017), to which we remind the interested reader for further details.
Revised: PPO, GPPO, and ERL First, PPO presents two versions and we choose to implement the clipped one, which was recommended in the original work (Schulman et al., 2017), to which we remind the interested reader for further details.

Original: We choose to use a buffer of 256 elements with mini-batches of size 64 for the considered algorithms, while we used the original ERL (Khadka & Tumer, 2018) for speciﬁc algorithm parameters.
Revised: We choose to use a buffer of 256 elements with mini-batches of size 64 for the considered algorithms, while we used the original ERL (Khadka & Tumer, 2018) for speciﬁc algorithm parameters.

Original: E R OBUSTNESS OF S UPE -RL
Revised: E R OBUSTNESS OF S UPE -RL

Original: A more detailed analysis of our results reveals that the high variance of GA, PPO, and ERL is due to a speciﬁc seed that results in a pathological performance of the approach (a similar phenomenon occurs for Rainbow).
Revised: A more detailed analysis of our results reveals that the high variance of GA, PPO, and ERL is due to a speciﬁc seed that results in a pathological performance of the approach (a similar phenomenon occurs for Rainbow).

Original: In particular, Figure 8A, B, C, D report the success rate and total reward of the only bad initialization seed that causes GA, Rainbow, PPO, and ERL to perform poorly, compared to the previous runs.
Revised: In particular, Figure 8A, B, C, D report the success rate and total reward of the only bad initialization seed that causes GA, Rainbow, PPO, and ERL to perform poorly, compared to the previous runs.

Original: On the other hand, GPPO, GRainbow, and SGRainbow do not present this issue, reaching similar results to the other training phases.
Revised: On the other hand, GPPO, GRainbow, and SGRainbow do not present this issue, reaching similar results to the other training phases.

Original: Even if this analysis is not as relevant as previous results, as it is not averaged over multiple seeds, we do believe that this is an important aspect of our framework and will need further investigation.
Revised: Even if this analysis is not as relevant as previous results, as it is not averaged over multiple seeds, we do believe that this is an important aspect of our framework and will need further investigation.

Original: Crucially, Figure 9A, B, C, D show that SGRainbow, and GPPO perform better than other approaches even in the selected runs without such detrimental seed.
Revised: Crucially, Figure 9A, B, C, D show that SGRainbow, and GPPO perform better than other approaches even in the selected runs without such detrimental seed.