Original: S OCIAL N ETWORK S TRUCTURE S HAPES I NNOVATION : E XPERIENCE SHARING IN RL WITH SAPIENS
Revised: S OCIAL N ETWORK S TRUCTURE S HAPES I NNOVATION : E XPERIENCE SHARING IN RL WITH SAPIENS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The human cultural repertoire relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones.
Revised: The human cultural repertoire relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones.

Original: Innovation is not solitary, it relies on collective accumulation and merging of previous solutions.
Revised: Innovation is not solitary, it relies on collective accumulation and merging of previous solutions.

Original: Reinforcement learning approaches commonly assume that fully-connected topologies are best suited for innovation.
Revised: Reinforcement learning approaches commonly assume that fully-connected topologies are best suited for innovation.

Original: However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures.
Revised: However, human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures.

Original: In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others.
Revised: In dynamic settings, humans oscillate between innovating individually or in small clusters, and then sharing outcomes with others.

Original: To our knowledge, the role of social network structure on innovation has not been systematically studied in reinforcement learning.
Revised: To our knowledge, the role of social network structure on innovation has not been systematically studied in reinforcement learning.

Original: It remains unclear a) which social network structures are optimal for which innovation tasks, and b) which properties of experience sharing improve multi-level innovation.
Revised: It remains unclear a) which social network structures are optimal for which innovation tasks, and b) which properties of experience sharing improve multi-level innovation.

Original: Here we use a multi-level problem setting (WordCraft), with three different innovation tasks.
Revised: Here we use a multi-level problem setting (WordCraft), with three different innovation tasks.

Original: We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully connected, small world, dynamic, ring).
Revised: We systematically design networks of DQNs sharing experiences from their replay buffers in varying structures (fully connected, small world, dynamic, ring).

Original: Comparing the level of innovation achieved by different experience-sharing social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation across tasks.
Revised: Comparing the level of innovation achieved by different experience-sharing social network structures across different tasks shows that, first, consistent with human findings, experience sharing within a dynamic structure achieves the highest level of innovation across tasks.

Original: Second, experience sharing is not as helpful when there is a single clear path to innovation.
Revised: Second, experience sharing is not as helpful when there is a single clear path to innovation.

Original: Third, two metrics we propose, conformity and diversity of shared experience, can help understand the success of different social network structures on different tasks.
Revised: Third, two metrics we propose, conformity and diversity of shared experience, can help understand the success of different social network structures on different tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unlike herds or swarms, human social networks solve different tasks with topologies (Momennejad, 2022).
Revised: Unlike herds or swarms, human social networks solve different tasks with topologies (Momennejad, 2022).

Original: Human and computational studies show that, to optimize the performance of collective search by choosing who should communicate with whom, one needs to take into account the inter-play between task and structure properties: social network structures with high connectivity are better suited for quick convergence in problems with clear global optima (Coman et al., 2016;
Revised: Human and computational studies show that, to optimize the performance of collective search by choosing who should communicate with whom, one needs to take into account the inter-play between task and structure properties: social network structures with high connectivity are better suited for quick convergence in problems with clear global optima (Coman et al., 2016;

Original: Momennejad et al., 2019) while partially-connected structures perform best in deceptive tasks, where acting greedily in the short-term leads to missing the optimal solution (Derex & Boyd, 2016; Lazer & Friedman, 2007;
Revised: Momennejad et al., 2019) while partially-connected structures perform best in deceptive tasks, where acting greedily in the short-term leads to missing the optimal solution (Derex & Boyd, 2016; Lazer & Friedman, 2007;

Original: Cantor et al., 2021; Du et al., 2021; Adjodah et al., 2019).
Revised: Cantor et al., 2021; Du et al., 2021; Adjodah et al., 2019).

Original: Despite this evidence, works in distributed reinforcement learning (RL) have focused on fully-connected architectures (Mnih et al., 2016; Horgan et al., 2018; Espeholt et al., 2018;
Revised: Despite this evidence, works in distributed reinforcement learning (RL) have focused on fully-connected architectures (Mnih et al., 2016; Horgan et al., 2018; Espeholt et al., 2018;

Original: Nair et al., 2015; Christianos et al., 2020;
Revised: Nair et al., 2015; Christianos et al., 2020;

Original: Schmitt et al., 2019; Jaderberg et al., 2018).
Revised: Schmitt et al., 2019; Jaderberg et al., 2018).

Original: Here, we test the performance of different social network structures in groups of RL agents that share their individual experiences in a distributed RL learning paradigm.
Revised: Here, we test the performance of different social network structures in groups of RL agents that share their individual experiences in a distributed RL learning paradigm.

Original: We refer to such groups as multi-agent topologies, introduce SAPIENS , a learning framework for
Revised: We refer to such groups as multi-agent topologies, introduce SAPIENS , a learning framework for

Original: Structuring multi-Agent toPologies for Innovation through ExperieNce Sharing 1 , and evaluate it on a deceptive task that models collective innovation.
Revised: Structuring multi-Agent toPologies for Innovation through ExperieNce Sharing 1 , and evaluate it on a deceptive task that models collective innovation.

Original: Innovations represent the expansion of an agent’s behavioral repertoire with new problem-solving abilities and are, therefore, a necessary ingredient of continuous learning (Leibo et al., 2019).
Revised: Innovations represent the expansion of an agent’s behavioral repertoire with new problem-solving abilities and are, therefore, a necessary ingredient of continuous learning (Leibo et al., 2019).

Original: The solitary genius narrative often obscures the collective nature of human innovations, which arise from tinkering, recombination and adoption of existing innovations (Sol´e et al., 2013; Derex &
Revised: The solitary genius narrative often obscures the collective nature of human innovations, which arise from tinkering, recombination and adoption of existing innovations (Sol´e et al., 2013; Derex &

Original: Boyd, 2016).
Revised: Boyd, 2016).

Original: Cognitive science and ecology studies of human innovation characterize it as a type of combinatorial search constrained by the semantics dictating the feasible combinations of innovation (Sol´e et al., 2013; Derex & Boyd, 2016).
Revised: Cognitive science and ecology studies of human innovation characterize it as a type of combinatorial search constrained by the semantics dictating the feasible combinations of innovation (Sol´e et al., 2013; Derex & Boyd, 2016).

Original: We adopt this definition, so that innovations are a type of collective search task with : a) a multi-level search space, where innovations arise out of recombination of existing ones(Hafner, 2021) b) rewards that increase monotonically with the level of innovation, in order to capture the human intrinsic motivation for progress (Sol´e et al., 2013).
Revised: We adopt this definition, so that innovations are a type of collective search task with : a) a multi-level search space, where innovations arise out of recombination of existing ones(Hafner, 2021) b) rewards that increase monotonically with the level of innovation, in order to capture the human intrinsic motivation for progress (Sol´e et al., 2013).

Original: Laboratory and field studies of human groups have shown that collective innovation is highly contingent on the social network structure (Momennejad, 2022; Migliano & Vinicius, 2022; Derex &
Revised: Laboratory and field studies of human groups have shown that collective innovation is highly contingent on the social network structure (Momennejad, 2022; Migliano & Vinicius, 2022; Derex &

Original: Boyd, 2016).
Revised: Boyd, 2016).

Original: The reason for this lies in the exploration versus exploitation dynamics of social networks.
Revised: The reason for this lies in the exploration versus exploitation dynamics of social networks.

Original: High clustering and long shortest path in partially-connected structures help maintain diversity in the collective at the benefit of exploration, while high connectivity quickly leads to conformity, which benefits exploitation (Lazer & Friedman, 2007).
Revised: High clustering and long shortest path in partially-connected structures help maintain diversity in the collective at the benefit of exploration, while high connectivity quickly leads to conformity, which benefits exploitation (Lazer & Friedman, 2007).

Original: Of particular interest are social network structures that achieve a balance in this trade-off: small-worlds are static graphs that, due a modular structure with long-range connections, achieve both high clustering and small shortest path Watts & Strogatz (1998).
Revised: Of particular interest are social network structures that achieve a balance in this trade-off: small-worlds are static graphs that, due a modular structure with long-range connections, achieve both high clustering and small shortest path Watts & Strogatz (1998).

Original: Another example are dynamic structures , where agents are able to periodically switch neighbors (Volz & Meyers, 2007).
Revised: Another example are dynamic structures , where agents are able to periodically switch neighbors (Volz & Meyers, 2007).

Original: These two families of graphs have the attractive property that they both locally protect innovations, thus maintaining diversity, and quickly disseminate good solutions (Derex & Boyd, 2016).
Revised: These two families of graphs have the attractive property that they both locally protect innovations, thus maintaining diversity, and quickly disseminate good solutions (Derex & Boyd, 2016).

Original: Despite progress on multiple fronts, many open questions remain before we get a clear understanding of how social network structure shapes innovation.
Revised: Despite progress on multiple fronts, many open questions remain before we get a clear understanding of how social network structure shapes innovation.

Original: On the cognitive science side, computational and human laboratory studies of collective innovation are few and have studied a single type of innovation task where two innovations are combined to create a new one (Derex & Boyd, 2016; Cantor et al., 2021), while a big part of the literature has studied other types of collective search that do not resemble innovation (Mason & Watts, 2012;
Revised: On the cognitive science side, computational and human laboratory studies of collective innovation are few and have studied a single type of innovation task where two innovations are combined to create a new one (Derex & Boyd, 2016; Cantor et al., 2021), while a big part of the literature has studied other types of collective search that do not resemble innovation (Mason & Watts, 2012;

Original: Mason et al., 2008; Lazer & Friedman, 2007; Fang et al., 2010).
Revised: Mason et al., 2008; Lazer & Friedman, 2007; Fang et al., 2010).

Original: Furthermore, laboratory studies have collected purely behavioral data (Mason et al., 2008; Derex & Boyd, 2016), while studies of collective memory have shown significant influence of social interactions on individual cognition (Coman et al., 2016).
Revised: Furthermore, laboratory studies have collected purely behavioral data (Mason et al., 2008; Derex & Boyd, 2016), while studies of collective memory have shown significant influence of social interactions on individual cognition (Coman et al., 2016).

Original: This lack of mnemonic data makes it hard to further analyze the effect of social network structure on collective search.
Revised: This lack of mnemonic data makes it hard to further analyze the effect of social network structure on collective search.

Original: On the side of the distributed RL community, studies are hypothesizing that the reason why groups outperform single agents, not just in terms of speed, but also in terms of final performance is the increased diversity of experiences collected by heterogeneous agents (Nair et al., 2015; Horgan et al., 2018).
Revised: On the side of the distributed RL community, studies are hypothesizing that the reason why groups outperform single agents, not just in terms of speed, but also in terms of final performance is the increased diversity of experiences collected by heterogeneous agents (Nair et al., 2015; Horgan et al., 2018).

Original: Nevertheless, studies do not measure this diversity.
Revised: Nevertheless, studies do not measure this diversity.

Original: Tn this case two steps seem natural: introducing appropriate metrics of diversity and increasing it, not only through heterogeneity, but also through the social network topology.
Revised: Tn this case two steps seem natural: introducing appropriate metrics of diversity and increasing it, not only through heterogeneity, but also through the social network topology.

Original: To achieve this we propose SAPIENS , a distributed RL learning framework for modeling a group of agents acting in their own copies of their environment and exchanging experiences according to a social network topology.
Revised: To achieve this we propose SAPIENS , a distributed RL learning framework for modeling a group of agents acting in their own copies of their environment and exchanging experiences according to a social network topology.

Original: We study instantiations of SAPIENS where multiple DQN learners (Mnih et al., 2013) share experience tuples from their replay buffers with their neighbors in different static and dynamic social network structures.
Revised: We study instantiations of SAPIENS where multiple DQN learners (Mnih et al., 2013) share experience tuples from their replay buffers with their neighbors in different static and dynamic social network structures.

Original: We employ Wordcraft (Jiang et al., 2020b) as a test-bed and design three custom tasks (Figures 1 and 2) covering innovation challenges of different complexity: (i) a task with a single innovation path to an easy-to-find global optimum.
Revised: We employ Wordcraft (Jiang et al., 2020b) as a test-bed and design three custom tasks (Figures 1 and 2) covering innovation challenges of different complexity: (i) a task with a single innovation path to an easy-to-find global optimum.

Original: This type of task can be used to model a linear innovation structure, such as the evolution of the fork from knife to having, two-, three- and, eventually four tines.
Revised: This type of task can be used to model a linear innovation structure, such as the evolution of the fork from knife to having, two-, three- and, eventually four tines.

Original: (Sol´e et al., 2013) and is not a deceptive task.
Revised: (Sol´e et al., 2013) and is not a deceptive task.

Original: (ii) a task with two paths that individually lead to local optima, but when combined, can merge toward the global optimum.
Revised: (ii) a task with two paths that individually lead to local optima, but when combined, can merge toward the global optimum.

Original: With this task, inspired from previous studies in cognitive science (Derex & Boyd, 2016;
Revised: With this task, inspired from previous studies in cognitive science (Derex & Boyd, 2016;

Original: Cantor et al., 2021) we can capture innovations that were repurposed after their invention, such as
Revised: Cantor et al., 2021) we can capture innovations that were repurposed after their invention, such as

Original: Gutenberg’s screw press leading to the print press (Sol´e et al., 2013).
Revised: Gutenberg’s screw press leading to the print press (Sol´e et al., 2013).

Original: (iii) a task with ten paths, only one of which leads to the global optimum, which captures search in vast spaces..
Revised: (iii) a task with ten paths, only one of which leads to the global optimum, which captures search in vast spaces..

Original: In addition to the two deceptive tasks in Wordcraft, we also evaluate SAPIENS algorithms on a deceptive task implemented in a grid world.
Revised: In addition to the two deceptive tasks in Wordcraft, we also evaluate SAPIENS algorithms on a deceptive task implemented in a grid world.

Original: We empirically show that the performance of SAPIENS depends on the inter-play between social network structure and task demands.
Revised: We empirically show that the performance of SAPIENS depends on the inter-play between social network structure and task demands.

Original: Dynamic structures perform most robustly, exhibiting an impressive ability to converge quickly in the easy task, avoid local optima in the second task, and explore efficiently in the third task.
Revised: Dynamic structures perform most robustly, exhibiting an impressive ability to converge quickly in the easy task, avoid local optima in the second task, and explore efficiently in the third task.

Original: To interpret these findings, we propose and compute novel behavioral and mnemonic metrics that, among others, quantify diversity and conformity (Momennejad et al., 2019) and compare the performance of SAPIENS with other distributed RL algorithms (Mnih et al., 2016; Nair et al., 2015).
Revised: To interpret these findings, we propose and compute novel behavioral and mnemonic metrics that, among others, quantify diversity and conformity (Momennejad et al., 2019) and compare the performance of SAPIENS with other distributed RL algorithms (Mnih et al., 2016; Nair et al., 2015).

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: Contributions Our contributions are two-fold.
Revised: Contributions Our contributions are two-fold.

Original: From a neuroscience and cognitive science perspective , SAPIENS is, to our knowledge, the first computational study of hypotheses derived with humans regarding the effect of social network structure on collective innovation, that employs deep
Revised: From a neuroscience and cognitive science perspective , SAPIENS is, to our knowledge, the first computational study of hypotheses derived with humans regarding the effect of social network structure on collective innovation, that employs deep

Original: RL as the individual learning mechanism.
Revised: RL as the individual learning mechanism.

Original: Compared to previous computational studies that employed simple agent-based models and evolutionary search (Lazer & Friedman, 2007; Cantor et al., 2021; Mason & Watts, 2012), deep RL provides three main advantages : i) it enables empirical experiments with more complex test-beds and larger search spaces ii) agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents iii) by using the replay buffer as a proxy of the memories of agents, we can directly measure properties such as diversity and alignment of experiences that are challenging to estimate with humans.
Revised: Compared to previous computational studies that employed simple agent-based models and evolutionary search (Lazer & Friedman, 2007; Cantor et al., 2021; Mason & Watts, 2012), deep RL provides three main advantages : i) it enables empirical experiments with more complex test-beds and larger search spaces ii) agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents iii) by using the replay buffer as a proxy of the memories of agents, we can directly measure properties such as diversity and alignment of experiences that are challenging to estimate with humans.

Original: Aside these methodological contributions, as we will see later our empirical study leads to clear hypotheses for future experiments with humans.
Revised: Aside these methodological contributions, as we will see later our empirical study leads to clear hypotheses for future experiments with humans.

Original: From an RL perspective , our work extends upon the distributed RL paradigm by systematically analyzing the effect of static and dynamic social network structures on different types of innovation tasks and analyzing both in terms of performance and novel behavioral and mnemonic metrics.
Revised: From an RL perspective , our work extends upon the distributed RL paradigm by systematically analyzing the effect of static and dynamic social network structures on different types of innovation tasks and analyzing both in terms of performance and novel behavioral and mnemonic metrics.

Original:  M ETHODS 2. W ORDCRAFT : A TEST - BED FOR INNOVATION
Revised:  M ETHODS 2. W ORDCRAFT : A TEST - BED FOR INNOVATION

Original: We perform experiments on Wordcraft (Jiang et al., 2020b), an environment inspired from the game
Revised: We perform experiments on Wordcraft (Jiang et al., 2020b), an environment inspired from the game

Original: Little Alchemy 2 2 .
Revised: Little Alchemy 2 2 .

Original: Tasks start with an initial set of elements and the player explores combinations of two elements in order to create new ones.
Revised: Tasks start with an initial set of elements and the player explores combinations of two elements in order to create new ones.

Original: Figure 1 (Left) provides an illustrative example.
Revised: Figure 1 (Left) provides an illustrative example.

Original: In Wordcraft one can create different types of tasks using a recipe book ( X valid ), which is a dictionary containing the valid element combinations with entries in the form ( x, y )
Revised: In Wordcraft one can create different types of tasks using a recipe book ( X valid ), which is a dictionary containing the valid element combinations with entries in the form ( x, y )

Original: → z , where x and y indicate the two chosen elements and z the newly crafted one.
Revised: → z , where x and y indicate the two chosen elements and z the newly crafted one.

Original: When queried with a non-valid combination X valid returns no element.
Revised: When queried with a non-valid combination X valid returns no element.

Original: In addition to the recipe book, a task requires a reward function
Revised: In addition to the recipe book, a task requires a reward function

Original: R valid that returns a scalar reward associated with crafting element z .
Revised: R valid that returns a scalar reward associated with crafting element z .

Original: A valid combination returns a new element and reward only the first time it is chosen.
Revised: A valid combination returns a new element and reward only the first time it is chosen.

Original: When queried with a non-valid combination,
Revised: When queried with a non-valid combination,

Original: R valid returns a reward of zero.
Revised: R valid returns a reward of zero.

Original: Thus, a task can be be described by a tuple ( X 0 , X valid , R valid , T ) , where X 0 denotes the initial set of elements and T is the number of time steps available to an agent before the environment resets.
Revised: Thus, a task can be be described by a tuple ( X 0 , X valid , R valid , T ) , where X 0 denotes the initial set of elements and T is the number of time steps available to an agent before the environment resets.

Original: The environment can therefore be modeled as a fully-observable
Revised: The environment can therefore be modeled as a fully-observable

Original: Markov Decision Process (MDP) that we define in detail in Appendix A.1.
Revised: Markov Decision Process (MDP) that we define in detail in Appendix A.

Original: In addition to Wordcraft, we also show the applicability of our method to another domain: a deceptive game implemented in a grid world (Appendix A.6).
Revised: In addition to Wordcraft, we also show the applicability of our method to another domain: a deceptive game implemented in a grid world (Appendix E.7).

Original: Combination
Revised: Combination

Original: I nn o v a t i o n l e v e l
Revised: I nn o v a t i o n l e v e l

Original: Result Reward ...
Revised: Result Reward ...

Original: S i n g l e p a t h
Revised: S i n g l e p a t h

Original: M e r g i n g p a t h s
Revised: M e r g i n g p a t h s

Original: B e s t - o ft e n p a t h s ...
Revised: B e s t - o ft e n p a t h s ...

Original: 14 + + = = ...
Revised: 14 + + = = ...

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: + + + + =
Revised: + + + + =

Original: = 
Revised: = = = a

Original: =
Revised: [Table] A A [Table]

Original: =
Revised: A [Table] A [Table]

Original: a
Revised: A

Original: [Table]
Revised: [Table]

Original: A

Original: A

Original: [Table]

Original: A

Original: [Table]

Original: A

Original: [Table]

Original: A

Original: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a...
Revised: a...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: a
Revised: a

Original: A A
Revised: A A

Original: A A
Revised: A A

Original: A A
Revised: A A

Original: A... aaaa...
Revised: A... aaaa...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: a
Revised: a

Original: A A
Revised: A A

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: C C
Revised: C C

Original: A A
Revised: A A

Original: [Table]
Revised: [Table]

Original: A A
Revised: A A

Original: A
Revised: A

Original: A... a
Revised: A... a

Original: [Table]
Revised: [Table]

Original: c
Revised: c

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: aa...
Revised: aa...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: b
Revised: b

Original: B
Revised: B

Original: B B
Revised: B B

Original: B B
Revised: B B

Original: B
Revised: B

Original: B... bbbb...
Revised: B... bbbb...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: j J
Revised: j J

Original: J J
Revised: J J

Original: J J
Revised: J J

Original: J
Revised: J

Original: J... jjjj
Revised: J... jjjj

Original: [Figure]
Revised: [Figure]

Original: 2. I NNOVATION TASKS
Revised: 2. I NNOVATION TASKS

Original: We design three innovation tasks, shown in Figure 2, that pose different challenges:
Revised: We design three innovation tasks, shown in Figure 2, that pose different challenges:

Original: Single innovation path The recipe book consists of an initial set of 3 elements ( X valid = { a 1 , a 2 , a 3 } ) and 8 innovation levels.
Revised: Single innovation path The recipe book consists of an initial set of 3 elements ( X valid = { a 1 , a 2 , a 3 } ) and 8 innovation levels.

Original: An agent needs to first combine two of them to create the first element and then progresses further by combining the most recently created element with an appropriate one from the initial set.
Revised: An agent needs to first combine two of them to create the first element and then progresses further by combining the most recently created element with an appropriate one from the initial set.

Original: This optimization problem contains a single global optimum.
Revised: This optimization problem contains a single global optimum.

Original: Merging paths There are two paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward to the end path A, moving forward to the end of path B, or combining elements from path A and B to progress on path C.
Revised: Merging paths There are two paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward to the end path A, moving forward to the end of path B, or combining elements from path A and B to progress on path C.

Original: The latter is more rewarding and is, thus, the optimal choice.
Revised: The latter is more rewarding and is, thus, the optimal choice.

Original: This task is particularly challenging: to explore path C
Revised: This task is particularly challenging: to explore path C

Original: the player needs to first go to lower innovation levels instead of just progressing on a single path.
Revised: the player needs to first go to lower innovation levels instead of just progressing on a single path.

Original: It, thus, exhibits two local optimum and one global optimum
Revised: It, thus, exhibits two local optimum and one global optimum

Original: Best-of-ten paths Here, one of the ten paths is the most rewarding.
Revised: Best-of-ten paths Here, one of the ten paths is the most rewarding.

Original: The optimal strategy is to move only on the most rewarding path but, to do so, the player must first explore and reject the other paths.
Revised: The optimal strategy is to move only on the most rewarding path but, to do so, the player must first explore and reject the other paths.

Original: This optimization task is characterized by a single global optimum and 9 local one.
Revised: This optimization task is characterized by a single global optimum and 9 local one.

Original: We introduce the following concepts to characterize the structure of innovation tasks.
Revised: We introduce the following concepts to characterize the structure of innovation tasks.

Original: An innovation task can contain one or more paths, which can potentially be connected to each other (as in the merging paths task).
Revised: An innovation task can contain one or more paths, which can potentially be connected to each other (as in the merging paths task).

Original: In our proposed tasks, an innovation path X is defined as a sequence of elements [ X 1 , ..., X n ] , where crafting an element
Revised: In our proposed tasks, an innovation path X is defined as a sequence of elements [ X 1 , ..., X n ] , where crafting an element

Original: X
Revised: X

Original: i ( i > 1) requires to combine the previously crafted element X i − 1 and a base element from the initial set.
Revised: i ( i > 1) requires to combine the previously crafted element X i − 1 and a base element from the initial set.

Original: The first element X 1 in the sequence requires combining elements from the initial set or from other paths.
Revised: The first element X 1 in the sequence requires combining elements from the initial set or from other paths.

Original: The innovation level of an element corresponds to the length of the path it belongs to, plus the sum of the path lengths required to craft its first item ( 0 if the first item is crafted from base elements).
Revised: The innovation level of an element corresponds to the length of the path it belongs to, plus the sum of the path lengths required to craft its first item ( 0 if the first item is crafted from base elements).

Original: For example, the innovation level of A 3 in the single path task is 3 , while the innovation level of element C 1 in the merging paths task is 5 ( 1 on C + 2 on A + 2 on B).
Revised: For example, the innovation level of A 3 in the single path task is 3 , while the innovation level of element C 1 in the merging paths task is 5 ( 1 on C + 2 on A + 2 on B).

Original: Within an innovation task, the trajectory of an agent is defined
Revised: Within an innovation task, the trajectory of an agent is defined

Original: [Figure]
Revised: [Figure]

Original: as the sequence of crafted elements it produces.
Revised: as the sequence of crafted elements it produces.

Original: Finally, the optimal trajectory of an innovation task is the trajectory that returns the highest cumulative reward within the problem horizon T .
Revised: Finally, the optimal trajectory of an innovation task is the trajectory that returns the highest cumulative reward within the problem horizon T .

Original: 2. L EARNING FRAMEWORK
Revised: 2. L EARNING FRAMEWORK

Original: SAPIENS considers a group of K DQN agents, where each agent interacts with its own copy of the environment and can share experiences with others by sampling transition tuples from its own replay buffer and directly inserting them to their replay buffer.
Revised: SAPIENS considers a group of K DQN agents, where each agent interacts with its own copy of the environment and can share experiences with others by sampling transition tuples from its own replay buffer and directly inserting them to their replay buffer.

Original: An undirected graph G determines who shares information with whom and consists of a set of nodes representing agents and a set of edges indicating that two agents share experiences with each other.
Revised: An undirected graph G determines who shares information with whom and consists of a set of nodes representing agents and a set of edges indicating that two agents share experiences with each other.

Original: We define the neighborhood of agent k as the set of nodes connected to it and denote it as N k .
Revised: We define the neighborhood of agent k as the set of nodes connected to it and denote it as N k .

Original: At the end of each episode the agent shares experiences with each of its neighbors with probability p s : sharing consists of sampling a subset of experiences of length L s from its own buffer B k and inserting it in the buffers of all neighbors B n , n ∈ N k .
Revised: At the end of each episode the agent shares experiences with each of its neighbors with probability p s : sharing consists of sampling a subset of experiences of length L s from its own buffer B k and inserting it in the buffers of all neighbors B n , n ∈ N k .

Original: Thus, an agent communicates distinct experiences with each of its neighbors.
Revised: Thus, an agent communicates distinct experiences with each of its neighbors.

Original: We present a schematic of two DQN agents sharing experiences on the right of Figure 3 and the pseudocode of SAPIENS in Appendix A.3. We also provide more implementation details about the DQN in Appendix A.1.
Revised: We present a schematic of two DQN agents sharing experiences on the right of Figureand the pseudocode of SAPIENS in Appendix C. We also provide more implementation details about the DQN in Appendix A.

Original: Thus, SAPIENS is a distributed RL learning paradigm where all agents are both actors and learners.
Revised: Thus, SAPIENS is a distributed RL learning paradigm where all agents are both actors and learners.

Original: This setting differs from multi-agent RL (Garnelo et al., 2021;
Revised: This setting differs from multi-agent RL (Garnelo et al., 2021;

Original: Christianos et al., 2020; Jiang et al., 2020a), where agents co-exist in the same environment and from parallelised RL (Steinkraus et al., 2005), where there need to be multiple agents.
Revised: Christianos et al., 2020; Jiang et al., 2020a), where agents co-exist in the same environment and from parallelised RL (Steinkraus et al., 2005), where there need to be multiple agents.

Original: It should also be distinguished from distributed RL paradigms with a single learner and multiple actors(Horgan et al., 2018; Espeholt et al., 2018;
Revised: It should also be distinguished from distributed RL paradigms with a single learner and multiple actors(Horgan et al., 2018; Espeholt et al., 2018;

Original: Nair et al., 2015; Garnelo et al., 2021), as multiple policies are learned simultaneously.
Revised: Nair et al., 2015; Garnelo et al., 2021), as multiple policies are learned simultaneously.

Original: We visualize the social network structures studied in our work in Figure 3: fully-connected, smallworld, ring and dynamic.
Revised: We visualize the social network structures studied in our work in Figure 3: fully-connected, smallworld, ring and dynamic.

Original: We construct the latter structure by grouping the agents in pairs and then allowing agents to visit other groups for a pre-determined duration, T v with probability p v .
Revised: We construct the latter structure by grouping the agents in pairs and then allowing agents to visit other groups for a pre-determined duration, T v with probability p v .

Original: This is a common type of dynamic topology that has been used in human laboratory and field studies Derex & Boyd (2016); Migliano & Vinicius (2022) and studied for its graph-theoretic properties (Volz & Meyers, 2007).
Revised: This is a common type of dynamic topology that has been used in human laboratory and field studies Derex & Boyd (2016); Migliano & Vinicius (2022) and studied for its graph-theoretic properties (Volz & Meyers, 2007).

Original: We provide more information about this dynamic topology in Appendix A.4, where we study how its behavior changes with different values of its hyper-parameters T v , p v .
Revised: We provide more information about this dynamic topology in Appendix D, where we study how its behavior changes with different values of its hyper-parameters T v , p v .

Original: We also present and analyze an alternative type of dynamic topology where the group oscillates between phases of full and no connectivity.
Revised: We also present and analyze an alternative type of dynamic topology where the group oscillates between phases of full and no connectivity.

Original: 2. E VALUATION FRAMEWORK
Revised: 2. E VALUATION FRAMEWORK

Original: During evaluation trials, which take place every E eval training steps with experience sharing deactivated, we compute performance-based metrics, which describe the quality of the final solution and the speed of cIn addition, we measure properties that are not directly related to performance but can help us analyze the effect of social network structure.
Revised: During evaluation trials, which take place every E eval training steps with experience sharing deactivated, we compute performance-based metrics, which describe the quality of the final solution and the speed of cIn addition, we measure properties that are not directly related to performance but can help us analyze the effect of social network structure.

Original: We introduce the following behavioral metrics , characterizing the policies followed by the agents, and mnemonic metrics , characterizing the replay buffers of agents, either at an individual or group level.
Revised: We introduce the following behavioral metrics , characterizing the policies followed by the agents, and mnemonic metrics , characterizing the replay buffers of agents, either at an individual or group level.

Original: onvergence to it, the two main properties of interest when evaluating RL algorithms.
Revised: onvergence to it, the two main properties of interest when evaluating RL algorithms.

Original: Performance-based metrics: (i) R + t : the maximum reward of the group at training step t ; (ii) R ∗ t : the average reward of the group at training step t ; (iii) T + , Time to first success: the first training step at which at least one of the agents collected the maximum achievable reward; (iv) T ∗ , Time to all successes: the first training step at which all of the agents found the optimal solution (v) T > ,
Revised: Performance-based metrics: (i) R + t : the maximum reward of the group at training step t ; (ii) R ∗ t : the average reward of the group at training step t ; (iii) T + , Time to first success: the first training step at which at least one of the agents collected the maximum achievable reward; (iv) T ∗ , Time to all successes: the first training step at which all of the agents found the optimal solution (v) T > ,

Original: Spread time: number of training steps required for the optimal solution to spread to the whole group, once at least one member discovered it (equals T ∗ − T + ) (vi) S , group success, a binary variable denoting whether at least one agent in a group found the optimal solution
Revised: Spread time: number of training steps required for the optimal solution to spread to the whole group, once at least one member discovered it (equals T ∗ − T + ) (vi) S , group success, a binary variable denoting whether at least one agent in a group found the optimal solution

Original: Behavioral metrics: (i) conformity C t is a group-level metric that denotes the percentage of agents in a group that end up with the same element at the end of a given evaluation trial.
Revised: Behavioral metrics: (i) conformity C t is a group-level metric that denotes the percentage of agents in a group that end up with the same element at the end of a given evaluation trial.

Original: Thus, agents conform to each other even if they follow alternative trajectories.
Revised: Thus, agents conform to each other even if they follow alternative trajectories.

Original: ; (ii) volatility V t is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by an agent across episodes.
Revised: ; (ii) volatility V t is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by an agent across episodes.

Original: Mnemonic metrics: (i) diversity D k t is an agent-level metric that denotes the number of unique experiences in an agent’s replay buffer; (ii) D G t is a group-level metric that captures the diversity of the aggregated group buffer.
Revised: Mnemonic metrics: (i) diversity D k t is an agent-level metric that denotes the number of unique experiences in an agent’s replay buffer; (ii) D G t is a group-level metric that captures the diversity of the aggregated group buffer.

Original: To compute it we concatenate the replay buffers of all agents into a single one and compute its diversity.
Revised: To compute it we concatenate the replay buffers of all agents into a single one and compute its diversity.

Original:  R ESULTS
Revised:  R ESULTS

Original: We now evaluate the social network structures presented in Figure 3 on the innovation tasks described in Section 2.2 and visualized in Figure 2.
Revised: We now evaluate the social network structures presented in Figure 3 on the innovation tasks described in Section 2.2 and visualized in Figure 2.

Original: Specifically, methods ring, small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with
Revised: Specifically, methods ring, small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with

Original: DQNs.
Revised: DQNs.

Original: In all experiments shared experiences are sampled randomly from the replay buffers.
Revised: In all experiments shared experiences are sampled randomly from the replay buffers.

Original: We benchmark SAPIENS against: a) no-sharing, a setting with 10 DQN agents without experience sharing b) single, a single DQN agent c) A2C, a distributed policy gradient RL algorithm that employs a single learner and 10 workers sharing gradients (Mnih et al., 2016) and d) Ape-X, a distributed
Revised: We benchmark SAPIENS against: a) no-sharing, a setting with 10 DQN agents without experience sharing b) single, a single DQN agent c) A2C, a distributed policy gradient RL algorithm that employs a single learner and 10 workers sharing gradients (Mnih et al., 2016) and d) Ape-X, a distributed

Original: RL algorithm with 10 workers that share experience samples with a single DQN learner (Horgan et al., 2018).
Revised: RL algorithm with 10 workers that share experience samples with a single DQN learner (Horgan et al., 2018).

Original: To test for statistical significance we perform ANOVA for multiple and Tukey range tests for pairwise comparisons.
Revised: To test for statistical significance we perform ANOVA for multiple and Tukey range tests for pairwise comparisons.

Original: We refer readers to Appendix A.5.1 for more information about our experimental setup.
Revised: We refer readers to Appendix E.1 for more information about our experimental setup.

Original: 3. O VERALL COMPARISON
Revised: 3. O VERALL COMPARISON

Original: In Figure 4 we compare performance across tasks and methods in terms of the the metrics of group success S and Time to first success T + , where we also indicate the p values with asterisks (more asterisks denote higher significance).
Revised: In Figure 4 we compare performance across tasks and methods in terms of the the metrics of group success S and Time to first success T + , where we also indicate the p values with asterisks (more asterisks denote higher significance).

Original: For completeness, we provide the data and figures corresponding to all evaluation metrics that we introduced in Section 2.4 in Appendix A.5.
Revised: For completeness, we provide the data and figures corresponding to all evaluation metrics that we introduced in Section 2.4 in Appendix E.

Original: We observe that performance varies across tasks and topologies.
Revised: We observe that performance varies across tasks and topologies.

Original: The single path task is optimally solved by all methods, except for single and Ape-X, which failed in 20% and 15% of the trials respectively due to slow convergence.
Revised: The single path task is optimally solved by all methods, except for single and Ape-X, which failed in 20% and 15% of the trials respectively due to slow convergence.

Original: ANOVA showed no significant difference in terms of group success S ( p = 0 . 43 ) but
Revised: ANOVA showed no significant difference in terms of group success S ( p = 0 . 43 ) but

Original: methods differed significantly in terms of T
Revised: methods differed significantly in terms of T

Original: + ( p = 0 .
Revised: + ( p = 0 .

Original: 2 e − 13 ).
Revised: 2 e − 13 ).

Original: In particular, single is significantly slower than all other methods ( T + = 648 e 3 for single) and A2C is significantly quicker than other methods ( T + = 36200 for A2C), with the Tukey’s range test indicating significance with p = 0 .
Revised: In particular, single is significantly slower than all other methods ( T + = 648 e 3 for single) and A2C is significantly quicker than other methods ( T + = 36200 for A2C), with the Tukey’s range test indicating significance with p = 0 .

Original: 001 for all pairwise comparisons with these two methods.
Revised: 001 for all pairwise comparisons with these two methods.

Original: This is in agreement with the expectation that a single agent learns slower due to having less data and that the policy-gradient algorithm A2C is more sample efficient than value function algorithms like DQNs.
Revised: This is in agreement with the expectation that a single agent learns slower due to having less data and that the policy-gradient algorithm A2C is more sample efficient than value function algorithms like DQNs.

Original: In the merging paths task there were significant differences among methods both for group success S ( p = 0 .
Revised: In the merging paths task there were significant differences among methods both for group success S ( p = 0 .

Original: 4 e − 4 ) and convergence speed T + ( p = 0 . 0095 ).
Revised: 4 e − 4 ) and convergence speed T + ( p = 0 . 0095 ).

Original: The group success of dynamic ( S = 0 . 65 ) is significantly higher compared to Ape-X ( S = 0 .
Revised: The group success of dynamic ( S = 0 . 65 ) is significantly higher compared to Ape-X ( S = 0 .

Original: 05 , p = 0 .
Revised: 05 , p = 0 .

Original: 001 ), A2C ( S = 0 .
Revised: 001 ), A2C ( S = 0 .

Original: 0 , p = 0 . 00101 ), fully-connected ( S = 0 .
Revised: 0 , p = 0 . 00101 ), fully-connected ( S = 0 .

Original: 0 , p = 0 . 00101 ) and ring ( S = 0 .
Revised: 0 , p = 0 . 00101 ) and ring ( S = 0 .

Original: 2 , p = 0 . 0105 ).
Revised: 2 , p = 0 . 0105 ).

Original: The single, no-sharing and small-world structures performed comparably well, but did not show statistically significant differences with other methods.
Revised: The single, no-sharing and small-world structures performed comparably well, but did not show statistically significant differences with other methods.

Original: In terms of T + , we see that dynamic is quicker than other methods
Revised: In terms of T + , we see that dynamic is quicker than other methods

Original: , G r o u p s u cc e ss
Revised: , G r o u p s u cc e ss

Original: Single path 0.
Revised: Single path 0.

Original: Merging paths 0.
Revised: Merging paths 0.

Original: Best-of-ten paths 0.
Revised: Best-of-ten paths 0.

Original: T + , T i m e t
Revised: T + , T i m e t

Original: o f i r s t s u cc e ss 1e ** ** *** ** ** ** ** ** **** ** ** 0.
Revised: o f i r s t s u cc e ss 1e ** ** *** ** ** ** ** ** **** ** ** 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: * * * singleno
Revised: * * * singleno

Original: [Figure]
Revised: [Figure]

Original: dynamicfully-
Revised: dynamicfully-

Original: [Figure]
Revised: [Figure]

Original: ringsmall-world
Revised: ringsmall-world

Original: A2CApe-X    1e ** ** ** **** ** ** with positive S , which leads to it being the only method with statistically significant differences with
Revised: A2CApe-X    1e ** ** ** **** ** ** with positive S , which leads to it being the only method with statistically significant differences with

Original: Ape-X ( p = 0 . 0292 ), fully-connected ( p = 0 . 0421 )
Revised: Ape-X ( p = 0 . 0292 ), fully-connected ( p = 0 . 0421 )

Original: and A2C ( p = 0 . 0421 ), the methods that failed in almost all trials and hence have T + equal to the budget of the experiment.
Revised: and A2C ( p = 0 . 0421 ), the methods that failed in almost all trials and hence have T + equal to the budget of the experiment.

Original: Thus, our main conclusion in the merging-paths task is that methods with fully-connected topologies (fully-connected,
Revised: Thus, our main conclusion in the merging-paths task is that methods with fully-connected topologies (fully-connected,

Original: A2C, Ape-X) fail dramatically and that the dynamic structure succeeds with higher probability.
Revised: A2C, Ape-X) fail dramatically and that the dynamic structure succeeds with higher probability.

Original: Finally, in the best-of-ten paths task, differences are also significant both for T + ( p = 0 .
Revised: Finally, in the best-of-ten paths task, differences are also significant both for T + ( p = 0 .

Original: 9 e − 7 ) and S ( p = 0 .
Revised: 9 e − 7 ) and S ( p = 0 .

Original: 15 e − 6 ).
Revised: 15 e − 6 ).

Original: Here, dynamic outperforms all methods in terms of S with all Tukey range tests indicating significance with p = 0 .
Revised: Here, dynamic outperforms all methods in terms of S with all Tukey range tests indicating significance with p = 0 .

Original: 001 , which leads to it also having significantly higher convergence rate ( T + = 14 e 6 for dynamic) compared to the other methods that exhausted their time budget.
Revised: 001 , which leads to it also having significantly higher convergence rate ( T + = 14 e 6 for dynamic) compared to the other methods that exhausted their time budget.

Original: In additional experiments we have also: a) observed that these conclusions are consistent across network sizes, with the performance of partially-connected social network structures improving with increasing network size and the performance of the fully-connected decreasing (see Appendix A.5.5) b) observed a drop in performance under prioritized experience sharing, where the DQNs employ prioritized replay buffers and experiences are selected for sharing proportionally to their priority (see Appendix A.5.6) .
Revised: In additional experiments we have also: a) observed that these conclusions are consistent across network sizes, with the performance of partially-connected social network structures improving with increasing network size and the performance of the fully-connected decreasing (see Appendix E.5) b) observed a drop in performance under prioritized experience sharing, where the DQNs employ prioritized replay buffers and experiences are selected for sharing proportionally to their priority (see Appendix E.6) .

Original: This behavior has also been observed in other works (Souza et al., 2019) and we hypothesize that it is due to the fact that priorities computed by an agent do not necessarily agree with the priorities of the other c) analyzed how the performance of the dynamic topology varies with its hyper-parameters and derived suggestions for tuning it (see Appendix A.4) d) monitored the robustness of social network structures to different learning hyper-parameters and observed that the dynamic topology is more robust than fully-connected and no-sharing (see Section A.5.4) e) computed measures of the alignment of experiences within and across groups that further support our hypothesis that the content of replay buffers is highly contingent on the social network structure (see Appendix A.5.3) f) performed simulations in another deceptive test-bed and derived similar conclusions to what we observed in Wordcraft, namely that partially-connected structures are better at avoiding local optima (See Appendix A.6)
Revised: This behavior has also been observed in other works (Souza et al., 2019) and we hypothesize that it is due to the fact that priorities computed by an agent do not necessarily agree with the priorities of the other c) analyzed how the performance of the dynamic topology varies with its hyper-parameters and derived suggestions for tuning it (see Appendix D) d) monitored the robustness of social network structures to different learning hyper-parameters and observed that the dynamic topology is more robust than fully-connected and no-sharing (see Section E.4) e) computed measures of the alignment of experiences within and across groups that further support our hypothesis that the content of replay buffers is highly contingent on the social network structure (see Appendix E.3) f) performed simulations in another deceptive test-bed and derived similar conclusions to what we observed in Wordcraft, namely that partially-connected structures are better at avoiding local optima (See Appendix E.7)

Original: To explain the differences for SAPIENS under different structures we now study each task in isolation for no-sharing, fully-connected, ring, small-world and dynamic, focusing on the behavioral and mnemonic metrics described in Section 2.4.
Revised: To explain the differences for SAPIENS under different structures we now study each task in isolation for no-sharing, fully-connected, ring, small-world and dynamic, focusing on the behavioral and mnemonic metrics described in Section 2.4.

Original: 3. T ASK : S INGLE PATH
Revised: 3. T ASK : S INGLE PATH

Original: Previous human (Mason et al., 2008) and computational (Lazer & Friedman, 2007) studies have indicated that, when it comes to problems with a single global optimum, fully-connected topologies
Revised: Previous human (Mason et al., 2008) and computational (Lazer & Friedman, 2007) studies have indicated that, when it comes to problems with a single global optimum, fully-connected topologies

Original: [Figure]
Revised: [Figure]

Original: perform best.
Revised: perform best.

Original: Our analysis here, however, indicates no statistically significant difference between methods: the fully-connected topology was actually the slowest ( T + = 311 e 3 ).
Revised: Our analysis here, however, indicates no statistically significant difference between methods: the fully-connected topology was actually the slowest ( T + = 311 e 3 ).

Original: To shed light into this behavior, we turn towards diversity and volatility.
Revised: To shed light into this behavior, we turn towards diversity and volatility.

Original: We compare average individual diversity ( ¯ D t ) and group diversity D G t , where for each method we create two samples by looking at these two metrics at the timestep at which diversity starts decreasing.
Revised: We compare average individual diversity ( ¯ D t ) and group diversity D G t , where for each method we create two samples by looking at these two metrics at the timestep at which diversity starts decreasing.

Original: ANOVA tests indicate that differences are significant both for ¯ D T
Revised: ANOVA tests indicate that differences are significant both for ¯ D T

Original: + ( p = 0 .
Revised: + ( p = 0 .

Original: 16 e − 8 ), where all pairwise comparisons are significant based on the Tukey range test, and D G t ( p = 0 . 0005 ), where the significantly different pairs are (no-sharing, dynamic, p = 0 . 02 ), (dynamic, fully-connected, p = 0 . 001 ), (dynamic, small-world, p = 0 . 0087 ), (fully-connected, ring, p = 0 . 008476 ).
Revised: 16 e − 8 ), where all pairwise comparisons are significant based on the Tukey range test, and D G t ( p = 0 . 0005 ), where the significantly different pairs are (no-sharing, dynamic, p = 0 . 02 ), (dynamic, fully-connected, p = 0 . 001 ), (dynamic, small-world, p = 0 . 0087 ), (fully-connected, ring, p = 0 . 008476 ).

Original: Intuitively, fully-connected exhibits the highest average individual diversity ¯ D t (left in Figure 5) and the lowest group diversity D G t (left in Figure 5): sharing experiences with others diversifies an individual’s experiences but also homogenizes the group.
Revised: Intuitively, fully-connected exhibits the highest average individual diversity ¯ D t (left in Figure 5) and the lowest group diversity D G t (left in Figure 5): sharing experiences with others diversifies an individual’s experiences but also homogenizes the group.

Original: This task does not require group diversity, but we expect that high individual diversity should be indicative of quicker exploration.
Revised: This task does not require group diversity, but we expect that high individual diversity should be indicative of quicker exploration.

Original: So why does the higher ¯ D t of fully-connected not act to its benefit?
Revised: So why does the higher ¯ D t of fully-connected not act to its benefit?

Original: To answer this we turn to volatility V t (top left of Figure 5) and observe that it is highest for the fully-connected topology.
Revised: To answer this we turn to volatility V t (top left of Figure 5) and observe that it is highest for the fully-connected topology.

Original: We tested for statistically significant differences among methods in terms of volatility at convergence ( V T train ) and found that these pairs differ significantly: (small-world, ring, p = 0 . 002387 ), (small-world, dynamic, p = 0 . 001 ), (small-world, no-sharing, p = 0 . 001 ), (small-world, fully-connected, p = 0 . 001 ), (ring, fully-connected, p = 0 . 001 ), (dynamic, fullyconnected, p = 0 . 001 ) and (no-sharing, fully-connected, p = 0 . 001 ).
Revised: We tested for statistically significant differences among methods in terms of volatility at convergence ( V T train ) and found that these pairs differ significantly: (small-world, ring, p = 0 . 002387 ), (small-world, dynamic, p = 0 . 001 ), (small-world, no-sharing, p = 0 . 001 ), (small-world, fully-connected, p = 0 . 001 ), (ring, fully-connected, p = 0 . 001 ), (dynamic, fullyconnected, p = 0 . 001 ) and (no-sharing, fully-connected, p = 0 . 001 ).

Original: We hypothesize that the higher individual diversity and volatility of the fully-connected structure are linked and indicate that, although experience sharing helps speed up exploration, it also destabilizes agents, so that no net gain is observed in terms of convergence rate.
Revised: We hypothesize that the higher individual diversity and volatility of the fully-connected structure are linked and indicate that, although experience sharing helps speed up exploration, it also destabilizes agents, so that no net gain is observed in terms of convergence rate.

Original: Notably the fact that experience sharing degrades performance in RL has been observed but not understood (Souza et al., 2019;
Revised: Notably the fact that experience sharing degrades performance in RL has been observed but not understood (Souza et al., 2019;

Original: Schmitt et al., 2019), while as we see here, this becomes possible by analyzing diversity and volatility.
Revised: Schmitt et al., 2019), while as we see here, this becomes possible by analyzing diversity and volatility.

Original: 3. T ASK : M ERGING PATHS
Revised: 3. T ASK : M ERGING PATHS

Original: The hypothesis that motivated this task was that partially connected groups will perform better than fully-connected structures due to their ability to explore diverse trajectories and avoid the two local optima(Derex & Boyd, 2016).
Revised: The hypothesis that motivated this task was that partially connected groups will perform better than fully-connected structures due to their ability to explore diverse trajectories and avoid the two local optima(Derex & Boyd, 2016).

Original: This was indeed the case as we saw in our discussion of Figure 4 in
Revised: This was indeed the case as we saw in our discussion of Figure 4 in

Original: Section 3.1.
Revised: Section 3.1.

Original: If we also look at the diversities in this task, then we see that the group diversity D G of fully connected (illustrated in the middle bottom plot in Figure 5) differs significantly to the one of (no-sharing, p = 0 .
Revised: If we also look at the diversities in this task, then we see that the group diversity D G of fully connected (illustrated in the middle bottom plot in Figure 5) differs significantly to the one of (no-sharing, p = 0 .

Original: 001 ), (ring, p = 0 . 0067 )
Revised: 001 ), (ring, p = 0 . 0067 )

Original: and dynamic ( p = 0 . 0177 ).
Revised: and dynamic ( p = 0 . 0177 ).

Original: The group diversity in small-world, which was relatively successful ( S = 0 . 3 ), follows that of the other partially connected structures but peaks at a lower level ( D G ≈ 1700 ).
Revised: The group diversity in small-world, which was relatively successful ( S = 0 . 3 ), follows that of the other partially connected structures but peaks at a lower level ( D G ≈ 1700 ).

Original: We believe that this is due to the dual ability of this topology to both protect and spread information and that its diversity and performance will improve as we increase the size of the group, enabling higher clustering (we illustrate this behavior in Figure 15 in Appendix A.5.5 where we increase the size of the network to 50 agents).
Revised: We believe that this is due to the dual ability of this topology to both protect and spread information and that its diversity and performance will improve as we increase the size of the group, enabling higher clustering (we illustrate this behavior in Figure 15 in Appendix E.5 where we increase the size of the network to 50 agents).

Original: 3. T ASK : B EST - OF - TEN PATH
Revised: 3. T ASK : B EST - OF - TEN PATH

Original: Which social network topology is most appropriate in large search spaces?
Revised: Which social network topology is most appropriate in large search spaces?

Original: Our analysis in Figure 4 clearly indicates that the dynamic topology explores more robustly and achieves the highest performance in this case.
Revised: Our analysis in Figure 4 clearly indicates that the dynamic topology explores more robustly and achieves the highest performance in this case.

Original: Differently from the two previous tasks where fully-connected exhibited the highest volatility, dynamic and ring, the topologies with the lowest number of connections, are the most volatile here (top row of Figure 5).
Revised: Differently from the two previous tasks where fully-connected exhibited the highest volatility, dynamic and ring, the topologies with the lowest number of connections, are the most volatile here (top row of Figure 5).

Original: This is not surprising: volatility indicates that an agent is exploring more paths, which is necessary to optimally solve the task.
Revised: This is not surprising: volatility indicates that an agent is exploring more paths, which is necessary to optimally solve the task.

Original: In terms of group diversity, we found that no-sharing scored significantly lower than all other methods with p = 0 .
Revised: In terms of group diversity, we found that no-sharing scored significantly lower than all other methods with p = 0 .

Original: 001 , the small-world surpassed all methods, with its difference from dynamic being marginally statistically significant and fully-connected did not score last.
Revised: 001 , the small-world surpassed all methods, with its difference from dynamic being marginally statistically significant and fully-connected did not score last.

Original: Thus, differently from the other two tasks with small search spaces, quick spreading of information increases group diversity here.
Revised: Thus, differently from the other two tasks with small search spaces, quick spreading of information increases group diversity here.

Original: However, to solve the task quickly, group diversity needs to be combined with quick, local exploitation of the optimal path, which is possible under structures with large shortest path, such as dynamic and ring.
Revised: However, to solve the task quickly, group diversity needs to be combined with quick, local exploitation of the optimal path, which is possible under structures with large shortest path, such as dynamic and ring.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In distributed RL, agents share information in the form of experience tuples, gradients or neural network parameters, with the former being preferred due to instabilities and latency concerns in the sharing of gradients (Horgan et al., 2018;
Revised: In distributed RL, agents share information in the form of experience tuples, gradients or neural network parameters, with the former being preferred due to instabilities and latency concerns in the sharing of gradients (Horgan et al., 2018;

Original: Mnih et al., 2016; Schmitt et al., 2019; Nair et al., 2015;
Revised: Mnih et al., 2016; Schmitt et al., 2019; Nair et al., 2015;

Original: Garnelo et al., 2021).
Revised: Garnelo et al., 2021).

Original: A common architecture in this area is that of multiple actors and a single learner (Mnih et al., 2016; Horgan et al., 2018;
Revised: A common architecture in this area is that of multiple actors and a single learner (Mnih et al., 2016; Horgan et al., 2018;

Original: Schmitt et al., 2019; Garnelo et al., 2021), while the
Revised: Schmitt et al., 2019; Garnelo et al., 2021), while the

Original: Gorila framework (Nair et al., 2015) is a more general multi-learner architecture, that differs from ours as agents are sharing parameters rather than experience tuples.
Revised: Gorila framework (Nair et al., 2015) is a more general multi-learner architecture, that differs from ours as agents are sharing parameters rather than experience tuples.

Original: Networked Evolutionary Strategy (NetES) enable the study of the effect of network structure on evolutionary strategies (Adjodah et al., 2019), where, multiple actors are sharing gradients with a single learner.
Revised: Networked Evolutionary Strategy (NetES) enable the study of the effect of network structure on evolutionary strategies (Adjodah et al., 2019), where, multiple actors are sharing gradients with a single learner.

Original: NetES was applied on continuous control tasks, whose optimization landscape differs from our deceptive and discrete innovation tasks.
Revised: NetES was applied on continuous control tasks, whose optimization landscape differs from our deceptive and discrete innovation tasks.

Original: In MARL settings social network structure can either determine who co-exists with whom in the environment (Garnelo et al., 2021) or which agents in its neighborhood an agent takes into account (Jiang et al., 2020a;
Revised: In MARL settings social network structure can either determine who co-exists with whom in the environment (Garnelo et al., 2021) or which agents in its neighborhood an agent takes into account (Jiang et al., 2020a;

Original: Du et al., 2021).
Revised: Du et al., 2021).

Original: Here dynamic topologies that are adapted to maximize a group’s reward have been shown to maximize strategic diversity (Garnelo et al., 2021) and help the agents coordinate on demand (Du et al., 2021).
Revised: Here dynamic topologies that are adapted to maximize a group’s reward have been shown to maximize strategic diversity (Garnelo et al., 2021) and help the agents coordinate on demand (Du et al., 2021).

Original: In contrast, our dynamic topologies vary periodically independently of the group’s performance, which is important for avoiding convergence to local optima.
Revised: In contrast, our dynamic topologies vary periodically independently of the group’s performance, which is important for avoiding convergence to local optima.

Original: In population-based training, another learning paradigm where multiple agents are trained to learn a single policy, policies are compared against the whole population, thus only considering a fully-connected social network structure (Jaderberg et al., 2018).
Revised: In population-based training, another learning paradigm where multiple agents are trained to learn a single policy, policies are compared against the whole population, thus only considering a fully-connected social network structure (Jaderberg et al., 2018).

Original: Admittedly, the literature on the effect of social network structure on collective search is diverse, with different fields making different assumptions and asking different questions; to illustrate this we provide a non-exhaustive summary of our literature review in Table 1 of Appendix A.2.
Revised: Admittedly, the literature on the effect of social network structure on collective search is diverse, with different fields making different assumptions and asking different questions; to illustrate this we provide a non-exhaustive summary of our literature review in Table 1 of Appendix B.

Original:  D ISCUSSION AND F UTURE W ORK
Revised:  D ISCUSSION AND F UTURE W ORK

Original: We tested the hypothesis that the topology of experience sharing in a group of deep RL agents can shape its performance using our proposed learning framework, SAPIENS and showed that, in line with human studies, both social network topology and task structure affect performance.
Revised: We tested the hypothesis that the topology of experience sharing in a group of deep RL agents can shape its performance using our proposed learning framework, SAPIENS and showed that, in line with human studies, both social network topology and task structure affect performance.

Original: Based on our experimental results, we can provide general recommendations on which topology to use for which task class.
Revised: Based on our experimental results, we can provide general recommendations on which topology to use for which task class.

Original: The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks (Gupta et al., 2019)).
Revised: The single-path task is an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks (Gupta et al., 2019)).

Original: In this case, our experimental results show no benefit of experience sharing.
Revised: In this case, our experimental results show no benefit of experience sharing.

Original: The merging-path task exhibits strong local optima that have to be explored up to a certain point in order to discover the global optimum (in the spirit of hard exploration tasks as in ( Baker et al.,  Ecoffet et al., 
Revised: The merging-path task exhibits strong local optima that have to be explored up to a certain point in order to discover the global optimum (in the spirit of hard exploration tasks as in (Baker et al., 2022; Ecoffet et al., 2021)).

Original: Our results show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths.
Revised: Our results show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) performs best here by improving the exploration of different innovation paths.

Original: The dynamic topology shows the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one.
Revised: The dynamic topology shows the highest performance, allowing different groups to reach the merging innovation level in non-optimal paths before sharing their experience during visits to other groups to find the optimal one.

Original: Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones (in the spirit of combinatorial optimization tasks as in (Mazyavkina et al., 2021)).
Revised: Finally, the best-of-ten task is an instance of a class of tasks with a large search space, many local optima and a few global ones (in the spirit of combinatorial optimization tasks as in (Mazyavkina et al., 2021)).

Original: In this case, our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.
Revised: In this case, our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.

Original: We hope that our work will contribute to the fields of cognitive science and DRL in multiple ways.
Revised: We hope that our work will contribute to the fields of cognitive science and DRL in multiple ways.

Original: First, our empirical observations of the behavior of different topologies in the single path and bestof-ten-path tasks provide concrete hypotheses for future experiments studying human collective innovation, which has so far been studied only in a task that inspired our merging-paths task (Derex & Boyd, 2016).
Revised: First, our empirical observations of the behavior of different topologies in the single path and bestof-ten-path tasks provide concrete hypotheses for future experiments studying human collective innovation, which has so far been studied only in a task that inspired our merging-paths task (Derex & Boyd, 2016).

Original: By continuing the dialogue that has been initiated between human and computation studies (Fang et al., 2010; Lazer & Friedman, 2007; Cantor et al., 2021) to include DRL methods, we believe that cognitive science will benefit from tools that, as we show here, can learn in realistic problem set-ups and can be analyzed not just in terms of their behavior, but also in terms of their memories.
Revised: By continuing the dialogue that has been initiated between human and computation studies (Fang et al., 2010; Lazer & Friedman, 2007; Cantor et al., 2021) to include DRL methods, we believe that cognitive science will benefit from tools that, as we show here, can learn in realistic problem set-ups and can be analyzed not just in terms of their behavior, but also in terms of their memories.

Original: Second, we hope that studies in distributed RL will extend their evaluation methodology by analyzing not just rewards, but also behavioral and mnemonic metrics such as diversity, conformity and volatility that, as we show here, correlate with success.
Revised: Second, we hope that studies in distributed RL will extend their evaluation methodology by analyzing not just rewards, but also behavioral and mnemonic metrics such as diversity, conformity and volatility that, as we show here, correlate with success.

Original: Aside this, the effect of social network structure in distributed RL can be extended beyond evolutionary strategies (Adjodah et al., 2019) and beyond our current instantiation of SAPIENS , by considering other off-policy algorithms than DQNs and other types of information sharing.
Revised: Aside this, the effect of social network structure in distributed RL can be extended beyond evolutionary strategies (Adjodah et al., 2019) and beyond our current instantiation of SAPIENS , by considering other off-policy algorithms than DQNs and other types of information sharing.

Original: Finally, considering the effectiveness of the dynamic topologies observed in this study, we envision future works that investigate more types of them, as well as meta-learning or online-adaptation algorithms where the social network structure is optimized for a desired performance metric.
Revised: Finally, considering the effectiveness of the dynamic topologies observed in this study, we envision future works that investigate more types of them, as well as meta-learning or online-adaptation algorithms where the social network structure is optimized for a desired performance metric.

Original: A A PPENDIX

Original: This supplementary material provides additional methods, results and discussion, as well as implementation details.

Original: • Section A.1 describes in detail the MDP formulation of Wordcraft; • Section A.3 contains the pseudocde of SAPIENS ; • Section A.4 explains how we model dynamic social network structures and how their performance varies with their hyper-paramaters; • Section A.5 provides more information about our experimental setup and results (effect of group size, intra-group and inter-group alignment, robustness to learning hyper-parameters and effect of prioritized experience sharing).

Original: We also provide tables and figures for all metrics presented in Section 2.4 and reward plots.

Original: • Section A.6 contains simulations with another testbed, the Deceptive Coins game.

Original: A. D ETAILS OF W ORDCRAFT AS A M ARKOV D ECISION P ROCESS
Revised: A D ETAILS OF W ORDCRAFT AS A M ARKOV D ECISION P ROCESS

Original: We consider the episodic setting, where the environment resets at the end of each episode and an agent is trained for E train episodes.
Revised: We consider the episodic setting, where the environment resets at the end of each episode and an agent is trained for E train episodes.

Original: At each time step t , the agent observes the state s t and selects an action a t from a set of possible actions A according to its policy π θ , where π θ is a mapping from states to actions, parameterized by a neural network with weights θ .
Revised: At each time step t , the agent observes the state s t and selects an action a t from a set of possible actions A according to its policy π θ , where π θ is a mapping from states to actions, parameterized by a neural network with weights θ .

Original: In return, the agent receives the next state s t +1 and a scalar reward r t .
Revised: In return, the agent receives the next state s t +1 and a scalar reward r t .

Original: Each DQN agent collects experience tuples of the form [ s t , a t , s t +1 , r t ] in its replay buffer.
Revised: Each DQN agent collects experience tuples of the form [ s t , a t , s t +1 , r t ] in its replay buffer.

Original: Figure 6 offers a visualization of the states and actions encountered during an episode in Wordcraft, where the chosen actions and elements are chosen so as to reproduce the example of Figure 1.
Revised: Figure 6 offers a visualization of the states and actions encountered during an episode in Wordcraft, where the chosen actions and elements are chosen so as to reproduce the example of Figure 1.

Original: In order to solve the innovation task described in Section 2.1 we compute the maximum number of elements a player can craft within horizon T for recipe book X valid and initial set X 0 , which we[Table] denote as | X | .
Revised: In order to solve the innovation task described in Section 2.1 we compute the maximum number of elements a player can craft within horizon T for recipe book X valid and initial set X 0 , which we denote as | X | .

Original: We, then, encode each element as an integer in [0 , | X | ) .
Revised: We, then, encode each element as an integer in [0 , | X | ) .

Original: Thus, the action space is
Revised: Thus, the action space is

Original:  A = [0 , | X | ) , with action a t indicating the index of the currently chosen element.
Revised: [Table] A = [0 , | X | ) , with action a t indicating the index of the currently chosen element.

Original: The state s t contains two sets of information: a binary vector of length | X | with non-zero entries for elements already crafted by the agent within the current episode (we refer to this as inventory i ) and another binary vector of length | X | where an index is non-zero if it is currently selected by the agent (we refer to this as current c ).
Revised: The state s t contains two sets of information: a binary vector of length | X | with non-zero entries for elements already crafted by the agent within the current episode (we refer to this as inventory i ) and another binary vector of length | X | where an index is non-zero if it is currently selected by the agent (we refer to this as current c ).

Original: An agent begins with an inventory having non-zero element only for the initial set X 0 and an all-zero selection.
Revised: An agent begins with an inventory having non-zero element only for the initial set X 0 and an all-zero selection.

Original: With the first action a 0 , the selected item becomes non-zero in the selection.
Revised: With the first action a 0 , the selected item becomes non-zero in the selection.

Original: With the second action, a 1 , we check if the combination ( a 1 , c 0 ) is valid under the recipe book and, if so, return the newly crafted element (corresponding entry in i becomes non-zero) and the reward.
Revised: With the second action, a 1 , we check if the combination ( a 1 , c 0 ) is valid under the recipe book and, if so, return the newly crafted element (corresponding entry in i becomes non-zero) and the reward.

Original: This two-step procedure continues until the end of the episode.
Revised: This two-step procedure continues until the end of the episode.

Original: A. S UMMARY OF RELATED WORKS
Revised: B S UMMARY OF RELATED WORKS

Original: In this appendix we provide a non-comprehensive summary of the literature on the topic of the effect of social network topology on collective search in Table 1, where our objective is to highlight similarities and differences within and across the fields of cognitive science and DRL.
Revised: In this appendix we provide a non-comprehensive summary of the literature on the topic of the effect of social network topology on collective search in Table 1, where our objective is to highlight similarities and differences within and across the fields of cognitive science and DRL.

Original: A. P SEUDOCODE OF SAPIENS
Revised: C P SEUDOCODE OF SAPIENS

Original: We present the pseudocode of our proposed algorithm SAPIENS in Algorithm 1.
Revised: We present the pseudocode of our proposed algorithm SAPIENS in Algorithm 1.

Original: SAPIENS works similarly to an off-policy reinforcement learning algorithm, with the difference that, after each episode, an experience sharing phase takes place between agents that belong in the same group.do 18: 19: end
Revised: SAPIENS works similarly to an off-policy reinforcement learning algorithm, with the difference that, after each episode, an experience sharing phase takes place between agents that belong in the same group.

Original: 20: end
Revised: 20: end

Original: 21: end

Original: 22: end
Revised: 22: end

Original: A. A NALYSIS OF DYNAMIC NETWORK TOPOLOGIES
Revised: D A NALYSIS OF DYNAMIC NETWORK TOPOLOGIES

Original: In the main paper we presented results for a single type of dynamic topolgoy.
Revised: In the main paper we presented results for a single type of dynamic topolgoy.

Original: Here we present another type and analyze how they both behave for different values of their hyper-parameters.
Revised: Here we present another type and analyze how they both behave for different values of their hyper-parameters.

Original: The two dynamic topologies are:
Revised: The two dynamic topologies are:

Original: • Inspired by graphs employed in human laboratory studies (Derex & Boyd,  we designed graphs where the macro structure of the graph is constant but agents can randomly change their position.
Revised: • Inspired by graphs employed in human laboratory studies (Derex & Boyd, 2016), we designed graphs where the macro structure of the graph is constant but agents can randomly change their position.

Original: In particular, we divide a group of agents into sub-groups of two agents and, at the end of each episode, move an agent to another group with a probability p v for a duration of T v episodes (for a visualization see Figure 3).
Revised: In particular, we divide a group of agents into sub-groups of two agents and, at the end of each episode, move an agent to another group with a probability p v for a duration of T v episodes (for a visualization see Figure 3).

Original: To reduce the complexity of the implementation, we assume that only one visit can take place at a time.
Revised: To reduce the complexity of the implementation, we assume that only one visit can take place at a time.

Original: In the main paper we employ p v = 0 .
Revised: In the main paper we employ p v = 0 .

Original: 01 and T v = 10 across conditions and present results with different values in Appendix A.4, where we refer to this topology as dynamic-Boyd.
Revised: 01 and T v = 10 across conditions and present results with different values in Appendix D, where we refer to this topology as dynamic-Boyd.

Original: • Human behavioral ecology emphasize the importance of periodic variation in human social networks encountered throughout our evolutionary trajectory Wiessner (2014); Dunbar (2014).
Revised: • Human behavioral ecology emphasize the importance of periodic variation in human social networks encountered throughout our evolutionary trajectory Wiessner (2014); Dunbar (2014).

Original: Due to ecological constraints human groups oscillate between phases of high and low connectivity: low-connectivity phases arise when individuals need to individually collected resources (e.g. day-time hunting) while high-connectivity phases arise when humans are idle and “forced” to be in proximity with others (e
Revised: Due to ecological constraints human groups oscillate between phases of high and low connectivity: low-connectivity phases arise when individuals need to individually collected resources (e g. day-time hunting) while high-connectivity phases arise when humans are idle and “forced” to be in proximity with others (e

Original: g. fireside chats).
Revised: g. fireside chats).

Original: Although these high-connectivity phases do not bare a direct evolutionary advantage, they may have played an important role by creating the conditions for the evolution of human language and culture.
Revised: Although these high-connectivity phases do not bare a direct evolutionary advantage, they may have played an important role by creating the conditions for the evolution of human language and culture.

Original: Inspired by this hypothesis, we have designed dynamic graphs that oscillate between a fully-connected topology that lasts for T h episodes and a topology without sharing that lasts for T l episodes.
Revised: Inspired by this hypothesis, we have designed dynamic graphs that oscillate between a fully-connected topology that lasts for T h episodes and a topology without sharing that lasts for T l episodes.

Original: We present results for various values of T h and T d of this topology in Appendix A.4, where we refer to this topology as dynamic-periodic.
Revised: We present results for various values of T h and T d of this topology in Appendix D, where we refer to this topology as dynamic-periodic.

Original: In Figure 8, we observe the % of group success ( S G ) with the dynamic-Boyd topology for different probabilities of visit ( p v ) and visit duration T v (the sub-group size is 2 in all cases).
Revised: In Figure 8, we observe the % of group success ( S G ) with the dynamic-Boyd topology for different probabilities of visit ( p v ) and visit duration T v (the sub-group size is 2 in all cases).

Original: We note that, due to our implementation choice that a visit can take place only if no other agent is currently on a visit, the visit duration also affects the mixing of the group: longer visits mean that fewer visitswill take place in total.
Revised: We note that, due to our implementation choice that a visit can take place only if no other agent is currently on a visit, the visit duration also affects the mixing of the group: longer visits mean that fewer visits

Original: In the merging paths task (left), two hyper-parameter settings have a clear
Revised: will

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  effect: (i) short visits with of high probability lead to bad performance.
Revised: take place in total. In the merging paths task (left), two hyper-parameter settings have a clear effect: (i) short visits with of high probability lead to bad performance.

Original: As such settings lead to a quick mixing of the population, they lead to convergence to the local optimum (ii) long visits with high probability work well.
Revised: As such settings lead to a quick mixing of the population, they lead to convergence to the local optimum (ii) long visits with high probability work well.

Original: Due to the high visit probability, this setting effectively leads to topology where exactly one agent is always on a long visit.
Revised: Due to the high visit probability, this setting effectively leads to topology where exactly one agent is always on a long visit.

Original: Thus, it ensures that sub-groups stay isolated for at least 1000 episodes, after which inter sub-group sharing needs to takes place to ensure that the sub-groups can progress quickly.
Revised: Thus, it ensures that sub-groups stay isolated for at least 1000 episodes, after which inter sub-group sharing needs to takes place to ensure that the sub-groups can progress quickly.

Original: In the best-of-ten paths task (right), this structure has a clear optimal hyper-parameterization: short visits with high probability are preferred, which maximizes the mixing of the group and makes early exploration more effective.
Revised: In the best-of-ten paths task (right), this structure has a clear optimal hyper-parameterization: short visits with high probability are preferred, which maximizes the mixing of the group and makes early exploration more effective.

Original: In Figure [Figure] we observe the % of group success ( S G ) of the dynamic-periodic topology for various values of T h and T l .
Revised: In Figure 9, we observe the % of group success ( S G ) of the dynamic-periodic topology for various values of T h and T l .

Original: In the merging paths task (left of Figure 9) medium values for the period of both phases works best, while there is some success when the low connectivity phase lasts long ( T l = 1000 ).
Revised: In the merging paths task (left of Figure 9) medium values for the period of both phases works best, while there is some success when the low connectivity phase lasts long ( T l = 1000 ).

Original: In the best-of-ten paths task (rightof Figure 9), we observe the same medium values for the period of both phases work best: thus both the absolute value and their ratio is important to ensure that exploration is efficient.
Revised: In the best-of-ten paths task (rightof Figure 9), we observe the same medium values for the period of both phases work best: thus both the absolute value and their ratio is important to ensure that exploration is efficient.

Original: The optimal configuration is the same between the two tasks ( T l = 100 , T h = 10 ), which is a good indication of the robustness of this structure.
Revised: The optimal configuration is the same between the two tasks ( T l = 100 , T h = 10 ), which is a good indication of the robustness of this structure.

Original: A. E MPIRICAL RESULTS
Revised: E E MPIRICAL RESULTS

Original: To ensure that all methods have the same number of samples, we assume that, for trials where a method did not find the optimal solution, and, hence, T ∗ is undefined, T ∗ is equal to the total number of timesteps the method was trained for, T train .
Revised: To ensure that all methods have the same number of samples, we assume that, for trials where a method did not find the optimal solution, and, hence, T ∗ is undefined, T ∗ is equal to the total number of timesteps the method was trained for, T train .

Original: For each task, all methods have been trained for an equal duration of time: T train = 1 e 6 for the single path , T train = 7 e 6 for the merging paths task and T train = 

Revised: For each task, all methods have been trained for an equal duration of time: T train = 1 e 6 for the single path , T train = 7 e 6 for the merging paths task and T train =

Original: e 7 for the best-of-ten paths task.
Revised: e 7 for the best-of-ten paths task.

Original: We perform 20 independent trials for each task and method and visualize our proposed metrics with barplots and line plots of averages across trials with error bars indicating 95% confidence intervals.

Original: We test for statistical significance of our evaluation metrics separately for each task by applying

Original: [Figure]
Revised: [Figure]

Original: o nn e c t i v i
Revised: o nn e c t i v i

Original: t
Revised: t

Original: y p
Revised: y p

Original: e r i o d
Revised: e r i o d

Original: S 
Revised: S , % of group success

Original: [Figure]
Revised: [Figure]

Original: % of group success
Revised: period

Original: [Figure]
Revised: [Figure]

Original:  o w - c o nn e c t i v i
Revised: T l , L o w - c o nn e c t i v i

Original: t
Revised: t

Original: y p
Revised: y p

Original: e r i o d
Revised: e r i o d

Original: S , % of
Revised: S , % of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: We perform 20 independent trials for each task and method and visualize our proposed metrics with barplots and line plots of averages across trials with error bars indicating 95% confidence intervals.

Revised: We test for statistical significance of our evaluation metrics separately for each task by applying

Original: ANOVA tests 8 to detect whether at least one method differs from the rest and, subsequently, employing the Tukey’s range test 9 to detect which pairs of methods that differ significantly.
Revised: ANOVA tests 8 to detect whether at least one method differs from the rest and, subsequently, employing the Tukey’s range test 9 to detect which pairs of methods that differ significantly.

Original: We report the exact p values of theses tests in the text and, when applicable, illustrate them in figures using a set of asterisks whose number indicates the significance level 10 .
Revised: We report the exact p values of theses tests in the text and, when applicable, illustrate them in figures using a set of asterisks whose number indicates the significance level 10 .

Original: We presented the major results of our evaluation of SAPIENS in Section 3.
Revised: We presented the major results of our evaluation of SAPIENS in Section 3.

Original: We now present additional information regarding the implementation of the different components (Appendix A.5.1), the values of all performance metrics and additional plots for experiments discussed in 3 (Appendix A.5.2), results on intra-group and inter-group alignment (AppendixA.5.3), results for groups of varying sizes (Appendix A.5.5) and results on various dynamic topologies (Appendix A.4)
Revised: We now present additional information regarding the implementation of the different components (Appendix E.1), the values of all performance metrics and additional plots for experiments discussed in 3 (Appendix E.2), results on intra-group and inter-group alignment (AppendixE.3), results for groups of varying sizes (Appendix E.5) and results on various dynamic topologies (Appendix D)

Original: A.5. I MPLEMENTATION DETAILS
Revised: E. I MPLEMENTATION DETAILS

Original: Implementation of DQN We employ the same hyper-parameter for each DQN across all studied tasks and topologies: discount factor γ = 0 .
Revised: Implementation of DQN We employ the same hyper-parameter for each DQN across all studied tasks and topologies: discount factor γ = 0 .

Original: 9 , the Adam optimizer with learning rate α = 0 .(Kingma & Ba, 2014; Dunbar, 2014), ϵ -greedy exploration with ϵ = 0 .
Revised: 9 , the Adam optimizer with learning rate α = 0 .(Kingma & Ba, 2014; Dunbar, 2014), ϵ -greedy exploration with ϵ = 0 .

Original: We employ a feedforward network with two layers with 64 neurons each.
Revised: We employ a feedforward network with two layers with 64 neurons each.

Original: We implemented SAPIENS by extending the DQN implementation in the stable-baselines3 framework 11 .
Revised: We implemented SAPIENS by extending the DQN implementation in the stable-baselines3 framework 11 .

Original: Implementation of A2C We used the stable-baselines3 implementation of A2C 12 and tuned the hyper-parameters: learning rate, number of steps, discount factor, the entropy coefficient and the value function coefficient.
Revised: Implementation of A2C We used the stable-baselines3 implementation of A2C 12 and tuned the hyper-parameters: learning rate, number of steps, discount factor, the entropy coefficient and the value function coefficient.

Original: This gave us the best-performing values 0.001, 5, 0.99, 0.1 and 0.25, respectively, that we also employed in the other tasks.
Revised: This gave us the best-performing values 0.001, 5, 0.99, 0.1 and 0.25, respectively, that we also employed in the other tasks.

Original: Implementation of Ape-X
Revised: Implementation of Ape-X

Original: We used the ray implementation of Ape-X DQN 13 and tuned the hyper-parameters: learning rate, discount factor, replay buffer capacity and ϵ -greedy exploration.
Revised: We used the ray implementation of Ape-X DQN 13 and tuned the hyper-parameters: learning rate, discount factor, replay buffer capacity and ϵ -greedy exploration.

Original: This gave us the best-performing values in the single path task 0.001, 0.9, 5000 and 0.02, respectively.
Revised: This gave us the best-performing values in the single path task 0.001, 0.9, 5000 and 0.02, respectively.

Original:  Implementation of graphs used as social network structures We construct small-worlds using the Watts–Strogatz model ( watts strogatz graph method of the networkx package 14 ).
Revised: [Table] [Table] Implementation of graphs used as social network structures We construct small-worlds using the Watts–Strogatz model ( watts strogatz graph method of the networkx package 14 ).

Original: This model first builds a ring lattice where each node has n neighbors and then rewires an edge with probability β .
Revised: This model first builds a ring lattice where each node has n neighbors and then rewires an edge with probability β .

Original: Compared to other techniques used in previous works studying the effect of topology
Revised: Compared to other techniques used in previous works studying the effect of topology

Original: [Table]

Original: [Table]

Original: Mason et al.
Revised: Mason et al.

Original: (2008), this way of constructing small-worlds ensures that the average path lengths is short and clustering is high.
Revised: (2008), this way of constructing small-worlds ensures that the average path lengths is short and clustering is high.

Original: These two properties are what differentiates small-worlds from random (short average path length and small clustering) and regular (long average path length and high clustering) graphs.
Revised: These two properties are what differentiates small-worlds from random (short average path length and small clustering) and regular (long average path length and high clustering) graphs.

Original: We employ n = 4 and β = 0 .
Revised: We employ n = 4 and β = 0 .

Original: 2 in our experiments, which we empirically found to lead to good values of average path length and clustering.
Revised: 2 in our experiments, which we empirically found to lead to good values of average path length and clustering.

Original: We have described the generation process of dynamic topologies in Appendix A.4. In the main paper we employ the dynamic-Boyd topology with T v = 10 and p v = 0 .
Revised: We have described the generation process of dynamic topologies in Appendix D. In the main paper we employ the dynamic-Boyd topology with T v = 10 and p v = 0 .

Original: 001 across tasks.
Revised: 001 across tasks.

Original: A.5. O VERALL COMPARISON
Revised: E. O VERALL COMPARISON

Original: Tables 2, 3 and 4 contain the values of all metrics discussed in Section 2.4 for the single path, merging paths and best-of-ten paths, respectively.
Revised: Tables 2, 3 and 4 contain the values of all metrics discussed in Section 2.4 for the single path, merging paths and best-of-ten paths, respectively.

Original: We denote values computed after convergence of the group with underscore ∞ and values averaged over all training steps with underscore avg (note that we use ¯ over variables to denote averaging over agents in a single training step).
Revised: We denote values computed after convergence of the group with underscore ∞ and values averaged over all training steps with underscore avg (note that we use ¯ over variables to denote averaging over agents in a single training step).

Original: Cells with a dash (-) indicate that we could not compute the corresponding metrics because a group failed to find a solution in all trials.
Revised: Cells with a dash (-) indicate that we could not compute the corresponding metrics because a group failed to find a solution in all trials.

Original: We also provide the plots of volatility and average diversity for the merging paths and best-of-10 paths task (that were not included in Figure 5) due to page limit constraints).
Revised: We also provide the plots of volatility and average diversity for the merging paths and best-of-10 paths task (that were not included in Figure 5) due to page limit constraints).

Original: Figure 10 presents the reward curves for all methods in the single path, merging paths and best-often paths tasks respectively.
Revised: Figure 10 presents the reward curves for all methods in the single path, merging paths and best-often paths tasks respectively.

Original: Specifically, we plot the maximum reward of the group at training step t ( R + t ).
Revised: Specifically, we plot the maximum reward of the group at training step t ( R + t ).

Original: A.5. M EASURING INTER - GROUP AND INTRA - GROUP ALIGNMENT

Original: We have so far captures the agreement between agents in a group through the behavioral metric of conformity.

Original: Here, we present a mnemonic metric for agreement, which we term alignment.

Original: Alignment is a complementary metric to the diversity ( D kt ) and group diversity ( D G t ) metrics, that aims at capturing the effect of experience sharing on the replay buffers in a group.

Original: We propose a definition of alignment within a single group (intra-group alignment A G t ) and a definition of alignment between

Original: Topology
Revised: Topology

Original: R + ∞
Revised: R + ∞

Original: R ∗∞
Revised: R ∗∞

Original: T +
Revised: T +

Original: T ∗
Revised: T ∗

Original: T >
Revised: T >

Original: S G
Revised: S G

Original: C avg ¯ V avg no sharing 0.
Revised: C avg ¯ V avg no sharing 0.

Original: dynamic 0.
Revised: dynamic 0.

Revised: fully-connected 0.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: E. M EASURING INTER - GROUP AND INTRA - GROUP ALIGNMENT

Original:  two different groups ( A G j , G j t ).
Revised: We have so far captures the agreement between agents in a group through the behavioral metric of conformity. Here, we present a mnemonic metric for agreement, which we term alignment. Alignment is a complementary metric to the diversity ( D kt ) and group diversity ( D G t ) metrics, that aims at capturing the effect of experience sharing on the replay buffers in a group. We propose a definition of alignment within a single group (intra-group alignment A G t ) and a definition of alignment between two different groups ( A G j , G j t ).

Original: Such metrics of mnemonic convergence have been linked to social network topology (Coman et al., 2016) and, as we show here, they can prove useful in analyzing groups of reinforcement learning agents.
Revised: Such metrics of mnemonic convergence have been linked to social network topology (Coman et al., 2016) and, as we show here, they can prove useful in analyzing groups of reinforcement learning agents.

Original: Specifically: (i)
Revised: Specifically: (i)

Original: A G t is the intra-group alignment.
Revised: A G t is the intra-group alignment.

Original: This metric captures the similarity in terms of content between the replay buffers of agents belonging to the same group.
Revised: This metric captures the similarity in terms of content between the replay buffers of agents belonging to the same group.

Original: To compute this we compute the size of the common subset of experiences for each pair of agents and, then, average over all these pairs, normalizing in [0,1].
Revised: To compute this we compute the size of the common subset of experiences for each pair of agents and, then, average over all these pairs, normalizing in [0,1].

Original: (ii) inter-group alignment A G j , G j t is a similar notion of alignment but employed between different groups (e.g. how different is a group of fully-connected and a dynamic group of agents in terms of the content of their group replay buffers).
Revised: (ii) inter-group alignment A G j , G j t is a similar notion of alignment but employed between different groups (e.g. how different is a group of fully-connected and a dynamic group of agents in terms of the content of their group replay buffers).

Original: To compute it we concatenate all replay buffers of a group into a single one and then compute the size of the common subset of the two replay buffers.
Revised: To compute it we concatenate all replay buffers of a group into a single one and then compute the size of the common subset of the two replay buffers.

Original: Figure 12 presents intra-group alignment in the three tasks.
Revised: Figure 12 presents intra-group alignment in the three tasks.

Original: We observe that, in all tasks, intragroup alignment increases with connectivity and that it reduces when the agents enter the exploitation phase.
Revised: We observe that, in all tasks, intragroup alignment increases with connectivity and that it reduces when the agents enter the exploitation phase.

Original: Thus, intra-group alignment can prove useful in characterizing the exploration behavior of a group.
Revised: Thus, intra-group alignment can prove useful in characterizing the exploration behavior of a group.

Original: In Figure 13, we present the inter-group alignment in the single path, merging paths and best-of-ten paths tasks.
Revised: In Figure 13, we present the inter-group alignment in the single path, merging paths and best-of-ten paths tasks.

Original: We observe that the topologies do not differ significantly in the single path task.
Revised: We observe that the topologies do not differ significantly in the single path task.

Original: In the merging task, we observe that inter-group alignment is lower during the exploration phase, compared to other tasks, and that the small-world is the slowest to align with all other structures.
Revised: In the merging task, we observe that inter-group alignment is lower during the exploration phase, compared to other tasks, and that the small-world is the slowest to align with all other structures.

Original: Perhaps this explains why this topology finds the optimal solution with the least probability: by propagating information quickly, the group early onconverges to the local optimum in this task.
Revised: Perhaps this explains why this topology finds the optimal solution with the least probability: by propagating information quickly, the group early on

Original: In the best-of-ten task, the no-sharing setting has the smallest alignment with all other structures.

Original: This reinforces our main conclusion in this work: experience sharing affects individuals and different topologies do so in different ways.

Original: A.5. R OBUSTNESS TO LEARNING HYPER - PARAMETERS

Original: In Figure 14 we present how the performance of SAPIENS varies for different values of the learning hyperparameters learning rate and disocunt factor in the single path task under a fully-connected and

Original: [Figure]
Revised: [Figure]

Original: C o n f o r m i t y ,
Revised: C o n f o r m i t y ,

Original: C 
Revised: C t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I n t r ag ro u p ali gn m e n t
Revised: r a g e D i v e r

Original: [Figure]

Original: t
Revised: s

Original: [Figure] li g n m e n t ,
Revised: i

Original: A t 
Revised: t y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] I n t e r - g r o u p a li g n m e n t , A j , j t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: no

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: C

Original: [Figure]
Revised: [Figure]

Original: I n t e r - g r o u p a li g n m e n t  ,
Revised: n f o r m i t y ,

Original: A j , j t
Revised: C t

Original: [Figure]
Revised: [Figure]

Original: I n t e r - g r o u p a li g n m e n t , A j , j t
Revised: i

Original: [Figure]
Revised: [Figure]

Revised: 3 Training step, t 1e 0.

Original: I n te r 
Revised: I n t r ag r o u p a li g n m e n t ,

Original: [Figure]
Revised: A t   3 Training step, t 1e 0. I n t r ag r o u p a li g n m e n t , A t 0. 2.0 Training step, t 1e 0.

Original:  u p a li g n [Figure] n t [Figure]
Revised: I n t r ag r o u p a li g n m e n t ,

Original: [Figure]
Revised: A t converges to the local optimum in this task. In the best-of-ten task, the no-sharing setting has the smallest alignment with all other structures. This reinforces our main conclusion in this work: experience sharing affects individuals and different topologies do so in different ways. E. R OBUSTNESS TO LEARNING HYPER - PARAMETERS

Original:  a dynamic topolgoy, as well as the no-sharing condition.
Revised: In Figure 14 we present how the performance of SAPIENS varies for different values of the learning hyperparameters learning rate and disocunt factor in the single path task under a fully-connected and a dynamic topolgoy, as well as the no-sharing condition.

Original: We observe that, although convergence to the optimal solution is not always achieved, the dynamic topology is at least as effective as the others either in terms of convergence rate and/or final performance in all conditions.
Revised: We observe that, although convergence to the optimal solution is not always achieved, the dynamic topology is at least as effective as the others either in terms of convergence rate and/or final performance in all conditions.

Original: A.5. E FFECT OF GROUP SIZE
Revised: E. E FFECT OF GROUP SIZE

Original: We here examine the effect of the group size for all social network structures in the merging-paths and best-of-ten paths task.
Revised: We here examine the effect of the group size for all social network structures in the merging-paths and best-of-ten paths task.

Original: To visualize the progression of a group on the paths of the different tasks, we focus on specific elements in the tasks: (i) ( [ A 8 , B 8 , C 2 ] in the merging-paths task.
Revised: To visualize the progression of a group on the paths of the different tasks, we focus on specific elements in the tasks: (i) ( [ A 8 , B 8 , C 2 ] in the merging-paths task.

Original: The first two correspond to reaching the end of the paths corresponding to the two local optima.
Revised: The first two correspond to reaching the end of the paths corresponding to the two local optima.

Original: To reduce the computational complexity of experiments, we do not study the last element of the optimal path ( C 4 ), but focus on C 2 instead.
Revised: To reduce the computational complexity of experiments, we do not study the last element of the optimal path ( C 4 ), but focus on C 2 instead.

Original: This is sufficient to detect whether a group has discovered the optimum path.
Revised: This is sufficient to detect whether a group has discovered the optimum path.

Original: Here, we observe that the fully-connected topology fails to find the optimal path regardless of its size (with a small success probability for N = 10 ).
Revised: Here, we observe that the fully-connected topology fails to find the optimal path regardless of its size (with a small success probability for N = 10 ).

Original: We observe that the ability of the ring , small-world and dynamict topologies to avoid the local optima improves with the group size (ii)
Revised: We observe that the ability of the ring , small-world and dynamict topologies to avoid the local optima improves with the group size (ii)

Original: [ B 4 , A 2 , E 2 ] in the best-of-ten tasks.
Revised: [ B 4 , A 2 , E 2 ] in the best-of-ten tasks.

Original: B 4 is the fourth element on the optimal path (again we do not study the last element to reduce complexity).
Revised: B 4 is the fourth element on the optimal path (again we do not study the last element to reduce complexity).

Original: To avoid cluttering the visualization we only present two of the nine sub-optimal paths.
Revised: To avoid cluttering the visualization we only present two of the nine sub-optimal paths.

Original: In this task, we again observe that the fully-connected network fails to discover the optimal task.
Revised: In this task, we again observe that the fully-connected network fails to discover the optimal task.

Original: Among all structures and group sizes, the large dynamic network performs best, while the performance of ring and small-world is also best for N = 50 .
Revised: Among all structures and group sizes, the large dynamic network performs best, while the performance of ring and small-world is also best for N = 50 .

Original: We observe that small networks sizes ( N = 2 , N = 6 ) are slower at exploring (we can see that as they rarely find the second element of the sub-optimal paths, which is required to conclude that path B is the optimal choice).
Revised: We observe that small networks sizes ( N = 2 , N = 6 ) are slower at exploring (we can see that as they rarely find the second element of the sub-optimal paths, which is required to conclude that path B is the optimal choice).

Original: Overall, this scaling analysis indicates that increasing the group size in a fully-connected topology will not improve performance, while benefits are expected for low-connectivity structures, particularly for the dynamic topology.
Revised: Overall, this scaling analysis indicates that increasing the group size in a fully-connected topology will not improve performance, while benefits are expected for low-connectivity structures, particularly for the dynamic topology.

Original: We believe that this observation is crucial.
Revised: We believe that this observation is crucial.

Original: In studies of groups of both human and artificial agents, we often encounter the conviction that, larger groups perform better and that size is a more important determinant than connectivity, the latter justifying why connectivity is often ignored Kline & Boyd (2010); Horgan et al.
Revised: In studies of groups of both human and artificial agents, we often encounter the conviction that, larger groups perform better and that size is a more important determinant than connectivity, the latter justifying why connectivity is often ignored Kline & Boyd (2010); Horgan et al.

Original: ; Mnih et al. (2016);
Revised: ; Mnih et al. (2016);

Original: Schmitt et al.
Revised: Schmitt et al.

Original: ; Nair et al.
Revised: ; Nair et al.

Original: Our results here point to the contrary.
Revised: Our results here point to the contrary.

Original: [Figure]
Revised: We now examine how sharing prioritized experiences instead of randomly sampled ones affects the performance of SAPIENS .

Original: t [Figure] i
Revised: In Figure 16 we repeat the same experiment with Figure 4, with the difference that all methods compute priorities, which they employ both for implementing a prioritized

Original: [Figure]
Revised: [Figure]

Original: l

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We now examine how sharing prioritized experiences instead of randomly sampled ones affects the performance of SAPIENS . In Figure 16 we repeat the same experiment with Figure 4, with the difference that all methods compute priorities, which they employ both for implementing a prioritized replay buffer and sharing experiences by sampling them in proportion to their priorities.
Revised: I n t e r - g r o u p a li g n m e n t , A j , j t

Original: As we see, using priorities negatively impacts experience sharing
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: I n t e r - g r o u p a li g n m e n t ,

Revised: A j , j t

Revised: [Figure]

Revised: I n t e r - g r o u p a li g n m e n t ,

Revised: A j , j t

Revised: [Figure]

Revised: I n t e r

Revised: [Figure]

Revised: u p a li g n

Revised: [Figure]

Revised: n t

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: i t h e l

Revised: [Figure]

Revised: m

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: Single path

Revised: Merging paths

Revised: Best-of-ten paths

Original: [Figure]
Revised: [Figure]

Original:  while it helps speed up the performance of the single agent in the single path task.
Revised: i [Figure] replay buffer and sharing experiences by sampling them in proportion to their priorities. As we see, using priorities negatively impacts experience sharing, while it helps speed up the performance of the single agent in the single path task.

Original: This behavior has been observed in previous works Souza et al.
Revised: This behavior has been observed in previous works Souza et al.

Original: and can be attributed to the fact that the priorities of the sender do not necessarily agree with the priorities of the receiver and, therefore, destabilize learning.
Revised: and can be attributed to the fact that the priorities of the sender do not necessarily agree with the priorities of the receiver and, therefore, destabilize learning.

Original: A. A DDITIONAL TEST - BED : THE DECEPTIVE COINS GAME
Revised: E. A DDITIONAL TEST - BED : THE DECEPTIVE COINS GAME

Original: Deceptive games are grid-world tasks introduced to test the ability of deep RL agents to avoid local optima.
Revised: Deceptive games are grid-world tasks introduced to test the ability of deep RL agents to avoid local optima.

Original: (Bontrager et al., 2019).
Revised: (Bontrager et al., 2019).

Original: Here, we perform preliminary experiments with our own JAXbased implementation of one of the games: the first difficulty level of the deceptive coins game (see Figure 17 for an illustration).
Revised: Here, we perform preliminary experiments with our own JAXbased implementation of one of the games: the first difficulty level of the deceptive coins game (see Figure 17 for an illustration).

Original: Here, the agent can navigate in the grid-world during an episode and collect diamonds, which give a unit of reward.
Revised: Here, the agent can navigate in the grid-world during an episode and collect diamonds, which give a unit of reward.

Original: The game finishes once the agent reaches the fire, which offers an additional reward, or when a timeout of 14 time steps is reached.
Revised: The game finishes once the agent reaches the fire, which offers an additional reward, or when a timeout of 14 time steps is reached.

Original: There are two possible paths the agent can follow: moving left and reaching the fire will give a reward of two while moving right and reaching the fire will give a reward of five.The second path is more rewarding [Figure] [Figure]
Revised: There are two possible paths the agent can follow: moving left and reaching the fire will give a reward of two while moving right and reaching the fire will give a reward of five.

Original: [Figure] [Figure] [Figure] but is harder to complete because, once an agent discovers the easier-to-find diamond on the left, it is deceived into following the left path.
Revised: The second path is more rewarding but is harder to complete because, once an agent discovers the easier-to-find diamond on the left, it is deceived into following the left path.

Original: Once an agent commits on a path (reaches the edge of the grid-world) a barrier is raised so that the agent cannot go back within that episode.
Revised: Once an agent commits on a path (reaches the edge of the grid-world) a barrier is raised so that the agent cannot go back within that episode.

Original: We now examine the performance of SAPIENS under different social network structures (fullyconnected, small-world, ring, dynamic), as well as the no-sharing, A2C and Ape-X baselines for three group sizes: 6 , 10 and 20 agents.
Revised: We now examine the performance of SAPIENS under different social network structures (fullyconnected, small-world, ring, dynamic), as well as the no-sharing, A2C and Ape-X baselines for three group sizes: 6 , 10 and 20 agents.

Original: We present the reward plots for the 3 sizes in Figures 18,and 20, respectively, and present an overall comparison in Figure 21 (equivalent to Figure 4 for the
Revised: We present the reward plots for the 3 sizes in Figures 18,and 20, respectively, and present an overall comparison in Figure 21 (equivalent to Figure 4 for the

Original: Wordcraft tasks).
Revised: Wordcraft tasks).

Original: We observe that all conditions found either the local or the global optimum and that : a) A2C fails for all network sizes.
Revised: We observe that all conditions found either the local or the global optimum and that : a) A2C fails for all network sizes.

Original: This behavior has been observed in previous works (Bontrager et al., 2019) and can be attributed to the fact that policy-gradient methods are more susceptible to local minima b) no-sharing gets stuck in the local optimum in half of the trials when the group size is small.
Revised: This behavior has been observed in previous works (Bontrager et al., 2019) and can be attributed to the fact that policy-gradient methods are more susceptible to local minima b) no-sharing gets stuck in the local optimum in half of the trials when the group size is small.

Original: Increasing the group size increases the probability that at least one agent in the group will escape the local minima by ϵ -greedy exploration c) partially connected structures find the global minima across network sizes d) fully-connected converges to the local optimum for the large group size, although the global optimum was discovered at the early exploration phase (see Figure 20).
Revised: Increasing the group size increases the probability that at least one agent in the group will escape the local minima by ϵ -greedy exploration c) partially connected structures find the global minima across network sizes d) fully-connected converges to the local optimum for the large group size, although the global optimum was discovered at the early exploration phase (see Figure 20).

Original: Thus, too much experience sharing is harmful e) Ape-X fails with high probability for all network sizes. 
Revised: Thus, too much experience sharing is harmful e) Ape-X fails with high probability for all network sizes. [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: cc

Original: In general, our conclusions in this task are consistent with what we observe in Wordcraft, in particular the merging paths task that has a similar deceptive nature.
Revised: [Figure] ss ******** * **** ** **** [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: In general, our conclusions in this task are consistent with what we observe in Wordcraft, in particular the merging paths task that has a similar deceptive nature.