Original: S OCIAL N ETWORK S TRUCTURE S HAPES I NNOVATION : E XPERIENCE SHARING IN RL WITH SAPIENS
Revised: S OCIAL N ETWORK S TRUCTURE S HAPES I NNOVATION : E XPERIENCE SHARING IN RL WITH SAPIENS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones
Revised: Human culture relies on innovation: our ability to continuously explore how existing elements can be combined to create new ones

Original: Innovation is not solitary it relies on collective search and accumulation.
Revised: Innovation is not solitary it relies on collective search and accumulation.

Original: Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation.
Revised: Reinforcement learning (RL) approaches commonly assume that fully-connected groups are best suited for innovation.

Original: However human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures
Revised: However human laboratory and field studies have shown that hierarchical innovation is more robustly achieved by dynamic social network structures

Original: In dynamic settings humans oscillate between innovating individually or in small clusters and then sharing outcomes with others.
Revised: In dynamic settings humans oscillate between innovating individually or in small clusters and then sharing outcomes with others.

Original: To our knowledge the role of social network structure on innovation has not been systematically studied in
Revised: To our knowledge the role of social network structure on innovation has not been systematically studied in

Original: RL
Revised: RL

Original: Here, we use a multi- level problem setting (WordCraft with three different in- novation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms
Revised: Here, we use a multi- level problem setting (WordCraft with three different in- novation tasks to test the hypothesis that the social network structure affects the performance of distributed RL algorithms

Original: We systematically design networks of
Revised: We systematically design networks of

Original: DQNs sharing experiences from their replay buffers in varying structures
Revised: DQNs sharing experiences from their replay buffers in varying structures

Original: (fully
Revised: (fully

Original: - connected small world dynamic ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward focused evaluation framework of RL
Revised: - connected small world dynamic ring) and introduce a set of behavioral and mnemonic metrics that extend the classical reward focused evaluation framework of RL

Original: Comparing the level of innovation achieved by different social network structures across different tasks shows that, first
Revised: Comparing the level of innovation achieved by different social network structures across different tasks shows that, first

Original: , consistent with human findings experience sharing within a dynamic structure achieves the highest level of inno vation in tasks with a deceptive nature and large search spaces
Revised: , consistent with human findings experience sharing within a dynamic structure achieves the highest level of inno vation in tasks with a deceptive nature and large search spaces

Original: Second experience sharing is not as helpful when there is a single clear path to innovation
Revised: Second experience sharing is not as helpful when there is a single clear path to innovation

Original: Third
Revised: Third

Original: , the metrics we propose
Revised: , the metrics we propose

Original: , can help understand the success of different social network structures on different tasks
Revised: , can help understand the success of different social network structures on different tasks

Original: , with the diversity of experiences on an individual and group level lending crucial insights
Revised: , with the diversity of experiences on an individual and group level lending crucial insights

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unlike herds or swarms, human social networks solve different tasks with different topologies (Momennejad, 2022).
Revised: Unlike herds or swarms, human social networks solve different tasks with different topologies (Momennejad, 2022).

Original: Human and computational studies show that properties of both the social network structure and task affect the abilities of groups to search collectively: social network structures with high connectivity are better suited for quick convergence in problems with clear global optima (Coman et al., 2016; Momennejad et al., 2019), while partially-connected structures perform best in deceptive tasks, where acting greedily in the short-term leads to missing the optimal solution (Derex & Boyd, 2016; Lazer & Friedman, 2007; Cantor et al., 2021; Du et al., 2021; Adjodah et al., 2019).
Revised: Human and computational studies show that properties of both the social network structure and task affect the abilities of groups to search collectively: social network structures with high connectivity are better suited for quick convergence in problems with clear global optima (Coman et al., 2016; Momennejad et al., 2019), while partially-connected structures perform best in deceptive tasks, where acting greedily in the short-term leads to missing the optimal solution (Derex & Boyd, 2016; Lazer & Friedman, 2007; Cantor et al., 2021; Du et al., 2021; Adjodah et al., 2019).

Original: Despite this evidence, works in distributed reinforcement learning (RL) have focused on fully-connected architectures (Mnih et al., 2016; Horgan et al., 2018; Espeholt et al., 2018;
Revised: Despite this evidence, works in distributed reinforcement learning (RL) have focused on fully-connected architectures (Mnih et al., 2016; Horgan et al., 2018; Espeholt et al., 2018;

Original: Nair et al., 2015; Christianos et al., 2020;
Revised: Nair et al., 2015; Christianos et al., 2020;

Original: Schmitt et al., 2019; Jaderberg et al., 2018).
Revised: Schmitt et al., 2019; Jaderberg et al., 2018).

Original: Here, we test the performance of different social network structures in groups of RL agents that share their experiences in a distributed RL learning paradigm.
Revised: Here, we test the performance of different social network structures in groups of RL agents that share their experiences in a distributed RL learning paradigm.

Original: We refer to such groups as multi-agent topologies, introduce SAPIENS , a learning framework for Structuring multi-Agent toPologies for Innovation through ExperieNce Sharing 1 , and evaluate it on a deceptive task that models collective innovation.
Revised: We refer to such groups as multi-agent topologies, introduce SAPIENS , a learning framework for Structuring multi-Agent toPologies for Innovation through ExperieNce Sharing 1 , and evaluate it on a deceptive task that models collective innovation.

Original: Innovations represent the expansion of an agent’s behavioral repertoire with new problem-solving abilities and are, therefore, a necessary ingredient of continuous learning (Leibo et al., 2019).
Revised: Innovations represent the expansion of an agent’s behavioral repertoire with new problem-solving abilities and are, therefore, a necessary ingredient of continuous learning (Leibo et al., 2019).

Original: They arise from tinkering, recombination and adoption of existing innovations (Sol´e et al., 2013; Derex &
Revised: They arise from tinkering, recombination and adoption of existing innovations (Sol´e et al., 2013; Derex &

Original: Boyd, 2016) and have been characterized as a type of combinatorial search constrained by semantics dictating the feasible combinations of innovations (Sol´e et al., 2013; Derex & Boyd, 2016).
Revised: Boyd, 2016) and have been characterized as a type of combinatorial search constrained by semantics dictating the feasible combinations of innovations (Sol´e et al., 2013; Derex & Boyd, 2016).

Original: We adopt this definition: innovations are a type of collective search task with : a) a multi-level search space, where innovations arise out of recombination of existing ones (Hafner, 2021) b) rewards that increase monotonically with the level of innovation, in order to capture the human intrinsic motivation for progress (Sol´e et al., 2013).
Revised: We adopt this definition: innovations are a type of collective search task with : a) a multi-level search space, where innovations arise out of recombination of existing ones (Hafner, 2021) b) rewards that increase monotonically with the level of innovation, in order to capture the human intrinsic motivation for progress (Sol´e et al., 2013).

Original: Laboratory and field studies of human groups have shown that collective innovation is highly contingent on the social network structure (Momennejad, 2022; Migliano & Vinicius, 2022; Derex &
Revised: Laboratory and field studies of human groups have shown that collective innovation is highly contingent on the social network structure (Momennejad, 2022; Migliano & Vinicius, 2022; Derex &

Original: Boyd, 2016).
Revised: Boyd, 2016).

Original: The reason for this lies in the exploration versus exploitation dynamics of social networks.
Revised: The reason for this lies in the exploration versus exploitation dynamics of social networks.

Original: High clustering and long shortest path in partially-connected structures help maintain diversity in the collective at the benefit of exploration, while high connectivity quickly leads to conformity, which benefits exploitation (Lazer & Friedman, 2007).
Revised: High clustering and long shortest path in partially-connected structures help maintain diversity in the collective at the benefit of exploration, while high connectivity quickly leads to conformity, which benefits exploitation (Lazer & Friedman, 2007).

Original: Of particular interest are structures that achieve a balance in this trade-off: small-worlds are static graphs that, due a modular structure with long-range connections, achieve both high clustering and small shortest path (Watts & Strogatz, 1998).
Revised: Of particular interest are structures that achieve a balance in this trade-off: small-worlds are static graphs that, due a modular structure with long-range connections, achieve both high clustering and small shortest path (Watts & Strogatz, 1998).

Original: Another example are dynamic structures , where agents are able to periodically change neighbors (Volz & Meyers, 2007).
Revised: Another example are dynamic structures , where agents are able to periodically change neighbors (Volz & Meyers, 2007).

Original: These two families of graphs have the attractive property that they both locally protect innovations and quickly disseminate good solutions (Derex & Boyd, 2016).
Revised: These two families of graphs have the attractive property that they both locally protect innovations and quickly disseminate good solutions (Derex & Boyd, 2016).

Original: Despite progress on multiple fronts, many open questions remain before we get a clear understanding of how social network structure shapes innovation.
Revised: Despite progress on multiple fronts, many open questions remain before we get a clear understanding of how social network structure shapes innovation.

Original: On the cognitive science side, computational and human laboratory studies of collective innovation are few and have studied a single task where two innovations are combined to create a new one (Derex & Boyd, 2016; Cantor et al., 2021), while most works study other types of collective search that do not resemble innovation (Mason & Watts, 2012; Mason et al., 2008; Lazer & Friedman, 2007; Fang et al., 2010).
Revised: On the cognitive science side, computational and human laboratory studies of collective innovation are few and have studied a single task where two innovations are combined to create a new one (Derex & Boyd, 2016; Cantor et al., 2021), while most works study other types of collective search that do not resemble innovation (Mason & Watts, 2012; Mason et al., 2008; Lazer & Friedman, 2007; Fang et al., 2010).

Original: Furthermore, laboratory studies have collected purely behavioral data (Mason et al., 2008; Derex & Boyd, 2016), while studies of collective memory have shown significant influence of social interactions on individual memory (Coman et al., 2016), indicating that mnemonic data may be another good source of information.
Revised: Furthermore, laboratory studies have collected purely behavioral data (Mason et al., 2008; Derex & Boyd, 2016), while studies of collective memory have shown significant influence of social interactions on individual memory (Coman et al., 2016), indicating that mnemonic data may be another good source of information.

Original: In distributed RL, studies are hypothesizing that the reason why groups outperform single agents not just in terms of speed, but also in terms of performance, is the increased diversity of experiences collected by heterogeneous agents but not explicitly measure it (Nair et al., 2015; Horgan et al., 2018).
Revised: In distributed RL, studies are hypothesizing that the reason why groups outperform single agents not just in terms of speed, but also in terms of performance, is the increased diversity of experiences collected by heterogeneous agents but not explicitly measure it (Nair et al., 2015; Horgan et al., 2018).

Original: In this case two steps seem natural: introducing appropriate metrics of diversity, and increasing it, not only through heterogeneity, but also through the social network topology.
Revised: In this case two steps seem natural: introducing appropriate metrics of diversity, and increasing it, not only through heterogeneity, but also through the social network topology.

Original: To achieve this we propose SAPIENS , a distributed RL learning framework for modeling a group of agents exchanging experiences according to a social network topology.
Revised: To achieve this we propose SAPIENS , a distributed RL learning framework for modeling a group of agents exchanging experiences according to a social network topology.

Original: We study instantiations of
Revised: We study instantiations of

Original: SAPIENS where multiple DQN learners (Mnih et al., 2013) share experience tuples from their replay buffers with their neighbors in different static and dynamic social network structures and compare them to other distributed RL algorithms (Mnih et al., 

Revised: SAPIENS where multiple DQN learners (Mnih et al., 2013) share experience tuples from their replay buffers with their neighbors in different static and dynamic social network structures and compare them to other distributed RL algorithms (Mnih et al.,

Original: ; Nair et al., 2015).
Revised: ; Nair et al., 2015).

Original: We employ Wordcraft (Jiang et al., 2020b) as a test-bed and design three custom tasks (Figures 1 and 2) covering innovation challenges of different complexity: (i) a task with a single innovation path to an easy-to-find global optimum.
Revised: We employ Wordcraft (Jiang et al., 2020b) as a test-bed and design three custom tasks (Figures 1 and 2) covering innovation challenges of different complexity: (i) a task with a single innovation path to an easy-to-find global optimum.

Original: This type of task can be used to model a linear innovation structure, such as the evolution of the fork from knife to having, two-, three- and, eventually four tines.
Revised: This type of task can be used to model a linear innovation structure, such as the evolution of the fork from knife to having, two-, three- and, eventually four tines.

Original: (Sol´e et al., 2013) and is not a deceptive task.
Revised: (Sol´e et al., 2013) and is not a deceptive task.

Original: (ii) a task with two paths that individually lead to local optima, but when combined, can merge toward the global optimum.
Revised: (ii) a task with two paths that individually lead to local optima, but when combined, can merge toward the global optimum.

Original: Ispired from previous studies in cognitive science (Derex &
Revised: Ispired from previous studies in cognitive science (Derex &

Original: Boyd, 2016; Cantor et al., 2021) this task we can capture innovations that were repurposed after their invention, such as Gutenberg’s screw press leading to the print press (Sol´e et al., 2013).
Revised: Boyd, 2016; Cantor et al., 2021) this task we can capture innovations that were repurposed after their invention, such as Gutenberg’s screw press leading to the print press (Sol´e et al., 2013).

Original: (iii) a task with ten paths, only one of which leads to the global optimum, which captures search in vast spaces.
Revised: (iii) a task with ten paths, only one of which leads to the global optimum, which captures search in vast spaces.

Original: In addition to the two deceptive tasks in Wordcraft, we also evaluate
Revised: In addition to the two deceptive tasks in Wordcraft, we also evaluate

Original: SAPIENS algorithms on a deceptive task implemented in a grid world.
Revised: SAPIENS algorithms on a deceptive task implemented in a grid world.

Original: We empirically show that the performance of SAPIENS depends on the inter-play between social network structure and task demands.
Revised: We empirically show that the performance of SAPIENS depends on the inter-play between social network structure and task demands.

Original: Dynamic structures perform most robustly, converging quickly in the easy task, avoiding local optima in the second task, and exploring efficiently in the third task.
Revised: Dynamic structures perform most robustly, converging quickly in the easy task, avoiding local optima in the second task, and exploring efficiently in the third task.

Original: To interpret these findings, we propose and compute novel behavioral and mnemonic metrics that quantify, among others, the diversity of experiences.
Revised: To interpret these findings, we propose and compute novel behavioral and mnemonic metrics that quantify, among others, the diversity of experiences.

Original: Contributions
Revised: Contributions

Original: Our contributions are two-fold.
Revised: Our contributions are two-fold.

Original: From a cognitive science perspective , SAPIENS is, to our knowledge, the first computational study of hypotheses in human studies relating social network structure to collective innovation, that employs deep RL as the individual learning mechanism.
Revised: From a cognitive science perspective , SAPIENS is, to our knowledge, the first computational study of hypotheses in human studies relating social network structure to collective innovation, that employs deep RL as the individual learning mechanism.

Original: Compared to the simple agent-based models employed by previous computational studies (Lazer & Friedman, 2007; Cantor et al., 2021; Mason & Watts, 2012), deep RL offers three main advantages : i) it enables empirical experiments with more complex test-beds and larger search spaces
Revised: Compared to the simple agent-based models employed by previous computational studies (Lazer & Friedman, 2007; Cantor et al., 2021; Mason & Watts, 2012), deep RL offers three main advantages : i) it enables empirical experiments with more complex test-beds and larger search spaces

Original: [Figure]
Revised: [Figure]

Original: + =
Revised: + =

Original: [Figure]
Revised: [Figure]

Original: ii) agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents, such as the majority rule (Lazer & Friedman, 2007; Cantor et al., 2021) iii) by using the replay buffer as a proxy of the memories of agents, we can directly measure properties such as the diversity of experiences that are challenging to estimate with humans.
Revised: ii) agents can share their experience by simply exchanging transitions from their respective replay buffers, without requiring ad-hoc mechanisms for copying the behaviors of other agents, such as the majority rule (Lazer & Friedman, 2007; Cantor et al., 2021) iii) by using the replay buffer as a proxy of the memories of agents, we can directly measure properties such as the diversity of experiences that are challenging to estimate with humans.

Original: Aside these methodological contributions, as we will see later, our empirical study leads to clear hypotheses for future experiments with humans.
Revised: Aside these methodological contributions, as we will see later, our empirical study leads to clear hypotheses for future experiments with humans.

Original: From an RL perspective , our work extends upon the distributed RL paradigm by systematically analyzing the effect of static and dynamic social network structures on different types of innovation tasks both in terms of performance and novel behavioral and mnemonic metrics.
Revised: From an RL perspective , our work extends upon the distributed RL paradigm by systematically analyzing the effect of static and dynamic social network structures on different types of innovation tasks both in terms of performance and novel behavioral and mnemonic metrics.

Original:  M ETHODS 2. W ORDCRAFT : A TEST - BED FOR INNOVATION
Revised:  M ETHODS 2. W ORDCRAFT : A TEST - BED FOR INNOVATION

Original: We perform experiments on Wordcraft (Jiang et al., 2020b), an RL environment inspired from the game Little Alchemy 2 2 .
Revised: We perform experiments on Wordcraft (Jiang et al., 2020b), an RL environment inspired from the game Little Alchemy 2 2 .

Original: As we illustrate on the left of Figure 1, tasks start with an initial set of elements and the player explores combinations of two elements in order to create new ones.
Revised: As we illustrate on the left of Figure 1, tasks start with an initial set of elements and the player explores combinations of two elements in order to create new ones.

Original: In Wordcraft one can create different types of tasks using a recipe book ( X valid ), which is a dictionary containing the valid element combinations of two elements and the newly crafted one.
Revised: In Wordcraft one can create different types of tasks using a recipe book ( X valid ), which is a dictionary containing the valid element combinations of two elements and the newly crafted one.

Original: In addition to the recipe book, a task requires a reward function R valid that returns a scalar reward associated with crafting element z .
Revised: In addition to the recipe book, a task requires a reward function R valid that returns a scalar reward associated with crafting element z .

Original: A valid combination returns a new element and reward only the first time it is chosen.
Revised: A valid combination returns a new element and reward only the first time it is chosen.

Original: When queried with a non-valid combination, R valid returns a reward of zero.
Revised: When queried with a non-valid combination, R valid returns a reward of zero.

Original: Thus, a task can be be described by a tuple ( X 0 , X valid , R valid , T ) , where X 0 denotes the initial set of elements and T is the number of time steps available to an agent before the environment resets, and can be modelled as a fully-observable Markov Decision process (see Appendix A).
Revised: Thus, a task can be be described by a tuple ( X 0 , X valid , R valid , T ) , where X 0 denotes the initial set of elements and T is the number of time steps available to an agent before the environment resets, and can be modelled as a fully-observable Markov Decision process (see Appendix A).

Original: 2. I NNOVATION TASKS
Revised: 2. I NNOVATION TASKS

Original: We introduce the following concepts to characterize the structure of innovation tasks.
Revised: We introduce the following concepts to characterize the structure of innovation tasks.

Original: An innovation task can contain one or more paths, which can potentially be connected to each other (as in the merging paths task).
Revised: An innovation task can contain one or more paths, which can potentially be connected to each other (as in the merging paths task).

Original: In our proposed tasks, an innovation path X is defined as a sequence of elements [ X 1 , ..., X n ] , where crafting an element
Revised: In our proposed tasks, an innovation path X is defined as a sequence of elements [ X 1 , ..., X n ] , where crafting an element

Original: X
Revised: X

Original: i ( i > 1) requires to combine the previously crafted element X i − 1 and a base element from the initial set.
Revised: i ( i > 1) requires to combine the previously crafted element X i − 1 and a base element from the initial set.

Original: The first element X 1 in the sequence requires combining elements from the initial set or from other paths.
Revised: The first element X 1 in the sequence requires combining elements from the initial set or from other paths.

Original: The innovation level of an element corresponds to the length of the path it belongs to, plus the sum of the path lengths required to craft its first item ( 0 if the first item is crafted from base elements).
Revised: The innovation level of an element corresponds to the length of the path it belongs to, plus the sum of the path lengths required to craft its first item ( 0 if the first item is crafted from base elements).

Original: For example, the innovation level of A 3 in the single path task is 3 , while the innovation level of element C 1 in the merging paths task is 5 ( 1 on C + 2 on A + 2 on B).
Revised: For example, the innovation level of A 3 in the single path task is 3 , while the innovation level of element C 1 in the merging paths task is 5 ( 1 on C + 2 on A + 2 on B).

Original: Within an innovation task, the trajectory of an agent is defined
Revised: Within an innovation task, the trajectory of an agent is defined

Original: Combination
Revised: Combination

Original: I nn o v a t i o n l e v e l
Revised: I nn o v a t i o n l e v e l

Original: Result Reward ...
Revised: Result Reward ...

Original: S i n g l e p a t h
Revised: S i n g l e p a t h

Original: M e r g i n g p a t h s
Revised: M e r g i n g p a t h s

Original: B e s t - o ft e n p a t h s ...
Revised: B e s t - o ft e n p a t h s ...

Original: 14 + + = = ...
Revised: 14 + + = = ...

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: A A
Revised: A A

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: a...
Revised: a...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: a
Revised: a

Original: A A
Revised: A A

Original: A A
Revised: A A

Original: A A
Revised: A A

Original: A... aaaa...
Revised: A... aaaa...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: a
Revised: a

Original: A A
Revised: A A

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: C C
Revised: C C

Original: A A
Revised: A A

Original: [Table]
Revised: [Table]

Original: A A
Revised: A A

Original: A
Revised: A

Original: A... a
Revised: A... a

Original: [Table]
Revised: [Table]

Original: c
Revised: c

Original: [Table]
Revised: [Table]

Original: a
Revised: a

Original: [Table]
Revised: [Table]

Original: aa...
Revised: aa...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: B
Revised: B

Original: [Table]
Revised: [Table]

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: b
Revised: b

Original: [Table]
Revised: [Table]

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: b
Revised: b

Original: B
Revised: B

Original: B B
Revised: B B

Original: B B
Revised: B B

Original: B
Revised: B

Original: B... bbbb...
Revised: B... bbbb...

Original: + + + + =
Revised: + + + + =

Original: =
Revised: =

Original: = =
Revised: = =

Original: j J
Revised: j J

Original: J J
Revised: J J

Original: J J
Revised: J J

Original: J
Revised: J

Original: J... jjjj
Revised: J... jjjj

Original: [Figure]
Revised: [Figure]

Original: as the sequence of crafted elements it produces.
Revised: as the sequence of crafted elements it produces.

Original: Finally, the optimal trajectory of an innovation task is the trajectory that returns the highest cumulative reward within the problem horizon T .
Revised: Finally, the optimal trajectory of an innovation task is the trajectory that returns the highest cumulative reward within the problem horizon T .

Original: We design three innovation tasks, shown in Figure 2, that pose different challenges:
Revised: We design three innovation tasks, shown in Figure 2, that pose different challenges:

Original: Single innovation path The initial set contains three elements ( X valid = { a 1 , a 2 , a 3 } ) and .
Revised: Single innovation path The initial set contains three elements ( X valid = { a 1 , a 2 , a 3 } ) and .

Original: An agent needs to first combine two of them to create the first element and then progresses further by combining the most recently created element with an appropriate one from the initial set.
Revised: An agent needs to first combine two of them to create the first element and then progresses further by combining the most recently created element with an appropriate one from the initial set.

Original: This optimization problem contains a single global optimum.
Revised: This optimization problem contains a single global optimum.

Original: Merging paths There are two paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward to the end path A, moving forward to the end of path B, or combining elements from path A and B to progress on path C.
Revised: Merging paths There are two paths, A and B, and at level 2 there is a cross-road that presents the player with three options: moving forward to the end path A, moving forward to the end of path B, or combining elements from path A and B to progress on path C.

Original: The latter is more rewarding and is, thus, the optimal choice.
Revised: The latter is more rewarding and is, thus, the optimal choice.

Original: This task is particularly challenging because the player needs to avoid two local optima before finding the global optimum.
Revised: This task is particularly challenging because the player needs to avoid two local optima before finding the global optimum.

Original: Best-of-ten paths Here, one of the ten paths is the most rewarding, but, to find it, the player must first explore and reject the other nine paths.
Revised: Best-of-ten paths Here, one of the ten paths is the most rewarding, but, to find it, the player must first explore and reject the other nine paths.

Original: This optimization task is characterized by a single global optimum and 9 local one and its challenge lies in its large search space.
Revised: This optimization task is characterized by a single global optimum and 9 local one and its challenge lies in its large search space.

Original: 2. L EARNING FRAMEWORK
Revised: 2. L EARNING FRAMEWORK

Original: SAPIENS considers a group of K DQN agents, where each agent interacts with its own copy of the environment and can share experiences with others.
Revised: SAPIENS considers a group of K DQN agents, where each agent interacts with its own copy of the environment and can share experiences with others.

Original: An undirected graph G , with nodes indicating agents and edges indicating that two agents share experiences with each other, determines who shares information with whom .
Revised: An undirected graph G , with nodes indicating agents and edges indicating that two agents share experiences with each other, determines who shares information with whom .

Original: We define the neighborhood of agent k , N k , as the set of nodes connected to it.
Revised: We define the neighborhood of agent k , N k , as the set of nodes connected to it.

Original: At the end of each episode the agent shares experiences with each of its neighbors with probability p s : sharing consists of sampling a subset of experiences of length L s from its own buffer B k and inserting it in the buffers of all neighbors B n , n ∈ N k .
Revised: At the end of each episode the agent shares experiences with each of its neighbors with probability p s : sharing consists of sampling a subset of experiences of length L s from its own buffer B k and inserting it in the buffers of all neighbors B n , n ∈ N k .

Original: Thus, an agent communicates distinct experiences with each of its neighbors.
Revised: Thus, an agent communicates distinct experiences with each of its neighbors.

Original: We present a schematic of two DQN agents sharing
Revised: We present a schematic of two DQN agents sharing

Original: [Figure]
Revised: [Figure]

Original: experiences on the right of Figure 3 and the pseudocode of SAPIENS in Appendix C. We also provide more implementation details about the DQN in Appendix A.
Revised: experiences on the right of Figure 3 and the pseudocode of SAPIENS in Appendix C. We also provide more implementation details about the DQN in Appendix A.

Original: Thus, SAPIENS is a distributed RL learning paradigm where all agents are both actors and learners, a setting distinct from multi-agent RL (Garnelo et al., 2021; Christianos et al., 2020; Jiang et al., 2020a), where agents co-exist in the same environment and from parallelised RL (Steinkraus et al., 2005), where there need to be multiple agents.
Revised: Thus, SAPIENS is a distributed RL learning paradigm where all agents are both actors and learners, a setting distinct from multi-agent RL (Garnelo et al., 2021; Christianos et al., 2020; Jiang et al., 2020a), where agents co-exist in the same environment and from parallelised RL (Steinkraus et al., 2005), where there need to be multiple agents.

Original: It should also be distinguished from distributed RL paradigms with a single learner and multiple actors (Horgan et al., 2018; Espeholt et al., 2018;
Revised: It should also be distinguished from distributed RL paradigms with a single learner and multiple actors (Horgan et al., 2018; Espeholt et al., 2018;

Original: Nair et al., 2015; Garnelo et al., 2021), as multiple policies are learned simultaneously.
Revised: Nair et al., 2015; Garnelo et al., 2021), as multiple policies are learned simultaneously.

Original: We visualize the social network structures studied in our work on the left of Figure 3: fullyconnected, small-world, ring and dynamic.
Revised: We visualize the social network structures studied in our work on the left of Figure 3: fullyconnected, small-world, ring and dynamic.

Original: We construct the latter by grouping the agents in pairs and then allowing agents to visit other groups for a pre-determined duration, T v with probability p v .
Revised: We construct the latter by grouping the agents in pairs and then allowing agents to visit other groups for a pre-determined duration, T v with probability p v .

Original: This is a common type of dynamic topology that has been used in human laboratory and field studies (Derex & Boyd, 2016; Migliano & Vinicius, 2022) and thoeretically studied (Volz & Meyers, 2007).
Revised: This is a common type of dynamic topology that has been used in human laboratory and field studies (Derex & Boyd, 2016; Migliano & Vinicius, 2022) and thoeretically studied (Volz & Meyers, 2007).

Original: We provide more information about it in Appendix D, where we study how its behavior changes with different values of its hyper-parameters T v , p v .
Revised: We provide more information about it in Appendix D, where we study how its behavior changes with different values of its hyper-parameters T v , p v .

Original: We also present and analyze an alternative type of dynamic topology where the group oscillates between phases of full and no connectivity.
Revised: We also present and analyze an alternative type of dynamic topology where the group oscillates between phases of full and no connectivity.

Original: 2. E VALUATION FRAMEWORK
Revised: 2. E VALUATION FRAMEWORK

Original: During evaluation trials with experience sharing deactivated, we measure the quality of the final solution and the convergence speed.
Revised: During evaluation trials with experience sharing deactivated, we measure the quality of the final solution and the convergence speed.

Original: In addition, we define metrics that are not directly related to performance but can help us analyze the effect of social network structure.
Revised: In addition, we define metrics that are not directly related to performance but can help us analyze the effect of social network structure.

Original: These are behavioral metrics , characterizing the policies followed by the agents, and mnemonic metrics , characterizing the replay buffers of agents, either at an individual or group level.
Revised: These are behavioral metrics , characterizing the policies followed by the agents, and mnemonic metrics , characterizing the replay buffers of agents, either at an individual or group level.

Original: Performance-based metrics: (i) S , group success, a binary variable denoting whether at least one agent in a group found the optimal solution (ii) R + t : the maximum reward of the group at training step t ; (iii) R ∗ t : the average reward of the group at training step t ; (iv) T + , Time to first success: the first training step at which at least one of the agents found the optimal solution; (v) T ∗ , Time to all successes: the first training step at which all of the agents found the optimal solution (vi) T > ,
Revised: Performance-based metrics: (i) S , group success, a binary variable denoting whether at least one agent in a group found the optimal solution (ii) R + t : the maximum reward of the group at training step t ; (iii) R ∗ t : the average reward of the group at training step t ; (iv) T + , Time to first success: the first training step at which at least one of the agents found the optimal solution; (v) T ∗ , Time to all successes: the first training step at which all of the agents found the optimal solution (vi) T > ,

Original: Spread time: number of training steps required for the optimal solution to spread to the whole group, once at least one member discovered it (equals T ∗ − T + ).
Revised: Spread time: number of training steps required for the optimal solution to spread to the whole group, once at least one member discovered it (equals T ∗ − T + ).

Original: Behavioral metrics: (i) conformity C t denotes the percentage of agents in a group that end up with the same element at the end of a given evaluation trial.
Revised: Behavioral metrics: (i) conformity C t denotes the percentage of agents in a group that end up with the same element at the end of a given evaluation trial.

Original: Thus, agents conform to each other even if they follow alternative trajectories.
Revised: Thus, agents conform to each other even if they follow alternative trajectories.

Original: ; (ii) volatility V t is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by an agent across episodes.
Revised: ; (ii) volatility V t is an agent-level metric that denotes the cumulative number of changes in the trajectory followed by an agent across episodes.

Original: Mnemonic metrics: (i) diversity D kt is an agent-level metric that denotes the number of unique experiences in an agent’s replay buffer; (ii) D G t is a group-level metric that captures the diversity of the aggregated group buffer..
Revised: Mnemonic metrics: (i) diversity D kt is an agent-level metric that denotes the number of unique experiences in an agent’s replay buffer; (ii) D G t is a group-level metric that captures the diversity of the aggregated group buffer..

Original:  R ESULTS
Revised:  R ESULTS

Original: We now evaluate the social network structures presented in Figure 3 on the innovation tasks described in Section 2.2 and visualized in Figure 2.
Revised: We now evaluate the social network structures presented in Figure 3 on the innovation tasks described in Section 2.2 and visualized in Figure 2.

Original: Specifically, methods ring, small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with
Revised: Specifically, methods ring, small-world, dynamic and fully-connected are instantiations of SAPIENS for different social network structures with

Original: DQNs where shared experiences are sampled randomly from the replay buffers.
Revised: DQNs where shared experiences are sampled randomly from the replay buffers.

Original: We benchmark
Revised: We benchmark

Original: SAPIENS against: a) no-sharing, a setting with 10 DQN agents without experience sharing b) single, a single DQN agent c) A2C, a distributed policy-gradient algorithm where 10 workers share gradients with a single (Mnih et al., 2016) and d) Ape-X, a distributed RL algorithm with 10 workers that share experience samples with a single DQN learner (Horgan et al., 2018).
Revised: SAPIENS against: a) no-sharing, a setting with 10 DQN agents without experience sharing b) single, a single DQN agent c) A2C, a distributed policy-gradient algorithm where 10 workers share gradients with a single (Mnih et al., 2016) and d) Ape-X, a distributed RL algorithm with 10 workers that share experience samples with a single DQN learner (Horgan et al., 2018).

Original: To test for statistical significance we perform ANOVA for multiple and Tukey range tests for pairwise comparisons.
Revised: To test for statistical significance we perform ANOVA for multiple and Tukey range tests for pairwise comparisons.

Original: We refer readers to Appendix E for more information about our experimental setup, including tables 2, 2 and 2 that contain the means and standard deviations for evaluation metrics in the three tasks.
Revised: We refer readers to Appendix E for more information about our experimental setup, including tables 2, 2 and 2 that contain the means and standard deviations for evaluation metrics in the three tasks.

Original: 3. O VERALL COMPARISON
Revised: 3. O VERALL COMPARISON

Original: In Figure 4 we compare methods across tasks in terms of group success S and Time to first success T + , where we also indicate statistically significant pairwise comparisons with asterisks (more asterisks denote higher significance).
Revised: In Figure 4 we compare methods across tasks in terms of group success S and Time to first success T + , where we also indicate statistically significant pairwise comparisons with asterisks (more asterisks denote higher significance).

Original: We observe that performance varies across tasks and topologies.
Revised: We observe that performance varies across tasks and topologies.

Original: The single path task is optimally solved by all methods, except for single and Ape-X, which failed in 20% and 15% of the trials respectively due to slow convergence.
Revised: The single path task is optimally solved by all methods, except for single and Ape-X, which failed in 20% and 15% of the trials respectively due to slow convergence.

Original: ANOVA showed no significant difference in terms of group success S ( p = 0 . 43 ) but
Revised: ANOVA showed no significant difference in terms of group success S ( p = 0 . 43 ) but

Original: methods differed significantly in terms of T
Revised: methods differed significantly in terms of T

Original: + ( p = 0 .
Revised: + ( p = 0 .

Original: 2 e − 13 ).
Revised: 2 e − 13 ).

Original: In particular, single is significantly slower than all other methods ( T + = 648 e 3 for single) and A2C is significantly quicker than other methods ( T + = 36200 for A2C), with the Tukey’s range test indicating significance with p = 0 .
Revised: In particular, single is significantly slower than all other methods ( T + = 648 e 3 for single) and A2C is significantly quicker than other methods ( T + = 36200 for A2C), with the Tukey’s range test indicating significance with p = 0 .

Original: 001 for all pairwise comparisons with these two methods.
Revised: 001 for all pairwise comparisons with these two methods.

Original: This is in agreement with the expectation that a single agent learns slower due to having less data and that the policy-gradient algorithm A2C is more sample efficient than value function algorithms like DQNs.
Revised: This is in agreement with the expectation that a single agent learns slower due to having less data and that the policy-gradient algorithm A2C is more sample efficient than value function algorithms like DQNs.

Original: In the merging paths task there were significant differences among methods both for group success S ( p = 0 .
Revised: In the merging paths task there were significant differences among methods both for group success S ( p = 0 .

Original: 4 e − 4 ) and convergence speed T + ( p = 0 . 0095 ).
Revised: 4 e − 4 ) and convergence speed T + ( p = 0 . 0095 ).

Original: The group success of dynamic ( S = 0 . 65 ) is significantly higher compared to Ape-X ( S = 0 .
Revised: The group success of dynamic ( S = 0 . 65 ) is significantly higher compared to Ape-X ( S = 0 .

Original: 05 , p = 0 .
Revised: 05 , p = 0 .

Original: 001 ), A2C ( S = 0 .
Revised: 001 ), A2C ( S = 0 .

Original: 0 , p = 0 . 00101 ), fully-connected ( S = 0 .
Revised: 0 , p = 0 . 00101 ), fully-connected ( S = 0 .

Original: 0 , p = 0 . 00101 ) and ring ( S = 0 .
Revised: 0 , p = 0 . 00101 ) and ring ( S = 0 .

Original: 2 , p = 0 . 0105 ).
Revised: 2 , p = 0 . 0105 ).

Original: The single, no-sharing and small-world structures performed comparably well, but did not show statistically significant differences with other methods.
Revised: The single, no-sharing and small-world structures performed comparably well, but did not show statistically significant differences with other methods.

Original: In terms of T + , we see that dynamic is quicker than other methods with positive S , which leads to it being the only method with statistically significant differences with
Revised: In terms of T + , we see that dynamic is quicker than other methods with positive S , which leads to it being the only method with statistically significant differences with

Original: Ape-X ( p = 0 . 0292 ), fully-connected ( p = 0 . 0421 )
Revised: Ape-X ( p = 0 . 0292 ), fully-connected ( p = 0 . 0421 )

Original: and A2C ( p = 0 . 0421 ), the methods that failed in almost all trials and hence have T + equal to the budget of the experiment.
Revised: and A2C ( p = 0 . 0421 ), the methods that failed in almost all trials and hence have T + equal to the budget of the experiment.

Original: Thus, our main conclusion in the merging-paths task is that methods with fully-connected topologies (fully-connected,
Revised: Thus, our main conclusion in the merging-paths task is that methods with fully-connected topologies (fully-connected,

Original: A2C, Ape-X) fail dramatically and that the dynamic structure succeeds with higher probability.
Revised: A2C, Ape-X) fail dramatically and that the dynamic structure succeeds with higher probability.

Original: Finally, in the best-of-ten paths task, differences are also significant both for T + ( p = 0 .
Revised: Finally, in the best-of-ten paths task, differences are also significant both for T + ( p = 0 .

Original: 9 e − 7 ) and S ( p = 0 .
Revised: 9 e − 7 ) and S ( p = 0 .

Original: 15 e − 6 ).
Revised: 15 e − 6 ).

Original: Here, dynamic outperforms all methods in terms of S with all Tukey range tests indicating significance with p = 0 .
Revised: Here, dynamic outperforms all methods in terms of S with all Tukey range tests indicating significance with p = 0 .

Original: 001 , which leads to it also having significantly higher convergence rate ( T + = 14 e 6 for dynamic) compared to the other methods that exhausted their time budget.
Revised: 001 , which leads to it also having significantly higher convergence rate ( T + = 14 e 6 for dynamic) compared to the other methods that exhausted their time budget.

Original: In additional experiments we have also: a) observed that these conclusions are consistent across network sizes, with the performance of partially-connectedsocial network structures improving with increasing network size and the performance of the fully-connected decreasing (see Appendix E.5) b) observed a drop in performance under prioritized experience sharing, where the DQNs employ prioritized replay buffers  and experiences are selected for sharing proportionally to their priority  (see Appendix E.6) .
Revised: In additional experiments we have also: a) observed that these conclusions are consistent across group sizes, with increasing sizes leading to better performance in partially-connected and worse in fully-connected structures (see Appendix E.5) b) observed a drop in performance under prioritized experience sharing (Souza et al., 2019; Horgan et al., 2018), where the DQNs employ prioritized replay buffers (Schaul et al., 2016) and experiences with higher priority are shared more often (see Appendix E.6).

Original: This behavior has also been observed in other works (Souza et al., 2019) and we hypothesize that it is due to the fact that priorities computed by an agent do not necessarily agree with the priorities of the other  c) analyzed how the performance ofthe dynamictopology varies with its hyper-parameters and derived suggestions for tuning it (see Appendix D) d) monitored the robustness of social network structures to different learning hyper-parameters and observed thatthe dynamictopology is more robust than fully-connected and no-sharing (see Section E.4)
Revised: In agreement with previous works (Souza et al., 2019), we observe that performance degrades in all methods that share experiences. This does not happen for no-sharing, which indicates that prioritized experiences are detrimental only when they are shared. To address this, agents can recompute priorities upon receiving them from other agents to ensure they agree with their own experience (Horgan et al., 2018). c) analyzed how the performance of dynamic varies with its hyperparameters and derived suggestions for tuning it (see Appendix D) d) monitored the robustness of social network structures to different learning hyper-parameters and observed that dynamic is more robust than fully-connected and no-sharing (see Appendix E.4)

Original: e) computed measures of the alignment of experiences within and across groups that further support our hypothesis that the content of replay buffersis highly contingent on the social network structure (see Appendix E.3) f) performed simulations in another deceptive test-bed and derived similar con-
Revised: e) measured alignment of experiences within and across groups that further support our hypothesis that the content of replay buffers

Original: , G r o u p s u cc e ss
Revised: , G r o u p s u cc e ss

Original: Single path 0.
Revised: Single path 0.

Original: Merging paths 0.
Revised: Merging paths 0.

Original: Best-of-ten paths 0.
Revised: Best-of-ten paths 0.

Original: T + , T i m e t
Revised: T + , T i m e t

Original: o f i r s t s u cc e ss 1e ** *** *** *** *** *** *** *** ***** ** ** 0.
Revised: o f i r s t s u cc e ss 1e ** *** *** *** *** *** *** *** ***** ** ** 0.

Original: 3.0 1e * * * singleno-sharingdynamicfully-connected ringsmall-worldA2CApe-X   3 1e *** *** *** ****** *** *** clusions to what we observed in Wordcraft, namely that partially-connected structures are better at avoiding local optima (See Appendix E.7) 
Revised: 3.0 1e * * * singleno-sharingdynamicfully-connected ringsmall-worldA2CApe-X   3 1e *** *** *** ****** *** *** is highly contingent on the social network structure (see Appendix E.3) f) performed simulations in another deceptive test-bed and derived similar conclusions to what we observed in Wordcraft, namely that partially-connected structures are better at avoiding local optima (See Appendix E.7) g) tested for the robustness of SAPIENS methods to the amount of sharing (hyper-parameters L s and p s introduced in Section 2.3) in Appendix E.8), where we observe sub-optimal performance for low amounts of sharing in dynamic and for large shared batches in fully-connected structures.

Original: To explain the differences for SAPIENS under different structures we now study each task in isolation for no-sharing, fully-connected, ring, small-world and dynamic, focusing on the behavioral and mnemonic metrics described in Section 2.4.
Revised: To explain the differences for SAPIENS under different structures we now study each task in isolation for nosharing, fully-connected, ring, small-world and dynamic, focusing on the behavioral and mnemonic metrics described in Section 2.4.

Original: 3. T ASK : S INGLE PATH
Revised: 3. T ASK : S INGLE PATH

Original: Previous human (Mason et al., 2008) and computational (Lazer & Friedman, 2007) studies have indicated that, when it comes to problems with a single global optimum, fully-connected topologies perform best.
Revised: Previous human (Mason et al., 2008) and computational (Lazer & Friedman, 2007) studies have indicated that, when it comes to problems with a single global optimum, fully-connected topologies perform best.

Original: Our analysis here, however, indicates no statistically significant difference between methods: the fully-connected topology was actually the slowest ( T + = 311 e 3 ).
Revised: Our analysis here, however, indicates no statistically significant difference between methods: the fully-connected topology was actually the slowest ( T + = 311 e 3 ).

Original: To shed light into this behavior, we turn towards diversity and volatility.
Revised: To shed light into this behavior, we turn towards diversity and volatility.

Original: We compare average individual diversity ( ¯ D t ) and group diversity D G t , where for each method we create two samples by looking at these two metrics at the timestep at which diversity starts decreasing.
Revised: We compare average individual diversity ( ¯ D t ) and group diversity D G t , where for each method we create two samples by looking at these two metrics at the timestep at which diversity starts decreasing.

Original: ANOVA tests indicate that differences are significant both for ¯ D T
Revised: ANOVA tests indicate that differences are significant both for ¯ D T

Original: + ( p = 0 .
Revised: + ( p = 0 .

Original: 16 e − 8 ), where all pairwise comparisons are significant based on the Tukey range test, and D G t ( p = 0 . 0005 ), where the significantly different pairs are (no-sharing, dynamic, p = 0 . 02 ), (dynamic, fully-connected, p = 0 . 001 ), (dynamic, small-world, p = 0 . 0087 ), (fully-connected, ring, p = 0 . 008476 ).
Revised: 16 e − 8 ), where all pairwise comparisons are significant based on the Tukey range test, and D G t ( p = 0 . 0005 ), where the significantly different pairs are (no-sharing, dynamic, p = 0 . 02 ), (dynamic, fully-connected, p = 0 . 001 ), (dynamic, small-world, p = 0 . 0087 ), (fully-connected, ring, p = 0 . 008476 ).

Original: Intuitively, fully-connected exhibits the highest average individual diversity ¯ D t (left in Figure 5) and the lowest group diversity D G t (left in Figure 5): sharing experiences with others diversifies an individual’s experiences but also homogenizes the group.
Revised: Intuitively, fully-connected exhibits the highest average individual diversity ¯ D t (left in Figure 5) and the lowest group diversity D G t (left in Figure 5): sharing experiences with others diversifies an individual’s experiences but also homogenizes the group.

Original: This task does not require group diversity, but we expect that high individual diversity should be indicative of quicker exploration.
Revised: This task does not require group diversity, but we expect that high individual diversity should be indicative of quicker exploration.

Original: So why does the higher ¯ D t of fully-connected not act to its benefit?
Revised: So why does the higher ¯ D t of fully-connected not act to its benefit?

Original: To answer this we turn to volatility V t (top left of Figure 5) and observe that it is highest for the fully-connected topology.
Revised: To answer this we turn to volatility V t (top left of Figure 5) and observe that it is highest for the fully-connected topology.

Original: We tested for statistically significant differences among methods in terms of volatility at convergence ( V T train ) and found that these pairs differ significantly: (small-world, ring, p = 0 . 002387 ), (small-world, dynamic, p = 0 . 001 ), (small-world, no-sharing, p = 0 . 001 ), (small-world, fully-connected, p = 0 . 001 ), (ring, fully-connected, p = 0 . 001 ), (dynamic, fullyconnected, p = 0 . 001 ) and (no-sharing, fully-connected, p = 0 . 001 ).
Revised: We tested for statistically significant differences among methods in terms of volatility at convergence ( V T train ) and found that these pairs differ significantly: (small-world, ring, p = 0 . 002387 ), (small-world, dynamic, p = 0 . 001 ), (small-world, no-sharing, p = 0 . 001 ), (small-world, fully-connected, p = 0 . 001 ), (ring, fully-connected, p = 0 . 001 ), (dynamic, fullyconnected, p = 0 . 001 ) and (no-sharing, fully-connected, p = 0 . 001 ).

Original: We hypothesize that the higher individual diversity and volatility of the fully-connected structure are linked and indicate that, although experience sharing helps speed up exploration, it also destabilizes agents, so that no  net gain is observed in terms of convergence rate.
Revised: We hypothesize that the higher individual diversity and volatility of the fully-connected structure are linked and indicate that, although experience sharing helps speed up exploration, it also destabilizes agents, so that no [Figure] net gain is observed in terms of convergence rate.

Original: Notably the fact that experience sharing degrades performance in RL has been observed but not understood (Souza et al., 2019;
Revised: Notably the fact that experience sharing degrades performance in RL has been observed but not understood (Souza et al., 2019;

Original: Schmitt et al., 2019), while as we see here, this becomes possible by analyzing diversity and volatility.[Figure]
Revised: Schmitt et al., 2019), while as we see here, this becomes possible by analyzing diversity and volatility.

Original:  T ASK : M ERGING PATHS
Revised: 3. T ASK : M ERGING PATHS

Original: The hypothesis that motivated this task was that partially connected groups will perform better than fully-connected structures due to their ability to explore diverse trajectories and avoid the two local optima(Derex & Boyd, 2016).
Revised: The hypothesis that motivated this task was that partially connected groups will perform better than fully-connected structures due to their ability to explore diverse trajectories and avoid the two local optima (Derex & Boyd, 2016).

Original: This was indeed the case as we saw in our discussion of Figure 4 in
Revised: This was indeed the case as we saw in our discussion of Figure 4 in

Original: Section 3.1.
Revised: Section 3.1.

Original: If we also look at the diversities in this task, then we see that the group diversity D G of fully connected (illustrated in the middle bottom plot in Figure 5) differs significantly to the one of (no-sharing, p = 0 .
Revised: If we also look at the diversities in this task, then we see that the group diversity D G of fully connected (illustrated in the middle bottom plot in Figure 5) differs significantly to the one of (no-sharing, p = 0 .

Original: 001 ), (ring, p = 0 . 0067 )
Revised: 001 ), (ring, p = 0 . 0067 )

Original: and dynamic ( p = 0 . 0177 ).
Revised: and dynamic ( p = 0 . 0177 ).

Original: The group diversity in small-world, which was relatively successful ( S = 0 . 3 ), follows that of the other partially connected structures but peaks at a lower level ( D G ≈ 1700 ).
Revised: The group diversity in small-world, which was relatively successful ( S = 0 . 3 ), follows that of the other partially connected structures but peaks at a lower level ( D G ≈ 1700 ).

Original: We believe that this is due to the dual ability of this topology to both protect and spread information and that its diversity and performance will improve as we increase the size of the group, enabling higher clustering (we illustrate this behavior in Figure 15 in Appendix E.5 where we increase the size of the network to 50 agents).
Revised: We believe that this is due to the dual ability of this topology to both protect and spread information and that its diversity and performance will improve as we increase the size of the group, enabling higher clustering (we illustrate this behavior in Figure 15 in Appendix E.5 where we increase the size of groups to 50 agents).

Original: 3. T ASK : B EST - OF - TEN PATH
Revised: 3. T ASK : B EST - OF - TEN PATH

Original: Which social network topology is most appropriate in large search spaces?
Revised: Which social network topology works best in large search spaces?

Original: Our analysis in Figure 4 clearly indicates thatthe dynamictopology explores more robustly and achieves the highest performance in this case.
Revised: Our analysis in Figure 4 clearly indicates that dynamic achieves the highest performance.

Original: Differently from the two previous tasks where fully-connected exhibited the highest volatility, dynamic and ring, the topologies with the lowest number of connections, are the most volatile here (top row of Figure 5). This is not surprising: volatility indicates that an agent is exploring more paths, which is necessary to optimally solve the task.
Revised: Differently from the two previous tasks where fully-connected exhibited the highest volatility, dynamic and ring, the topologies with the lowest number of connections, are the most volatile here (top row of Figure 5), as agents in them are exploring quickly and, hence, performing better.

Original: In terms of group diversity, we found that no-sharing scored significantly lower than all other methods with p = 0 .
Revised: In terms of group diversity, we found that no-sharing scored significantly lower than all other methods with p = 0 .

Original: 001 , the small-world surpassed all methods, with its difference from dynamic being marginally statistically significant and fully-connected did not score last.
Revised: 001 , the small-world surpassed all methods, with its difference from dynamic being marginally statistically significant and fully-connected did not score last.

Original: Thus, differently from the other two tasks with small search spaces, quick spreading of information increases group diversity here.
Revised: Thus, differently from the other two tasks with small search spaces, quick spreading of information increases group diversity here.

Original: However, to solve the task quickly, group diversity needs to be combined with quick, local exploitation of the optimal path, which is possible under structures with large shortest path, such as dynamic and ring.
Revised: However, to solve the task quickly, group diversity needs to be combined with quick, local exploitation of the optimal path, which is possible under structures with large shortest path, such as dynamic and ring.

Original: Another interesting observation in this task, as well as in the single-path task is that, as we reported above, dynamic  exhibits significantly higher group diversity than no-sharing and it is the only structure to do so: this indicates that shared experiences can foster group exploration.
Revised: Another interesting observation here and in the single-path task is that dynamic is the only structure that exhibits significantly higher group diversity than no-sharing, indicating that it fosters group exploration.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: In distributed RL, shared information has the form of experience tuples or gradients, with the former being preferred due to instabilities and latency concerns in the latter (Horgan et al., 2018;
Revised: In distributed RL, shared information has the form of experience tuples or gradients, with the former being preferred due to instabilities and latency concerns in the latter (Horgan et al., 2018;

Original: Mnih et al., 2016; Schmitt et al., 2019; Nair et al., 2015; Garnelo et al., 2021).
Revised: Mnih et al., 2016; Schmitt et al., 2019; Nair et al., 2015; Garnelo et al., 2021).

Original: A common social network structure in this area is that of multiple actors and a single learner (Mnih et al., 2016; Horgan et al., 2018;
Revised: A common social network struc- ture is that of multiple actors and a single learner (Mnih et al., 2016; Horgan et al., 2018;

Original: Schmitt et al., 2019; Garnelo et al., 2021), while the Gorila framework (Nair et al., 2015) is a more general multi-learner architecture, that differs from ours as agents are sharing parameters rather than experience tuples.
Revised: Schmitt et al., 2019; Garnelo et al., 2021), while the Gorila framework (Nair et al., 2015) is a more general multi-learner architecture, that differs from ours as agents are sharing parameters.

Original: Networked Evolutionary Strategies (NetES) enable the study of the effect of network structure on evolutionary strategies (Adjodah et al., 2019), where, multiple actors are sharing gradients with a single learner.
Revised: Networked Evolutionary Strategies (NetES) consider the effect of network structure on evolutionary strategies (Adjodah et al., 2019), where multiple actors share gradients with a single learner.

Original: NetES was applied on continuous control tasks, whose optimization landscape differs from our deceptive and discrete innovation tasks.
Revised: NetES was applied on continuous control tasks that differ from our deceptive and discrete innovation tasks.

Original: In MARL social network structureusually determines who co-exists with whom in the environment (Garnelo et al., 2021) or which agents in its neighborhood an agent takes into account (Jiang et al., 2020a;
Revised: In MARL, social network structure determines who co-exists with whom in the environment (Garnelo et al., 2021) or which agents in its neighborhood an agent attends to (Jiang et al., 2020a;

Original: Du et al., 2021).
Revised: Du et al., 2021).

Original: Here dynamic topologies that are adapted to maximize a group’s reward have been shown to maximize strategic diversity (Garnelo et al., 2021) and help the agents coordinate on demand (Du et al., 2021).
Revised: Here, dynamic topologies that are adapted to maximize a group’s reward have been shown to maximize strategic diversity (Garnelo et al., 2021) and help the agents coordinate on demand (Du et al., 2021).

Original: In contrast, our dynamic topologies vary periodically independently of the group’s performance, which is important for avoidingconvergence to local optima.
Revised: In contrast, our dynamic topologies vary periodically independently of the group’s performance, which is important for avoiding local optima.

Original: In population-based training, policies are compared against the whole population, thus only considering a fully-connected social network structure (Jaderberg et al., 2018).
Revised: In population-based training, policies are compared against the whole population, thus only considering a fully-connected social network structure (Jaderberg et al., 2018).

Original: Admittedly, the literature on the effect of social network structure on collective search is diverse, with different fields making different design choices; to illustrate this we provide a non-exhaustive summary of our literature review in Table 1 of Appendix B.
Revised: Admittedly, the literature on the effect of social network structure on collective search is diverse, with different fields making different design choices; to illustrate this we provide a non-exhaustive summary of our literature review in Table 1 of Appendix B.

Original:  D ISCUSSION AND F UTURE W ORK
Revised:  D ISCUSSION AND F UTURE W ORK

Original: We tested the hypothesis that the social network structure of experience sharing can shape the performance of a group of RL agents using our proposed learning framework, SAPIENS , and showed that, in line with human studies, both social network topology and task structure affect performance.
Revised: We tested the hypothesis that the social network structure of experience sharing can shape the performance of a group of RL agents using our proposed learning framework, SAPIENS , and showed that, in line with human studies, both social network topology and task structure affect performance.

Original: Based on our experimental results, we can provide general recommendations on which topology to use for which task class.
Revised: Based on our experimental results, we can provide general recommendations on which topology to use for which task class.

Original: In the single-path task, an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks (Gupta et al., 2019)), our results show no benefit of experience sharing.
Revised: In the single-path task, an instance of a class of tasks with no strong local optima (similarly to long-horizon tasks (Gupta et al., 2019)), our results show no benefit of experience sharing.

Original: In the merging-path task which exhibits strong local optima that have to be explored up to a certain point in order to discover the global optimum (in the spirit of hard exploration tasks (Baker et al., 2022; Ecoffet et al., 2021)), our results show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) perform best.
Revised: In the merging-path task which exhibits strong local optima that have to be explored up to a certain point in order to discover the global optimum (in the spirit of hard exploration tasks (Baker et al., 2022; Ecoffet et al., 2021)), our results show that topologies with low initial connectivity (such as no-sharing, small world and dynamic) perform best.

Original: The dynamic topology shows the highest performance, allowing different groups to explore non-optimal paths before sharing their experience during visits to other groups to find the optimal one.
Revised: The dynamic topology shows the highest performance, allowing different groups to explore non-optimal paths before sharing their experience during visits to other groups to find the optimal one.

Original: Finally, in the case of large search space with many local optimas (in the spirit of combinatorial optimization tasks (Mazyavkina et al., 2021)), our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.
Revised: Finally, in the case of large search space with many local optimas (in the spirit of combinatorial optimization tasks (Mazyavkina et al., 2021)), our results show that the dynamic topology performs best, allowing different groups to first explore different paths, then spread the optimal solution to other groups once discovered.

Original: When adopting RL algorithms as computational models for replicating experiments with humans, one needs to acknowledge that their communication and decision-making mechanisms may not faithfully replicate the ones used by humans.
Revised: When adopting RL algorithms as computational models for replicating experiments with humans, one needs to acknowledge that their communication and decision-making mechanisms may not faithfully replicate the ones used by humans.

Original: One notable difference is that humans may exhibit normative behavior, adopting information not for its utility in the task but for social approval (Mason et al., 2008).
Revised: One notable difference is that humans may exhibit normative behavior, adopting information not for its utility in the task but for social approval (Mason et al., 2008).

Original: From an RL perspective, our study is limited in including experiments only in a few symbolic tasks and a simple navigation task; in the future we plan to extend to more complex environments like Crafter (Hafner, 2021).
Revised: From an RL perspective, our study is limited in including experiments only in a few symbolic tasks and a simple navigation task; in the future we plan to study more complex environments like Crafter (Hafner, 2021).

Original: We hope that our work will contribute to the fields of cognitive science and DRL in multiple ways.
Revised: We hope that our work will contribute to the fields of cognitive science and DRL in multiple ways.

Original: First, our empirical observations in the single path and best-of-ten-path tasks provide concrete hypotheses for future experiments studying human innovation, which has so far been studied only in a task that inspired our merging-paths task (Derex & Boyd, 2016).
Revised: First, our empirical observations in the single path and best-of-ten-path tasks provide concrete hypotheses for future experiments studying human innovation, which has so far been studied only in a task that inspired our merging-paths task (Derex & Boyd, 2016).

Original: By continuing the dialogue that has been initiated between human and computational studies (Fang et al., 2010; Lazer & Friedman, 2007; Cantor et al., 2021) to include DRL methods, we believe that cognitive science will benefit from tools that, as we show here, can learn in realistic problem set-ups and can be analyzed not just in terms of their behavior, but also in terms of their memories.
Revised: By continuing the dialogue that has been initiated between human and computational studies (Fang et al., 2010; Lazer & Friedman, 2007; Cantor et al., 2021) to include DRL methods, we believe that cognitive science will benefit from tools that, as we show here, can learn in realistic problem set-ups and can be analyzed not just in terms of their behavior, but also in terms of their memories.

Original: Second, we hope that studies in distributed RL will extend their evaluation methodology by analyzing not just rewards, but also behavioral and mnemonic metrics such as diversity, conformity and volatility that, as we show here, correlate with success.
Revised: Second, we hope that studies in distributed RL will extend their evaluation methodology by analyzing not just rewards, but also behavioral and mnemonic metrics such as diversity, conformity and volatility that, as we show here, correlate with success.

Original: Aside this, the effect of social network structure in distributed RL can be extended beyond evolutionary strategies (Adjodah et al., 2019) and beyond our current instantiation of SAPIENS , by considering other off-policy algorithms than DQNs and other types of information sharing.
Revised: Aside this, the effect of social network structure in distributed RL can be extended beyond evolutionary strategies (Adjodah et al., 2019) and beyond our current instantiation of SAPIENS , by considering other off-policy algorithms than DQNs and other types of information sharing.

Original: Finally, considering the effectiveness of the dynamic topologies observed in this study, we envision future works that investigate more types of them, as well as meta-learning or onlineadaptation algorithms where the social network structure is optimized for a desired objective.
Revised: Finally, considering the effectiveness of the dynamic topologies observed in this study, we envision future works that investigate more types of them, as well as meta-learning or onlineadaptation algorithms where the social network structure is optimized for a desired objective.

Original: A D ETAILS OF W ORDCRAFT AS A M ARKOV D ECISION P ROCESS
Revised: A D ETAILS OF W ORDCRAFT AS A M ARKOV D ECISION P ROCESS

Original: We consider the episodic setting, where the environment resets at the end of each episode and an agent is trained for E train episodes.
Revised: We consider the episodic setting, where the environment resets at the end of each episode and an agent is trained for E train episodes.

Original: At each time step t , the agent observes the state s t and selects an action a t from a set of possible actions A according to its policy π θ , where π θ is a mapping from states to actions, parameterized by a neural network with weights θ .
Revised: At each time step t , the agent observes the state s t and selects an action a t from a set of possible actions A according to its policy π θ , where π θ is a mapping from states to actions, parameterized by a neural network with weights θ .

Original: In return, the agent receives the next state s t +1 and a scalar reward r t .
Revised: In return, the agent receives the next state s t +1 and a scalar reward r t .

Original: Each DQN agent collects experience tuples of the form [ s t , a t , s t +1 , r t ] in its replay buffer.
Revised: Each DQN agent collects experience tuples of the form [ s t , a t , s t +1 , r t ] in its replay buffer.

Original: Figure 6 offers a visualization of the states and actions encountered during an episode in Wordcraft, where the chosen actions and elements are chosen so as to reproduce the example of Figure 1.
Revised: Figure 6 offers a visualization of the states and actions encountered during an episode in Wordcraft, where the chosen actions and elements are chosen so as to reproduce the example of Figure 1.

Original: In order to solve the innovation task described in Section 2.1 we compute the maximum number of elements a player can craft within horizon T for recipe book X valid and initial set X 0 , which we denote as | X | .
Revised: In order to solve the innovation task described in Section 2.1 we compute the maximum number of elements a player can craft within horizon T for recipe book X valid and initial set X 0 , which we denote as | X | .

Original: We, then, encode each element as an integer in [0 , | X | ) .
Revised: We, then, encode each element as an integer in [0 , | X | ) .

Original: Thus, the action space is
Revised: Thus, the action space is

Original: [Table]
Revised: [Table]

Original: A = [0 , | X | ) , with action a t indicating the index of the currently chosen element.
Revised: A = [0 , | X | ) , with action a t indicating the index of the currently chosen element.

Original: The state s t contains two sets of information: a binary vector of length | X | with non-zero entries for elements already crafted by the agent within the current episode (we refer to this as inventory i ) and another binary vector of length | X | where an index is non-zero if it is currently selected by the agent (we refer to this as current c ).
Revised: The state s t contains two sets of information: a binary vector of length | X | with non-zero entries for elements already crafted by the agent within the current episode (we refer to this as inventory i ) and another binary vector of length | X | where an index is non-zero if it is currently selected by the agent (we refer to this as current c ).

Original: An agent begins with an inventory having non-zero element only for the initial set X 0 and an all-zero selection.
Revised: An agent begins with an inventory having non-zero element only for the initial set X 0 and an all-zero selection.

Original: With the first action a 0 , the selected item becomes non-zero in the selection.
Revised: With the first action a 0 , the selected item becomes non-zero in the selection.

Original: With the second action, a 1 , we check if the combination ( a 1 , c 0 ) is valid under the recipe book and, if so, return the newly crafted element (corresponding entry in i becomes non-zero) and the reward.
Revised: With the second action, a 1 , we check if the combination ( a 1 , c 0 ) is valid under the recipe book and, if so, return the newly crafted element (corresponding entry in i becomes non-zero) and the reward.

Original: This two-step procedure continues until the end of the episode.
Revised: This two-step procedure continues until the end of the episode.

Original: B S UMMARY OF RELATED WORKS
Revised: B S UMMARY OF RELATED WORKS

Original: In this appendix we provide a non-comprehensive summary of the literature on the topic of the effect of social network topology on collective search in Table 1, where our objective is to highlight similarities and differences within and across the fields of cognitive science and DRL.
Revised: In this appendix we provide a non-comprehensive summary of the literature on the topic of the effect of social network topology on collective search in Table 1, where our objective is to highlight similarities and differences within and across the fields of cognitive science and DRL.

Original: C P SEUDOCODE OF SAPIENS
Revised: C P SEUDOCODE OF SAPIENS

Original: We present the pseudocode of our proposed algorithm SAPIENS in Algorithm 1.
Revised: We present the pseudocode of our proposed algorithm SAPIENS in Algorithm 1.

Original: SAPIENS works similarly to an off-policy reinforcement learning algorithm, with the difference that, after each episode, an experience sharing phase takes place between agents that belong in the same group.
Revised: SAPIENS works similarly to an off-policy reinforcement learning algorithm, with the difference that, after each episode, an experience sharing phase takes place between agents that belong in the same group.

Original: 20: end
Revised: 20: end

Original: 22: end
Revised: 22: end

Original: D A NALYSIS OF DYNAMIC NETWORK TOPOLOGIES
Revised: D A NALYSIS OF DYNAMIC NETWORK TOPOLOGIES

Original: In the main paper we presented results for a single type of dynamic topolgoy.
Revised: In the main paper we presented results for a single type of dynamic topolgoy.

Original: Here we present another type and analyze how they both behave for different values of their hyper-parameters.
Revised: Here we present another type and analyze how they both behave for different values of their hyper-parameters.

Original: The two dynamic topologies are:
Revised: The two dynamic topologies are:

Original: • Inspired by graphs employed in human laboratory studies (Derex & Boyd, 2016), we designed graphs where the macro structure of the graph is constant but agents can randomly change their position.
Revised: • Inspired by graphs employed in human laboratory studies (Derex & Boyd, 2016), we designed graphs where the macro structure of the graph is constant but agents can randomly change their position.

Original: In particular, we divide a group of agents into sub-groups of two agents and, at the end of each episode, move an agent to another group with a probability p v for a duration of T v episodes (for a visualization see Figure 3).
Revised: In particular, we divide a group of agents into sub-groups of two agents and, at the end of each episode, move an agent to another group with a probability p v for a duration of T v episodes (for a visualization see Figure 3).

Original: To reduce the complexity of the implementation, we assume that only one visit can take place at a time.
Revised: To reduce the complexity of the implementation, we assume that only one visit can take place at a time.

Original: In the main paper we employ p v = 0 .
Revised: In the main paper we employ p v = 0 .

Original: 01 and T v = 10 across conditions and present results with different values in Appendix D, where we refer to this topology as dynamic-Boyd.
Revised: 01 and T v = 10 across conditions and present results with different values in Appendix D, where we refer to this topology as dynamic-Boyd.

Original: • Human behavioral ecology emphasize the importance of periodic variation in human social networks encountered throughout our evolutionary trajectory Wiessner (2014); Dunbar (2014).
Revised: • Human behavioral ecology emphasize the importance of periodic variation in human social networks encountered throughout our evolutionary trajectory Wiessner (2014); Dunbar (2014).

Original: Due to ecological constraints human groups oscillate between phases of high and low connectivity: low-connectivity phases arise when individuals need to individually collected resources (e
Revised: Due to ecological constraints human groups oscillate between phases of high and low connectivity: low-connectivity phases arise when individuals need to individually collected resources (e

Original: g. day-time hunting) while high-connectivity phases arise when humans are idle and “forced” to be in proximity with others (e
Revised: g. day-time hunting) while high-connectivity phases arise when humans are idle and “forced” to be in proximity with others (e

Original: g. fireside chats).
Revised: g. fireside chats).

Original: Although these high-connectivity phases do not bare a direct evolutionary advantage, they may have played an important role by creating the conditions for the evolution of human language and culture.
Revised: Although these high-connectivity phases do not bare a direct evolutionary advantage, they may have played an important role by creating the conditions for the evolution of human language and culture.

Original: Inspired by this hypothesis, we have designed dynamic graphs that oscillate between a fully-connected topology that lasts for T h episodes and a topology without sharing that lasts for T l episodes.
Revised: Inspired by this hypothesis, we have designed dynamic graphs that oscillate between a fully-connected topology that lasts for T h episodes and a topology without sharing that lasts for T l episodes.

Original: We present results for various values of T h and T d of this topology in Appendix D, where we refer to this topology as dynamic-periodic.
Revised: We present results for various values of T h and T d of this topology in Appendix D, where we refer to this topology as dynamic-periodic.

Original: In Figure 8, we observe the % of group success ( S G ) with the dynamic-Boyd topology for different probabilities of visit ( p v ) and visit duration T v (the sub-group size is 2 in all cases).
Revised: In Figure 8, we observe the % of group success ( S G ) with the dynamic-Boyd topology for different probabilities of visit ( p v ) and visit duration T v (the sub-group size is 2 in all cases).

Original: We note that, due to our implementation choice that a visit can take place only if no other agent is currently on a visit, the visit duration also affects the mixing of the group: longer visits mean that fewer visits
Revised: We note that, due to our implementation choice that a visit can take place only if no other agent is currently on a visit, the visit duration also affects the mixing of the group: longer visits mean that fewer visits

Original: will
Revised: will

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: take place in total.
Revised: take place in total.

Original: In the merging paths task (left), two hyper-parameter settings have a clear effect: (i) short visits with of high probability lead to bad performance.
Revised: In the merging paths task (left), two hyper-parameter settings have a clear effect: (i) short visits with of high probability lead to bad performance.

Original: As such settings lead to a quick mixing of the population, they lead to convergence to the local optimum (ii) long visits with high probability work well.
Revised: As such settings lead to a quick mixing of the population, they lead to convergence to the local optimum (ii) long visits with high probability work well.

Original: Due to the high visit probability, this setting effectively leads to topology where exactly one agent is always on a long visit.
Revised: Due to the high visit probability, this setting effectively leads to topology where exactly one agent is always on a long visit.

Original: Thus, it ensures that sub-groups stay isolated for at least 1000 episodes, after which inter sub-group sharing needs to takes place to ensure that the sub-groups can progress quickly.
Revised: Thus, it ensures that sub-groups stay isolated for at least 1000 episodes, after which inter sub-group sharing needs to takes place to ensure that the sub-groups can progress quickly.

Original: In the best-of-ten paths task (right), this structure has a clear optimal hyper-parameterization: short visits with high probability are preferred, which maximizes the mixing of the group and makes early exploration more effective.
Revised: In the best-of-ten paths task (right), this structure has a clear optimal hyper-parameterization: short visits with high probability are preferred, which maximizes the mixing of the group and makes early exploration more effective.

Original: In Figure 9, we observe the % of group success ( S G ) of the dynamic-periodic topology for various values of T h and T l .
Revised: In Figure 9, we observe the % of group success ( S G ) of the dynamic-periodic topology for various values of T h and T l .

Original: In the merging paths task (left of Figure 9) medium values for the period of both phases works best, while there is some success when the low connectivity phase lasts long ( T l = 1000 ).
Revised: In the merging paths task (left of Figure 9) medium values for the period of both phases works best, while there is some success when the low connectivity phase lasts long ( T l = 1000 ).

Original: In the best-of-ten paths task (rightof Figure 9), we observe the same medium values for the period of both phases work best: thus both the absolute value and their ratio is important to ensure that exploration is efficient.
Revised: In the best-of-ten paths task (rightof Figure 9), we observe the same medium values for the period of both phases work best: thus both the absolute value and their ratio is important to ensure that exploration is efficient.

Original: The optimal configuration is the same between the two tasks ( T l = 100 , T h = 10 ), which is a good indication of the robustness of this structure.
Revised: The optimal configuration is the same between the two tasks ( T l = 100 , T h = 10 ), which is a good indication of the robustness of this structure.

Original: E E MPIRICAL RESULTS
Revised: E E MPIRICAL RESULTS

Original: To ensure that all methods have the same number of samples, we assume that, for trials where a method did not find the optimal solution, and, hence, T + is undefined, T + is equal to the total number of timesteps the method was trained for, T train .
Revised: To ensure that all methods have the same number of samples, we assume that, for trials where a method did not find the optimal solution, and, hence, T + is undefined, T + is equal to the total number of timesteps the method was trained for, T train .

Original: For each task, all methods have been trained for an equal duration of time: T train = 1 e 6 for the single path , T train = 7 e 6 for the merging paths task and T train = 

Revised: For each task, all methods have been trained for an equal duration of time: T train = 1 e 6 for the single path , T train = 7 e 6 for the merging paths task and T train =

Original: e 7 for the best-of-ten paths task.
Revised: e 7 for the best-of-ten paths task.

Original: [Figure]
Revised: [Figure]

Original: o nn e c t i v i
Revised: o nn e c t i v i

Original: t
Revised: t

Original: y p
Revised: y p

Original: e r i o d
Revised: e r i o d

Original: S , % of group success
Revised: S , % of group success

Original: [Figure]
Revised: [Figure]

Original: period
Revised: period

Original: [Figure]
Revised: [Figure]

Original: T l ,
Revised: T l ,

Original: [Figure]
Revised: [Figure]

Original: o w - c o nn e c t i v i
Revised: o w - c o nn e c t i v i

Original: t
Revised: t

Original: y p
Revised: y p

Original: e r i o d
Revised: e r i o d

Original: S , % of
Revised: S , % of

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We perform 20 independent trials for each task and method and visualize our proposed metrics with barplots and line plots of averages across trials with error bars indicating 95% confidence intervals.
Revised: We perform 20 independent trials for each task and method and visualize our proposed metrics with barplots and line plots of averages across trials with error bars indicating 95% confidence intervals.

Original: We test for statistical significance of our evaluation metrics separately for each task by applying
Revised: We test for statistical significance of our evaluation metrics separately for each task by applying

Original: ANOVA tests 8 to detect whether at least one method differs from the rest and, subsequently, employing the Tukey’s range test 9 to detect which pairs of methods that differ significantly.
Revised: ANOVA tests 8 to detect whether at least one method differs from the rest and, subsequently, employing the Tukey’s range test 9 to detect which pairs of methods that differ significantly.

Original: We report the exact p values of theses tests in the text and, when applicable, illustrate them in figures using a set of asterisks whose number indicates the significance level ( p < = 0 . 

Revised: We report the exact p values of theses tests in the text and, when applicable, illustrate them in figures using a set of asterisks whose number indicates the significance level ( p < = 0 .

Original: : *, p < = 0 .
Revised: : *, p < = 0 .

Original: **, p < = 0 .
Revised: **, p < = 0 .

Original: ***, p < = 0 . 0001 :
Revised: ***, p < = 0 . 0001 :

Original: We presented the major results of our evaluation of SAPIENS in Section 3.
Revised: We presented the major results of our evaluation of SAPIENS in Section 3.

Original: We now present additional information regarding the implementation of the different components (Appendix E.1), the values of all performance metrics and additional plots for experiments discussed in 3 (Appendix E.2), results on intra-group and inter-group alignment (AppendixE.3), results for groups of varying sizes (Appendix E.5) and results on various dynamic topologies (Appendix D)
Revised: We now present additional information regarding the implementation of the different components (Appendix E.1), the values of all performance metrics and additional plots for experiments discussed in 3 (Appendix E.2), results on intra-group and inter-group alignment (AppendixE.3), results for groups of varying sizes (Appendix E.5) and results on various dynamic topologies (Appendix D)

Original: E.
Revised: E.

Original: I MPLEMENTATION DETAILS
Revised: I MPLEMENTATION DETAILS

Original: Implementation of DQN We employ the same hyper-parameter for each DQN across all studied tasks and topologies: discount factor γ = 0 .
Revised: Implementation of DQN We employ the same hyper-parameter for each DQN across all studied tasks and topologies: discount factor γ = 0 .

Original: 9 , the Adam optimizer with learning rate α = 0 .(Kingma & Ba, 2014; Dunbar, 2014), ϵ -greedy exploration with ϵ = 0 .
Revised: 9 , the Adam optimizer with learning rate α = 0 .(Kingma & Ba, 2014; Dunbar, 2014), ϵ -greedy exploration with ϵ = 0 .

Original: We employ a feedforward network with two layers with 64 neurons each.
Revised: We employ a feedforward network with two layers with 64 neurons each.

Original: We implemented SAPIENS by extending the DQN implementation in the stable-baselines3 framework.
Revised: We implemented SAPIENS by extending the DQN implementation in the stable-baselines3 framework.

Original: Implementation of A2C We used the stable-baselines3 implementation of A2C 11 and tuned the hyper-parameters: learning rate, number of steps, discount factor, the entropy coefficient and the value function coefficient.
Revised: Implementation of A2C We used the stable-baselines3 implementation of A2C 11 and tuned the hyper-parameters: learning rate, number of steps, discount factor, the entropy coefficient and the value function coefficient.

Original: This gave us the best-performing values 0.001, 5, 0.99, 0.1 and 0.25, respectively, that we also employed in the other tasks.
Revised: This gave us the best-performing values 0.001, 5, 0.99, 0.1 and 0.25, respectively, that we also employed in the other tasks.

Original: Implementation of Ape-X
Revised: Implementation of Ape-X

Original: We used the ray implementation of Ape-X DQN 12 and tuned the hyper-parameters: learning rate, discount factor, replay buffer capacity and ϵ -greedy exploration.
Revised: We used the ray implementation of Ape-X DQN 12 and tuned the hyper-parameters: learning rate, discount factor, replay buffer capacity and ϵ -greedy exploration.

Original: This gave us the best-performing values in the single path task 0.001, 0.9, 5000 and 0.02, respectively.
Revised: This gave us the best-performing values in the single path task 0.001, 0.9, 5000 and 0.02, respectively.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Implementation of graphs used as social network structures We construct small-worlds using the Watts–Strogatz model ( watts strogatz graph method of the networkx package 13 ).
Revised: Implementation of graphs used as social network structures We construct small-worlds using the Watts–Strogatz model ( watts strogatz graph method of the networkx package 13 ).

Original: This model first builds a ring lattice where each node has n neighbors and then rewires an edge with probability β .
Revised: This model first builds a ring lattice where each node has n neighbors and then rewires an edge with probability β .

Original: Compared to other techniques used in previous works studying the effect of topology
Revised: Compared to other techniques used in previous works studying the effect of topology

Original: Mason et al.
Revised: Mason et al.

Original: (2008), this way of constructing small-worlds ensures that the average path lengths is short and clustering is high.
Revised: (2008), this way of constructing small-worlds ensures that the average path lengths is short and clustering is high.

Original: These two properties are what differentiates small-worlds from random (short average path length and small clustering) and regular (long average path length and high clustering) graphs.
Revised: These two properties are what differentiates small-worlds from random (short average path length and small clustering) and regular (long average path length and high clustering) graphs.

Original: We employ n = 4 and β = 0 .
Revised: We employ n = 4 and β = 0 .

Original: 2 in our experiments, which we empirically found to lead to good values of average path length and clustering.
Revised: 2 in our experiments, which we empirically found to lead to good values of average path length and clustering.

Original: We have described the generation process of dynamic topologies in Appendix D. In the main paper we employ the dynamic-Boyd topology with T v = 10 and p v = 0 .
Revised: We have described the generation process of dynamic topologies in Appendix D. In the main paper we employ the dynamic-Boyd topology with T v = 10 and p v = 0 .

Original: 001 across tasks.
Revised: 001 across tasks.

Original: These parameters have been tuned for the merging-paths task.
Revised: These parameters have been tuned for the merging-paths task.

Original: E. O VERALL COMPARISON
Revised: E. O VERALL COMPARISON

Original: Tables 2, 3 and 4 contain the values of all metrics discussed in Section 2.4 for the single path, merging paths and best-of-ten paths, respectively.
Revised: Tables 2, 3 and 4 contain the values of all metrics discussed in Section 2.4 for the single path, merging paths and best-of-ten paths, respectively.

Original: We denote values computed after convergence of the group with underscore ∞ and values averaged over all training steps with underscore avg (note that we use ¯ over variables to denote averaging over agents in a single training step).
Revised: We denote values computed after convergence of the group with underscore ∞ and values averaged over all training steps with underscore avg (note that we use ¯ over variables to denote averaging over agents in a single training step).

Original: Cells with a dash (-) indicate that we could not compute the corresponding metrics because a group failed to find a solution in all trials.
Revised: Cells with a dash (-) indicate that we could not compute the corresponding metrics because a group failed to find a solution in all trials.

Original: We also provide the plots of volatility and average diversity for the merging paths and best-of-10 paths task (that were not included in Figure 5) due to page limit constraints).
Revised: We also provide the plots of volatility and average diversity for the merging paths and best-of-10 paths task (that were not included in Figure 5) due to page limit constraints).

Original: Figure 10 presents the reward curves for all methods in the single path, merging paths and best-often paths tasks respectively.
Revised: Figure 10 presents the reward curves for all methods in the single path, merging paths and best-often paths tasks respectively.

Original: Specifically, we plot the maximum reward of the group at training step t ( R + t ).
Revised: Specifically, we plot the maximum reward of the group at training step t ( R + t ).

Original: 13 https://networkx.org/
Revised: 13 https://networkx.org/

Original: Topology
Revised: Topology

Original: R + ∞
Revised: R + ∞

Original: R ∗∞
Revised: R ∗∞

Original: T +
Revised: T +

Original: T ∗
Revised: T ∗

Original: T >
Revised: T >

Original: S
Revised: S

Original: C avg ¯ V avg no-sharing (0.2124,
Revised: C avg ¯ V avg no-sharing (0.2124,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: dynamic
Revised: dynamic

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: E. M EASURING INTER - GROUP AND INTRA - GROUP ALIGNMENT
Revised: E. M EASURING INTER - GROUP AND INTRA - GROUP ALIGNMENT

Original: We have so far captures the agreement between agents in a group through the behavioral metric of conformity.
Revised: We have so far captures the agreement between agents in a group through the behavioral metric of conformity.

Original: Here, we present a mnemonic metric for agreement, which we term alignment.
Revised: Here, we present a mnemonic metric for agreement, which we term alignment.

Original: Alignment is a complementary metric to the diversity ( D kt ) and group diversity ( D G t ) metrics, that aims at capturing the effect of experience sharing on the replay buffers in a group.
Revised: Alignment is a complementary metric to the diversity ( D kt ) and group diversity ( D G t ) metrics, that aims at capturing the effect of experience sharing on the replay buffers in a group.

Original: We propose a definition of alignment within a single group (intra-group alignment A G t ) and a definition of alignment between two different groups ( A G j , G j t ).
Revised: We propose a definition of alignment within a single group (intra-group alignment A G t ) and a definition of alignment between two different groups ( A G j , G j t ).

Original: Such metrics of mnemonic convergence have been linked to social network topology (Coman et al., 2016) and, as we show here, they can prove useful in analyzing groups of reinforcement learning agents.
Revised: Such metrics of mnemonic convergence have been linked to social network topology (Coman et al., 2016) and, as we show here, they can prove useful in analyzing groups of reinforcement learning agents.

Original: Specifically: (i)
Revised: Specifically: (i)

Original: A G t is the intra-group alignment.
Revised: A G t is the intra-group alignment.

Original: This metric captures the similarity in terms of content between the replay buffers of agents belonging to the same group.
Revised: This metric captures the similarity in terms of content between the replay buffers of agents belonging to the same group.

Original: To compute this we compute the size of the common subset of experiences for each pair of agents and, then, average over all these pairs, normalizing in [0,1].
Revised: To compute this we compute the size of the common subset of experiences for each pair of agents and, then, average over all these pairs, normalizing in [0,1].

Original: (ii) inter-group alignment A G j , G j t is a similar notion of alignment but employed between different groups (e.g. how different is a group of fully-connected and a dynamic group of agents in terms of the content of their group replay buffers).
Revised: (ii) inter-group alignment A G j , G j t is a similar notion of alignment but employed between different groups (e.g. how different is a group of fully-connected and a dynamic group of agents in terms of the content of their group replay buffers).

Original: To compute it we concatenate all replay buffers of a group into a single one and then compute the size of the common subset of the two replay buffers.
Revised: To compute it we concatenate all replay buffers of a group into a single one and then compute the size of the common subset of the two replay buffers.

Original: Figure 12 presents intra-group alignment in the three tasks.
Revised: Figure 12 presents intra-group alignment in the three tasks.

Original: We observe that, in all tasks, intragroup alignment increases with connectivity and that it reduces when the agents enter the exploitation phase.
Revised: We observe that, in all tasks, intragroup alignment increases with connectivity and that it reduces when the agents enter the exploitation phase.

Original: Thus, intra-group alignment can prove useful in characterizing the exploration behavior of a group.
Revised: Thus, intra-group alignment can prove useful in characterizing the exploration behavior of a group.

Original: In Figure 13, we present the inter-group alignment in the single path, merging paths and best-of-ten paths tasks.
Revised: In Figure 13, we present the inter-group alignment in the single path, merging paths and best-of-ten paths tasks.

Original: We observe that the topologies do not differ significantly in the single path task.
Revised: We observe that the topologies do not differ significantly in the single path task.

Original: In the merging task, we observe that inter-group alignment is lower during the exploration phase, compared to other tasks, and that the small-world is the slowest to align with all other structures.
Revised: In the merging task, we observe that inter-group alignment is lower during the exploration phase, compared to other tasks, and that the small-world is the slowest to align with all other structures.

Original: Perhaps this explains why this topology finds the optimal solution with the least probability: by propagating information quickly, the group early on
Revised: Perhaps this explains why this topology finds the optimal solution with the least probability: by propagating information quickly, the group early on

Original: [Figure]
Revised: [Figure]

Original: C o n f o r m i t y ,
Revised: C o n f o r m i t y ,

Original: C t
Revised: C t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r a g e
Revised: r a g e

Original: D i v e r
Revised: D i v e r

Original: s
Revised: s

Original: i
Revised: i

Original: t y
Revised: t y

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: n f o r m i t y ,
Revised: n f o r m i t y ,

Original: C t
Revised: C t

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: 3 Training step, t 1e 0.
Revised: 3 Training step, t 1e 0.

Original: I n t r ag r o u p a li g n m e n t ,
Revised: I n t r ag r o u p a li g n m e n t ,

Original: A t   3 Training step, t 1e 0.
Revised: A t   3 Training step, t 1e 0.

Original: I n t r ag r o u p a li g n m e n t ,
Revised: I n t r ag r o u p a li g n m e n t ,

Original: A t 0.
Revised: A t 0.

Original: 2.0 Training step, t 1e 0.
Revised: 2.0 Training step, t 1e 0.

Original: I n t r ag r o u p a li g n m e n t ,
Revised: I n t r ag r o u p a li g n m e n t ,

Original: A t converges to the local optimum in this task.
Revised: A t converges to the local optimum in this task.

Original: In the best-of-ten task, the no-sharing setting has the smallest alignment with all other structures.
Revised: In the best-of-ten task, the no-sharing setting has the smallest alignment with all other structures.

Original: This reinforces our main conclusion in this work: experience sharing affects individuals and different topologies do so in different ways.
Revised: This reinforces our main conclusion in this work: experience sharing affects individuals and different topologies do so in different ways.

Original: E. R OBUSTNESS TO LEARNING HYPER - PARAMETERS
Revised: E. R OBUSTNESS TO LEARNING HYPER - PARAMETERS

Original: In Figure 14 we present how the performance of SAPIENS varies for different values of the learning hyperparameters learning rate and disocunt factor in the single path task under a fully-connected and a dynamic topolgoy, as well as the no-sharing condition.
Revised: In Figure 14 we present how the performance of SAPIENS varies for different values of the learning hyperparameters learning rate and disocunt factor in the single path task under a fully-connected and a dynamic topolgoy, as well as the no-sharing condition.

Original: We observe that, although convergence to the optimal solution is not always achieved, the dynamic topology is at least as effective as the others either in terms of convergence rate and/or final performance in all conditions.
Revised: We observe that, although convergence to the optimal solution is not always achieved, the dynamic topology is at least as effective as the others either in terms of convergence rate and/or final performance in all conditions.

Original: E. E FFECT OF GROUP SIZE
Revised: E. E FFECT OF GROUP SIZE

Original: We here examine the effect of the group size for all social network structures in the merging-paths and best-of-ten paths task.
Revised: We here examine the effect of the group size for all social network structures in the merging-paths and best-of-ten paths task.

Original: To visualize the progression of a group on the paths of the different tasks, we focus on specific elements in the tasks: (i) ( [ A 8 , B 8 , C 2 ] in the merging-paths task.
Revised: To visualize the progression of a group on the paths of the different tasks, we focus on specific elements in the tasks: (i) ( [ A 8 , B 8 , C 2 ] in the merging-paths task.

Original: The first two correspond to reaching the end of the paths corresponding to the two local optima.
Revised: The first two correspond to reaching the end of the paths corresponding to the two local optima.

Original: To reduce the computational complexity of experiments, we do not study the last element of the optimal path ( C 4 ), but focus on C 2 instead.
Revised: To reduce the computational complexity of experiments, we do not study the last element of the optimal path ( C 4 ), but focus on C 2 instead.

Original: This is sufficient to detect whether a group has discovered the optimum path.
Revised: This is sufficient to detect whether a group has discovered the optimum path.

Original: Here, we observe that the fully-connected topology fails to find the optimal path regardless of its size (with a small success probability for N = 10 ).
Revised: Here, we observe that the fully-connected topology fails to find the optimal path regardless of its size (with a small success probability for N = 10 ).

Original: We observe that the ability of the ring , small-world and dynamict topologies to avoid the local optima improves with the group size (ii)
Revised: We observe that the ability of the ring , small-world and dynamict topologies to avoid the local optima improves with the group size (ii)

Original: [ B 4 , A 2 , E 2 ] in the best-of-ten tasks.
Revised: [ B 4 , A 2 , E 2 ] in the best-of-ten tasks.

Original: B 4 is the fourth element on the optimal path (again we do not study the last element to reduce complexity).
Revised: B 4 is the fourth element on the optimal path (again we do not study the last element to reduce complexity).

Original: To avoid cluttering the visualization we only present two of the nine sub-optimal paths.
Revised: To avoid cluttering the visualization we only present two of the nine sub-optimal paths.

Original: In this task, we again observe that the fully-connected network fails to discover the optimal task.
Revised: In this task, we again observe that the fully-connected network fails to discover the optimal task.

Original: Among all structures and group sizes, the large dynamic network performs best, while the performance of ring and small-world is also best for N = 50 .
Revised: Among all structures and group sizes, the large dynamic network performs best, while the performance of ring and small-world is also best for N = 50 .

Original: We observe that small networks sizes ( N = 2 , N = 6 ) are slower at exploring (we can see that as they rarely find the second element of the sub-optimal paths, which is required to conclude that path B is the optimal choice).
Revised: We observe that small networks sizes ( N = 2 , N = 6 ) are slower at exploring (we can see that as they rarely find the second element of the sub-optimal paths, which is required to conclude that path B is the optimal choice).

Original: Overall, this scaling analysis indicates that increasing the group size in a fully-connected topology will not improve performance, while benefits are expected for low-connectivity structures, particularly for the dynamic topology.
Revised: Overall, this scaling analysis indicates that increasing the group size in a fully-connected topology will not improve performance, while benefits are expected for low-connectivity structures, particularly for the dynamic topology.

Original: We believe that this observation is crucial.
Revised: We believe that this observation is crucial.

Original: In studies of groups of both human and artificial agents, we often encounter the conviction that, larger groups perform better and that size is a more important determinant than connectivity, the latter justifying why connectivity is often ignored Kline & Boyd (2010); Horgan et al.
Revised: In studies of groups of both human and artificial agents, we often encounter the conviction that, larger groups perform better and that size is a more important determinant than connectivity, the latter justifying why connectivity is often ignored Kline & Boyd (2010); Horgan et al.

Original: ; Mnih et al. (2016);
Revised: ; Mnih et al. (2016);

Original: Schmitt et al.
Revised: Schmitt et al.

Original: ; Nair et al.
Revised: ; Nair et al.

Original: Our results here point to the contrary.
Revised: Our results here point to the contrary.

Original: E. P RIORITIZED EXPERIENCE SHARING
Revised: E. P RIORITIZED EXPERIENCE SHARING

Original: We now examine how sharing prioritized experiences instead of randomly sampled ones affects the performance of SAPIENS .
Revised: We now examine how sharing prioritized experiences instead of randomly sampled ones affects the performance of SAPIENS .

Original: In Figure 16 we repeat the same experiment with Figure 4, with the difference that all methods compute priorities, which they employ both for implementing a prioritized
Revised: In Figure 16 we repeat the same experiment with Figure 4, with the difference that all methods compute priorities, which they employ both for implementing a prioritized

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: I n t e r - g r o u p a li g n m e n t ,
Revised: I n t e r - g r o u p a li g n m e n t ,

Original: A j , j t
Revised: A j , j t

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: I n t e r - g r o u p a li g n m e n t ,
Revised: I n t e r - g r o u p a li g n m e n t ,

Original: A j , j t
Revised: A j , j t

Original: [Figure]
Revised: [Figure]

Original: I n t e r - g r o u p a li g n m e n t ,
Revised: I n t e r - g r o u p a li g n m e n t ,

Original: A j , j t
Revised: A j , j t

Original: [Figure]
Revised: [Figure]

Original: I n t e r
Revised: I n t e r

Original: [Figure]
Revised: [Figure]

Original: u p a li g n
Revised: u p a li g n

Original: [Figure]
Revised: [Figure]

Original: n t
Revised: n t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: i t h e l
Revised: i t h e l

Original: [Figure]
Revised: [Figure]

Original: m
Revised: m

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Single path
Revised: Single path

Original: Merging paths
Revised: Merging paths

Original: Best-of-ten paths
Revised: Best-of-ten paths

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: replay buffer and sharing experiences by sampling them in proportion to their priorities.
Revised: replay buffer and sharing experiences by sampling them in proportion to their priorities.

Original: As we see, using priorities negatively impacts experience sharing, while it helps speed up the performance of the single agent in the single path task.
Revised: As we see, using priorities negatively impacts experience sharing, while it helps speed up the performance of the single agent in the single path task.

Original: This behavior has been observed in previous works Souza et al.
Revised: This behavior has been observed in previous works Souza et al.

Original: and can be attributed to the fact that the priorities of the sender do not necessarily agree with the priorities of the receiver and, therefore, destabilize learning.
Revised: and can be attributed to the fact that the priorities of the sender do not necessarily agree with the priorities of the receiver and, therefore, destabilize learning.

Original: E. A DDITIONAL TEST - BED : THE DECEPTIVE COINS GAME
Revised: E. A DDITIONAL TEST - BED : THE DECEPTIVE COINS GAME

Original: Deceptive games are grid-world tasks introduced to test the ability of deep RL agents to avoid local optima.
Revised: Deceptive games are grid-world tasks introduced to test the ability of deep RL agents to avoid local optima.

Original: (Bontrager et al., 2019).
Revised: (Bontrager et al., 2019).

Original: Here, we perform preliminary experiments with our own JAXbased implementation of one of the games: the first difficulty level of the deceptive coins game (see Figure 17 for an illustration).
Revised: Here, we perform preliminary experiments with our own JAXbased implementation of one of the games: the first difficulty level of the deceptive coins game (see Figure 17 for an illustration).

Original: Here, the agent can navigate in the grid-world during an episode and collect diamonds, which give a unit of reward.
Revised: Here, the agent can navigate in the grid-world during an episode and collect diamonds, which give a unit of reward.

Original: The game finishes once the agent reaches the fire, which offers an additional reward, or when a timeout of 14 time steps is reached.
Revised: The game finishes once the agent reaches the fire, which offers an additional reward, or when a timeout of 14 time steps is reached.

Original: There are two possible paths the agent can follow: moving left and reaching the fire will give a reward of two while moving right and reaching the fire will give a reward of five.
Revised: There are two possible paths the agent can follow: moving left and reaching the fire will give a reward of two while moving right and reaching the fire will give a reward of five.

Original: The second path is more rewarding but is harder to complete because, once an agent discovers the easier-to-find diamond on the left, it is deceived into following the left path.
Revised: The second path is more rewarding but is harder to complete because, once an agent discovers the easier-to-find diamond on the left, it is deceived into following the left path.

Original: Once an agent commits on a path (reaches the edge of the grid-world) a barrier is raised so that the agent cannot go back within that episode.
Revised: Once an agent commits on a path (reaches the edge of the grid-world) a barrier is raised so that the agent cannot go back within that episode.

Original: We now examine the performance of SAPIENS under different social network structures (fullyconnected, small-world, ring, dynamic), as well as the no-sharing, A2C and Ape-X baselines for three group sizes: 6 , 10 and 20 agents.
Revised: We now examine the performance of SAPIENS under different social network structures (fullyconnected, small-world, ring, dynamic), as well as the no-sharing, A2C and Ape-X baselines for three group sizes: 6 , 10 and 20 agents.

Original: We present the reward plots for the 3 sizes in Figures 18,and 20, respectively, and present an overall comparison in Figure 21 (equivalent to Figure 4 for the
Revised: We present the reward plots for the 3 sizes in Figures 18,and 20, respectively, and present an overall comparison in Figure 21 (equivalent to Figure 4 for the

Original: Wordcraft tasks).
Revised: Wordcraft tasks).

Original: We observe that all conditions found either the local or the global optimum and that : a) A2C fails for all network sizes.
Revised: We observe that all conditions found either the local or the global optimum and that : a) A2C fails for all network sizes.

Original: This behavior has been observed in previous works (Bontrager et al., 2019) and can be attributed to the fact that policy-gradient methods are more susceptible to local minima b) no-sharing gets stuck in the local optimum in half of the trials when the group size is small.
Revised: This behavior has been observed in previous works (Bontrager et al., 2019) and can be attributed to the fact that policy-gradient methods are more susceptible to local minima b) no-sharing gets stuck in the local optimum in half of the trials when the group size is small.

Original: Increasing the group size increases the probability that at least one agent in the group will escape the local minima by ϵ -greedy exploration c) partially connected structures find the global minima across network sizes d) fully-connected converges to the local optimum for the large group size, although the global optimum was discovered at the early exploration phase (see Figure 20).
Revised: Increasing the group size increases the probability that at least one agent in the group will escape the local minima by ϵ -greedy exploration c) partially connected structures find the global minima across network sizes d) fully-connected converges to the local optimum for the large group size, although the global optimum was discovered at the early exploration phase (see Figure 20).

Original: Thus, too much experience sharing is harmful e) Ape-X fails with high probability for all network sizes.
Revised: Thus, too much experience sharing is harmful e) Ape-X fails with high probability for all network sizes.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: cc

Original: [Figure]
Revised: [Figure]

Original: ss ******** * **** ** ****
Revised: ss ******** * **** ** ****

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: In general, our conclusions in this task are consistent with what we observe in Wordcraft, in particular the merging paths task that has a similar deceptive nature.
Revised: In general, our conclusions in this task are consistent with what we observe in Wordcraft, in particular the merging paths task that has a similar deceptive nature.

Revised: E. R OBUSTNESS TO AMOUNT OF SHARING ( p s AND L s

Revised: In Section 2.3 we formulated SAPIENS and described two hyper-parameters: p s is the probability of sharing a batch of experience tuples at the end of an episode and L s is the length of this batch.

Revised: Here, we test the robustness of SAPIENS to these two hyper-parameters, which both control the amount of shared information and, therefore, interact with hyper-parameters of the DQNs (in particular the learning rate) to control the rate at which information is shared to the rate of individual learning.

Revised: Specifically, we evaluate the dynamic topology (with the same hyper-parameters employed in the main paper, i.e., visit duration T v = 10 and probability of visit p v = 0 . 05 ) and the fully-connected topology in the deceptive coins game (described in Appendix E.7) with 20 DQN agents.

Revised: In Figure 22 we present group success ( S ) averaged across tasks for a parametric analysis over

Revised: L s ∈ (1 , 6 , 36) and p s ∈ (0 . 35 , 0 .

Revised: We observe that the dynamic topology finds the optimal solution across conditions except for a small probability of failure for L s = 1 , p s = 0 .

Revised: 35 and

Revised: L s = 1 , p s = 0 .

Revised: These values correspond to low amounts of information sharing.

Revised: In this case, the dynamic structure becomes more similar to a no-sharing structure: the amount of shared information is not enough to help the agents avoid local optima they fall into due to individual exploration.

Revised: For the fully-connected topology we observe that performance degrades for high amounts of information (( L = 36 , p s = 0 . 35 ), ( L = 36 , p s = 0 . 7 ), ( L = 36 , p s = 1 )).

Revised: This is in accordance with our expectation that fully-connected topologies lead to convergence to local optima.

Revised: Interestingly, this structure performs well when p s = 1 and L s ≤ 6 .

Revised: Thus, sharing more frequently is better than sharing longer batches: we hypothesize that this is because longer batches have more correlated data, making convergence to local optima more probable.

Revised: [Figure]