Reinforcement Communication Learning in Different Social Network Structures
Abstract
Social network structure is one of the key determinants of human language evolution.
Previous work has shown that the network of social interactions shapes decentralized learning in human groups, leading to the emergence of different kinds of communicative conventions.
We examined the effects of social network organization on the properties of communication systems emerging in decentralized, multi-agent reinforcement learning communities.
We found that the global connectivity of a social network drives the convergence of populations on shared and symmetric communication systems, preventing the agents from forming many local “dialects”.
Moreover, the agent’s degree is inversely related to the consistency of its use of communicative conventions.
These results show the importance of the basic properties of social network structure on reinforcement communication learning and suggest a new interpretation of ﬁndings on human convergence on word conventions.
1. Introduction
Human languages evolve as complex adaptive systems, driven by micro-level processes and constraints (such as individual learning mechanisms and perceptual biases), macrolevel factors (such as a topology of social interactions) and the history of their development (Steels, 2000; Christiansen & Chater, 2008; “Five Graces Group” et al., 2009).
Linguistic communication depends on the shared knowledge of word-to-meaning mapping conventions (Lewis, 1975), upon which the population converges through the local interactions between agents, often with no central controller
Accepted to the 1 st Workshop on Language in Reinforcement Learning , ICML 2020, Copyright 2020 by the author(s). available (Baronchelli, 2018).
Empirical studies of human learning demonstrate that groups quickly converge on new communicative conventions in “decentralized” settings (Garrod & Doherty, 1994; Selten & Warglien, 2007).
Multi-agent reinforcement learning to communicate (MARLC), however, faces instability challenges if no central optimization is introduced (Bernstein et al., 2002; Laurent et al., 2011; Matignon et al., 2012).
This inﬂuences the ability of groups consisting of reinforcement learners to converge on efﬁcient and stable communication systems which are shared by all their members.
Therefore, different methods of centralized control and optimization have been proposed to stabilize MARLC (Sukhbaatar et al., 2016;
Foerster et al., 2016; Lowe et al., 2017; Pesce & Montana, 2020; Foerster et al., 2018).
Central optimization makes the simulations more brittle and less ﬂexibly adaptive, and, potentially, less promising in developing communication systems as freely expressive and well-optimized for their users (Gibson et al., 2019) as natural languages.
We argue that empirical evidence on individual- and population-level factors that drive decentralized learning in human groups can guide simulations of language evolution in MARLC settings.
In this work, we explore whether the social network organization shapes the properties of communication systems that arise through decentralized
MARLC in simpliﬁed settings.
1.1. Human Learning in Different Social Network Structures
Convergence of human groups on word conventions is dramatically affected by the social topology that determines the possible interactions between participants, as demonstrated by the naming game experiment (Centola & Baronchelli, 2015).
In particular, when arranged in a social network with many local connections (e.g. “ring” topology (Fig. 1d)) or a randomly-connected network (Fig. 1a), large groups converge on many local word conventions, reaching no global consensus.
However, if each person is equally likely to interact with any other person in the group (“clique” (Fig. 1b)) , global consensus is easily achieved with no centralization.
Other studies on decentralized problem solving in human groups demonstrated that social network organization shapes the multi-agent optimization process, with different network types being beneﬁcial for different types of optimization landscapes.
High local connectivity supports independent local exploration, whereas high global connectivity helps groups to converge on a shared solution, choosing among the local ones (Fang et al., 2010; Mason et al., 2008; Mason & Watts, 2012; Lazer & Friedman, 2007; Wisdom & Goldstone, 2011).
In this study, we looked at how the type of social network organization, its average degree, and local connectivity affect the results of communication learning in groups of deep reinforcement learning agents.
2. Method 2.1. Coordination Game
Every game episode involves two agents, randomly assigned to speaker and listener roles.
The speaker produces a message, which is transmitted to the listener. Then, both agents independently choose an action 1 .
If the actions match, the agents receive a reward.
We add an additional penalty for overusing any speciﬁc action to avoid degenerate solutions that ignore the communication channel. This setting encapsulates the most basic form of a coordination game that beneﬁts from the formation of communicative conventions.
Please, see supplementary materials for more detail.
2.2. Types of Social Network Organization
Social network determines how the agents are sampled to play games with one another.
For each game round, one agent is selected randomly, and then its partner is selected from its neighbors.
Thus, an agent can only play a game with one of its immediate neighbors in the social network.
In all the simulations, the social network size was set to
The networks were undirected, and the self-connections were not allowed.
We tested 4 types of social networks in our experiments (Fig. 1):
Random (ER).
Random graph is generated by connecting the nodes with equal probability p (Erdos, 1959).
Fully-connected (clique).
In clique, all the nodes are connected to each other.
Ring.
In the ring network, all the nodes have exactly two neighbors, and connections form a single continuous path.
Small-world.
The small-world network is generated by adding new, “global”, connections to the ring network with constant probability p (Newman & Watts,
2.3. Agents
We used simple feed-forward neural networks to represent the agents.
The networks were trained using a vanilla deep
Q-learning algorithm (Mnih et al., 2015) with an added proportion of bottom-up driven “supervising” feedback (see supplementary materials).
We tried to avoid any centralized optimization, popular in MARLC settings, to study the ability of different networks to self-organize conventions.
2.4. Metrics
We used a number of information-theoretic metrics, developed in Dubova & Moskvichev (2020) and Lowe et al.
(2019) to comprehensively evaluate the communication protocols: speaker & listener consistency, between- and withinagent signal-action mapping divergence, signaling divergence, and behavioral predictability (see supp. materials).
3. Results 3.1. Experiment 1: types of social network organization
We simulated multi-agent learning in 4 types of social networks: ring (avg degree=2, var=0), random (avg degree=2, var=1.6, p(connection)=0.2), small-world (avg degree=2.2, p(add global connection)=0.2), and clique (avg degree=9, var=0).
We also varied the supervision rate (from 0.0 to 0.
with a step of 0.1).
MARLC in each combination of these two conditions was simulated 10 times to get statistical estimates of the communication metrics.
In all the experiments, each simulation consisted of 120000 game rounds.
The average speaker and listener consistencies were highest in the ring and small-world networks, and the lowest in the clique structure.
Consistency scores did not vary with the supervision rate (Fig. 2d).
These patterns indicated the potential dependency of the consistency scores on a single factor: agent’s degree (the clique had the highest possible average degree).
This hypothesis is tested in Experiment 2.
The agents in random and fully-connected social networks developed almost perfectly symmetric and homogeneous communication systems according to all three communication asymmetry metrics (Fig. 2b and 2c).
Small-world and ring social networks lead the populations to develop asymmetrical and local communication patterns.
Moreover, supervised feedback helped the agents in all topology conditions to develop more shared and symmetric communication systems.
We hypothesized that the effect of network type on asymmetry scores is driven by local connectivity of the populations: fully-connected and random networks do not form local communities, whereas the ring-shaped and smallworld networks in our simulations mainly consisted of local connections.
We tested this hypothesis in Experiment 3.
Lastly, agents in fully-connected and random networks produced less diversiﬁed actions, but their action and signaling distributions were much more coordinated than in the ringshaped and small-world network structures (see Fig. 2).
It suggests that agents in the last two conditions overﬁtted to the “action diversity” part of the reward function, while failing to coordinate using the communication channel.
3.2. Experiment 2: average degree
To test the effect of network’s average degree on speaker and listener consistencies, we focused on the random (ER) network, which produces desired variation in the average degrees of the nodes, while keeping most of the other network properties constant.
We varied the probability of connecting nodes in the network from 0.2 to 0.9, and supervision rate from 0 to 0.9 with a step of 0.1.
This resulted in 80 conditions total, each of which was simulated 5 times to obtain consistency estimates with conﬁdence intervals.
We found that the average degree negatively affected listener ( p < 0 . 001 ) and speaker ( p < 0 .
001 ) consistencies (Fig. 3).
Consistency scores for the average degrees higher than 5. stayed similar to the estimate we obtained for the fullyconnected topology in Experiment 1.
3.3. Experiment 3: global and local connections
We examined the effect of locally-connected groups in a social network on homogeneity and symmetry of the developed communication systems.
For this, we tested MARLC in the small-world network with different probabilities of adding a “global” connection to the initial ring-shaped struc- ture.
We classify a link as “global” if it connects agents further than one link apart in the ring network.
We varied both the probability of global connection and the supervision rate from 0 to 0.9 with a step of 0.1.
Each combination of these two factors was simulated 5 times.
Global connection probability was inversely related to the signaling divergence ( p < 0 .
001 ), between- ( p < 0 .
001 ) and within-agent signal-action mapping divergence ( p < 0 .
001 ) of the developed communication systems (Fig. 3).
Higher proportion of global connections led the groups to converge on communication systems that are shared by all their agents.
Supervised feedback also helped the agents to develop homogeneous and symmetric communication patterns.
The cumulative effect of both high supervision rate and probability of global connections lead to communication systems that are shared by all the agents in a group.
4. Discussion and Conclusion
We conducted three experiments to explore and isolate the factors of social network organization that drive the effectiveness, homogeneity, and symmetry of communication systems developed in MARLC settings.
Following the suggestions of Lowe et al.
(2019), we used a set of informationtheoretic metrics to evaluate our results.
This allowed us to determine which particular properties of communication systems are affected by our interventions.
The results of our ﬁrst experiment partially replicated the effects of social network type on learning word conventions by human groups (Centola & Baronchelli, 2015).
This suggests that the simple domain-general reinforcement learning model can capture core regularities found in human convergence on linguistic conventions.
Our further experiments provide a more detailed account of particular social network properties that are responsible for these patterns in communication learning.
The second experiment demonstrated that the average degree of a social network is a central factor affecting how deterministically agents use the communication channel.
The more communication partners an agent needs to adapt to, the more variable are its signaling and listening patterns.
Key ﬁnding of the third experiment is that the proportion of global connections is the primary determining factor for three variables: homogeneity, within- and between-agent symmetry of the developed communication system.
In particular, high proportion of local social connections led to the emergence of local communication patterns (“dialects”) within the population, whereas adding more global connections forced the agents to ﬁnd global consensus.
Similarly, high local connectivity resulted in asymmetric communicative patterns, where even the same agent in two different roles uses completely different vocabularies.
At present, social network effects on MARLC have been largely under-investigated.
To the best of our knowledge, there is only one work in this direction; L. Graesser and her colleagues (2019) looked at how communication systems developed within communities change if one community is introduced to another, depending on their inter- and intraconnectivity.
Unfortunately, the insights from that work are only applicable for the speciﬁc scenario of community merging.
In our study, we focused on the social network factors that are applicable to almost any MARLC setting.
The results of our analysis corroborate Lowe et al.’s
(2019) suggestion that overall scores do not reﬂect whether the agents use the communication channel to solve the task, and how efﬁciently they do so (see Fig. 2).
For example, the performance scores of the agents learning to coordinate in a ring-shaped social network were the highest among all the network types.
However, more detailed analysis revealed that agents in this condition developed many local communication patterns, which varied across agents and their roles.
This, again, illustrates the importance of thorough analysis of communication protocols learned in MARLC settings.
Our approach has a number of limitations that are important to mention.
Firstly, while we aimed to test a broad spectrum of social network factors, our experiments are by no means comprehensive.
There are other important aspects of social network structure that may play a key role in determining the properties of emerging communication protocols.
We believe that studying the effects of variance of network’s degree distribution and its modularity on MARLC is especially promising.
Secondly, we used a very simpliﬁed setting of vanilla Q-learning in an “amodal” coordination game to minimize the number of assumptions that might make our results less representative to the MARLC problem in general.
We suggest testing these results on more advanced reinforcement learning models and realistically perceptually grounded game settings.
5. Data Availability
The code and data for this work are available online at the project’s github repository. 6. Acknowledgements Authors thank Yong-Yeol Ahn, Andrei Amatuni, Thomas
Gorman, Jack Avery, Ben Kovitz, all the attendees of the PCL lab meetings, and two anonymous reviewers for their valuable feedback on this project. This research was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute. https://kb.iu.edu/d/anwt#carbonate
Each game consists of two-step episodes. Every episode of the game involves two agents: a speaker and a listener who need to coordinate their actions.
On the ﬁrst step, the speaker produces an action and a message.
On the second step, the listener receives the message and outputs an action.
If the actions match, agents receive a ﬁxed “coordination” reward R c .
There are four unique actions and messages that the agents choose from.
In order to avoid trivial solutions when agents converge on a single action and ignore the communication channel, we introduced a penalty for repeating an action too often.
Speciﬁcally, an additional reward of min (0 , 1 / 4 − ˆ p A ) is added when an agent chooses action A .
Here ˆ p A denotes an empirical proportion of selecting action A during the last H steps.
We used a ﬁxed length history H = 100 .
Current ˆ p A are added to the agent’s input state spaces to aid convergence.
On every episode, each agent can be assigned to play either the speaker or the listener role.
The current role type is provided as a binary input.
Overall, the agents receive, as inputs:
A binary role indicator 2.
Random input (for speakers, in order to allow for nondeterministic policies) or speaker’s message (for listeners)
Proportions of different actions in agent’s recent his tory
The agents have two output layers both stemming from the last hidden
layer Action output layer (for neurons, and the action is determined as argmax) 2.
Message output layer which has the same structure as the action output layer.
Message outputs are ignored if the agents plays the listener role
B. Agents
Agents are implemented using simple feed-forward neural networks with two hidden layers (hidden sizes of 25 and 15) and ReLU activation function.
We used vanilla deep Q-learning algorithm with additional supervised updates to train the agents.
Supervised feedback corresponds to ”peeking” the other agent’s action and storing it in memory (with the signaling experience from that game) as a ”correct” response.
In this case, the action-signal mapping is not superimposed, and the ”correct” answers in the supervising trials are bottom-up driven and reﬂect the dynamics of the agents themselves.
Supervised feedback was implemented by changing a certain proportion of negative (“miscoordinated”) experiences to the “supervising” ones.
We decided to control for supervision in our experiments because partial supervising feedback in different forms is often available in naturalistic language learning situations.
By amplifying the reinforcement signal, supervising feedback may help to overcome the “sparcity of rewards” problem in multi-agent reinforcement learning.
C. Metrics
We follow the same set of metrics as used in (Dubova &
Moskvichev, 2020).
1. Speaking Consistency and Listening Consistency .
This metric provides a quantitative measure of whether the actions that the agents perform are related to the signals that they send or receive.
We follow Lowe et al.
(2019) who proposed to use the normalized mutual information between the distributions over messages and actions induced by an agent.
The metric can be formally described as follows:
[Equation]
Here, Z is the average entropy of the two marginal distributions: Z
=
( H ( p a )+ H
( p m
[Equation]
A l and A c denote the set of available actions and the set of available messages respectively.
This metric is computed twice for every agent, conditioned on the role played by the agent (speaker or listener).
We average these metrics across agents in every simulation to obtain the speaking and listening consistency metrics that we report.
2. Communication Asymmetry metrics :
a.
Between-agent signal-action mapping divergence .
While the previous metric aimed to measure whether an agent is using communication channel (i.e. whether its actions correspond to the signals in any way), the second metric aims to measure whether the communication patterns differ between agents.
We compute the average Jensen-Shannon pairwise divergence between distributions of agents’ actions following a speciﬁc signal (averaged over all signals and pairs of agents).
For a pair of agents, the metric is deﬁned as follows
[Equation]
Here, p ( a 1 | m ) are action distributions of agent 1 conditional on the message (received or sent) being equal to m .
b. Within-agent signal-action mapping divergence .
This metric aims to capture internal inconsistencies in agent’s behaviors when it switches between roles: whether the agent’s behaviors differ depending on whether it receive or send a speciﬁc message.
For that, we use the average Jensen-Shannon divergence between the distributions over agent’s actions (conditioned on receiving or sending a speciﬁc message) when the agent plays the speaker and the listener role.
Formally, we use the same deﬁnition as in Equation 2, but now p a 1 | m and p a 2 | m correspond to the same agent’s distributions when this agent plays different roles (as opposed to distributions of a pair of different agents playing the same role).
We average the metric scores across all agents to obtain the ﬁnal measure that we report.
c. Signaling divergence .
This metric aims to measure the difference in individual agents’ messaging preferences.
We deﬁne signal- ing divergence as an average pairwise Jensen-Shannon divergence of marginal signaling distributions of dif
- ferent agents
Behavioral Predictability .
This last metric is created to assess the general diversity in agent’s actions.
When the agents’ actions are less diverse (and hence, more predictable), it is easier to achieve successful multiagent coordination without using the communication channel.
To look at whether the diversity of actions corresponds to the diversity of signals, we also compute the predictability of agents’ signaling patterns. We deﬁne behavioural action/message
predictability as Jensen- Shannon divergence between marginal distributions of agent’s actions/messages and the uniform distribution.
Many of these metrics require knowing probability distributions.
We estimate all such distributions empirically.
As a short summary, if we see the learned language as a simple probabilistic dictionary that maps messages to actions, the metrics can be summarized as follows (note that every agent deﬁnes two such dictionaries: one for the speaker and one for the listener role):
Speaking Consistency and Listening Consistency .
Are the dictionaries reliable?
I
e. if we look up a speciﬁc message, do we consistently get the same action, or is there a lot of randomness?
Communication Asymmetry : a. Between-agent signal-action mapping divergence .
Are the dictionaries similar for different agents?
b. Within-agent signal-action mapping divergence .
How different are the “speaker” and “listener” dictionaries that each agent deﬁnes?
c.
Talking divergence .
Do different agents show different patterns in their dictionary lookups?
Behavioral Predictability .
How uniformly do agents look up different words in the dictionary (speaking predictability)?
How uniform are the results of their queries (behavioral predictability)?
D. Statistical analysis
All hypotheses were tested using a linear regression model with robust covariance estimation, controlling for supervi- sion rate.
Excluding the supervision rate did not qualitatively change the results, however.