Original: M ODELING F AKE N EWS IN S OCIAL N ETWORKS WITH D EEP M ULTI - AGENT R EINFORCEMENT L EARNING
Revised: M ODELING F AKE N EWS IN S OCIAL N ETWORKS WITH D EEP M ULTI - AGENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Over the last few years deep multi-agent reinforcement learning (DMARL) has become an increasingly active area of research with hundreds of papers submitted to top machine learning conferences every year.
Revised: Over the last few years deep multi-agent reinforcement learning (DMARL) has become an increasingly active area of research with hundreds of papers submitted to top machine learning conferences every year.

Original: However, so far there have been few real world use cases that beneﬁted from the progress in the ﬁeld.
Revised: However, so far there have been few real world use cases that beneﬁted from the progress in the ﬁeld.

Original: We use DMARL to develop apractical and ﬂexible computational model of fake news on social networks in which agents act according to learned best response functions.
Revised: We use DMARL to develop a ﬂexible computational model of fake news on social networks in which agents act according to learned best response functions.

Original: We achieve this by extending an information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN).
Revised: We achieve this by extending an information aggregation game to allow for fake news and by representing agents as recurrent deep Q-networks (DQN).

Original: In the game, agents repeatedly guess whether a claim is true or false taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period.
Revised: In the game, agents repeatedly guess whether a claim is true or false taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period.

Original: We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to, or takes over, a subset of agents.
Revised: We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to, or takes over, a subset of agents.

Original: The attacker can follow either a hand-tuned or trained policy.
Revised: The attacker can follow either a hand-tuned or trained policy.

Original: Our model allows us to tackle questions that are analytically intractable in fully rational models, while ensuring that agents follow reasonable best response functions.
Revised: Our model allows us to tackle questions that are analytically intractable in fully rational models, while ensuring that agents follow reasonable best response functions.

Original: Our results highlight the importance of awareness, privacy and social connectivity in curbing the adverse effects of fake news and open an entire new real world application area for DMARL.
Revised: Our results highlight the importance of awareness, privacy and social connectivity in curbing the adverse effects of fake news and open an entire new real world application area for DMARL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Problem setting At least since the 2016 US presidential election, fake news on social networks, aimed at manipulating the users’ perception of facts, has been recognized as a major issue in open societies.
Revised: Problem setting At least since the 2016 US presidential election, fake news on social networks, aimed at manipulating the users’ perception of facts, has been recognized as a major issue in open societies.

Original: Yet, to date little is understood about how fake news spread on social networks, when a piece of fake news is effective in swaying public opinion and what interventions might be successful in mitigating the effect of fake news.
Revised: Yet, to date little is understood about how fake news spread on social networks, when a piece of fake news is effective in swaying public opinion and what interventions might be successful in mitigating the effect of fake news.

Original: Our lack of understanding of this important phenomenon is in large part due to the difﬁculty of modeling complex decision making processes on social networks.
Revised: Our lack of understanding of this important phenomenon is in large part due to the difﬁculty of modeling complex decision making processes on social networks.

Original: To tackle this issue, we extend a standard game of information aggregation on a social network, see
Revised: To tackle this issue, we extend a standard game of information aggregation on a social network, see

Original: Mossel et al.
Revised: Mossel et al.

Original: (2015), to accommodate fake news and solve this, analytically intractable, game using deep multi-agent reinforcement learning.
Revised: (2015), to accommodate fake news and solve this, analytically intractable, game using deep multi-agent reinforcement learning.

Original: This approach is ﬂexible, practical, and allows us to work towards answering the questions raised above.
Revised: This approach is ﬂexible, computationally tractable, and allows us to work towards answering the questions raised above.

Original: Related work Our work relates to two main strands of literature: (i) models of information aggregation in social networks and (ii) deep multi-agent reinforcement learning as a tool to solve games.
Revised: Related work Our work relates to two main strands of literature: (i) models of information aggregation in social networks and (ii) deep multi-agent reinforcement learning as a tool to solve games.

Original: The standard information aggregation game, outlined in Section 2.1, involves agents (repeatedly) guessing an unknown state of the world taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period.
Revised: The standard information aggregation game, outlined in Section 2.1, involves agents (repeatedly) guessing an unknown state of the world taking into account an informative private signal and observations of actions of their neighbors on the social network in the previous period.

Original: Note that this model involves no fake news .
Revised: Note that this model involves no fake news .

Original: There are two main approaches to studying this game.
Revised: There are two main approaches to studying this game.

Original: In the rational agent approach, agents’ decision are best responses and the analysis focuses on the Nash equilibria of the game.
Revised: In the rational agent approach, agents’ decision are best responses and the analysis focuses on the Nash equilibria of the game.

Original: In an important contribution, Mossel et al.
Revised: In an important contribution, Mossel et al.

Original: (2015) show that under certain conditions on the social network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct state of the world.
Revised: (2015) show that under certain conditions on the social network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct state of the world.

Original: This approach is not tractable for ﬁnite T and n and therefore does not allow, for example, the study of transient behavior.
Revised: This approach is not tractable for ﬁnite T and n and therefore does not allow, for example, the study of transient behavior.

Original: An alternative approach to studying the game of information aggregation is heuristic, see Golub & Sadler (2016) for a review of many important contributions.
Revised: An alternative approach to studying the game of information aggregation is heuristic, see Golub & Sadler (2016) for a review of many important contributions.

Original: While heuristic approaches are more ﬂexible to changes in the model setup and allow the study of transient behavior, agent strategies can be too simplistic.
Revised: While heuristic approaches are more ﬂexible to changes in the model setup and allow the study of transient behavior, agent strategies can be too simplistic.

Original: For example, agents may treat repeated observations of a neighbor’s guess as conveying additional information, even when this is not the case (this occurs for example when this neighbor’s only information is his private signal).
Revised: For example, agents may treat repeated observations of a neighbor’s guess as conveying additional information, even when this is not the case (this occurs for example when this neighbor’s only information is his private signal).

Original: Furthermore, heuristic behavior does not adapt to changing circumstances and would therefore not allow us to study how agent behavior changes in the presence of fake news.
Revised: Furthermore, heuristic behavior does not adapt to changing circumstances and would therefore not allow us to study how agent behavior changes in the presence of fake news.

Original: Our deep multi-agent reinforcement learning approach combines both worlds.
Revised: Our deep multi-agent reinforcement learning approach combines both worlds.

Original: Our approach is ﬂexible since it does not require the model to be analytically tractable and ﬁnds near-optimal policies through reinforcement learning.
Revised: Our approach is ﬂexible since it does not require the model to be analytically tractable and ﬁnds near-optimal policies through reinforcement learning.

Revised: Our work is also related to studies of competitive information diffusion, see Alon et al.

Revised: Etesami & Bas¸ar (2014).

Revised: However, while the literature on competitive information diffusion focuses on the strategic seeding of a pre-determined diffusion process, our work focuses on modeling the information diffusion process itself through the agents’ learned actions.

Original: Deep multi-agent reinforcement learning (DMARL) has been applied to solve games of different structure.
Revised: Deep multi-agent reinforcement learning (DMARL) has been applied to solve games of different structure.

Original: Policies or value functions are approximated by deep neural nets and trained via backpropagation.
Revised: Policies or value functions are approximated by deep neural nets and trained via backpropagation.

Original: Some recent examples are Letcher et al.
Revised: Some recent examples are Letcher et al.

Original: ; Foerster et al.
Revised: ; Foerster et al.

Original: ; Lerer &
Revised: ; Lerer &

Original: Peysakhovich (2017); Balduzzi et al.
Revised: Peysakhovich (2017); Balduzzi et al.

Original: Often the focus is on methods that allow agents to converge on a particular, socially desirable policy, such as cooperation in social dilemmas without changing the payout structure.
Revised: Often the focus is on methods that allow agents to converge on a particular, socially desirable policy, such as cooperation in social dilemmas without changing the payout structure.

Original: Since this is not an issue for most of our analyses, we make use of
Revised: Since this is not an issue for most of our analyses, we make use of

Original: Independent Q-learning, see Tampuu et al.
Revised: Independent Q-learning, see Tampuu et al.

Original: However, for some of our more advanced analyses, convergence does become an issue and applying more advanced DMARL methods provides a promising avenue of future research.
Revised: However, for some of our more advanced analyses, convergence does become an issue and applying more advanced DMARL methods provides a promising avenue of future research.

Original: Contribution To the best of our knowledge, we are the ﬁrst to develop apractical and ﬂexible computational model of fake news on social networks in which agents act according to learned best response functions.
Revised: Contribution To the best of our knowledge, we are the ﬁrst to develop a ﬂexible computational model of fake news on social networks in which agents act according to learned best response functions.

Original: We achieve this by extending a standard information aggregation model to allow for fake news and by representing agents as recurrent deep Q-networks (DQN).
Revised: We achieve this by extending a standard information aggregation model to allow for fake news and by representing agents as recurrent deep Q-networks (DQN).

Original: We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents.
Revised: We incorporate fake news into the model by adding an adversarial agent, the attacker, that either provides biased private signals to or takes over a subset of agents.

Original: We conduct a number of experiments to answer three main questions.
Revised: We conduct a number of experiments to answer three main questions.

Original: (i)
Revised: (i)

Original: What determines the effectiveness of the attacker in reducing the accuracy of information aggregation?
Revised: What determines the effectiveness of the attacker in reducing the accuracy of information aggregation?

Original: (ii) How well can agents learn adapt to the presence of an attacker?
Revised: (ii) How well can agents learn adapt to the presence of an attacker?

Original: (iii)
Revised: (iii)

Original: When attacker and agents learn simultaneously, how do policies evolve over time?
Revised: When attacker and agents learn simultaneously, how do policies evolve over time?

Original: Our results suggest three potential interventions to curb the effectiveness of fake news on social networks.
Revised: Our results suggest three potential interventions to curb the effectiveness of fake news on social networks.

Original: First, “vaccinate” users of social networks by making them aware of the presence of fake news.
Revised: First, “vaccinate” users of social networks by making them aware of the presence of fake news.

Original: Second, keep private information on social networks private so that attackers cannot target poorly informed or well connected users.
Revised: Second, keep private information on social networks private so that attackers cannot target poorly informed or well connected users.

Original: Third, encourage the formation of “balanced” rather than clustered or fragmented social networks, so that information can “ﬂow” effectively across the network.
Revised: Third, encourage the formation of “balanced” rather than clustered or fragmented social networks, so that information can “ﬂow” effectively across the network.

Original: Lastly, we contribute to the MARL literature by providing an important and concrete real-world application of methods that to date have found limited practical application.
Revised: Lastly, we contribute to the MARL literature by providing an important and concrete real-world application of methods that to date have found limited practical application.

Original:  B ACKGROUND 2. A N INFORMATION AGGREGATION GAME ON A SOCIAL NETWORK
Revised:  B ACKGROUND 2. A N INFORMATION AGGREGATION GAME ON A SOCIAL NETWORK

Original: We consider a sequential game of information aggregation over T periods indexed by t = 1 , . .
Revised: We consider a sequential game of information aggregation over T periods indexed by t = 1 , . .

Original: , T with a set of agents N , see Mossel et al.
Revised: , T with a set of agents N , see Mossel et al.

Original: ; Golub & Sadler (2016) for details.
Revised: ; Golub & Sadler (2016) for details.

Original: Agents interact via a ﬁxed, possibly directed, graph G ( N, E ) , where E denotes the set of edges connecting the nodes N ; see Figure 1 for an illustration of the networks used in this paper.
Revised: Agents interact via a ﬁxed, possibly directed, graph G ( N, E ) , where E denotes the set of edges connecting the nodes N ; see Figure 1 for an illustration of the networks used in this paper.

Original: For concreteness, one can think of the agents as users of a social network, such as Twitter.
Revised: For concreteness, one can think of the agents as users of a social network, such as Twitter.

Original: A directed link from agent A to agent B would then imply that A follows B on Twitter.
Revised: A directed link from agent A to agent B would then imply that A follows B on Twitter.

Original: Agents are presented with a claim that is either true or false.
Revised: Agents are presented with a claim that is either true or false.

Original: Again, for concreteness, one can think of the claim as a factual statement encountered in a piece of news that has been shared among the agents on the social network.
Revised: Again, for concreteness, one can think of the claim as a factual statement encountered in a piece of news that has been shared among the agents on the social network.

Original: 1 True claims are denoted as θ = 1 and false claims as θ = 0 .
Revised: 1 True claims are denoted as θ = 1 and false claims as θ = 0 .

Original: True and false claims occur with equal probability; that is, the unconditional probability of a claim being {+Barabasi Albert graph 135 
 Clustered graph    
 Rewired form of clustered graph+} true or false is P ( θ = 0)=
Revised: True and false claims occur with equal probability; that is, the unconditional probability of a claim being {+Barabasi Albert graph 135 
 Clustered graph    
 Rewired form of clustered graph+} true or false is P ( θ = 0)

Original:  P ( θ = 1) = 1 / 2 .
Revised: = P ( θ = 1) = 1 / 2 .

Original: This choice simpliﬁes our analysis without affecting any of our qualitative results.
Revised: This choice simpliﬁes our analysis without affecting any of our qualitative results.

Original: Agents receive private signals s i ∼ F θ once at the beginning of an episode.
Revised: Agents receive private signals s i ∼ F θ once at the beginning of an episode.

Original: In the baseline F θ is such that private signals are always informative but noisy.
Revised: In the baseline F θ is such that private signals are always informative but noisy.

Original: In each period agents choose a binary action a it ∈ { 0 , 1 } .
Revised: In each period agents choose a binary action a it ∈ { 0 , 1 } .

Original: Agents observe the actions of their neighbors on the graph from the previous period.
Revised: Agents observe the actions of their neighbors on the graph from the previous period.

Original: At the end of the episode, agents are rewarded for actions that[-Barabasi Albert graph 135 
 Clustered graph    
 Rewired form of clustered graph-] matched the veracity of the claim presented.
Revised: At the end of the episode, agents are rewarded for actions that[-Barabasi Albert graph 135 
 Clustered graph    
 Rewired form of clustered graph-] matched the veracity of the claim presented.

Original: For a given claim θ and sequence of actions { a it } , an agent’s total discounted reward is r i =
Revised: For a given claim θ and sequence of actions { a it } , an agent’s total discounted reward is r i =

Original: (cid:80)
Revised: (cid:80)

Original: Tt γ t 1 { a it = θ } .
Revised: Tt γ t 1 { a it = θ } .

Original: Note that in this game, for t < T , it is not necessarily optimal to choose the action that matches the maximum of the posterior likelihood of θ .
Revised: Note that in this game, for t < T , it is not necessarily optimal to choose the action that matches the maximum of the posterior likelihood of θ .

Original: This is because an agent wants to prevent others from simply copying its action, since that reveals no information about the others’ private signal.
Revised: This is because an agent wants to prevent others from simply copying its action, since that reveals no information about the others’ private signal.

Original: Injecting noise into actions may therefore be beneﬁcial.
Revised: Injecting noise into actions may therefore be beneﬁcial.

Original: Despite these concerns, it can be shown that, under certain conditions on the network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct θ , see Mossel et al. (2015).
Revised: Despite these concerns, it can be shown that, under certain conditions on the network, in the limit T → ∞ (number of periods) and n → ∞ (number of agents), agents will converge and will agree on the correct θ , see Mossel et al. (2015).

Original: 2. M ULTI - AGENT REINFORCEMENT LEARNING : INDEPENDENT Q- LEARNING
Revised: 2. M ULTI - AGENT REINFORCEMENT LEARNING : INDEPENDENT Q- LEARNING

Original: In principle, reinforcement learning (RL) can be used to learn the agents’ policy (or response functions) in any game.
Revised: In principle, reinforcement learning (RL) can be used to learn the agents’ policy (or response functions) in any game.

Original: Many methods have been proposed to learn reasonable best response functions in various multi-agent reinforcement learning (MARL) settings, see Foerster et al.
Revised: Many methods have been proposed to learn reasonable best response functions in various multi-agent reinforcement learning (MARL) settings, see Foerster et al.

Original: ; Letcher et al.
Revised: ; Letcher et al.

Original: ; Lerer & Peysakhovich (2017).
Revised: ; Lerer & Peysakhovich (2017).

Original: In this paper, the agents’ response functions will be learned via independent Q-learning (IQL) and Q functions will be approximated by deep recurrent neural networks, see Tampuu et al.
Revised: In this paper, the agents’ response functions will be learned via independent Q-learning (IQL) and Q functions will be approximated by deep recurrent neural networks, see Tampuu et al.

Original: ; Foerster et al.
Revised: ; Foerster et al.

Original: We relegate a discussion of the RL and IQL framework to Appendix A.1.
Revised: We relegate a discussion of the RL and IQL framework to Appendix A.1.

Original:  M ETHODS
Revised:  M ETHODS

Original: Graphs We consider three types of undirected graphs, see Figure 1 for an illustration.
Revised: Graphs We consider three types of undirected graphs, see Figure 1 for an illustration.

Original: An instance of a Barabasi-Albert random graph with | N | = 10 , see Barab´asi & Albert (1999); a clustered graph with three clusters ( n c = 3 ) of fully connected nodes with four nodes each ( s c = 4 ) such that | N | = 12 ; and a balanced graph ( | N | = 12 ) that is obtained by randomly rewiring the clustered graph.
Revised: An instance of a Barabasi-Albert random graph with | N | = 10 , see Barab´asi & Albert (1999); a clustered graph with three clusters ( n c = 3 ) of fully connected nodes with four nodes each ( s c = 4 ) such that | N | = 12 ; and a balanced graph ( | N | = 12 ) that is obtained by randomly rewiring the clustered graph.

Original: Random rewiring involves repeatedly selecting two edges uniformly at random and swapping the terminal nodes of these edges.
Revised: Random rewiring involves repeatedly selecting two edges uniformly at random and swapping the terminal nodes of these edges.

Original: This preserves the degree sequence, i.e. the number of neighbors of each node, but removes the clusters such that the rewired graph is “balanced”.
Revised: This preserves the degree sequence, i.e. the number of neighbors of each node, but removes the clusters such that the rewired graph is “balanced”.

Original: The BarabasiAlbert random graph is often used to model social networks and is therefore a natural choice for our application.
Revised: The BarabasiAlbert random graph is often used to model social networks and is therefore a natural choice for our application.

Original: The clustered graph is a stereotypical example of a social network which is comprised of cliques that are weakly connected.
Revised: The clustered graph is a stereotypical example of a social network which is comprised of cliques that are weakly connected.

Original: It thereby represents a more “fragmented” or “polarized” social network.
Revised: It thereby represents a more “fragmented” or “polarized” social network.

Original: The balanced graph removes these clusters from the graph while maintaining the degree sequence and can thus help isolate the effect of clusters on information aggregation.
Revised: The balanced graph removes these clusters from the graph while maintaining the degree sequence and can thus help isolate the effect of clusters on information aggregation.

Original: Fake news We extend the model of information aggregation in order to study the effects of fake news on information aggregation.
Revised: Fake news We extend the model of information aggregation in order to study the effects of fake news on information aggregation.

Original: We introduce an adversary, the so called attacker, who wants to reduce the other agents’ total reward (cid:80) i r i .
Revised: We introduce an adversary, the so called attacker, who wants to reduce the other agents’ total reward (cid:80) i r i .

Original: To disambiguate between the attacker and other agents, we often refer to these other agents as citizens.
Revised: To disambiguate between the attacker and other agents, we often refer to these other agents as citizens.

Original: The attacker wants to persuade citizens to support false claims and dissuade them from supporting true claims.
Revised: The attacker wants to persuade citizens to support false claims and dissuade them from supporting true claims.

Original: The attacker can achieve this in one of two ways.
Revised: The attacker can achieve this in one of two ways.

Original: First, in the biased signal attack , he can distribute a ﬁxed budget of bias β = (cid:80) i β i across a subset of the agents such that citizen
Revised: First, in the biased signal attack , he can distribute a ﬁxed budget of bias β = (cid:80) i β i across a subset of the agents such that citizen

Original: i ’s private signal is drawn from s i ∼ F θ ( β i ) .
Revised: i ’s private signal is drawn from s i ∼ F θ ( β i ) .

Original:  We choose s i ∼ N ( θ + β i (1 − 2 θ ) , σ 2 ) .
Revised: 2 We choose s i ∼ N ( θ + β i (1 − 2 θ ) , σ 2 ) .

Original: We set σ 2 = 1 throughout.
Revised: We set σ 2 = 1 throughout.

Original: In this case the choice of { β i } i ∈ N is hand-tuned , i.e. set by the experimenter and not optimized.
Revised: In this case the choice of { β i } i ∈ N is hand-tuned , i.e. set by the experimenter and not optimized.

Original: Second, in the agent takeover attack , he can directly take over an agent and act on his behalf.
Revised: Second, in the agent takeover attack , he can directly take over an agent and act on his behalf.

Original: In this case the attacker optimizes his attack policy by IQL.
Revised: In this case the attacker optimizes his attack policy by IQL.

Original: Learning to aggregate information We hold the network ﬁxed across episodes.
Revised: Learning to aggregate information We hold the network ﬁxed across episodes.

Original: At the beginning of each episode θ and all private signals are drawn at random according to the distributions given above.
Revised: At the beginning of each episode θ and all private signals are drawn at random according to the distributions given above.

Original: For citizens we train a single DQN.
Revised: For citizens we train a single DQN.

Original: In period t , the citizen observes the episode private signal s i , his and the actions of his neighbors in the previous period { a jt − 1 } j ∈ B
Revised: In period t , the citizen observes the episode private signal s i , his and the actions of his neighbors in the previous period { a jt − 1 } j ∈ B

Original: i
Revised: i

Original: ( B i is the set of neighbors of agent i augmented by i ) and the agent id i which identiﬁes the agent’s network position.
Revised: ( B i is the set of neighbors of agent i augmented by i ) and the agent id i which identiﬁes the agent’s network position.

Original: To maintain a constant length input vector, unobserved actions are encoded as − 1 .
Revised: To maintain a constant length input vector, unobserved actions are encoded as − 1 .

Original: Thus, the corresponding DQN is a function Q : R × {− 1 , 0 , 1 } | N | × N × { 0 , 1 } (cid:55)→ R .
Revised: Thus, the corresponding DQN is a function Q : R × {− 1 , 0 , 1 } | N | × N × { 0 , 1 } (cid:55)→ R .

Original: In the agent takeover attack , the attacker is represented by a DQN.
Revised: In the agent takeover attack , the attacker is represented by a DQN.

Original: This DQN is distinct from the citizen DQN.
Revised: This DQN is distinct from the citizen DQN.

Original: At the beginning of each episode the attacker chooses a citizen uniformly at random from N and acts on his behalf.
Revised: At the beginning of each episode the attacker chooses a citizen uniformly at random from N and acts on his behalf.

Original: In period t , the attacker observes θ without noise, his and the actions of his neighbors in the previous period { a jt − 1 } j ∈ B i and the agent id i .
Revised: In period t , the attacker observes θ without noise, his and the actions of his neighbors in the previous period { a jt − 1 } j ∈ B i and the agent id i .

Original: Thus, the corresponding DQN is a function Q : { 0 , 1 } × {− 1 , 0 , 1 } | N | × N × { 0 , 1 } (cid:55)→ R .
Revised: Thus, the corresponding DQN is a function Q : { 0 , 1 } × {− 1 , 0 , 1 } | N | × N × { 0 , 1 } (cid:55)→ R .

Original: In the biased signal attack , the attacker is not represented by a DQN.
Revised: In the biased signal attack , the attacker is not represented by a DQN.

Original: Instead, the biases are hand-tuned.
Revised: Instead, the biases are hand-tuned.

Original: At the beginning of each episode the attacker chooses a set of citizens uniformly at random from N and delivers biased signals to them as outlined above.
Revised: At the beginning of each episode the attacker chooses a set of citizens uniformly at random from N and delivers biased signals to them as outlined above.

Original: Measuring information aggregation Intuitively, agents should be able learn to extract information from their private signals and their neighbors’ actions, such that at the end of an episode their expected reward ( E [ r iT ] ) should be signiﬁcantly higher than at the beginning of an episode ( E [ r i 0 ] ).
Revised: Measuring information aggregation Intuitively, agents should be able learn to extract information from their private signals and their neighbors’ actions, such that at the end of an episode their expected reward ( E [ r iT ] ) should be signiﬁcantly higher than at the beginning of an episode ( E [ r i 0 ] ).

Original: That is, agents should aggregate information over time.
Revised: That is, agents should aggregate information over time.

Original: In the following, we give a formal deﬁnition of information aggregation for ﬁnite N and T that is useful in our case.
Revised: In the following, we give a formal deﬁnition of information aggregation for ﬁnite N and T that is useful in our case.

Original: Deﬁnition 1 (Information aggregation) .
Revised: Deﬁnition 1 (Information aggregation) .

Original: Let F = { s i } i ∈
Revised: Let F = { s i } i ∈

Original: N denote the set of all realized signals in a given episode.
Revised: N denote the set of all realized signals in a given episode.

Original: Let ˆ θ = argmax θ P [ θ | F ] denote the maximum-a-posteriori estimator of θ given F .
Revised: Let ˆ θ = argmax θ P [ θ | F ] denote the maximum-a-posteriori estimator of θ given F .

Original: Then information is aggregated if for all i ∈ N , we have a iT
Revised: Then information is aggregated if for all i ∈ N , we have a iT

Original: =
Revised: =

Original: ˆ θ .
Revised: ˆ θ .

Original: That is, under this deﬁnition information is aggregated if agents act as if they had seen all private signals.
Revised: That is, under this deﬁnition information is aggregated if agents act as if they had seen all private signals.

Original: In reality, it is not reasonable to expect perfect information aggregation with actions learned via independent Q-learning.
Revised: In reality, it is not reasonable to expect perfect information aggregation with actions learned via independent Q-learning.

Original: We therefore deﬁne the following measure of information aggregation.
Revised: We therefore deﬁne the following measure of information aggregation.

Original: Deﬁnition 2 (Accuracy of information aggregation) .
Revised: Deﬁnition 2 (Accuracy of information aggregation) .

Original: The accuracy of information aggregation at time step t is A t =
Revised: The accuracy of information aggregation at time step t is A t =

Original: E [ 1 { a it = θ } ] .
Revised: E [ 1 { a it = θ } ] .

Original: A useful benchmark to compare A t to is of course E [ 1 { ˆ θ = θ } ] .
Revised: A useful benchmark to compare A t to is of course E [ 1 { ˆ θ = θ } ] .

Original: Another useful benchmark is E [ 1 { ˜ θ
Revised: Another useful benchmark is E [ 1 { ˜ θ

Original: = θ } ] , where
Revised: = θ } ] , where

Original: ˜ θ = argmax θ P [ θ | s i ] is the MAP estimator given only a single private signal.
Revised: ˜ θ = argmax θ P [ θ | s i ] is the MAP estimator given only a single private signal.

Original: A T can never exceed E [ 1 { ˆ θ = θ } ] and A 1 can never exceed E [ 1 { ˜ θ
Revised: A T can never exceed E [ 1 { ˆ θ = θ } ] and A 1 can never exceed E [ 1 { ˜ θ

Original: = θ } ] and of course E [ 1 { ˆ θ = θ } ] > E
Revised: = θ } ] and of course E [ 1 { ˆ θ = θ } ] > E

Original: [ 1 { ˜ θ
Revised: [ 1 { ˜ θ

Original: = θ } ] provided | N | > 1 .
Revised: = θ } ] provided | N | > 1 .

Original: Once we have trained the agents via
Revised: Once we have trained the agents via

Original: Q-learning it is easy to estimate A t by evaluating a set of batches of games B and computing ˆ
Revised: Q-learning it is easy to estimate A t by evaluating a set of batches of games B and computing ˆ

Original: A t = 1 / | B | 1 / | N | (cid:80) b ∈ B,i ∈ N 1 { a it,b
Revised: A t = 1 / | B | 1 / | N | (cid:80) b ∈ B,i ∈ N 1 { a it,b

Original: = θ  where 
Revised: = θ } , where a it,b is the action taken by agent i in batch b ∈ B .

Original: a
Revised: In the following we will simply refer to ˆ A t as the accuracy. Heuristic benchmark We compare the accuracy of the DQN against the DeGroot heuristic, DeGroot (1974).

Original: it ,b is the action taken by agent i in
Revised: Under this heuristic information aggregation rule at t = 0 , we set a i 0 =

Original: batch
Revised: ˜

Original: b ∈ B .
Revised: θ ( s i ) .

Original: In the following we will simply refer to ˆ A t as the accuracy.
Revised: In each subsequent period actions are set to a t =

Original:   E XPERIMENTS
Revised: Ua t − 1 , where a t is the vector of actions taken by citizens and U = D − 1 ( I + M ) , where I is the identity matrix, M is the graph adjacency matrix and D is a diagonal matrix with entries D ii being the number of neighbors of node i plus one (i.e. a self loop). Information is therefore aggregated through repeated averaging of past actions.  E XPERIMENTS

Original: Our objective is to improve our understanding of the propagation of fake news in social networks and discover potential interventions to curb its effectiveness.
Revised: Our objective is to improve our understanding of the propagation of fake news in social networks and discover potential interventions to curb its effectiveness.

Original: This goal translates into three main questions.
Revised: This goal translates into three main questions.

Original: (i)
Revised: (i)

Original: What determines the effectiveness of the attacker in reducing the accuracy of information aggregation?
Revised: What determines the effectiveness of the attacker in reducing the accuracy of information aggregation?

Original: In particular, how does the network structure and the choice of the attacked agent affect attack effectiveness?
Revised: In particular, how does the network structure and the choice of the attacked agent affect attack effectiveness?

Original: (ii) How well can agents learn adapt to the presence of an attacker?
Revised: (ii) How well can agents learn adapt to the presence of an attacker?

Original: (iii)
Revised: (iii)

Original: When attacker and citizen learn simultaneously (agent takeover attack), how do policies change over time?
Revised: When attacker and citizen learn simultaneously (agent takeover attack), how do policies change over time?

Original: To study these questions we set up a series of training and testing scenarios.
Revised: To study these questions we set up a series of training and testing scenarios.

Original: Training scenarios: We implement the following training scenarios.
Revised: Training scenarios: We implement the following training scenarios.

Original: (i) Baseline: We train citizens in the absence of any attack, i.e. all citizens receive unbiased signals and no citizen is taken over by the attacker.
Revised: (i) Baseline: We train citizens in the absence of any attack, i.e. all citizens receive unbiased signals and no citizen is taken over by the attacker.

Original: We train baseline models for the Barabasi-Albert, clustered and balanced graphs.
Revised: We train baseline models for the Barabasi-Albert, clustered and balanced graphs.

Original: This scenario establishes a benchmark for the ability of IQL agents to learn to aggregate information in the absence of attack.
Revised: This scenario establishes a benchmark for the ability of IQL agents to learn to aggregate information in the absence of attack.

Original: (ii) Biased signal attack: We train citizens in the presence of a biased signal attack where a single, randomly chosen citizen receives a biased signal ( β = 3 ).
Revised: (ii) Biased signal attack: We train citizens in the presence of a biased signal attack where a single, randomly chosen citizen receives a biased signal ( β = 3 ).

Original: We train this model for the Barabasi-Albert graph only.
Revised: We train this model for the Barabasi-Albert graph only.

Original: This scenario allows us to evaluate whether citizens can learn to adapt to the presence of a biased signal attack and thereby mitigate the adverse effects of the attack.
Revised: This scenario allows us to evaluate whether citizens can learn to adapt to the presence of a biased signal attack and thereby mitigate the adverse effects of the attack.

Original: (iii) Agent takeover attack: We train citizens and attacker simultaneously.
Revised: (iii) Agent takeover attack: We train citizens and attacker simultaneously.

Original: We train this model for the Barabasi-Albert graph only.
Revised: We train this model for the Barabasi-Albert graph only.

Original: This scenario allows us to understand the effectiveness of attack as it co-evolves with the citizen’s effort to mitigate the attacker’s ability to manipulate citizens.
Revised: This scenario allows us to understand the effectiveness of attack as it co-evolves with the citizen’s effort to mitigate the attacker’s ability to manipulate citizens.

Original: Note that IQL may not converge to stable policies in such a game.
Revised: Note that IQL may not converge to stable policies in such a game.

Original: It is interesting however, to understand the action dynamics that emerge under IQL as they can help us understand the coevolution of fake news attack and defense in real social networks.
Revised: It is interesting however, to understand the action dynamics that emerge under IQL as they can help us understand the coevolution of fake news attack and defense in real social networks.

Original: (iv) Random action attack: We train citizens in the presence of an attacker that selects a single citizen uniformly at random and then, in each period, picks an action uniformly at random.
Revised: (iv) Random action attack: We train citizens in the presence of an attacker that selects a single citizen uniformly at random and then, in each period, picks an action uniformly at random.

Original: We train this model for the Barabasi-Albert graph only.
Revised: We train this model for the Barabasi-Albert graph only.

Original: This scenario serves as a benchmark for the agent takeover attack.
Revised: This scenario serves as a benchmark for the agent takeover attack.

Original: The accuracy under the random action attack should exceed the accuracy under the agent takeover attack.
Revised: The accuracy under the random action attack should exceed the accuracy under the agent takeover attack.

Original: Testing scenarios: Given the models obtained for these different training scenarios we conduct a number of testing scenarios.
Revised: Testing scenarios: Given the models obtained for these different training scenarios we conduct a number of testing scenarios.

Original: First, for each attack training scenario, we evaluate accuracy, as deﬁned above, both in the presence of the corresponding attack (as trained) and in the absence of any attack.
Revised: First, for each attack training scenario, we evaluate accuracy, as deﬁned above, both in the presence of the corresponding attack (as trained) and in the absence of any attack.

Original: For the baseline scenario we consider two testing scenarios in addition to testing in the absence of any attack (as trained).
Revised: For the baseline scenario we consider two testing scenarios in addition to testing in the absence of any attack (as trained).

Original: First, we evaluate accuracy in the baseline model for the Barabasi-Albert graph in the presence of a biased signal attack with a single attacked agent and β = 3 .
Revised: First, we evaluate accuracy in the baseline model for the Barabasi-Albert graph in the presence of a biased signal attack with a single attacked agent and β = 3 .

Original: Second, we evaluate accuracy in the baseline model for the clustered and balanced graphs under two biased signal attack scenarios.
Revised: Second, we evaluate accuracy in the baseline model for the clustered and balanced graphs under two biased signal attack scenarios.

Original: In the focused scenario, a single, randomly chosen citizen receives a biased signal with β i = 3 .
Revised: In the focused scenario, a single, randomly chosen citizen receives a biased signal with β i = 3 .

Original: In the spread scenario, two, randomly chosen citizens receive biased signals with β i = 1 . 5 each.
Revised: In the spread scenario, two, randomly chosen citizens receive biased signals with β i = 1 . 5 each.

Original:  R ESULTS
Revised:  R ESULTS

Original: Our experiments yield four main results, which we will discuss in turn.
Revised: Our experiments yield four main results, which we will discuss in turn.

Original: Baseline information aggregation:
Revised: Baseline information aggregation:

Original: In the absence of attack, citizens learn to aggregate information with an accuracy close to the optimal benchmarks E [ 1 { ˆ θ = θ } ] as t → T and E [ 1 { ˜ θ
Revised: In the absence of attack, citizens learn to aggregate information with an accuracy close to the optimal benchmarks E [ 1 { ˆ θ = θ } ] as t → T and E [ 1 { ˜ θ

Original: = θ } ] for t = 0 .
Revised: = θ } ] for t = 0 .

Revised: Our method’s accuracy also substantially outperforms the accuracy achieved by DeGroot information aggregation heuristic.

Original: This highlights the appropriateness of IQL to solve the standard information aggregation game.
Revised: This highlights the appropriateness of IQL to solve the standard information aggregation game.

Original: In the presence of an attack, information aggregation is severely disrupted.
Revised: In the presence of an attack, information aggregation is severely disrupted.

Original: These results are illustrated in Figure 2 (A). 
Revised: These results are illustrated in Figure 2 (A). [Figure] C

Original:  Determinants of effectiveness of attack: Our experiments allow us to study the following determinants of attack effectiveness: network position of the attacked agent, signal strength of the attacked agent, network structure (clustered vs balanced) and distribution of bias across agents (focus vs spread).
Revised: D E [Figure] Determinants of effectiveness of attack: Our experiments allow us to study the following determinants of attack effectiveness: network position of the attacked agent, signal strength of the attacked agent, network structure (clustered vs balanced) and distribution of bias across agents (focus vs spread).

Original: Let us consider each in turn.
Revised: Let us consider each in turn.

Original: To study the effect of network position, we compute the accuracy in the ﬁnal time step ( t = T ) conditional on the network position of the attacked agent when evaluating the baseline model under the biased signal attack in the Barabasi-Albert network.
Revised: To study the effect of network position, we compute the accuracy in the ﬁnal time step ( t = T ) conditional on the network position of the attacked agent when evaluating the baseline model under the biased signal attack in the Barabasi-Albert network.

Original: Denote this conditional accuracy by ˆ A T ( i ) , where i is the attacked agent.
Revised: Denote this conditional accuracy by ˆ A T ( i ) , where i is the attacked agent.

Original: Let ˆ A T denote the baseline accuracy in the absence of attack.
Revised: Let ˆ A T denote the baseline accuracy in the absence of attack.

Original: We then deﬁne the decline in accuracy relative to baseline as ∆ A ( i ) =
Revised: We then deﬁne the decline in accuracy relative to baseline as ∆ A ( i ) =

Original: ˆ A T − ˆ A T ( i ) .
Revised: ˆ A T − ˆ A T ( i ) .

Original: We plot ∆
Revised: We plot ∆

Original: A ( i ) as a function of i in Figure 2 (C).
Revised: A ( i ) as a function of i in Figure 2 (C).

Original: It is clear that there is substantial heterogeneity in the attack effectiveness.
Revised: It is clear that there is substantial heterogeneity in the attack effectiveness.

Original: As the tight correlation between ∆ A ( i ) and the degree of i in Figure 2 (C) shows, the heterogeneity in ∆ A ( i ) arises from the network structure which implicitly gives more inﬂuence to those nodes with more neighbors (higher degree).
Revised: As the tight correlation between ∆ A ( i ) and the degree of i in Figure 2 (C) shows, the heterogeneity in ∆ A ( i ) arises from the network structure which implicitly gives more inﬂuence to those nodes with more neighbors (higher degree).

Original: If an attacker is able to target the biased signal to such highly connected nodes, his attack will be more effective.
Revised: If an attacker is able to target the biased signal to such highly connected nodes, his attack will be more effective.

Original: Next we study the effect of the signal strength of attacked agent.
Revised: Next we study the effect of the signal strength of attacked agent.

Original: Again, we restrict ourselves to evaluating the baseline model under the biased signal attack in the Barabasi-Albert network.A [Figure]
Revised: Again, we restrict ourselves to evaluating the baseline model under the biased signal attack in the Barabasi-Albert network.

Original: C natural measure of a signal’s strength, or informativeness, is the absolute value of the log likelihood ratio of the two states of θ conditional on the signal.
Revised: A natural measure of a signal’s strength, or informativeness, is the absolute value of the log likelihood ratio of the two states of θ conditional on the signal.

Original: Let f ( θ | s ) denote the posterior distribution over θ given the signal s .
Revised: Let f ( θ | s ) denote the posterior distribution over θ given the signal s .

Original: Then, the signal strength is given by L ( s ) =
Revised: Then, the signal strength is given by L ( s ) =

Original: | log[ f ( θ = 1 | s ) /f
Revised: | log[ f ( θ = 1 | s ) /f

Original: ( θ = 0 | s )] | .
Revised: ( θ = 0 | s )] | .

Original: Let L k denote the k th decile of the empirical distribution of signal strengths for a particular experiment.
Revised: Let L k denote the k th decile of the empirical distribution of signal strengths for a particular experiment.

Original: We deﬁne the conditional accuracy ˆ A ( L k ) as the average accuracy in the ﬁnal time step conditional on the attacked agent’s signal strength lying in [ L k , L k +1 ] .
Revised: We deﬁne the conditional accuracy ˆ A ( L k ) as the average accuracy in the ﬁnal time step conditional on the attacked agent’s signal strength lying in [ L k , L k +1 ] .

Original: ∆ A ( L k ) is deﬁned analogously to ∆ A ( i ) .
Revised: ∆ A ( L k ) is deﬁned analogously to ∆ A ( i ) .

Original: We plot ∆
Revised: We plot ∆

Original: A ( L k ) against L k in Figure 2 (D).
Revised: A ( L k ) against L k in Figure 2 (D).

Original: There exists a strong negative correlation between attack effectiveness as measured by ∆ A ( L k ) and the attacked agent’s signal strength.
Revised: There exists a strong negative correlation between attack effectiveness as measured by ∆ A ( L k ) and the attacked agent’s signal strength.

Original: This is intuitive.
Revised: This is intuitive.

Original: If an agent has received a strong private signal in support of some value of θ , a larger bias will be required to convince him of the contrary.
Revised: If an agent has received a strong private signal in support of some value of θ , a larger bias will be required to convince him of the contrary.

Original: Thus, if an attacker can target the biased signal to agents with weak private signals, his attack will be more effective.
Revised: Thus, if an attacker can target the biased signal to agents with weak private signals, his attack will be more effective.

Original:  Lastly, we consider the case of clustered vs rewired/balanced networks and spread our vs focused attacks.
Revised: [Table] [Table] Lastly, we consider the case of clustered vs rewired/balanced networks and spread our vs focused attacks.

Original: This analysis is done evaluating the baseline model under the biased signal attack in the clustered and balanced networks.
Revised: This analysis is done evaluating the baseline model under the biased signal attack in the clustered and balanced networks.

Original: A number of results are worth noting.
Revised: A number of results are worth noting.

Original: As can be seen in Figure 2 (E), for each scenario (baseline/no attack, focus and spread attacks) accuracy in the balanced network exceeds accuracy in the clustered network.
Revised: As can be seen in Figure 2 (E), for each scenario (baseline/no attack, focus and spread attacks) accuracy in the balanced network exceeds accuracy in the clustered network.

Original: This suggests that information aggregation is more effective in the balanced network.
Revised: This suggests that information aggregation is more effective in the balanced network.

Original: This is intuitive as the balanced networks has a shorter maximum path length between any two nodes thereby allowing information to “propagate” faster between nodes.
Revised: This is intuitive as the balanced networks has a shorter maximum path length between any two nodes thereby allowing information to “propagate” faster between nodes.

Original: In Figure 2 (E), we can also see that variation in the accuracy between training runs with different seeds (each run corresponds to one marker for a particular scenario) is smaller for the balanced network.
Revised: In Figure 2 (E), we can also see that variation in the accuracy between training runs with different seeds (each run corresponds to one marker for a particular scenario) is smaller for the balanced network.

Original: This suggests that in the balanced network learning good policies is an easier task than in the clustered network.
Revised: This suggests that in the balanced network learning good policies is an easier task than in the clustered network.

Original: We can also see from Figure 2 (E) that spread attacks are more effective both in the clustered and balanced networks.
Revised: We can also see from Figure 2 (E) that spread attacks are more effective both in the clustered and balanced networks.

Original: To summarize, targeting highly connected agents in the network or those with weak private signals, makes biased signal attacks more effective.
Revised: To summarize, targeting highly connected agents in the network or those with weak private signals, makes biased signal attacks more effective.

Original: Clustered networks are more susceptible to attack than balanced networks and spread attacks are more effective than focused ones.
Revised: Clustered networks are more susceptible to attack than balanced networks and spread attacks are more effective than focused ones.

Original: Adapting to attacks: So far we have evaluated models trained in the absence of attack.
Revised: Adapting to attacks: So far we have evaluated models trained in the absence of attack.

Original: However, one can expect that users of social networks will adapt over time to the presence of fake news.
Revised: However, one can expect that users of social networks will adapt over time to the presence of fake news.

Original: We therefore investigate to what extent citizens can learn to mitigate the effect of an attacker.
Revised: We therefore investigate to what extent citizens can learn to mitigate the effect of an attacker.

Original: For this purpose we train citizens in the presence of a biased signal attacker, a random attacker and an agent takeover attacker for the Barabasi-Albert network.We then evaluate the accuracy of each of these [Figure] [Figure]
Revised: For this purpose we train citizens in the presence of a biased signal attacker, a random attacker and an agent takeover attacker for the Barabasi-Albert network.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] models in the absence of an attack and in the presence of the attack under which they were trained.
Revised: We then evaluate the accuracy of each of these models in the absence of an attack and in the presence of the attack under which they were trained.

Original: We summarize our results in Table 1.
Revised: We summarize our results in Table 1.

Original: Let us ﬁrst contrast the accuracy of the baseline model (trained without attack) under a biased signal attacker with the accuracy of a model trained under this attack scenario.
Revised: Let us ﬁrst contrast the accuracy of the baseline model (trained without attack) under a biased signal attacker with the accuracy of a model trained under this attack scenario.

Original: When trained in the presence of an attacker the test accuracy under attack increases from 0 .
Revised: When trained in the presence of an attacker the test accuracy under attack increases from 0 .

Original: 78 to 0 .
Revised: 78 to 0 .

Original: 84 relative to the baseline.
Revised: 84 relative to the baseline.

Original: This can also be seen in Figure 2 (B).
Revised: This can also be seen in Figure 2 (B).

Original: We conclude that agents can indeed learn to adapt to the presence of fake news.
Revised: We conclude that agents can indeed learn to adapt to the presence of fake news.

Original: However, this adaptation comes at a cost.
Revised: However, this adaptation comes at a cost.

Original: When trained in the presence of an attacker the test accuracy without attack decreases from 0 .
Revised: When trained in the presence of an attacker the test accuracy without attack decreases from 0 .

Original: 92 to 0 .
Revised: 92 to 0 .

Original: 88 relative to the baseline.
Revised: 88 relative to the baseline.

Original: We conjecture that this is because agents learn to trust strong private signals less and are less likely to follow their neighbors’ actions.
Revised: We conjecture that this is because agents learn to trust strong private signals less and are less likely to follow their neighbors’ actions.

Original: A similar picture can be seen for the random attacker and the agent takeover attacker.
Revised: A similar picture can be seen for the random attacker and the agent takeover attacker.

Original: To summarize, agents can learn to adapt to the presence of an attacker which reduces attack effectiveness.
Revised: To summarize, agents can learn to adapt to the presence of an attacker which reduces attack effectiveness.

Original: However, adaptation comes at a cost.
Revised: However, adaptation comes at a cost.

Original: If trained under attack and tested in the absence of attack, accuracy is lower than in the baseline which is trained without attack.
Revised: If trained under attack and tested in the absence of attack, accuracy is lower than in the baseline which is trained without attack.

Original: Dynamics of attack and defense: We now take a closer look at the agent takeover attack.
Revised: Dynamics of attack and defense: We now take a closer look at the agent takeover attack.

Original: Recall that in this scenario, a separate DQN agent is trained to minimize the total reward of the citizens by acting on behalf of a randomly chosen agent.
Revised: Recall that in this scenario, a separate DQN agent is trained to minimize the total reward of the citizens by acting on behalf of a randomly chosen agent.

Original: This attacker knows θ and must then choose an action to convince the citizens of the opposite.
Revised: This attacker knows θ and must then choose an action to convince the citizens of the opposite.

Original: Simply choosing the opposite action would be a naive strategy that is likely to be detected quickly by the citizens as they learn in the presence of this strategy.
Revised: Simply choosing the opposite action would be a naive strategy that is likely to be detected quickly by the citizens as they learn in the presence of this strategy.

Original: Always choosing a random action in every period, as the random attacker does, is also  quite easily detectable by the citizens.
Revised: Always choosing a random action in every period, as the random attacker does, is also [Figure] quite easily detectable by the citizens.

Original: This intuition is conﬁrmed by results reported in Table 1 and
Revised: This intuition is conﬁrmed by results reported in Table 1 and

Original: Figure 3.
Revised: Figure 3.

Original: From Table 1 we see that the trained attacker clearly “outperforms” (i.e. more effective attack) the random action attack benchmark.
Revised: From Table 1 we see that the trained attacker clearly “outperforms” (i.e. more effective attack) the random action attack benchmark.

Original: The bottom panel of Figure 3 (B) shows the accuracy of the attacker in the ﬁrst time step as a simple summary statistic of its strategy.
Revised: The bottom panel of Figure 3 (B) shows the accuracy of the attacker in the ﬁrst time step as a simple summary statistic of its strategy.

Original: The accuracy is either 0 .indicating that the attacker “randomizes” or 0 indicating that the attacker “lies”.
Revised: The accuracy is either 0 .indicating that the attacker “randomizes” or 0 indicating that the attacker “lies”.

Original: While Q-learning only allows for deterministic policies the attacker can learn to lie if the attacked agent is in one partition of N and be truthful if the attacked agent is in another partition.
Revised: While Q-learning only allows for deterministic policies the attacker can learn to lie if the attacked agent is in one partition of N and be truthful if the attacked agent is in another partition.

Original: Since the attacked agent is chosen at random, this strategy amounts to a randomized attack strategy.
Revised: Since the attacked agent is chosen at random, this strategy amounts to a randomized attack strategy.

Original: Note further, that neither the attacker’s nor the citizens’ policy converges as can been seen in the highly variable reward and unstable TD-error in Figure 3 (A) .
Revised: Note further, that neither the attacker’s nor the citizens’ policy converges as can been seen in the highly variable reward and unstable TD-error in Figure 3 (A) .

Original: Indeed, as can be seen in the bottom panel of Figure 3 (B) , the attacker’s policy ﬂuctuates between randomization and lying between episodes.
Revised: Indeed, as can be seen in the bottom panel of Figure 3 (B) , the attacker’s policy ﬂuctuates between randomization and lying between episodes.

Original: These ﬂuctuations can be seen as irregular “cycles” in the policy space of the attacker and the citizen.
Revised: These ﬂuctuations can be seen as irregular “cycles” in the policy space of the attacker and the citizen.

Original: One such cycle in Figure 3 (B) shows the attacker switching to a lying strategy, while the citizens lose “trust” in their private signal (they nearly ignore it in the ﬁrst time step).
Revised: One such cycle in Figure 3 (B) shows the attacker switching to a lying strategy, while the citizens lose “trust” in their private signal (they nearly ignore it in the ﬁrst time step).

Original: The result of these two events coming together can lead to substantial, temporary declines in accuracy.
Revised: The result of these two events coming together can lead to substantial, temporary declines in accuracy.

Original: We don’t claim that these dynamics are accurate representations of the learning dynamics between users of social networks and pushers of fake news.
Revised: We don’t claim that these dynamics are accurate representations of the learning dynamics between users of social networks and pushers of fake news.

Original: However, we consider the failure of IQL to converge and the irregular dynamics that emerge more of a feature than a bug.
Revised: However, we consider the failure of IQL to converge and the irregular dynamics that emerge more of a feature than a bug.

Original: Indeed these dynamics suggest two interesting, qualitative take aways.
Revised: Indeed these dynamics suggest two interesting, qualitative take aways.

Original: First, attackers and citizens are unlikely to converge to stable policies in real life.
Revised: First, attackers and citizens are unlikely to converge to stable policies in real life.

Original: We should expect a continuing cycle of adaptation with periods of a high prevalence of fake news followed by periods of low prevalence.
Revised: We should expect a continuing cycle of adaptation with periods of a high prevalence of fake news followed by periods of low prevalence.

Original: Second, the co-evolution of attackers and citizens can have unexpected effects on the citizens’ ability to aggregate information in the social network.
Revised: Second, the co-evolution of attackers and citizens can have unexpected effects on the citizens’ ability to aggregate information in the social network.

Original: As we observe in Figure 3 (B) , the presence of neighbors that do not act in accordance with their private signals (i.e. the attacked agents) can lead citizens to learn to mistrust their own private signals.
Revised: As we observe in Figure 3 (B) , the presence of neighbors that do not act in accordance with their private signals (i.e. the attacked agents) can lead citizens to learn to mistrust their own private signals.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: The deliberate manipulation of the public’s perception of facts via fake news, in particular on social networks, has become a growing concern for policy makers and technology companies alike.
Revised: The deliberate manipulation of the public’s perception of facts via fake news, in particular on social networks, has become a growing concern for policy makers and technology companies alike.

Original: Despite its importance, little is understood about the spread of fake news on social networks.
Revised: Despite its importance, little is understood about the spread of fake news on social networks.

Original: This is in large part due to the technical difﬁculties involved in studying complex decision making behavior on social networks.
Revised: This is in large part due to the technical difﬁculties involved in studying complex decision making behavior on social networks.

Original: We seek to close this gap by developing the (to the best of our knowledge) ﬁrst framework for the study fake news on social networks that is theoretically grounded yet practical and ﬂexible.
Revised: We seek to close this gap by developing the (to the best of our knowledge) ﬁrst framework for the study fake news on social networks that is theoretically grounded yet com- putationally tractable and ﬂexible.

Original: We achieve this by (i) extending a standard game of information aggregation to accommodate fake news and (ii) applying state-of-the-art deep multi-agent reinforcement learning to solve the game.
Revised: We achieve this by (i) extending a standard game of information aggregation to accommodate fake news and (ii) applying state-of-the-art deep multi-agent reinforcement learning to solve the game.

Original: Our ﬁndings suggest a number of interventions that could contribute to making fake news attacks less effective.
Revised: Our ﬁndings suggest a number of interventions that could contribute to making fake news attacks less effective.

Original: First, if agents are made aware of the presence of fake news, they can learn to adapt and mitigate its effect.
Revised: First, if agents are made aware of the presence of fake news, they can learn to adapt and mitigate its effect.

Original: However, this adaptation is likely to be accompanied by evolving attack strategies, so any adaptation will not be lead to permanent mitigation.
Revised: However, this adaptation is likely to be accompanied by evolving attack strategies, so any adaptation will not be lead to permanent mitigation.

Original: In addition, the adaptation is likely to harm information aggregation in the absence of fake news.
Revised: In addition, the adaptation is likely to harm information aggregation in the absence of fake news.

Original: Second, keeping private information on social networks private, can make it harder for attackers to condition their attacks on network position and general informedness (as the real world analogue of signal strength).
Revised: Second, keeping private information on social networks private, can make it harder for attackers to condition their attacks on network position and general informedness (as the real world analogue of signal strength).

Original: This should reduce attack effectiveness.
Revised: This should reduce attack effectiveness.

Original: Third, encouraging well balanced social networks can improve information aggregation in general and make fake news attacks less effective in particular.
Revised: Third, encouraging well balanced social networks can improve information aggregation in general and make fake news attacks less effective in particular.

Original: Our current approach has two main shortcomings.
Revised: Our current approach has two main shortcomings.

Original: First, it does not scale well to larger populations of agents since it requires the entire social network to be trained simultaneously.
Revised: First, it does not scale well to larger populations of agents since it requires the entire social network to be trained simultaneously.

Original: One way forward is to train agents in sub-graphs feeding them an embedding of the graph, see for example Gilmer et al.
Revised: One way forward is to train agents in sub-graphs feeding them an embedding of the graph, see for example Gilmer et al.

Original: (2017), and then composing larger graphs at test time.
Revised: (2017), and then composing larger graphs at test time.

Original: The second shortcoming is the use of IQL in the agent takeover attack scenario.
Revised: The second shortcoming is the use of IQL in the agent takeover attack scenario.

Original: While the lack of convergence leads to interesting qualitative insights, MARL methods that encourage convergence of policies would be desirable here, at least as benchmarks.
Revised: While the lack of convergence leads to interesting qualitative insights, MARL methods that encourage convergence of policies would be desirable here, at least as benchmarks.

Original: One possible way to achieve this would be to apply stable opponent shaping to citizen and attacker, see Letcher et 
Revised: One possible way to achieve this would be to apply stable opponent shaping to citizen and attacker, see Letcher et al.

Original: A A PPENDIX A. M ULTI - AGENT REINFORCEMENT LEARNING
Revised: A A PPENDIX A. M ULTI - AGENT REINFORCEMENT LEARNING

Original: Single-Agent Reinforcement Learning Consider an MDP speciﬁed by the following tuple: (cid:104) S, A, P, r, γ (cid:105) .
Revised: Single-Agent Reinforcement Learning Consider an MDP speciﬁed by the following tuple: (cid:104) S, A, P, r, γ (cid:105) .

Original: In the fully observable case the environment has an observed true state, s ∈ S , and at each time step the agent chooses an action a ∈ A , which induces a probabilistic transition to the next state via the state transition function P ( s (cid:48) | s, a ) :
Revised: In the fully observable case the environment has an observed true state, s ∈ S , and at each time step the agent chooses an action a ∈ A , which induces a probabilistic transition to the next state via the state transition function P ( s (cid:48) | s, a ) :

Original: S ×
Revised: S ×

Original: A × S → [0 , 1] .
Revised: A × S → [0 , 1] .

Original: The reward function assigns an instantaneous reward to each state-action pair, r ( s, a ) : S ×
Revised: The reward function assigns an instantaneous reward to each state-action pair, r ( s, a ) : S ×

Original: A → R , and γ ∈ [0 , 1) is a discount factor.
Revised: A → R , and γ ∈ [0 , 1) is a discount factor.

Original: In single agent RL the task of the agent is to maximize the discounted reward per episode, R t = (cid:80) ∞
Revised: In single agent RL the task of the agent is to maximize the discounted reward per episode, R t = (cid:80) ∞

Original: l =

Revised: l =

Original: γ l r t + l .
Revised: γ l r t + l .

Original: The agent’s policy, π ( a | s ) : S ×
Revised: The agent’s policy, π ( a | s ) : S ×

Original: A → [0 , 1] , induces a state-action-value function (Qfunction), Q π ( s t , a t ) =
Revised: A → [0 , 1] , induces a state-action-value function (Qfunction), Q π ( s t , a t ) =

Original: E s t +1: ∞ ,a t +1: ∞
Revised: E s t +1: ∞ ,a t +1: ∞

Original: [ R t | s t , a t ] .
Revised: [ R t | s t , a t ] .

Original: Q-learning aims to estimate the optimal action- value function, Q ∗ ( x, a ) =
Revised: Q-learning aims to estimate the optimal action- value function, Q ∗ ( x, a ) =

Original: max π Q π ( x, a ) , via an estimated Q-function, Q ( s, a ) .
Revised: max π Q π ( x, a ) , via an estimated Q-function, Q ( s, a ) .

Original: Training is carried out by collecting samples from the environment in order to obtain MonteCarlo-Estimates of the expected return.
Revised: Training is carried out by collecting samples from the environment in order to obtain MonteCarlo-Estimates of the expected return.

Original: For any sampled transition, the current estimate, Q ( s, a ) , is compared to a greedy one-step lookahead using the Bellman optimality operator, T Q ( s, a ) =
Revised: For any sampled transition, the current estimate, Q ( s, a ) , is compared to a greedy one-step lookahead using the Bellman optimality operator, T Q ( s, a ) =

Original: E s [Equation]
Revised: E s (cid:48)

Original: [ r + γ max a [Equation] Q ( s (cid:48) , a (cid:48) )] .
Revised: [ r + γ max a (cid:48) Q ( s (cid:48) , a (cid:48) )] .

Original: We can use samples to evaluate the Bellman update:
Revised: We can use samples to evaluate the Bellman update:

Original: [Equation]
Revised: [Equation]

Original: Here k is the iteration number, α is the learning rate and r + γ max a (cid:48) Q ( s (cid:48) , a (cid:48) )
Revised: Here k is the iteration number, α is the learning rate and r + γ max a (cid:48) Q ( s (cid:48) , a (cid:48) )

Original: k − Q ( s, a ) k is commonly referred to as the temporal-difference or TD-error.
Revised: k − Q ( s, a ) k is commonly referred to as the temporal-difference or TD-error.

Original: In the tabular case the Q-values for each state-action pair are maintained separately and the Bellman update is a contraction mapping.
Revised: In the tabular case the Q-values for each state-action pair are maintained separately and the Bellman update is a contraction mapping.

Original: As a consequence at convergence this iterative process results in the optimal Q -function, Q ( s, a ) ∞ =
Revised: As a consequence at convergence this iterative process results in the optimal Q -function, Q ( s, a ) ∞ =

Original: Q ∗ ( s, a ) .
Revised: Q ∗ ( s, a ) .

Original: Finally, Q ∗ ( s, a ) trivially deﬁnes the optimal policy π ∗ ( x, a )
Revised: Finally, Q ∗ ( s, a ) trivially deﬁnes the optimal policy π ∗ ( x, a )

Original: =
Revised: =

Original: δ (arg max a (cid:48)
Revised: δ (arg max a (cid:48)

Original: Q ∗ ( x, a (cid:48) ) − a ) , where δ ( · ) is the Dirac-delta function.
Revised: Q ∗ ( x, a (cid:48) ) − a ) , where δ ( · ) is the Dirac-delta function.

Original: In contrast to the tabular case, DQN (Mnih et al., 2015) uses a neural network parametrized by a large number of weights, φ , to represent the Q -function.
Revised: In contrast to the tabular case, DQN (Mnih et al., 2015) uses a neural network parametrized by a large number of weights, φ , to represent the Q -function.

Original: In order to reduce the variance of the update process the average square of the TD-error across a large number of transitions (the batch ) is used as the DQN-loss.
Revised: In order to reduce the variance of the update process the average square of the TD-error across a large number of transitions (the batch ) is used as the DQN-loss.

Original: Using backpropagation the parameters of the neural network are updated to minimize the magnitude of the DQN error
Revised: Using backpropagation the parameters of the neural network are updated to minimize the magnitude of the DQN error

Original: L ( φ ) =
Revised: L ( φ ) =

Original: (cid:80) bj =
Revised: (cid:80) bj =

Original: 1 [( y DQNj − Q ( x j , a j ; φ )) 2 ] .
Revised: 1 [( y DQN j − Q ( x j , a j ; φ )) 2 ] .

Original: Here y DQNj =
Revised: Here y DQN j

Original: u j + γ max a (cid:48)
Revised: [Equation]

Original: j Q ( x (cid:48) j , a (cid:48) j ; φ − ) , is the target function and φ − is the target network, which contains a stale copy of the parameters.
Revised: ) , is the target function and φ − is the target network, which contains a stale copy of the parameters.

Original: This target network helps to stabilize the training.
Revised: This target network helps to stabilize the training.

Original: So far we have assumed that the agent has access to the Markov state, s , of the system.
Revised: So far we have assumed that the agent has access to the Markov state, s , of the system.

Original: In a partially observable settings we augment the MDP with an observation space, Z , and observation function
Revised: In a partially observable settings we augment the MDP with an observation space, Z , and observation function

Original: O ( s ) .
Revised: O ( s ) .

Original: In particular the observation z ∈ Z is produced by the observation function
Revised: In particular the observation z ∈ Z is produced by the observation function

Original: O ( s ) :
Revised: O ( s ) :

Original: S → Z .
Revised: S → Z .

Original: We further deﬁne an action-observation history τ ∈ T ≡ ( Z × U ) ∗ , which is used to condition a stochastic policy π ( a | τ ) : T × U → [0 , 1] .
Revised: We further deﬁne an action-observation history τ ∈ T ≡ ( Z × U ) ∗ , which is used to condition a stochastic policy π ( a | τ ) : T × U → [0 , 1] .

Original: In recurrent deep RL (Hausknecht & Stone, 2015), this can be achieved using recurrent neural networks, such as LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Cho et al., 2014) which we use here.
Revised: In recurrent deep RL (Hausknecht & Stone, 2015), this can be achieved using recurrent neural networks, such as LSTM (Hochreiter & Schmidhuber, 1997) or GRU (Cho et al., 2014) which we use here.

Original: Independent Q-Learning In multi-agent reinforcement learning, each agent i ∈ N receives a private observation O ( x, i ) , where i is the agent index and O is the observation function.
Revised: Independent Q-Learning In multi-agent reinforcement learning, each agent i ∈ N receives a private observation O ( x, i ) , where i is the agent index and O is the observation function.

Original: The agents also receive individual reward r it and take actions a it .
Revised: The agents also receive individual reward r it and take actions a it .

Original: Furthermore the state-transition conditions on the joined action a ∈ A ≡ A n .
Revised: Furthermore the state-transition conditions on the joined action a ∈ A ≡ A n .

Original: In independent Q-learning each agent further estimates a Q -function Q i ( τ i , a i ) , treating the other agents and their policies as part of a non-stationary environment.
Revised: In independent Q-learning each agent further estimates a Q -function Q i ( τ i , a i ) , treating the other agents and their policies as part of a non-stationary environment.

Original: A combination DQN with IQL commonly uses parameter-sharing across agents combined with an agents speciﬁc index in the observation function in order to accelerate learning while still allowing for specialization of policies.
Revised: A combination DQN with IQL commonly uses parameter-sharing across agents combined with an agents speciﬁc index in the observation function in order to accelerate learning while still allowing for specialization of policies.

Original: A. I MPLEMENTATION DETAILS
Revised: A. I MPLEMENTATION DETAILS

Original: Our code has been submitted.
Revised: Our code has been submitted.

Original: Our neural network architecture implements the IQL architecture in
Revised: Our neural network architecture implements the IQL architecture in

Original: Foerster et al.
Revised: Foerster et al.