Original: Peer Prediction for Learning Agents
Revised: Peer Prediction for Learning Agents

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Peer prediction refers to a collection of mechanisms for eliciting information fromhuman agents when direct verification of the obtained information is unavailable.
Revised: Peer prediction refers to a collection of mechanisms for eliciting information fromhuman agents when direct verification of the obtained information is unavailable.

Original: They are designed to have a game-theoretic equilibrium where everyone revealstheir private information truthfully.
Revised: They are designed to have a game-theoretic equilibrium where everyone revealstheir private information truthfully.

Original: This result holds under the assumption thatagents are Bayesian and they each adopt a fixed strategy across all tasks.
Revised: This result holds under the assumption thatagents are Bayesian and they each adopt a fixed strategy across all tasks.

Original: Humanagents however are observed in many domains to exhibit learning behavior insequential settings.
Revised: Humanagents however are observed in many domains to exhibit learning behavior insequential settings.

Original: In this paper, we explore the dynamics of sequential peerprediction mechanisms when participants are learning agents.
Revised: In this paper, we explore the dynamics of sequential peerprediction mechanisms when participants are learning agents.

Original: We first show thatthe notion of no regret alone for the agents’ learning algorithms cannot guarantee convergence to the truthful strategy.
Revised: We first show thatthe notion of no regret alone for the agents’ learning algorithms cannot guarantee convergence to the truthful strategy.

Original: We then focus on a family of learningalgorithms where strategy updates only depend on agents’ cumulative rewards andprove that agents’ strategies in the popular Correlated Agreement (CA) mechanismconverge to truthful reporting when they use algorithms from this family.
Revised: We then focus on a family of learningalgorithms where strategy updates only depend on agents’ cumulative rewards andprove that agents’ strategies in the popular Correlated Agreement (CA) mechanismconverge to truthful reporting when they use algorithms from this family.

Original: This family of algorithms is not necessarily no-regret, but includes several familiar no-regretlearning algorithms (e.g multiplicative weight update and Follow the Perturbed
Revised: This family of algorithms is not necessarily no-regret, but includes several familiar no-regretlearning algorithms (e.g multiplicative weight update and Follow the Perturbed

Original: Leader) as special cases.
Revised: Leader) as special cases.

Original: Simulation of several algorithms in this family as well asthe ϵ -greedy algorithm, which is outside of this family, shows convergence to thetruthful strategy in the CA mechanism.
Revised: Simulation of several algorithms in this family as well asthe ϵ -greedy algorithm, which is outside of this family, shows convergence to thetruthful strategy in the CA mechanism.

Original:  Introduction
Revised:  Introduction

Original: A fundamental challenge in many domains is to elicit high-quality information from people whendirectly verifying the acquired information is not feasible, either because the ground truth is notavailable or because it’s too costly to obtain.
Revised: A fundamental challenge in many domains is to elicit high-quality information from people whendirectly verifying the acquired information is not feasible, either because the ground truth is notavailable or because it’s too costly to obtain.

Original: Notable settings include asking people to label datafor machine learning, having students perform peer grading in education and soliciting customerfeedback for products and services.
Revised: Notable settings include asking people to label datafor machine learning, having students perform peer grading in education and soliciting customerfeedback for products and services.

Original: The peer prediction literature has made impressive progress on this challenge in the past two decades,with many mechanisms that have desirable incentive properties developed for this problem [20, 19,27, 5, 22, 8, 14, 25, 17, 16, 21, 18, 23, 15].The term peer prediction refers to a collection of rewardmechanisms that solicit information from human agents and reward each agent solely based on howthe agent’s reported information compares with that of the other agents, without having access to theground truth.
Revised: The peer prediction literature has made impressive progress on this challenge in the past two decades,with many mechanisms that have desirable incentive properties developed for this problem [19, 18,26, 5, 21, 8, 13, 24, 16, 15, 20, 17, 22, 14].The term peer prediction refers to a collection of rewardmechanisms that solicit information from human agents and reward each agent solely based on howthe agent’s reported information compares with that of the other agents, without having access to theground truth.

Original: Under some assumptions, many peer prediction mechanisms [20, 19, 27, 22, 8, 16, 21]guarantee that every agent truthfully reporting their information is a game-theoretic equilibrium, andthe more recent multi-task peer prediction mechanisms [5, 14, 25, 17, 18, 23, 15] further ensure thatagents receive the highest expected payoff at the truthful equilibrium, compared with other strategyprofiles.
Revised: Under some assumptions, many peer prediction mechanisms [19, 18, 26, 21, 8, 15, 20]guarantee that every agent truthfully reporting their information is a game-theoretic equilibrium, andthe more recent multi-task peer prediction mechanisms [5, 13, 24, 16, 17, 22, 14] further ensure thatagents receive the highest expected payoff at the truthful equilibrium, compared with other strategyprofiles.

Original: While achieving truthful reporting as a highest-payoff equilibrium is a victory to declare for thischallenging without-verification setting, there are however caveats associated with adopting thenotion of equilibrium as a solution concept.
Revised: While achieving truthful reporting as a highest-payoff equilibrium is a victory to declare for thischallenging without-verification setting, there are however caveats associated with adopting thenotion of equilibrium as a solution concept.

Original: The equilibrium results rely on the assumption thatparticipants are fully rational Bayesian agents.
Revised: The equilibrium results rely on the assumption thatparticipants are fully rational Bayesian agents.

Original: Equilibrium is a static notion and doesn’t address how agents, who act independently, jump to play their equilibrium strategies.
Revised: Equilibrium is a static notion and doesn’t address how agents, who act independently, jump to play their equilibrium strategies.

Original: Moreover, the equilibriumresults of multi-task peer prediction mechanisms heavily depend on a consistent strategy assumption,that is, each agent is assumed to adopt a fixed strategy across all tasks that she participates.
Revised: Moreover, the equilibriumresults of multi-task peer prediction mechanisms heavily depend on a consistent strategy assumption,that is, each agent is assumed to adopt a fixed strategy across all tasks that she participates.

Original: Alltogether these assumptions exclude the possibility that agents may explore and learn from previousexperience, a behavior that’s not only commonly observed in practice but also has been modeled instudying other strategic settings [3, 6, 4].
Revised: Alltogether these assumptions exclude the possibility that agents may explore and learn from previousexperience, a behavior that’s not only commonly observed in practice but also has been modeled instudying other strategic settings [3, 6, 4].

Original: This paper is the first theoretical study on the dynamics of sequential peer prediction mechanismswhen participants are learning agents.
Revised: This paper is the first theoretical study on the dynamics of sequential peer prediction mechanismswhen participants are learning agents.

Original: The main question that we explore is whether and when insequential peer prediction, learning agents will converge to all playing the truthful reporting strategy.
Revised: The main question that we explore is whether and when insequential peer prediction, learning agents will converge to all playing the truthful reporting strategy.

Original: We first consider agents adopting no-regret learning algorithms and prove that the notion of noregret alone cannot guarantee convergence to truthful reporting.
Revised: We first consider agents adopting no-regret learning algorithms and prove that the notion of noregret alone cannot guarantee convergence to truthful reporting.

Original: We then define a natural family ofreward-based learning algorithms where strategy updates only depend on agents’ cumulative rewards.
Revised: We then define a natural family ofreward-based learning algorithms where strategy updates only depend on agents’ cumulative rewards.

Original: While algorithms in this family is not necessarily no-regret (e.g. the Follow the Leader algorithm),this family includes some familiar no-regret learning algorithms, including the Multiplicative Weight
Revised: While algorithms in this family is not necessarily no-regret (e.g. the Follow the Leader algorithm),this family includes some familiar no-regret learning algorithms, including the Multiplicative Weight

Original: Update and the Follow the Perturbed Leader algorithms.
Revised: Update and the Follow the Perturbed Leader algorithms.

Original: Our main result shows that, for the binarysignal setting, agents’ strategies in the popular Correlated Agreement (CA) mechanism [5] convergeto truthful reporting when agents use any algorithm from this family.
Revised: Our main result shows that, for the binarysignal setting, agents’ strategies in the popular Correlated Agreement (CA) mechanism [5] convergeto truthful reporting when agents use any algorithm from this family.

Original: To prove theresult, we showthe process has a self-fulfilling property: once Alice and Bob have large accumulated rewards fortruth-telling, they are more likely to play truth-telling and resulting in larger accumulated rewards.
Revised: We track the process usingthe agents’ accumulated rewards in the CA mechanism, and discover the process has a self-fulfillingproperty: once Alice and Bob have large accumulated rewards for truth-telling, they are more likelyto play truth-telling and resulting in larger accumulated rewards.

Original: Theoretically, we carefully partition the process into three stages, bad, intermediate, and good eventsillustrated in fig.
Revised: Theoretically, we carefully partitionthe process into three stages, bad, intermediate, and good events illustrated in fig.

Original: 1, and use tools in martingale theory to argue the progress of the process.
Revised: 1, and use tools inmartingale theory to argue the progress of the process.

Original: Finally, wesimulate the strategy dynamics in the CA mechanism for several algorithms in this family as well asfor the ϵ -greedy algorithm, which doesn’t belong to this family.
Revised: Finally, we simulate the strategy dynamicsin the CA mechanism for several algorithms in this family as well as for the ϵ -greedy algorithm,which doesn’t belong to this family.

Original: We observe convergence to truthfulreporting for all algorithms considered in our simulation, suggesting an interesting future direction tocharacterize all learning algorithms that converge to truthful reporting.
Revised: We observe convergence to truthful reporting for all algorithmsconsidered in our simulation, suggesting an interesting future direction to characterize all learningalgorithms that converge to truthful reporting.

Original:  Related Works
Revised: 1.1 Related Works

Original: This paper relates to two lines of work, information elicitation and mechanisms forlearning agents.
Revised: This paper relates to two lines of work, information elicitation and mechanisms for learning agents.

Original: Information Elicitation Mechanisms The literature on information elicitation without verificationfocuses on capturing the strategic aspect of human agents.
Revised: Information Elicitation Mechanisms The literature on information elicitation without verificationfocuses on capturing the strategic aspect of human agents.

Original: In multi-task settings, Dasgupta and Ghosh[5] proposed a seminal informed truthful mechanism, the Correlated Agreement (CA) mechanism,for binary positively correlated signals.
Revised: In multi-task settings, Dasgupta and Ghosh[5] proposed a seminal informed truthful mechanism, the Correlated Agreement (CA) mechanism,for binary positively correlated signals.

Original: A series of works then relaxed the binary and positivelycorrelation assumptions [25, 17, 23, 15].
Revised: A series of works then relaxed the binary and positivelycorrelation assumptions [24, 16, 22, 14].

Original: Additionally, Zheng et al.
Revised: Additionally, Zheng et al.

Original: [28] study the limitation ofinformation elicitation in the multi-task setting.
Revised: [27] study the limitation ofinformation elicitation in the multi-task setting.

Original: However, all of the above works assume agents usingconsistent strategies that are identical across all tasks.
Revised: However, all of the above works assume agents usingconsistent strategies that are identical across all tasks.

Original: The consistent strategy assumption excludesthe possibility of agent learning.
Revised: The consistent strategy assumption excludesthe possibility of agent learning.

Original: Our work removes the consistent strategy assumption and explicitlyconsiders learning agents.
Revised: Our work removes the consistent strategy assumption and explicitlyconsiders learning agents.

Original: We theoretically prove truthful convergence of the CA mechanism whenagents using algorithms from a family of reward-based online learning algorithms.
Revised: We theoretically prove truthful convergence of the CA mechanism whenagents using algorithms from a family of reward-based online learning algorithms.

Original: Our work of considering learning agents can be viewed as a way of testing the robustness ofinformation elicitation mechanisms with respect to deviation from the rational Bayesian agent model.
Revised: Our work of considering learning agents can be viewed as a way of testing the robustness ofinformation elicitation mechanisms with respect to deviation from the rational Bayesian agent model.

Original: From this perspective, Shnayder et al.
Revised: From this perspective, Shnayder et al.

Original: [26] is closely related to ours.
Revised: [25] is closely related to ours.

Original: They consider sequentialinformation elicitation and empirically study if agents using replicator dynamics can converge totruth-telling in the CA mechanism and several other mechanisms.
Revised: They consider sequentialinformation elicitation and empirically study if agents using replicator dynamics can converge totruth-telling in the CA mechanism and several other mechanisms.

Original: Our work theoretically proves that,besides replicator dynamics, learning agents can converge to truth-telling in the CA mechanism whenthey use a general family of learning algorithms.
Revised: Our work theoretically proves that,besides replicator dynamics, learning agents can converge to truth-telling in the CA mechanism whenthey use a general family of learning algorithms.

Original: Additionally, Schoenebeck et al.
Revised: Additionally, Schoenebeck et al.

Original: [24] designed aninformation elicitation mechanism that was robust against a small fraction of adversarial agents.
Revised: [23] designed aninformation elicitation mechanism that was robust against a small fraction of adversarial agents.

Original: Mechanisms for Learning Agents Several works in economics and computer science try to designmechanisms for learning agents, rather than for rational, Bayesian ones.
Revised: Mechanisms for Learning Agents Several works in economics and computer science try to designmechanisms for learning agents, rather than for rational, Bayesian ones.

Original: Braverman et al.
Revised: Braverman et al.

Original: [3] studiedpricing mechanisms for learning agents with no external regrets.
Revised: [3] studiedpricing mechanisms for learning agents with no external regrets.

Original: Their work was generalized by
Revised: Their work was generalized by

Original: Deng et al.
Revised: Deng et al.

Original: [6] to consider repeated Stackelberg games in full-information settings.
Revised: [6] to consider repeated Stackelberg games in full-information settings.

Original: Camara et al.
Revised: Camara et al.

Original: [4]further proposed counterfactual internal regrets (CIR) together with no-CIR assumption, which wasproved to be a sufficient behavior assumption for no-regret principal mechanism design in repeatedstage games.
Revised: [4]further proposed counterfactual internal regrets (CIR) together with no-CIR assumption, which wasproved to be a sufficient behavior assumption for no-regret principal mechanism design in repeatedstage games.

Original: However, all of these works focus on a single agent or full-information games, whilepeer prediction is an incomplete-information game with multiple agents.
Revised: However, all of these works focus on a single agent or full-information games, whilepeer prediction is an incomplete-information game with multiple agents.

Original: Finally, our goal is slightlydifferent from that of most sequential mechanism design.
Revised: Finally, our goal is slightly different from that of most sequential mechanism design.

Original: Instead of maximizing the mechanismdesigner’s utility, our goal is to incentivize truthful reporting from agents.
Revised: Instead of maximizing the mechanismdesigner’s utility, our goal is to incentivize truthful reporting form agents.

Original:   Peer Prediction Settings
Revised:  Sequential Peer Prediction Settings

Original: For simplicity we consider two agents, Alice and Bob, who work on a sequence of tasks indexedby t ≥ 1 .
Revised: For simplicity we consider two agents, Alice and Bob, who work on a sequence of tasks indexedby t ≥ 1 .

Original: For round t , both agents work on task t , Alice receives a signal X t = x t in { 0 , 1 } ,and Bob a signal Y t =
Revised: For round t , both agents work on task t , Alice receives a signal X t = x t in { 0 , 1 } , and Bob a signal Y t =

Original: y t in { 0 , 1 } where X t and Y t denote random variables, and x t and y tare their realizations.
Revised: y t in { 0 , 1 } .

Original: Then Alice and Bob report ˆ X t = ˆ x t and ˆ Y t = ˆ y t in { 0 , 1 } .
Revised: Then Alice and Bob report ˆ X t = ˆ x t and ˆ Y t = ˆ y t in { 0 , 1 } .

Original: We define
Revised: Wedefine X ≤ t =

Original: X ≤ t = { X s : 1 ≤ s ≤ t } ∈ { 0 , 1 } t and ˆ X ≤ t = { ˆ X s : 1 ≤ s ≤ t } ∈ { 0 , 1 } t to denote Alice’ssignal profile and report profiles until t -th round respectively, and define Y ≤ t and ˆ Y ≤ t for Bobsimilarly.
Revised: { X s : 1 ≤ s ≤ t } ∈ { 0 , 1 } t and ˆ X ≤ t = { ˆ X s : 1 ≤ s ≤ t } ∈ { 0 , 1 } t to denote Alice’s signal profile and report profiles until t -th round respectively, and define Y ≤ t and ˆ Y ≤ t for Bob similarly.

Original: We use X , ˆ X , Y , and ˆ Y for the complete signal and report profiles.
Revised: We use X , ˆ X , Y , and ˆ Y for the complete signal and report profiles.

Original: Additionally, weconsider the signals are generated from some distribution P that satisfies the following assumptions:
Revised: Additionally, weconsider the signals are generated from some distribution P that satisfies the following assumptions:

Original: Assumption 2.1 (A priori similar tasks [5]) .
Revised: Assumption 2.1 (A priori similar tasks [5]) .

Original: Each pair of signal is identically and independently(i.i.d.)
Revised: Each pair of signal is identically and independently(i.i.d.)

Original: generated: there exists a distribution P X,Y over { 0 , 1 } 2 such that ( X t , Y t ) ∼ P X,Y for anyt ∈ N + .
Revised: generated: there exists a distribution P X,Y over { 0 , 1 } 2 such that ( X t , Y t ) ∼ P X,Y for anyt ∈ N + .

Original: Moreover, we assume the distribution has full support, P X,Y ( x, y ) > 0 for all x, y ∈ { 0 , 1 } .
Revised: Moreover, we assume the distribution has full supports, P X,Y ( x, y ) > 0 for all x, y ∈ { 0 , 1 } .

Original: Assumption 2.2 (Positively correlated signals) .
Revised: Assumption 2.2 (Positively correlated signals) .

Original: The distribution P X,Y is positively correlated,min { P X,Y (1 , 1) , P X,Y (0 , 0) }
Revised: The distribution P X,Y is positively correlated,min { P X,Y (1 , 1) , P X,Y (0 , 0) }

Original: > max { P X,Y (1 , 0) , P X,Y (0 , 1) } .
Revised: > max { P X,Y (1 , 0) , P X,Y (0 , 1) } .

Original: Now we introduce multi-task peer prediction mechanisms and sequential peer prediction mechanisms, and their relation.

Original: We will focus on the sequential setting.

Original: Multi-task peer predictionmechanisms work on a fixed number of tasks.

Original: Formally, a multi-task peer prediction mechanism on k tasks is a pair of payment functions ¯ M : { 0 , 1 } k

Original: → [0 , 1] 2 .

Original: For instance, the(multi-task) correlated agreement mechanism (CA mechanism) 2 [5, 25, 26] is ¯ M CA

Original: (ˆ x , ˆ y ) =(

Original: I [ˆ x 2 = ˆ y 2 ] − I [ˆ x 2 =

Original: [-ˆ y 1 ] , I [ˆ y 
-]
Revised: [-ˆ y 1 ] , I [ˆ y 
-]

Original: =

Original: ˆ x 2 ] − I [ˆ y 2 =

Original: ˆ x 1 ]) for all ˆ x , ˆ y ∈ { 0 , 1 } 2 .

Original: Intuitively, the CA mechanism rewards agreement on the same task and punishes agreement on uncorrelated tasks.

Original: A sequential information elicitation mechanism is a sequence of payment functions M =
Revised: A sequential information elicitation mechanism is a sequence of payment functions M =

Original: { M t :t ≥ 1 } where M t : { 0 , 1 } 2 × t → [ − 1 , 1] for all t .
Revised: { M t :t ≥ 1 } where M t : { 0 , 1 } 2 × t → [ − 1 , 1] for all t .

Original: After Alice and Bob reporting ˆ x ≤ t and ˆ y ≤ tin round t , the mechanism computes ( r t , s t ) :
Revised: After Alice and Bob reporting ˆ x ≤ t and ˆ y ≤ tin round t , the mechanism computes ( r t , s t ) :

Original: = M t (ˆ x ≤ t , ˆ y ≤ t ) and pay r t to Alice and s t to Bob.
Revised: = M t (ˆ x ≤ t , ˆ y ≤ t ) and pay r t to Alice and s t to Bob.

Original: Here we assume M t can only depends on a constant k round of reports so that M t (ˆ x ≤ t , ˆ y ≤ t ) =
Revised: Here we assume M t can only depends on a constant k round of reports so that M t (ˆ x ≤ t , ˆ y ≤ t ) =

Original: M t (ˆ x t − k +1 , ˆ x t − k +1 , . . .
Revised: M t (ˆ x t − k +1 , ˆ x t − k +1 , . . .

Original: , ˆ x t , ˆ y t − k +1 , ˆ y t − k +1 , . . .
Revised: , ˆ x t , ˆ y t − k +1 , ˆ y t − k +1 , . . .

Original: , ˆ y t ) for all t , ˆ x ≤ t , and ˆ y ≤ t , and we call such M 119 rank k mechanism .
Revised: , ˆ y t ) for all t , ˆ x ≤ t , and ˆ y ≤ t , and we call such M 112 rank k mechanism .

Original: For instance, the (multi-task) CA mechanism can be adopted as a sequential rank2 information elicitation mechanism: At round t , the payment is
Revised: For instance, previous work [5, 24, 25] study correlated agreement mechanism(CA mechanism) 1 that can be adopted as a sequential rank 2 information elicitation mechanism: Atround t , the payment is

Original: [Equation]
Revised: [Equation]

Original: where ˆ x 0 and ˆ y 0 are set as 0 .
Revised: We let ˆ x 0 and ˆ y 0 to be both 0 for the first round.

Original: Similarly, we say a sequential information elicitation mechanism M = ( M t ) t ≥ 1 is a sequential version of a multi-task information elicitation mechanism ¯ M if M t (ˆ x ≤ t , ˆ y ≤ t ) = ¯ M (ˆ x t − k
Revised: Note that multi-task information elicitationmechanisms on T task is a payment function that maps agents’ reports on all T tasks to payment, e.g.,the payment of CA mechanism is M T when the number of tasks is T .

Original: +1 , ˆ x t − k +1 , . .

Original: , ˆ x t , ˆ y t − k +1 , ˆ y t − k +1 , . . .

Original: , ˆ y t ) for all ˆ x ≤ t , ˆ y ≤ t and t ≥ k .

Original: The payment at round t ≥ k is ¯ M on the latest k reports.

Original: Conversely, a sequential informationelicitation mechanism can be seen as a sequence of multi-task information elicitation mechanisms.
Revised: Thus, multi-task informationelicitation mechanisms can be seen as a special case of the sequential one.

Original: Now we formally define agents’  strategies.
Revised: Now we formally define agents’ actions and strategies.

Original: Due to symmetry, we introduce notation for Alice andomit Bob’s.
Revised: Due to symmetry, we introduce notationfor Alice and omit Bob’s.

Original: Given an information elicitation mechanism M , at round t , Alice  observes her signalx t and decides on her report ˆ x t .
Revised: Given an information elicitation mechanism M , at round t , Alice canchoose an action observes her signal x t and reports ˆ x t .

Original: Thus, Alice has four options (pure strategies) : 1) opt 1 : report theprivate signal truthfully, 2) opt 2 : flip the private signal, 3) opt 3 : report 1 regardless of the signal, and4) opt 4 : report 0 regardless of the signal.
Revised: Thus, Alice has four options (actions) : 1)opt 1 : report the private signal truthfully, 2) opt 2 : flip the private signal, 3) opt 3 : report 1 regardlessof the signal, and 4) opt 4 : report 0 regardless of the signal.

Original: We call opt 3 and opt 4 uninformative strategies.
Revised: We call opt 3 and opt 4 as uninformativeactions.

Original: We useopt Xt to denote Alice’s pure strategy, and r t for her payoff at round t .
Revised: We use opt Xt to denote Alice’s actions, and r t for their payoff at round r .

Original: At each round t , Alice knowsher previous signals x ≤ t ∈ { 0 , 1 } t , her pure strategies opt X 1 , . .
Revised: At each roundt , Alice knows her previous signals x ≤ t ∈ { 0 , 1 } t , her actions opt X 1 , . .

Original: . , opt Xt − 1 , and Bob’s reports ˆ y ≤ t − 1 ,so we use F t = { x ≤ t , opt X ≤ t − 1 , ˆ y ≤ t − 1 } to denote Alice knowledge at round t .
Revised: opt Xt − 1 , and Bob’s reportsˆ y ≤ t − 1 , so we use F t = { x ≤ t , opt X ≤ t − 1 , ˆ y ≤ t − 1 } ⊂ { 0 , 1 } t × { opt 1 , opt 2 , opt 3 , opt 4 } t − 1 × { 0 , 1 } t − 1to denote Alice knowledge at round t .

Original: Thus, Alice’s mixedstrategy at round t is a stochastic mapping σ  Xt from F t to { opt 1 , opt 2 , opt 3 , opt 4 } . We’ll abuse ournation andalso use σ Xt = opt Xt to represent the realized pure strategy. Finally, a learning algorithmof Alice is a mapping from an information elicitation mechanism M to her strategies.
Revised: Thus, Alice’s strategy at round t is a stochastic mapping σ Xtso that σ Xt ( F t ) ∈ { opt 1 , opt 2 , opt 3 , opt 4 } , and a learning algorithm of Alice is a mapping from aninformation elicitation mechanism M to her strategy.

Original: Strongly Truthful for Rational and Bayesian agents Previous works on  information elicitationtry to ensure truth-telling opt 1 is the best strategy for rational and strategic agents.
Revised: Previous works on multi-task information elicitation try to ensure truth-telling opt 1 is the best actionfor rational and strategic agents.

Original: In particular, amechanism is strongly truthful if Alice and Bob report truthfully is a Bayesian Nash Equilibrium(BNE) and they get strictly higher paymentat this BNE thanat any other non-permutation BNE.
Revised: In particular, a mechanism is strongly truthful if Alice and Bob getstrictly higher payment than any other non-permutation Bayesian Nash equilibrium.

Original: We present the formal definitions in the appendix.
Revised: We present theformal definitions in the appendix.

Original: Informally, in a permutation BNE, every agent’sstrategy on each round is a permutation/bijection from his/her signals to reports.

Original: However,theequilibrium results of previous mechanisms not only require assumption 2.1 but further assumeagents using consistent strategies .
Revised: However, previous mechanisms not only require assumption 2.1but further assume that agents use consistent strategy .

Original: Specifically, Alice uses a consistent strategy if there is a fixeddistribution on { opt 1 , opt 2 , opt 3 , opt 4 } so that opt Xt is generated from a fixed distribution that isindependent of her private signals on other tasks and the round number.
Revised: Specifically, Alice uses a consistent strategy ifthere is a fixed distribution µ X on { opt 1 , opt 2 , opt 3 , opt 4 } so that her action at round t is generatedfrom µ X that is independent of her private signals on other tasks.

Original: For instance, when Alice and Bob are Bayesian and use consistent strategies under assumption 2.2 and 2.1, Dasgupta and Ghosh[5] show CA mechanism in eq.
Revised: For instance, when Alice and Bob are Bayesian and use consistent strategies under assumption 2.2 and 2.1, Dasgupta and Ghosh [5]show CA mechanism in eq.

Original: (1) is strongly truthful.
Revised: (1) is strongly truthful.

Original: Intuitively, positive correlation assumption 2.2guarantees that truthful reporting can maximize the chance of agreeing with the peer on the sametask while avoiding agreeing on reports on other tasks.
Revised: Intuitively, positive correlation assumption 2.2guarantees that truthful reporting can maximize the chance of agreeing with the peer on the sametask while avoiding agreeing on reports on other tasks.

Original: Furthermore, in appendix B we show CAmechanism merely has threetypes Bayesian Nash equilibria, at which both agents 1) play truth-tellingopt 1 , 2) flip the signal opt 2 , or 3)generate uninformative reports (mixture between opt 3 , opt 4 ) whenagents use consistent strategies and assumptions 2.1 and 2.2 hold.
Revised: Furthermore, in appendix B we show CAmechanism merely has three Bayesian Nash equilibria which are 1) truth-telling opt 1 , 2) flip thesignal opt 2 , or 3) uninformative reports (mixture between opt 3 , opt 4 ).

Original:  Truthful convergence for learning agents
Revised: 2.2 Truthful convergence for learning agents

Original: However, as we consider agents using a family ofonline learningalgorithms to decide their strategies, standard solution concepts like Bayesian Nashequilibrium no longer apply.
Revised: However, as we consider agents use a family of online learning to update their actions, standardsolution concepts like Bayesian Nash equilibrium no longer apply.

Original: Additionally, online learning algorithms often have exploration, so wecannot hope agentswill always use  the truth-telling strategy.
Revised: Additionally, online linearalgorithms often conduct exploration, so we cannot hope agents always use telling the truth.

Original: For learning agents, our goal is to testwhether existing mechanisms can ensure that agents will converge to truthful reporting when theydeploy certain learning behavior that goes beyond obliviously consistent strategies.
Revised: Forlearning agents, our goal is to design mechanism M so that agents will converge to truthful reportingwhen they deploy certain learning behavior that goes beyond obliviously consistent strategies.

Original: We now formalize the convergence of algorithms to truthful reporting.
Revised: We now formalize the convergence of algorithms to truthful reporting.

Original: Because we want to elicitinformation without verification, it is information-theoretically impossible for us to separate permutation equilibrium, where all agents play opt 2 , from truthful equilibrium, where all agents playopt 1 , without any additional information [17].
Revised: Because we want to elicitinformation without verification, it is information-theoretically impossible for us to separate permutation equilibrium from truthful equilibrium without any additional information[16].

Original: However, if we have an additional bit of informationon whether the prior of 0 is larger than 1 , we may tell apart these two equilibria.

Original: We hence defineconvergence to  truthful reporting as the limits ofboth opt Xt and opt Yt being truth-telling (opt 1 ) orflipping (opt 2 ).Note that definition 2.3 requires almost surely convergence which is very strongconvergence concept.
Revised: Analogous to thestrongly truthful guarantee, we ask if the limit of opt Xt and opt Yt can be both truth-telling (opt 1 ) orthe same permutation (opt 2 ).

Original: Definition 2.3.
Revised: Definition 2.3.

Original: An information elicitation mechanism M achieves truthful convergence for agents using algorithms A 1 and A 2respectively if and only if both sequences of pure strategies converge to  truth-telling or both flipping the reports.
Revised: An information elicitation mechanism M achieves truthful convergence for agents using algorithms A 1 , A 2 if and only if the sequence of actions converges to both truth-telling or both flipping the reports.

Original: [Equation]
Revised: [Equation]

Revised: Note that definition 2.3 requires almost surely convergence which is very strong convergence result.

Revised: Additionally, by using a single bit of information, e.g., whether the prior of 0 is larger than 1 , wecan remove the permutation and convert the agents’ report back to their private signal regardless ofwhether they using the truth-telling opt 1 or flipping action opt 2 .

Original:  Online Learning Algorithms
Revised:  Online Learning Algorithms

Original: In this section, we explore candidates to model agents’ learning behavior to replace Bayesian agents’consistent strategies in the literature.
Revised: In this section, we explore candidates to model agents’ learning behavior to replace Bayesian agents’oblivious consistent strategies in the literature.

Original: We first show the conventional no-regret assumption isanecessary but not a sufficient condition for truthful convergence in section 3.1.
Revised: We first show the conventional no-regret assumption is not enough in section 3.1.

Revised: In particular,theorem 3.1 shows that for any sequential information elicitation mechanism with bounded rank,there exist no regret agents that collide at the beginning of the game and decide a sequence ofuninformative actions.

Original: Then in section 3.2,we introduce a family of reward-based online learning algorithm to model agents’ learning behavior,and show that  the family of reward-based online learning algorithms contains several commonno-regret algorithms as special cases.
Revised: Then in section 3.2, we introduce our reward-based online learning algorithm to model agents’learning behavior. Informally, a reward-based online learning algorithm uses the accumulated rewardto decide each round’s action using a fixed update function that is closely related to mean-basedalgorithms by [3]. We show the family of reward-based online learning algorithms contains severalcommon no regret algorithms as special cases in theorem 3.5.

Original: 3.1 No-regret online learning algorithms
Revised: 3.1 No regret online learning algorithms

Original: We now investigate the relationship between no regret and truthful convergence. First, we showgeneral no-regret algorithms may not ensure truthful convergence (theorem 3.1). However, we showthe converse is almost true (theorem 3.2): If truthful convergence happens, the agents do not haveregret when the sequential mechanism is a sequential version of a strongly truthful mechanism.
Revised: One natural family of algorithms for Alice and Bob to use are no regret algorithms.

Original: Given a sequential information elicitation mechanism M= ( M Xt , M Yt ) t ≥ 1 , signals x , y , and reportsˆ x , ˆ y , we define r i,t = M Xt ( opt i ( x 1 ) , . .
Revised: Given a sequentialinformation elicitation mechanism M , we define r i,t be the payoff when Alice uses action opt i and

Original: , opt i ( x t ) , ˆ y ≤ t ) be the payoff when Alice uses strategyopt i and Bob’s choices are unchanged.
Revised: Bob’s choices are unchanged.

Original: Then Alice’s regret is Reg X ( T )=
Revised: Then Alice’s regret is Reg X ( T )

Original: max i (cid:80) t ≤ T r i,t −  (cid:80) t ≤ T r t . Finally, we say that Alice’s and Bob’s online learning algorithms are no regret(on M ) if E [ Reg X ( T )]
Revised: = (cid:80) t ≤ T r t − max i (cid:80) t ≤ T r i,t , andwe say her online learning algorithm is no regret if E [ Reg X ( T )]

Original: = E [ Reg Y ( T )]
Revised: = o ( T ) .

Original: = o ( T ) over the randomness of signals and the algorithms, and we say Alice and Bob are no regret for short. One may hope that no regret as a behavior assumption for agents is sufficient for achieving desirableoutcome in a mechanism.
Revised: In previous literature, one may use some no-regret as the behavior assumption for agents and showthat a mechanism can achieve good results for agents ruled by this behavior assumption.

Original: However, the following theorem shows that we cannot have an informationelicitation mechanism that achieves truthful convergence for all no-regret agents.
Revised: However,the following theorem shows that we cannot have an information elicitation mechanism that achievestruthful convergence for no regret agents.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any sequential information elicitation mechanism Mof rank k ∈ N , there existno-regret algorithms for Alice and Bob so that M cannot achieve truthful convergence.
Revised: For any rank k information elicitation mechanism M , there exist no-regret algorithmsfor Alice and Bob so that M does not have truthful convergence on the algorithms.

Original: The main idea of the proof is that the no-regret assumption cannot prevent Alice and Bob fromcolluding.
Revised: The main idea of the proof is that no regret assumption cannot prevent Alice and Bob from colluding.

Original: In our counterexample, Alice and Bob decide on a no-regret sequence of reports regardlessof their signals once the mechanism is announced.
Revised: In our counterexample, Alice and Bob decide on a no-regret sequence of reports regardless of theirsignals once the mechanism is announced.

Revised: Standard multi-task information elicitation settings excludesuch collusion by requiring agents to use consistent strategy.

Original: Technically, we use probabilistic method to showthe existence of a deterministic and no-regret sequence of strategies ( opt Xt , opt Yt ) t ≥ 1 that consists ofreporting 1 or 0 regardless of private signal, i.e. opt 3 or opt 4 .
Revised: Technically, we use probabilistic method to show the existence of a deterministic and no regretsequence of actions ( opt Xt , opt Yt ) t ≥ 1 that consists of reporting 1 and 0 regardless of private signalopt 3 , opt 4 .

Original: The formal proof is in appendix C.1.
Revised: The formal proof is in appendix C.

Original: The notion of truthful convergence in definition 2.3 provides an ideal truthful guarantee to themechanism designer.

Original: Here we show that truthful convergence also ensures no regret for agents whenthe sequential mechanism is a sequential version of a strongly truthful one-shot multi-task mechanism.

Original: That is, when the one-shot mechanism admits truthful reporting as a highest-payoff BNE.

Original: For instance, if a pair of algorithms exhibits truthful convergence on the sequential CA mechanism(eq.

Original: (1)), they are also no regret (on the game).

Original: Intuitively, if Bob converges to the truth-tellinglim t →∞ opt Yt = opt 1 , the average expected gain of Alice deviating to opt i is equal to the expectedgain of deviating to opt i when Bob always tells the truth.

Original: The gain is non positive because the

Original: CA mechanism is strictly truthful by lemma B.4.Therefore, the expected regrets E [ Reg X ( T )] and 205 E [ Reg Y ( T )] are small.

Original: Theorem 3.2 formalizes and extends the above idea to any strongly truthfulmulti-task information elicitation mechanism.

Original: Theorem 3.2.

Original: Let ¯ M be a strongly truthful multi-task information elicitation mechanism, and M bea sequential version of ¯ M .

Original: If M achieve truthful convergence for Alice and Bob using algorithms

Original: A 1and A 2 respectively, then Alice and Bob are no regret.

Original: 3.2 Reward-based Online Learning Algorithms
Revised: 3.2 Reward-based Online Learning Algorithms

Original: As shown in theorem 3.1, the no-regret assumption allow does not guarantee truthful convergence.
Revised: As we show in theorem 3.1, no regret assumption is not enough to tame agents’ unrealistic behavior,and they may collude in sequential information elicitation mechanisms.

Original: In this section, we introduce a general family of online learning algorithms, reward-based onlinelearning algorithm , under the general full feedback bandit setting, and we will apply these algorithmsin sequential information elicitation mechanisms later.. Informally, a reward-based online learningalgorithm decides each round’s strategy using a fixed update function that depends only on theaccumulative reward.
Revised: In this paper, we introducea general family of online learning algorithms that include several popular no regret algorithms asspecial cases (theorem 3.5). In this section, we introduce our learning algorithms under the general full feedback bandit setting,and we will apply these algorithms in sequential information elicitation mechanisms later.

Original: For simplicity, we only consider algorithms on four strategies { opt 1 , opt 2 , opt 3 , opt 4 } which can beextended easily.
Revised: Forsimplicity, we introduce a family A of learning algorithms on four actions { opt 1 , opt 2 , opt 3 , opt 4 }which can be extended easily.

Original: Recall that the payoff of choosing option opt i , i ∈ [4] at round t is r i,j when others’choices are unchanged.
Revised: Recall that the payoff of choosing option opt i , i ∈ [4] at round t is r i,jwhen others’ choices are unchanged.

Original: We denote the accumulated payoffs of these four options as R i,t = (cid:80) t j =1 r i,jfor i ∈ [4] .
Revised: We denote the accumulated payoffs of these four options as R i,t = (cid:80) tj =1 r i,j for i ∈ [4] .

Original: Symmetrically, we use S i,t and s i,t to represent accumulated payoffs and the payoffof turning to choose option opt i in the t th round for Bob.
Revised: Symmetrically, we use S i,t and s i,t to represent accumulated payoffsand the payoff of turning to choose option opt i in the t th round for Bob.

Original: For example, for our specific peerprediction game using CA mechanism in eq.
Revised: For example, for our specificpeer prediction game using CA mechanism in eq.

Original: (1), r 1 ,t
Revised: (1), r 1 ,t

Original: = I
Revised: = I

Original: [ x t = ˆ y t ] − I [ x t = ˆ y t − 1 ] and therefore,
Revised: [ x t = ˆ y t ] − I [ x t = ˆ y t − 1 ] and therefore,

Original: R 1 ,t = (cid:80) tj =1 ( I [ x t = ˆ y t ] − I [ x t = ˆ y t − 1 ]) for Alice.
Revised: R 1 ,t = (cid:80) tj =1 ( I [ x t = ˆ y t ] − I [ x t = ˆ y t − 1 ]) for Alice.

Original: We consider a family of reward-based online learning algorithms
Revised: We consider a family of reward-based online learning algorithms

Original: A that use an update functionf : R 4 → △ 3 , and choose opt i with probability f i ( R 1 ,t − 1 , R  2,t − 1 , R  3,t − 1 , R  4,t − 1 ) for i ∈ [4] in thet th round, where f i is the i th coordinate of f .
Revised: A that use a update functionf : R 4 → △ 3 , and choose opt i with probability f i ( R t 1 , R t 2 , R t 3 , R t 4 ) for i ∈ [4] , where f i is the i thcoordinate of f .

Original: A such mechanism based on accumulated payoffs isdenoted by A f .
Revised: A such mechanism based on accumulated payoffs is denoted by A f .

Original: We have three assumptions for the update function f , which are all very natural.
Revised: We have threeassumptions for the update function f , which are all very natural.

Original: Thefirst two require that f is exchangeable and preserves ordering.
Revised: The first two require that f isexchangeable 2 and preserves ordering.

Original: Assumption 3.3 (Exchangeability of f ) .
Revised: Assumption 3.2 (Exchangeability of f ) .

Original: For any R 1 , R 2 , R 3 , R 4 ∈ R and an arbitrary permutationof them
Revised: For any R 1 , R 2 , R 3 , R 4 ∈ R and an arbitrary permutationof them

Original: R i 1 , R i 2 , R i 3 , R i 4 ,  f i j ( R 1 , R 2 , R 3 , R 4 ) =
Revised: R i 1 , R i 2 , R i 3 , R i 4 , it holds for f that

Original: f j ( R i 1 , R i 2 , R i 3 , R i 4 ) for all j ∈ [4] .
Revised: [Equation]

Original: Assumption 3.4 (Order preservation of f ) .
Revised: Assumption 3.3 (Order preservation of f ) .

Original: For any R 1 , R 2 , R 3 , R 4 ∈ R and suppose that
Revised: For any R 1 , R 2 , R 3 , R 4 ∈ R and suppose that

Original: R i 1 , R i 2 , R i 3 , R i 4 is a non-increasing order of them, for f we have [-f i 
-]
Revised: R i 1 , R i 2 , R i 3 , R i 4 is a non-increasing order of them, for f we have [-f i 
-][Equation] R

Revised: [Equation]

Original: ( R 1 , R 2 , R 3 , R 4 ) ≥ 233 f i 2 ( R 1 , R 2 , R 3 , R 4 ) ≥ f i 3 ( R 1 , R2 , R 3 , R 4 ) ≥ f
Revised: R 2 , R 3 , R [Equation] [Equation] R

Original: i 4 ( R 1 , R 2 , R 3 , R 4 ) .
Revised: [Equation] , R 2 , R 3 , R [Equation] ( R 1 , R

Original:  Finally we consider thestrategy chosen by the update function f  when the accumulated payoff ofan strategy is much higher thanthat of other strategies (assumption 3.5).
Revised: [Equation] , R 3 , R [Equation] Finally we consider the update function f will choose an action when the accumulated payoff ismuch higher than other actions’ accumulated payoffs.

Original: Appendix D.2 shows thatthe assumption is necessary for reward-based online learning algorithms to achieve no regret for anyonline decision problem.

Original: Assumption 3.5 (Full exploitation of f ) 
Revised: Assumption 3.4 (Full exploitation of f ) .

Original: [Equation]
Revised: [Equation]

Revised: In Appendix D.2, we prove that assumption 3.4 is a necessary condition for reward-based onlinelearning algorithms to achieve no regret for any online decision problem.

Original: Now we showthat the family of reward-based online learning algorithms A  satisfying assumptions 3.3to 3.5 contains several classic no-regret online learning algorithms [10, 13].
Revised: Now we show the family of reward-based online learning algorithms A that use f satisfying assumptions 3.2 to 3.4 contains several classical no regret online learning algorithms [10, 12].

Original: Theorem 3.6.
Revised: Theorem 3.5.

Original: A contains Follow the Perturbed Leader (FPL) algorithm,and Multiplicative Weightsalgorithm as special cases.
Revised: A contains FPL algorithm, Multiplicative Weights algorithm and replicator dynamicsas special cases.

Original: Corresponding f
Revised: Corresponding f

Original: ’s for them are listed as below:
Revised: ’s for them are listed as below:

Original: Multiplicative Weight algorithm
Revised: Multiplicative Weight algorithm

Original: f hedge 1 i

Original: [Equation]
Revised: [Equation]

Original: FPL algorithm
Revised: FPL algorithm

Original: Given a noise distribution N  on scalars, [Equation]
Revised: Given a noise distribution N X on scalars, f FPL i ( R 1 , R 2 , R 3 , R 4 ) =

Original: [Equation]
Revised: [Equation]

Original: In the binary peer prediction problem using CA mechanism, we can further show that replicatordynamics and linear updating multiplicative weight algorithm
Revised: If we focus on the binary peer prediction problem using CA mechanism, we can furthermore provethat replicator dynamics and linear updating multiplicative weight algorithm

Original: [2] are both in A in appendix E.
Revised: [2] are both in A inappendix E.

Original: Remark 3.7.
Revised: Remark 3.6.

Original: Our reward based online learning is very similar to the mean-based algorithm in [3] andboth use the accumulated rewards to characterize the algorithm’s choice.
Revised: Our reward based online learning is very similar to the mean-based algorithm in [3]and both use the accumulated rewards to characterize the algorithm’s choice.

Original: Additionally, like the meanbased algorithms, our reward based online algorithm may contain algorithms with regret in genralgame, e.g., follow the leader.
Revised: Additionally, like themean-based algorithms, our reward based online algorithm may contain algorithms with regret, e.g.,follow the leader.

Original: The family of reward based online algorithms uses an identical updatefunction across all rounds.
Revised: However, the reward based online algorithm uses the identical update function across all rounds.

Original: Thus some no-regret algorithms, e.g., ϵ -greedy with time decreasing ϵ ,doesn’t belong to family.
Revised: Thus some no regret algorithm are not a reward based online algorithm, e.g., ϵ -greedy with timedecreasing ϵ .

Original: Finally, if a mean-based algorithm uses the same update function in eachround, then the mean-based algorithm belongs to reward based algorithms. Additionally, then theupdate function needs to satisfy assumption 3.5.
Revised: Finally, if a mean-based algorithm uses the same update function in each round, thenthe update function needs to satisfy assumption 3.4.

Original:  Truthful Convergence of CA Mechanism on Reward-based Algorithms
Revised:  Truthful Convergence of CA Mechanism on Reward-based Algorithms

Original: Now we present our main result.
Revised: Now we present our main result.

Original: We will show thatthe sequential CA mechanism can achieve truthfulconvergence if both agents use reward-based online learning algorithmsfrom A .
Revised: We will show that CA mechanism can achieve truthful convergenceif both agents use reward-based online learning algorithms A .

Original: This convergenceresult suggests that the classical CA is robust even when agents deviate from Bayesian rationalbehavior and use a general family of online learning algorithms.
Revised: This convergence result suggests thatthe classical CA is robust even when agents deviate from Bayesian rational behavior and use a generalfamily of online learning algorithms.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Under assumptions 2.1 and 2.2, the binary-signal, sequential CA mechanismasdefined in eq.
Revised: For a binary signal repeated peer prediction game under assumptions 2.1 and 2.2 andupdate functions f and g satisfying assumptions 3.2 to 3.4, CA mechanism in eq.

Original: (1) achieves truthful convergence when agents  usereward-based algorithms
Revised: (1) achieves truthfulconvergence when agents that use algorithms

Original: A f and A g , where the update functions f
Revised: A f and A g that are in A .

Original: and g satisfy assumptions 3.3 to 3.5. Note that given agents’ online learning algorithm and the payment function, the sequence of accumulated reward vector ( R 1 ,t , R 2 ,t , R 3 ,t , R 4 ,t , S 1 ,t , S 2 ,t , S 3 ,t , S 4 ,t ) t ≥ 1 forms a stochastic process and 266 we define H t as the game history of the rewards and private signals in the first t round.
Revised: Note that given agents’ online learning algorithm and the payment function, the sequence of accumulated reward vector ( R 1 ,t , R 2 ,t , R 3 ,t , R 4 ,t , S 1 ,t , S 2 ,t , S 3 ,t , S 4 ,t ) t ≥ 1 forms a stochastic process and 244 we define H t as the game history of the rewards and private signals in the first t round.

Original: Additionally,if the accumulated reward of truth-telling R 1 ,t , S 1 ,t is much larger than the others’, we can show
Revised: Additionally,if the accumulated reward of truth-telling R 1 ,t , S 1 ,t is much larger than the others’, we can show

Original: Alice and Bob converge to the truth-telling by assumption 3.4.
Revised: Alice and Bob converge to the truth-telling by assumption 3.3.

Original: Thus, it is sufficient for us to trackthe evolution of accumulated reward vector.
Revised: Thus, it is sufficient for us to track theevolution of accumulated reward vector.

Original: Though the process of accumulated reward vector is nota Markov chain because the payment function eq.
Revised: Though the process of accumulated reward vector is not a Markov chain because the payment function eq.

Original: (1) depends on reports in two rounds, we can stilluse ideas from semi-martingale to track the process.
Revised: (1) depends on reports in two rounds, we can stilluse ideas from semi-martingale to track the process.

Original: Before proving our main result, we first present two properties of CA mechanism for our binarysignal peer prediction problem.
Revised: Before proving our main result, we firstly present two properties of CA mechanism for our binarysignal peer prediction problem.

Original:  Lemma 4.2 shows that the accumulated payoffs of uninformativestrategies opt 3 , opt 4 R 3 ,t , R 4 ,t , S 3 ,t , and S 4 ,t are always bounded.
Revised: First Lemma 4.2 shows that the accumulated payoffs of uninformativeactions opt 3 , opt 4 R 3 ,t , R 4 ,t , S 3 ,t , and S 4 ,t are always bounded.

Original: Lemma 4.2.
Revised: Lemma 4.2.

Original: Given the game defined in theorem 4.1, for any round t , the accumulated payoffs
Revised: Given the game defined in theorem 4.1, for any round t , the accumulated payoffs

Original: R 3 ,t , R 4 ,t , S 3 ,t , and S 4 ,t for two agents are bounded by [ − 1 , 1] .
Revised: R 3 ,t , R 4 ,t , S 3 ,t , and S 4 ,t for two agents are bounded by [ − 1 , 1] .

Original: The second one, Lemma 4.3, tells us that the summation of accumulated payoffs of opt 1 and opt 2 isalways fixed and is equal to the summation of accumulated payoffs of uninformative ones opt 3 andopt 4 for both agents.
Revised: The second one, Lemma 4.3, tells us that the summations of accumulated payoffs of opt 1 and opt 2 isalways fixed and is equal to the summations of accumulated payoffs of uninformative ones opt 3 andopt 4 for both agents.

Original: The proofs of these two lemmas arein appendix F.1.
Revised: The proofs of these two lemmas are appendix F.1.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: Given the game defined in theorem 4.1, for any round t , Alice has R 1 ,t + R 2 ,t =
Revised: Given the game defined in theorem 4.1, for any round t , Alice has R 1 ,t + R 2 ,t =

Original: R 3 ,t + R 4 ,t = 0 , and Bob has S 1 ,t + S 2 ,t = S 3 ,t + S 4 ,t = 0 .
Revised: R 3 ,t + R 4 ,t = 0 , and Bob has S 1 ,t + S 2 ,t = S 3 ,t + S 4 ,t = 0 .

Original: We now sketch the proof that consists of four steps.
Revised: We now sketch the proof that consists of four steps.

Original: Informally, using lemmas 4.2 and 4.3, the firststep says that the uninformative strategies opt 3 and opt 4 can not completely dominate other strategies.
Revised: Informally, using lemmas 4.2 and 4.3, the firststep says that the uninformative actions opt 3 and opt 4 can not completely dominate other actions.

Original: Specifically, both agents can not choose an option between opt 3 and opt 4 with a probability largerthan 0 .
Revised: Specifically, both agents can not choose an option between opt 3 and opt 4 with a probability largerthan 0 .

Original: With the first step, lemma 4.2 and lemma 4.3, we only need to focus on agents’ reward ofthe truthtelling opt 1 shown in fig.
Revised: With the first step, lemma 4.2 and lemma 4.3, we only need to focus on agents’ reward ofthe truthtelling opt 1 shown in fig.

Original: We partition the space into three types of events.
Revised: We partition the space into three types of events.

Original: Good eventshappen when R 1 , S 1 are both very large or very small, bad events happen when one of R 1 , S 1 isvery large and one of them is very small, and intermediate states are the states between them.
Revised: Good eventshappen when R 1 , S 1 are both very large or very small, bad events happen when one of R 1 , S 1 isvery large and one of them is very small, and intermediate states are the states between them.

Original: Thesecond step removes the possibility that Alice and Bob continue using different reports opt 1 and opt 2 .
Revised: Thesecond step removes the possibility that Alice and Bob continue using different reports opt 1 and opt 2 .

Original: Therefore, we can always escape "bad events" and enter "intermediate states".
Revised: Therefore, we can always escape "bad events" and enter "intermediate states".

Original: The third step furthershows that if the game is in an intermediate state, there exists a constant probability that the gamewill get into a good events that leads to truthful convergence in a constant number of rounds.
Revised: The third step furthershows that if the game is in an intermediate state, there exists a constant probability that the gamewill get into a good events that leads to truthful convergence in a constant number of rounds.

Original: Hence,after the game enters "good events", which leaves us the final step: showing their strategies convergesto either both truth telling opt 1 or flipping opt 2 truthful convergence.
Revised: Hence, after the game enters "good events", which leaves us the final step: showing their action converges toeither both truth telling opt 1 or flipping opt 2 truthful convergence.

Original: Now we discuss each steps in more details but defer all the proofs to appendix F.Step 1:
Revised: Now we discuss each steps in more details but defer all the proofs to appendix F.

Original: Choosing opt 1 , opt 2 with Nonzero Probability

Original: Combining Lemmas 4.2 and 4.3, we have the following Lemma 4.4, which completes step 1.
Revised: Combining Lemmas 4.2 and 4.3, we have the following Lemma 4.4, which completes step 1.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Given the game defined in theorem 4.1, for any round t ≥ 0 and i ∈ { 1 , 2 } , if 299 (cid:80) tj =1 r i,j ≥ 0 , the probability for Alice to choose opt i is larger than 14 ; if (cid:80) tj =
Revised: Given the game defined in theorem 4.1, for any round t ≥ 0 and i ∈ { 1 , 2 } , if 277 (cid:80) tj =1 r i,j ≥ 0 , the probability for Alice to choose opt i is larger than 14 ; if (cid:80) tj =

Original: 1 s i,t ≥ 0 , theprobability for Bob to choose opt i is larger than 14 .
Revised: 1 s i,t ≥ 0 , theprobability for Bob to choose opt i is larger than 14 .

Original:  Step 2: Escaping Bad 
Revised: 4.2 Step 2: Escaping Bad Events

Original: Events

Original: Before introduce the formal statement of step 2, we define two "bad events".
Revised: Before introduce the formal statement of step 2, we define two "bad events".

Original: Given c 0 for all t ≥ 

Revised: Given c 0 for all t ≥

Original: E 1 , 2 t :=
Revised: E 1 , 2 t :=

Original: { R 1 ,t > c 0 , and S 2 ,t > c 0 } and E 2 , 1 t := { R 2 ,t > c 0 and S 1 ,t > c 0 } .
Revised: { R 1 ,t > c 0 , and S 2 ,t > c 0 } and E 2 , 1 t := { R 2 ,t > c 0 and S 1 ,t > c 0 } .

Original: We will specify c 0later.
Revised: We will specify c 0later.

Original: Intuitively, when c 0 is sufficiently large, E 1 , 2 t implies that Alice and Bob will choose opt 1 , opt 2respectively with a probability close to 1 in following rounds.
Revised: Intuitively, when c 0 is sufficiently large, E 1 , 2 t implies that Alice and Bob will choose opt 1 , opt 2respectively with a probability close to 1 in following rounds.

Original: For simplicity, we treat each event E asan indicator function, i.e., E happens if and only if E = 1 .
Revised: For simplicity, we treat each event E asan indicator function, i.e., E happens if and only if E = 1 .

Original: In order to prove that these two bad eventscannot go on forever, we want to show that when E 1 , 2 t happens, R 1 ,t − R 2 ,t will tend to decrease at arapid rate.
Revised: In order to prove that these two bad eventscannot go on forever, we want to show that when E 1 , 2 t happens, R 1 ,t − R 2 ,t will tend to decrease at arapid rate.

Original: Therefore, Alice will eventually deviate from opt 1 to choose opt 2 with a relatively highprobability.
Revised: Therefore, Alice will eventually deviate from opt 1 to choose opt 2 with a relatively highprobability.

Original: By assumption 3.5 and lemma 4.2, given δ > 0 there exists a constant c 1 such that when R 1 ,t > c 1 ,
Revised: By assumption 3.4 and lemma 4.2, given δ > 0 there exists a constant c 1 such that when R 1 ,t > c 1 ,

Original: Alice chooses opt 1 with probability larger than 1 − δ ; when S 2 ,t > c 1 , Bob chooses opt 2 withprobability larger than 1 − δ .
Revised: Alice chooses opt 1 with probability larger than 1 − δ ; when S 2 ,t > c 1 , Bob chooses opt 2 withprobability larger than 1 − δ .

Original: Let γ 1 =
Revised: Let γ 1 =

Original: P X,Y (1 , 1) + P X,Y (0 , 0)
Revised: P X,Y (1 , 1) + P X,Y (0 , 0)

Original: − P X,Y (1 , 0)
Revised: − P X,Y (1 , 0)

Original: − P X,Y (0 , 1)γ 2 =
Revised: − P X,Y (0 , 1)γ 2 =

Original: ( P X,Y (1 , 1) − P X,Y (0 , 0)) 2 − ( P X,Y (1 , 0) − P X,Y (0 , 1)) 2 .
Revised: ( P X,Y (1 , 1) − P X,Y (0 , 0)) 2 − ( P X,Y (1 , 0) − P X,Y (0 , 1)) 2 .

Original: Lemma 4.5.
Revised: Lemma 4.5.

Original: Given the game defined in theorem 4.1, there exists a δ > 0 and corresponding c 1 suchthat for any round t , E
Revised: Given the game defined in theorem 4.1, there exists a δ > 0 and corresponding c 1 suchthat for any round t , E

Original: [ r 1 ,t +1 − r 2 ,t +1 | S 1 ,t − 1 > c 1 + 1] ≥ γ 1 − γ 2> 0 .
Revised: [ r 1 ,t +1 − r 2 ,t +1 | r t − 1 > c 1 + 1] ≥ γ 1 − γ 2> 0 .

Original: Given such δ and c 1 in lemma 4.5, we set c 

Revised: Given such δ and c 1 in lemma 4.5, we set c

Original: = c 1 + ⌈ 1000 γ 1 − γ 2 ⌉ + 1 .
Revised: = c 1 + ⌈ 1000 γ 1 − γ 2 ⌉ + 1 .

Original: Because in each round R 1 ,t , R 2 ,tand S 1 ,t , S 2 ,t vary by at most 1 , if E 1 , 2 t happens Alice chooses opt 1 and Bob chooses opt 2 withprobability larger than 1 − δ independently for the next ⌈ 1000 γ 1 − γ 2 ⌉ + 1 rounds.
Revised: Because in each round R 1 ,t , R 2 ,tand S 1 ,t , S 2 ,t vary by at most 1 , if E 1 , 2 t happens Alice chooses opt 1 and Bob chooses opt 2 withprobability larger than 1 − δ independently for the next ⌈ 1000 γ 1 − γ 2 ⌉ + 1 rounds.

Original: Similar argument holdsfor E 2 , 1 t .
Revised: Similar argument holdsfor E 2 , 1 t .

Original: We use the above observation to show lemma 4.6.
Revised: We use the above observation to show lemma 4.6.

Original: Lemma 4.6.
Revised: Lemma 4.6.

Original: Given the game defined in theorem 4.1, for all t and history H t ∈ E 1 , 2 t , we have
Revised: Given the game defined in theorem 4.1, for all t and history H t ∈ E 1 , 2 t , we have

Original: [Equation]
Revised: [Equation]

Original: This lemma formalizes the blue arrows in Fig.
Revised: This lemma formalizes the blue arrows in Fig.

Original: With this lemma, we get the main result of step two.
Revised: With this lemma, we get the main result of step two.

Original: Lemma 4.7.
Revised: Lemma 4.7.

Original: Given the game defined in theorem 4.1, Pr (cid:110) lim
Revised: Given the game defined in theorem 4.1, Pr (cid:110) lim

Original: sup t →∞ E 1 , 2 t 
Revised: sup t →∞ E 1 , 2 t ∨ E 2 , 1 t = 1 (cid:111)

Original: [Equation]
Revised: = 1 .

Original: If we treat R 1 ,t − R 2 ,t as money of Alice, Lemma 4.7 is similar to the gambler’s ruin problem[7].
Revised: If we treat R 1 ,t − R 2 ,t as money of Alice, Lemma 4.7 is similar to the gambler’s ruin problem[7].

Original: More specifically, R 1 ,t − R 2 ,t has a negative expected growth each ⌈ 1000 γ 1 − γ 2 ⌉ + 1 rounds by
Revised: More specifically, R 1 ,t − R 2 ,t has a negative expected growth each ⌈ 1000 γ 1 − γ 2 ⌉ + 1 rounds by

Original: Lemma 4.6, so R 1 ,t − R 2 ,t will always become small enough to escape E 1 , 2 .
Revised: Lemma 4.6, so R 1 ,t − R 2 ,t will always become small enough to escape E 1 , 2 .

Original:  Step 3: From Intermediate States to Good Events
Revised: 4.3 Step 3: From Intermediate States to Good Events

Original: We define a series of "good events" at first.
Revised: We define a series of "good events" at first.

Original: For all u ∈ N + and t ≥ 1 , E 1 , 1 t ( u ) := { R 1 ,t ≥u, and S 1 ,t ≥ u } and E 2 , 2 t ( u ) := { R 2 ,t ≥ u and S 2 ,t ≥ u } .
Revised: For all u ∈ N + and t ≥ 1 , E 1 , 1 t ( u ) := { R 1 ,t ≥u, and S 1 ,t ≥ u } and E 2 , 2 t ( u ) := { R 2 ,t ≥ u and S 2 ,t ≥ u } .

Original: To the end, we want to show that ∨ t ∈ N + E 1 , 1 t ( u ) and ∨ t ∈ N + E 2 , 2 t ( u ) happen with probability 1 for anyu ∈ N + .
Revised: To the end, we want to show that ∨ t ∈ N + E 1 , 1 t ( u ) and ∨ t ∈ N + E 2 , 2 t ( u ) happen with probability 1 for anyu ∈ N + .

Original: Formally, we claim Lemma 4.8.
Revised: Formally, we claim Lemma 4.8.

Original: Lemma 4.8.
Revised: Lemma 4.8.

Original: Given the game defined in theorem 4.1, for all u there exists λ u so that for any T with history H T ∈ E 1 , 2 T ∨ E 2 , 1 T , we have
Revised: Given the game defined in theorem 4.1, for all u there exists λ u so that for any T withhistory H T ∈ E 1 , 2 T ∨ E 2 , 1 T , we have

Original: [Equation]
Revised: [Equation]

Original: This lemma generally says that when the agents are in an intermediate state that E 1 , 2 T ∨ E 2 , 1 T = 1 , theyhave a constant probability λ u to get into a good event in the next 4( u + c 0 ) + 100 rounds.
Revised: This lemma generally says that when the agents are in an intermediate state that E 1 , 2 T ∨ E 2 , 1 T = 1 , theyhave a constant probability λ u to get into a good event in the next 4( u + c 0 ) + 100 rounds.

Original: In orderto prove this lemma, we define a nice event P T such that P T happens with probability larger than λ uand P T implies the good events happen in no more than 4( u + c 0 ) + 100 rounds.
Revised: In orderto prove this lemma, we define a nice event P T such that P T happens with probability larger than λ uand P T implies the good events happen in no more than 4( u + c 0 ) + 100 rounds.

Original: Formally, event P Tis defined as the following: First, for j = T, . .
Revised: Formally, event P Tis defined as the following: First, for j = T, . .

Original: , T 1 until some round T 1 ≥ T such that S 1 ,T 1 ≥ 0 ,x j +1 = 1 − ˆ y j , y j +

Revised: , T 1 until some round T 1 ≥ T such that S 1 ,T 1 ≥ 0 ,x j +1 = 1 − ˆ y j , y j +

Original: = 1 − ˆ x j , Alice uses strategy opt 1 , Bob uses strategy opt 2 .
Revised: = 1 − ˆ x j , Alice uses strategy opt 1 , Bob uses strategy opt 2 .

Original: Then, Alice and
Revised: Then, Alice and

Original: Bob uses strategy opt 1 for 4 u + 50 rounds and signals are generated as x j +

Revised: Bob uses strategy opt 1 for 4 u + 50 rounds and signals are generated as x j +

Original: = y j +1 =
Revised: = y j +1 =

Original: − x j forj ≥ T 1 .
Revised: − x j forj ≥ T 1 .

Original: We can use Lemma 4.4 to prove that λ u
Revised: We can use Lemma 4.4 to prove that λ u

Original: = (min i,j ∈{ 0 , 1 } { P X,Y ( i, j ) } × 0 .
Revised: = (min i,j ∈{ 0 , 1 } { P X,Y ( i, j ) } × 0 .

Original: 25 2 ) 100+

Revised: 25 2 ) 100+

Original: ( u + c 0 )is a feasible lower bound.
Revised: ( u + c 0 )is a feasible lower bound.

Original:  Step 4: Good Events Lead to Truthful Convergence
Revised: 4.4 Step 4: Good Events Lead to Truthful Convergence

Original: Similar to step 2, we have lemma 4.9 to show that E 1 , 1 t ( u ) can lead to increasing of R 1 ,t − R 2 ,t atfirst.
Revised: Similar to step 2, we have lemma 4.9 to show that E 1 , 1 t ( u ) can lead to increasing of R 1 ,t − R 2 ,t atfirst.

Original: The idea is completely similar to Lemma 4.6 and it formalize the red arrows in Fig.
Revised: The idea is completely similar to Lemma 4.6 and it formalize the red arrows in Fig.

Original: Lemma 4.9.
Revised: Lemma 4.9.

Original: Given the game defined in theorem 4.1, for all u > 2 c 0 + 1 and t if H t ∈ E 1 , 1 t [Equation] ⌊ u 2 ⌋ (cid:1) ,we have E[Equation]
Revised: Given the game defined in theorem 4.1, for all u > 2 c 0 + 1 and t if H t ∈ E 1 , 1 t (cid:0) ⌊ u 2 ⌋ (cid:1) ,we have E

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 4.9, we are able to prove that for any ε , there exists a u such that when E 1 , 1 t ( u ) or E 2 , 2 t ( u ) happens, Alice and Bob will tend to choose ( opt 1 , opt 1 ) or ( opt 2 , opt 2 ) with increasinglyhigher probability.
Revised: Using Lemma 4.9, we are able to prove that for any ε , there exists a u such that when E at ( u ) or E bt ( u )happens, Alice and Bob will tend to choose ( opt 1 , opt 1 ) or ( opt 2 , opt 2 ) with increasingly higherprobability.

Original: Formally, we propose Lemma 4.10.
Revised: Formally, we propose Lemma 4.10.

Original: Lemma 4.10.
Revised: Lemma 4.10.

Original: Given the game defined in theorem 4.1, for all ϵ > 0 there exists u ∈ N + such that given a history H T ∈ [Equation] u 
Revised: Given the game defined in theorem 4.1, for all ϵ > 0 there exists u ∈ N + such thatgiven a history H T ∈ E 1 , 1 T ( u ) ∨ E 2 , 2 T ( u ) , we have

Original: [Equation]
Revised: [Equation]

Original: 2 , 2 T ( u ) , we have Pr [Equation] [Figure]
Revised: We can verify that M i

Original: [Equation]

Original: [Figure] [Equation] [Figure] [Equation]
Revised: = R 1 ,T + i (cid:16) ⌈

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Figure] (cid:12) H T (cid:27) ≥ 1 −ε .
Revised: 1 −

Original: We design a sub-martingale { D i } i ∈ N that is proportional to R 1 ,T + i (cid:16)

Original: [Figure]
Revised: [Equation]

Original: R 2,T + i (cid:16) [Figure] ⌉ +1 (cid:17) , and use Azuma-Hoeffding inequality to prove lemma 4.10.
Revised: 2 ⌉ +1 (cid:17) − 100 i is a sub-martingale, and use Azuma-Hoeffdinginequality to prove lemma 4.10.

Original:  Simulations
Revised:  Simulations

Original: We simulate the CA mechanism with various learning algorithms: the Hedge algorithms, follow theperturbed leader, follow the leader, and ϵ -greedy, and repeat the process 400 times with 800 roundson each algorithm each time.
Revised: In this section, we run simulations on the CA mechanism with various learning algorithms: the Hedge algorithms, follow the perturbed leader, follow the leader, and ϵ -greedy. We ran the processfor 400 times for 800 rounds on each algorithm.

Original: We define the converge proportion in round t  as the fraction of thesimulations where both agents report truthfully opt 1 (or both use opt 2 ) in all the subsequent rounds.
Revised: The converge proportion in round t is defined as the percentage of the simulations where both agents report truthfully opt 1 (or both use opt 2 ) inall the subsequent rounds.

Revised: We show that all our algorithms converge to truth-telling.

Revised: Note that allreward-based online learning algorithms (the Hedge algorithms, follow the perturbed leader, andfollow the leader) exhibit truthful convergence that aligns with our theoretical result, theorem 4.1.

Revised: We also find that the ϵ -greedy with time decreasing ϵ , which is not a reward-based online learningalgorithm, also shows truthful convergence.

Revised: This suggests that the CA mechanism may have truthfulconvergence beyond reward-based online learning algorithms.

Original: In our simulations, we use the following private signal distribution that satisfies assumption 2.2:
Revised: In our simulations, we use the following private signal distribution that satisfies assumption 2.2:

Original: P X,Y (0 , 0) =
Revised: P X,Y (0 , 0) =

Original: P X,Y (1 , 1) = 0 .
Revised: P X,Y (1 , 1) = 0 .

Original: 4 , P X,Y (1 , 0) =
Revised: 4 , P X,Y (1 , 0) =

Original: P X,Y (0 , 1) = 0 .
Revised: P X,Y (0 , 1) = 0 .

Original: Moreover, Alice and Bobare using the same learning algorithms in our simulations that are listed below: First, Follow the Leader algorithm (FTL) choosesopt i with probability proportional to I
Revised: Moreover, Alice and Bob areusing the same learning algorithms in our simulations that are listed below: First, Follow the Leaderalgorithm (FTL) chooses i with probability proportional to I

Original: [ R i = max { R 1 , R 2 , R 3 , R 4 } ] .
Revised: [ R i = max { R 1 , R 2 , R 3 , R 4 } ] .

Original: Follow the perturbed leader (FPL*, where * can be 1, 4 or 8) adds a uniform random noise between 0and ∗ and choose strategy opt i with probability
Revised: Followthe perturbed leader (FPL*, where * can be 1, 4 or 8) adds a uniform random noise between 0and ∗ and choose action i with probability

Original: Pr (cid:110) R i + p i = max j ∈ [4] { R j + p j } (cid:12)(cid:12)(cid:12) p
Revised: Pr (cid:110) R i + p i = max j ∈ [4] { R j + p j } (cid:12)(cid:12)(cid:12) p

Original: j iid ∼ U [0 , ∗ ] (cid:111) .
Revised: j iid ∼ U [0 , ∗ ] (cid:111) .

Original: We consider FPL1, FPL4, and FPL8.
Revised: Weconsider FPL1, FPL4, and FPL8.

Original: Hedge algorithm 1 choose i with probability proportional to3 R i / 2 that is an implementation by choosing ϵ = 0 .
Revised: Hedge algorithm 1 choose i with probability proportional to 3 R i / 2that is an implementation by choosing ϵ = 0 .

Original: 5 for the multiplicative weights algorithm of Aroraet al.
Revised: 5 for the multiplicative weights algorithm of Arora et al.[2].

Original: Hedge algorithm 2 chooses i with probability proportional to e R i that is an implementationby choosing β
Revised: Hedge algorithm 2 chooses i with probability proportional to e R i that is an implementation bychoosing β

Original: = 1 of exponentially weighted averaged forecaster introduced by Freund and Schapire[10].
Revised: = 1 of exponentially weighted averaged forecaster introduced by Freund and Schapire[10] .

Original: Finally, ϵ -greedy algorithm uses time varying ϵ = 1 ( t +1) 2 at round t .
Revised: Finally, ϵ -greedy algorithm uses time varying ϵ = 1 ( t +1) 2 at round t .

Original: Note that ϵ -greedy is notin A but still achieves truthful convergence.
Revised: Note that ϵ -greedy is notin A but still achieves truthful convergence.

Original: In Figure 2, allour algorithms  converge to truth-telling. First, all reward-based online learning
Revised: Figure 2 shows that all algorithms experimented converge to truthful reporting.

Original: [Figure] algorithms (the Hedge algorithms, follow the perturbed leader, and follow the leader) exhibit truthfulconvergence that aligns with our theoretical result, theorem 4.1.
Revised: Moreover, although

Original: Moreover, although FTL is generallynot no-regret, CA mechanism still works well with it.
Revised: FTL is generally not no-regret, CA mechanism still works well with it.

Original: Additionally, we observe that when an algorithmexplores less (e.g. FPL4 vs FPL8), it converges faster, but very little exploration does not furtherimprove the convergence rate (e.g. FTL vs FPL2).
Revised: Additionally, we observe thatwhen an algorithm explores less (e.g. FPL4 vs FPL8), it converges faster, but very little explorationdoes not further improve the convergence rate (e.g. FTL vs FPL1).

Original: Finally, we also find that the ϵ -greedy with timedecreasing ϵ , which is not a reward-based online learning algorithm, also shows truthful convergence.

Original: This suggests that the CA mechanism may have truthful convergence beyond reward-based onlinelearning algorithms.

Original:  Conclusions
Revised:  Conclusions

Original: In this paper, we study sequential peer prediction with learning agents and prove that the notion ofno-regret alone is not sufficient for truthful convergence.
Revised: In this paper, we study sequential peer prediction with learning agents and prove that the notion ofno-regret alone is not sufficient for truthful convergence.

Original: We then define a family of reward-basedlearning algorithms and show that the CA mechanism is able to achieve truthful convergence whenagents use algorithms in this family.
Revised: We then define a family of reward-basedlearning algorithms and show that the CA mechanism is able to achieve truthful convergence whenagents use algorithms in this family.

Original: Finally, we give a discussion on the converge rates of differentlearning agents based on simulations.
Revised: Finally, we give a discussion on the converge rates of differentlearning agents based on simulations.

Original: This is the first theoretical study on peer prediction with learning agents.
Revised: This is the first theoretical study on peer prediction with learning agents.

Original: There are many openproblems and future directions to extend this work.
Revised: There are many openproblems and future directions to extend this work.

Original: We believe similar proof techniques can be usedto extend our results to settings where agents’ private signals are generated by a Markov chain withsome assumptions on the transition matrix.
Revised: For the private signals, we believe the sametechniques can be used to prove truthful convergence of the CA mechanism used on reward-basedlearning agents when signals are generated by a Markov chain given some assumptions on thetransition matrix.

Original: Moreover, this work is only restricted to binary signalsand it is still an open problem whether there exists a mechanism for non-binary settings that canpromise truthful convergence.
Revised: Moreover, this work is only restricted to binary signals and it is still an open problemwhether there exists a mechanism for non-binary settings that can promise truthful convergence.

Original: For the learning agents, one could consider a more general family oflearning algorithms such as when f is time-varying.
Revised: Forthe learning agents, one could consider a more general family of learning algorithms such as when fis time-varying.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: Our work
Revised: Our work

Original: focuses on pure theoretical problems, thus it does not have any negative social impact.
Revised: focuses on pure theoretical problems, thus it does not have any negative social impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] Our code isincluded in supplementary materials.
Revised: [Yes] Our code is included in supplementary materials.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Training details are specified in section 5.
Revised: [Yes] Training details are specified in section 5.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] We give error bars in appendix G instead of in section
Revised: [Yes] We give error bars in appendix G instead of in section

Original: for clearness of fig. 2.
Revised: for clearness of fig. 2.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type of GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] We illustrate our hardware detailsin appendix G.
Revised: [Yes] We illustrate our hardware details in appendix G.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]