Original: HCMD- zero : L EARNING V ALUE A LIGNED M ECHANISMS FROM D ATA
Revised: HCMD- zero : L EARNING V ALUE A LIGNED M ECHANISMS FROM D ATA

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Artiﬁcial learning agents are mediating a larger and larger number of interactions among humans, ﬁrms, and organizations, and the intersection between mechanism design and machine learning has been heavily investigated in recent years.
Revised: Artiﬁcial learning agents are mediating a larger and larger number of interactions among humans, ﬁrms and organizations and the intersection between mechanism design and machine learning has been heavily investigated in recent years.

Original: However, mechanism design methods make strong assumptions on how participants behave (e.g. rationality), or on the kind of knowledge designers have access to a priori (e.g. access to strong baseline mechanisms).
Revised: However, mechanism design methods make strong assumptions on how participants behave (e.g. rationality), or on the kind of knowledge designers have access to a priori (e g. access to strong baseline mechanisms).

Original: Here we introduce HCMD- zero , a general purpose method to construct mechanism agents.
Revised: Here we introduce HCMD- zero , a general purpose method to construct mechanism agents.

Original: HCMD- zero learns by mediating interactions among participants, while remaining engaged in an electoral contest with copies of itself, thereby accessing direct feedback from participants.
Revised: HCMD- zero learns by mediating interactions among participants, while remaining engaged in an electoral contest with copies of itself, thereby accessing direct feedback from participants.

Original: Our results on the Public Investment Game, a stylized resource allocation game that highlights the tension between productivity, equality and the temptation to free-ride, show that HCMD- zero produces competitive mechanism agents that are consistently preferred by human participants over baseline alternatives, and does so automatically, without requiring human knowledge, and by using human data sparingly and effectively Our detailed analysis shows HCMD- zero elicits consistent improvements over the course of training, and that it results in a mechanism with an interpretable and intuitive policy.
Revised: Our results on the Public Investment Game, a stylized resource allocation game that highlights the tension between productivity, equality and the temptation to free-ride, show that HCMD- zero produces competitive mechanism agents that are consistently preferred by human participants over baseline alternatives, and does so automatically, without requiring human knowledge, and by using human data sparingly and effectively Our detailed analysis shows HCMD- zero elicits consistent improvements over the course of training, and that it results in a mechanism with an interpretable and intuitive policy.

Original:  I NTRODUCTIO
Revised:  I NTRODUCTIO

Original: Artiﬁcial learning agents are beginning to play a central role in our institutions.
Revised: Artiﬁcial learning agents are beginning to play a central role in our institutions.

Original: From social networks, to investment management, and trafﬁc routing, an ever growing number of interactions among humans, ﬁrms and organizations are mediated by adaptive systems.
Revised: From social networks, to investment management, and trafﬁc routing, an ever growing number of interactions among humans, ﬁrms and organizations are mediated by adaptive systems.

Original: While the intersection between mechanism design and machine learning has been heavily investigated in recent years, most methods make strong assumptions on either the behavior and preferences of participants (e.g. rationality), or on the kinds of knowledge, baseline mechanisms, or data we have access to before constructing a new mechanism agent for a given economic interaction.
Revised: While the intersection between mechanism design and machine learning has been heavily investigated in recent years, most methods make strong assumptions on either the behavior and preferences of participants (e.g. rationality), or on the kinds of knowledge, baseline mechanisms, or data we have access to before constructing a new mechanism agent for a given economic interaction.

Original: Here we address these restrictive assumptions and present a general method to design a mechanism agent that is able to mediate complex economics interactions among human participants, while requiring no access to alternative mechanisms, human knowledge of the underlying interaction dynamics, and making very few assumptions on the nature of participants’ preferences and strategies.
Revised: Here we address these restrictive assumptions and present a general method to design a mechanism agent that is able to mediate complex economics interactions among human participants, while requiring no access to alternative mechanisms, human knowledge of the underlying interaction dynamics, and making very few assumptions on the nature of participants’ preferences and strategies.

Original: Our method builds on the pipeline outlined in Koster et al.
Revised: Our method builds on the pipeline outlined in Koster et al.

Original: but extends it to scenarios with zero human knowledge.
Revised: but extends it to scenarios with zero human knowledge.

Original: It works by letting an adaptive agent repeatedly mediate interactions among participants, while remaining engaged in an electoral competition against a copy of itself.
Revised: It works by letting an adaptive agent repeatedly mediate interactions among participants, while remaining engaged in an electoral competition against a copy of itself.

Original: Letting the agent compete for participants’ votes, rather than chasing proxy measures such as welfare or equality, ensures that the mechanism remains aligned to the preferences of its constituents, even when these preferences are hard to specify.
Revised: Letting the agent compete for participants’ votes, rather than chasing proxy measures such as welfare or equality, ensures that the mechanism remains aligned to the preferences of its constituents, even when these preferences are hard to specify.

Original: More speciﬁcally, we set out to construct a mechanism agent that is able to mediate economic interactions among human participants, and that is preferred by humans over alternative mechanisms.
Revised: More speciﬁcally, we set out to construct a mechanism agent that is able to mediate economic interactions among human participants, and that is preferred by humans over alternative mechanisms.

Original: We test our method on a stylized investment game where human participants could earn real money and which is known to stress the tensions between welfare, equality and the temptation to free ride.
Revised: We test our method on a stylized investment game where human participants could earn real money and which is known to stress the tensions between welfare, equality and the temptation to free ride.

Original: Our results show that the method presented here was able to construct a complex mechanism policy based on a simple expression of preference; and that this policy is favored by novel participants over a baseline that was previously established as strong in this task.
Revised: Our results show that the method presented here was able to construct a complex mechanism policy based on a simple expression of preference; and that this policy is favored by novel participants over a baseline that was previously established as strong in this task.

Original: Moreover, our detailed analysis shows how the election-against-self curriculum pushes our agent towards interpretable mediation schemes with more and more pronounced punish / reward regions.
Revised: Moreover, our detailed analysis shows how the election-against-self curriculum pushes our agent towards interpretable mediation schemes with more and more pronounced punish / reward regions.

Original: The impact of AI on our institutions is growing rapidly; and as such the intersection of mechanism design and machine learning is receiving considerable attention.
Revised: The impact of AI on our institutions is growing rapidly; and as such the intersection of mechanism design and machine learning is receiving considerable attention.

Original: Here we show that merging the most basic democratic principle of “one person, one vote” with modern machine learning and game theory insights leads to a general method for designing mechanisms that are aligned with the preferences of their constituents, while requiring zero human knowledge [Figure]
Revised: Here we show that merging the most basic democratic principle of “one person, one vote” with modern machine learning and game theory insights leads to a general method for designing mechanisms that are aligned with the preferences of their constituents, while requiring zero human knowledge.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: [Figure]
Revised: [Figure]

Original: [Table] Value alignment and AI safety have been intensely investigated in recent years both from a normative perspective (Gabriel, 2020), and from a technical one (Dafoe et al., 2020); and there is a growing support for building participatory systems for AI ethics and governance (Rahwan, 2017; Lee et al., 2019).
Revised: Value alignment and AI safety have been intensely investigated in recent years both from a normative perspective (Gabriel, 2020), and from a technical one (Dafoe et al., 2020); and there is a growing support for building participatory systems for AI ethics and governance (Rahwan, 2017; Lee et al., 2019).

Original: Mechanism design is a sub-ﬁeld of economics that studies how to design the rules and incentives of multi-agent interactions, so that selfinterested participants will prefer certain strategies, often trading off their own welfare for that of the group.
Revised: Mechanism design is a sub-ﬁeld of economics that studies how to design the rules and incentives of multi-agent interactions, so that selfinterested participants will prefer certain strategies, often trading off their own welfare for that of the group.

Original: The ﬁeld has a long history to which it is near impossible to do justice, see (Maskin, 2008) for a review.
Revised: The ﬁeld has a long history to which it is near impossible to do justice, see (Maskin, 2008) for a review.

Original: More recently mechanism design has been studied from an algorithmic point of view (Conitzer & Sandholm, 2002; Nisan & Ronen, 2001), as well as a machine learning one (Dütting et al., 2017;
Revised: More recently mechanism design has been studied from an algorithmic point of view (Conitzer & Sandholm, 2002; Nisan & Ronen, 2001), as well as a machine learning one (Dütting et al., 2017;

Original: Manisha et al., 2018;
Revised: Manisha et al., 2018;

Original: Tacchetti et al., 2019; Koster et al., 2022).
Revised: Tacchetti et al., 2019; Koster et al., 2022).

Original: Finally, researchers have recently turned their attention to the role that mechanism design can play in our pursuit of social good (Abebe & Goldner, 2018).
Revised: Finally, researchers have recently turned their attention to the role that mechanism design can play in our pursuit of social good (Abebe & Goldner, 2018).

Original: Agent based models (ABMs), where a computer simulation predicts how autonomous agents will adapt to certain environment interventions, has been a tool used by policy makers to design new mechanisms since its inception.
Revised: Agent based models (ABMs), where a computer simulation predicts how autonomous agents will adapt to certain environment interventions, has been a tool used by policy makers to design new mechanisms since its inception.

Original: ABMs have received renewed attention after the 2008 Economic Crisis (Farmer & Foley, 2009; Hamill & Gilbert, 2015).
Revised: ABMs have received renewed attention after the 2008 Economic Crisis (Farmer & Foley, 2009; Hamill & Gilbert, 2015).

Original: The problem of building artiﬁcial agents that coordinate with human participants starting from “zero knowledge” had been investigated both in the computer game setting (Strouse et al., 2021), and in a simulated economy environment (Zheng et al., 2020).
Revised: The problem of building artiﬁcial agents that coordinate with human participants starting from “zero knowledge” had been investigated both in the computer game setting (Strouse et al., 2021), and in a simulated economy environment (Zheng et al., 2020).

Original: Similarly, self-play and no-human-knowledge methods have been successfully applied to challenging constant-sum two-player games in recent past (Silver et al., 2016; 2017; Vinyals et al., 2019).
Revised: Similarly, self-play and no-human-knowledge methods have been successfully applied to challenging constant-sum two-player games in recent past (Silver et al., 2016; 2017; Vinyals et al., 2019).

Original: The method we present here builds heavily on the pipeline outlined in Koster et al.
Revised: The method we present here builds heavily on the pipeline outlined in Koster et al.

Original: Human
Revised: Human

Original: Centered Mechanism Design (HCMD), which uses a similar approach to train a mechanism agent on the same stylized economic game we consider, and with the same goal of producing mechanisms that are preferred by human participants over baseline alternatives.
Revised: Centered Mechanism Design (HCMD), which uses a similar approach to train a mechanism agent on the same stylized economic game we consider, and with the same goal of producing mechanisms that are preferred by human participants over baseline alternatives.

Original: Here we provide a detailed comparison and highlight the substantial differences between HCMD and our contributions: First, HCMD does not rely on self-play, and rather trains mechanism agents using knowledge of the baseline mechanisms it needs to be preferred over, which need to be known a priori.
Revised: Here we provide a detailed comparison and highlight the substantial differences between HCMD and our contributions: First, HCMD does not rely on self-play, and rather trains mechanism agents using knowledge of the baseline mechanisms it needs to be preferred over, which need to be known a priori.

Original: Second, HCMD is not an iterative method, and it assumes that participants behavior and strategies do not depend on the mechanism they interact with, and can thus be modeled once and for all.
Revised: Second, HCMD is not an iterative method, and it assumes that participants behavior and strategies do not depend on the mechanism they interact with, and can thus be modeled once and for all.

Original: The method presented here, on the other hand, works iteratively and thus takes into account how human participants react to changes in the mechanism agent.
Revised: The method presented here, on the other hand, works iteratively and thus takes into account how human participants react to changes in the mechanism agent.

Original: Finally, HCMD uses ad-hoc knowledge about the underlying interaction dynamics to construct predictive models of participants’ voting behavior; our method does not.
Revised: Finally, HCMD uses ad-hoc knowledge about the underlying interaction dynamics to construct predictive models of participants’ voting behavior; our method does not.

Original: In summary,
Revised: In summary,

Original: HCMD is not a zero -knowledge method: it requires access to alternative mechanism policies for the speciﬁc interaction it sets out to mediate, it relies on knowledge of the voting dynamics to construct voting models
Revised: HCMD is not a zero -knowledge method: it requires access to alternative mechanism policies for the speciﬁc interaction it sets out to mediate, it relies on knowledge of the voting dynamics to construct voting models

Original: [Table]
Revised: [Table]

Original: and ﬁnally it assumes that humans strategies are unaffected by changes in the mechanism policy, which is at best something that needs to be veriﬁed empirically for each interaction
Revised: and ﬁnally it assumes that humans strategies are unaffected by changes in the mechanism policy, which is at best something that needs to be veriﬁed empirically for each interaction

Original: [Table]
Revised: [Table]

Original:  M ETHODS
Revised:  M ETHODS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We begin this section describing the Public Investment Game (PIG) we used for our experiments, and then proceed to introduce our methods in detail.
Revised: We begin this section describing the Public Investment Game (PIG) we used for our experiments, and then proceed to introduce our methods in detail.

Original: We highlight here that while we report results on the
Revised: We highlight here that while we report results on the

Original: PIG, and we use it to ground our exposition, our methods are applicable in very general settings.
Revised: PIG, and we use it to ground our exposition, our methods are applicable in very general settings.

Original: 3. P UBLIC I NVESTMENT G AME FOR PARTICIPANTS AND MECHANISMS
Revised: 3. P UBLIC I NVESTMENT G AME FOR PARTICIPANTS AND MECHANISMS

Original: From the point of view of the participants, the PIG is a general-sum 4 player game that unfolds over 10 identical rounds.
Revised: From the point of view of the participants, the PIG is a general-sum 4 player game that unfolds over 10 identical rounds.

Original: At the beginning of each round, each participant receives an endowment of e i,t “coins”, with i = 1 , . . .
Revised: At the beginning of each round, each participant receives an endowment of e i,t “coins”, with i = 1 , . . .

Original: , 4 , and t = 1 , . .
Revised: , 4 , and t = 1 , . .

Original: . , 10 , and decides what fraction of coins
Revised: . , 10 , and decides what fraction of coins

Original: ρ i,t they would like to invest in a public fund that grows with a ﬁxed multiplier of 1 . 6 .
Revised: ρ i,t they would like to invest in a public fund that grows with a ﬁxed multiplier of 1 . 6 .

Original: A mechanism agent then observes e i,t and ρ i,t , and determines 4 redistribution weights w i,t ≥ 0 , with (cid:80) i w i,t = 1 , according to which the fund is returned in its entirety: each participant receives w i,t (1 . 6 × (cid:80) i ρ i,t e i,t ) , and the game moves on to the next round.
Revised: A mechanism agent then observes e i,t and ρ i,t , and determines 4 redistribution weights w i,t ≥ 0 , with (cid:80) i w i,t = 1 , according to which the fund is returned in its entirety: each participant receives w i,t (1 . 6 × (cid:80) i ρ i,t e i,t ) , and the game moves on to the next round.

Original: In our experiments with human participants, at the end of the 10 rounds, each participant collected a monetary reward proportional to the funds they received from the public investment fund, and the endowments they decided not to contribute
Revised: In our experiments with human participants, at the end of the 10 rounds, each participant collected a monetary reward proportional to the funds they received from the public investment fund, and the endowments they decided not to contribute

Original: R i =
Revised: R i =

Original: (cid:80) t r i,t = (cid:80) t w i,t (1 . 6 ×
Revised: (cid:80) t r i,t = (cid:80) t w i,t (1 . 6 ×

Original: (cid:80) i ρ i,t e i,t )
Revised: (cid:80) i ρ i,t e i,t )

Original: +(1 − ρ i,t ) e i,t .
Revised: +(1 − ρ i,t ) e i,t .

Original: From the point of view of the participants, the redistribution decisions of the mechanism agent are folded in the game transition kernel.
Revised: From the point of view of the participants, the redistribution decisions of the mechanism agent are folded in the game transition kernel.

Original: In our human experiments, we let the same cohort of players face two mechanism agents in independent instances of the PIG played in sequence.

Original: We then asked each participant to cast a vote on

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: In our human experiments

Original: [Figure]
Revised: [Figure]

Revised: we let the same cohort of players face two mechanism agents in independent instances of the PIG played in sequence.

Original:  which mechanism they would like to re-experience in a third follow up game in which the mechanism would be decided by majority vote.
Revised: We then asked each participant to cast a vote on which mechanism they would like to re-experience in a third follow up game in which the mechanism would be decided by majority vote.

Original: It is worth noting that participants collected monetary rewards in all three games (two initial, and one follow up), and thus had “skin in the game” when reporting which mechanism they preferred.
Revised: It is worth noting that participants collected monetary rewards in all three games (two initial, and one follow up), and thus had “skin in the game” when reporting which mechanism they preferred.

Original: From the point of view of the mechanism agent, the PIG is a 2 player constant-sum game.
Revised: From the point of view of the mechanism agent, the PIG is a 2 player constant-sum game.

Original: First, the two candidate mechanisms face independent sequential decision making problems with states coinciding with endowments, contributions, and redistribution histories, and actions coinciding with redistribution weights.
Revised: First, the two candidate mechanisms face independent sequential decision making problems with states coinciding with endowments, contributions, and redistribution histories, and actions coinciding with redistribution weights.

Original: Second, the two mechanisms collect a reward based on the number of votes cast in their favor.
Revised: Second, the two mechanisms collect a reward based on the number of votes cast in their favor.

Original: Similarly to what happens with participants, the transition kernel implements the
Revised: Similarly to what happens with participants, the transition kernel implements the

Original: PIG game dynamics, as well as the participants contributing and voting behaviors.
Revised: PIG game dynamics, as well as the participants contributing and voting behaviors.

Original: 3. S ELF - PLAY LOOP OVERVIEW
Revised: 3. S ELF - PLAY LOOP OVERVIEW

Original: [Table]
Revised: [Table]

Original: =

Original: [Table]

Original: s
Revised: s

Original: =s =s
Revised: =s =s

Original: =s 
Revised: =s =s

Original: =s =s =
Revised: =s =s =

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Here we introduce HCMD- zero , a method to construct a mechanism agent that is able to mediate economic interactions among human participants, and that is preferred by humans over alternative mechanisms.
Revised: Here we introduce HCMD- zero , a method to construct a mechanism agent that is able to mediate economic interactions among human participants, and that is preferred by humans over alternative mechanisms.

Original: We build on the recent successes of self-play to train competitive agents in challenging 2-player constant sum games (Silver et al., 2016; 2017; Vinyals et al., 2019), and let our mechanism agent update its redistribution policy so as to win against a copy of itself in the PIG election game we outlined above (i.e. the two mechanism agents faced by participants in the two initial games were in fact copies of one another, this fact was not disclosed to participants).
Revised: We build on the recent successes of self-play to train competitive agents in challenging 2-player constant sum games (Silver et al., 2016; 2017; Vinyals et al., 2019), and let our mechanism agent update its redistribution policy so as to win against a copy of itself in the PIG election game we outlined above (i.e. the two mechanism agents faced by participants in the two initial games were in fact copies of one another, this fact was not disclosed to participants).

Original: Notably, we did not train our mechanism agent directly on experience acquired while interacting with human participants (this would require access to a prohibitive amount of data).
Revised: Notably, we did not train our mechanism agent directly on experience acquired while interacting with human participants (this would require access to a prohibitive amount of data).

Original: Instead we constructed “virtual participants” by training neural network models to closely imitate human contributing and voting behavior in the PIG, and let our mechanism agents interact with these instead.
Revised: Instead we constructed “virtual participants” by training neural network models to closely imitate human contributing and voting behavior in the PIG, and let our mechanism agents interact with these instead.

Original: The method we outline is thus composed of 3 phases , which are repeated in order over several iterations : 1) Acquire : collect behavioral data (contributions and voting) by letting participants interact with a mechanism agent in two sequential instances of the PIG, 2) Model : construct accurate models of participants’ voting and contributing behaviors, 3) Optimize : construct a simulated election game environment using “virtual participants” and optimize the mechanism agent’s redistribution policy in pursuit of participants preferences.
Revised: The method we outline is thus composed of 3 phases , which are repeated in order over several iterations : 1) Acquire : collect behavioral data (contributions and voting) by letting participants interact with a mechanism agent in two sequential instances of the PIG, 2) Model : construct accurate models of participants’ voting and contributing behaviors, 3) Optimize : construct a simulated election game environment using “virtual participants” and optimize the mechanism agent’s redistribution policy in pursuit of participants preferences.

Original: As stated above, the 3 phases Acquire , Model , and
Revised: As stated above, the 3 phases Acquire , Model , and

Original: Optimize are repeated in order within each iteration .
Revised: Optimize are repeated in order within each iteration .

Original: In particular, in each Acquire phase, we collect behavioral trajectories of human participants interacting with the mechanism agent produced by the most recent Optimize phase (we use a random mechanism on the ﬁrst iteration).
Revised: In particular, in each Acquire phase, we collect behavioral trajectories of human participants interacting with the mechanism agent produced by the most recent Optimize phase (we use a random mechanism on the ﬁrst iteration).

Original: The Model phase trains participants’ models using data from all previous iterations, and ﬁnally the Optimize phase relies on the most recent virtual participants models to construct the mechanism agent’s training environment.
Revised: The Model phase trains participants’ models using data from all previous iterations, and ﬁnally the Optimize phase relies on the most recent virtual participants models to construct the mechanism agent’s training environment.

Original: 3. Acquire STEP : D ATA A CQUISITION
Revised: 3. Acquire STEP : D ATA A CQUISITION

Original: We used a crowd-sourcing platform to acquire contributing and voting behavior data from human participants ( n = 1656 ).
Revised: We used a crowd-sourcing platform to acquire contributing and voting behavior data from human participants ( n = 1656 ).

Original: All participants gave informed consent to participate in the experiment.
Revised: All participants gave informed consent to participate in the experiment.

Original: In particular, during the Acquire step of each iteration s , groups of 4 human participants completed two episodes of the PIG game interacting with a mechanism agent endowed with the most recent parameters (see Sec.
Revised: In particular, during the Acquire step of each iteration s , groups of 4 human participants completed two episodes of the PIG game interacting with a mechanism agent endowed with the most recent parameters (see Sec.

Original: 3.5 for details on the mechanism agent), and voted for the episode they preferred.
Revised: 3.5 for details on the mechanism agent), and voted for the episode they preferred.

Original: We denote this data-set as D s .
Revised: We denote this data-set as D s .

Original: Since the mechanism and the conditions (e.g. the endowment) are identical in both episodes, the only difference between them is driven by the randomness in human behavior.
Revised: Since the mechanism and the conditions (e.g. the endowment) are identical in both episodes, the only difference between them is driven by the randomness in human behavior.

Original: 3. Model STEP : M ODEL PARTICIPANTS
Revised: 3. Model STEP : M ODEL PARTICIPANTS

Original: During the Model step for iteration s , we trained independent models to predict human contributions and votes (we jointly refer to these models as “virtual participants”) from all data-sets D 1 , . . .
Revised: During the Model step for iteration s , we trained independent models to predict human contributions and votes (we jointly refer to these models as “virtual participants”) from all data-sets D 1 , . . .

Original: , D s collected thus far.
Revised: , D s collected thus far.

Original: The contribution model is a neural network similar to that in (Koster et al., 2022), which takes as input each players normalized endowments and contributions: e i,t / 10 and c i,t / 10 , as well as each players fractional contribution ρ i,t = c
Revised: The contribution model is a neural network similar to that in (Koster et al., 2022), which takes as input each players normalized endowments and contributions: e i,t / 10 and c i,t / 10 , as well as each players fractional contribution ρ i,t = c

Original: i,t /e i,t , and outputs the log-likelihood of contributing 0 , 1 , . .
Revised: i,t /e i,t , and outputs the log-likelihood of contributing 0 , 1 , . .

Original: ,coins ( 10 coins being the maximum endowment).
Revised: ,coins ( 10 coins being the maximum endowment).

Original: The network was applied independently for each participant and composed of an input linear layer, a LSTM and an output linear layer.
Revised: The network was applied independently for each participant and composed of an input linear layer, a LSTM and an output linear layer.

Original: The contribution model was trained to minimize group-wise cross-entropy between predicted and actual contributions.
Revised: The contribution model was trained to minimize group-wise cross-entropy between predicted and actual contributions.

Original: The votes model is a simple linear layer, which we apply independently for each participant, that takes in the ﬂattened observations from a single episode (10 rounds × 3 endowment/contribution/payout × 4 participants) and produces a single output, which can be interpreted as the log-likelihood of voting for the current episode.
Revised: The votes model is a simple linear layer, which we apply independently for each participant, that takes in the ﬂattened observations from a single episode (10 rounds × 3 endowment/contribution/payout × 4 participants) and produces a single output, which can be interpreted as the log-likelihood of voting for the current episode.

Original: The same linear layer is applied to both episodes, and a softmax normalization produces the ﬁnal probabilities.
Revised: The same linear layer is applied to both episodes, and a softmax normalization produces the ﬁnal probabilities.

Original: We train this network to minimize group-wise cross-entropy between predicted and actual votes, with an additional l 2 regularization loss of the linear layer parameters.
Revised: We train this network to minimize group-wise cross-entropy between predicted and actual votes, with an additional l 2 regularization loss of the linear layer parameters.

Original: Since the amount of data available increases with every iteration, hyper-parameters must be adjusted each time.
Revised: Since the amount of data available increases with every iteration, hyper-parameters must be adjusted each time.

Original: In our experiments, we tuned the l 2 regularization and network size using cross validation with a random 70%-30% train/eval split.
Revised: In our experiments, we tuned the l 2 regularization and network size using cross validation with a random 70%-30% train/eval split.

Original: We reconstituted the original data-set for training (see Tab.for details).
Revised: We reconstituted the original data-set for training (see Tab.for details).

Original: 3. Optimize STEP : T RAIN MECHANISM AGENT
Revised: 3. Optimize STEP : T RAIN MECHANISM AGENT

Original: Similarly to HCMD, we parameterize the mechanism policy with a Graph Network (Battaglia et al., 2018) policy function π , with no memory and parameters θ , which takes as input the current endowment and contribution from each player (as nodes of a fully connected graph) and outputs deterministic redistribution weights of each player w i,t .
Revised: Similarly to HCMD, we parameterize the mechanism policy with a Graph Network (Battaglia et al., 2018) policy function π , with no memory and parameters θ , which takes as input the current endowment and contribution from each player (as nodes of a fully connected graph) and outputs deterministic redistribution weights of each player w i,t .

Original: During the Optimize step, we update the redistribution policy parameters θ s − 1 → θ s by letting our mechanism agent interact with the most current “virtual participant” models p s .
Revised: During the Optimize step, we update the redistribution policy parameters θ s − 1 → θ s by letting our mechanism agent interact with the most current “virtual participant” models p s .

Original: Speciﬁcally, we trained the mechanism by approximating the mechanism agent’s policy gradient through a bespoke low variance estimator based on Stochastic Computation Graphs (Schulman et al., 2016) that exploits the differentiable structure of the PIG while accounting for the stochastic nature of the player model’s contributions (similar to (Koster et al., 2022)).
Revised: Speciﬁcally, we trained the mechanism by approximating the mechanism agent’s policy gradient through a bespoke low variance estimator based on Stochastic Computation Graphs (Schulman et al., 2016) that exploits the differentiable structure of the PIG while accounting for the stochastic nature of the player model’s contributions (similar to (Koster et al., 2022)).

Original: We note that while this choice is suitable for our set up, the learning rule can be replaced by any Reinforcement Learning technique that ﬁts the problem at hand.
Revised: We note that while this choice is suitable for our set up, the learning rule can be replaced by any Reinforcement Learning technique that ﬁts the problem at hand.

Original: During training, we used batches of 1000 games equally split among the endowment condition we considered: [10 , 2 , 2 , 2] , [10 , 4 , 4 , 4] , [10 , 6 , 6 , 6] , [10 , 8 , 8 , 8] and [10 , 10 , 10 , 10] .
Revised: During training, we used batches of 1000 games equally split among the endowment condition we considered: [10 , 2 , 2 , 2] , [10 , 4 , 4 , 4] , [10 , 6 , 6 , 6] , [10 , 8 , 8 , 8] and [10 , 10 , 10 , 10] .

Original: The mechanism’s policy was trained using an ADAM optimizer with learning rate 4 e − 5 .
Revised: The mechanism’s policy was trained using an ADAM optimizer with learning rate 4 e − 5 .

Original: Finally, we ﬁxed the number of gradient updates to 2000 for intermediate iterations and 10000 for the ﬁnal one.
Revised: Finally, we ﬁxed the number of gradient updates to 2000 for intermediate iterations and 10000 for the ﬁnal one.

Original: This choice warrants a brief discussion: there is a trade off between how aggressively we require our participants model p s to extrapolate beyond its training distribution (recall that training data was collected using mechanism parameters θ 0 . . .
Revised: This choice warrants a brief discussion: there is a trade off between how aggressively we require our participants model p s to extrapolate beyond its training distribution (recall that training data was collected using mechanism parameters θ 0 . . .

Original: θ s − 1 ), and how many total iterations (and thus data collection steps) we prescribe.
Revised: θ s − 1 ), and how many total iterations (and thus data collection steps) we prescribe.

Original: Related to the choice of training updates within an iteration’s
Revised: Related to the choice of training updates within an iteration’s

Original: Optimize step, our method requires determining how many iterations , i.e. repetitions of our Acquire , Model and Optimize pipeline, we should complete.
Revised: Optimize step, our method requires determining how many iterations , i.e. repetitions of our Acquire , Model and Optimize pipeline, we should complete.

Original: Our proposed approach is to construct a meta-game, that is, recording the results of a round-robin election tournament among the mechanism agents produced at each iteration, constructed using the most recent “virtual participants” models (see Fig. 3).
Revised: Our proposed approach is to construct a meta-game, that is, recording the results of a round-robin election tournament among the mechanism agents produced at each iteration, constructed using the most recent “virtual participants” models (see Fig. 3).

Original: More precisely we constructed a meta-game as a two-player game deﬁned by a payoff matrix of size s × s with entry i, j corresponding to the proportion of votes collected by mechanisms playing with parameters θ i and θ j over 100 games.
Revised: More precisely we constructed a meta-game as a two-player game deﬁned by a payoff matrix of size s × s with entry i, j corresponding to the proportion of votes collected by mechanisms playing with parameters θ i and θ j over 100 games.

Original: Once the actions corresponding to later checkpoints no longer constitute a dominant strategy in the meta-game, or when their advantage becomes negligible, we can conclude that HCMD- zero has converged, since it no longer produces meaningful improvements 1 .
Revised: Once the actions corresponding to later checkpoints no longer constitute a dominant strategy in the meta-game, or when their advantage becomes negligible, we can conclude that HCMD- zero has converged, since it no longer produces meaningful improvements 1 .

Original:  R ESULTS
Revised:  R ESULTS

Original: In this section we show the results of applying our method in the Public Investment Game.
Revised: In this section we show the results of applying our method in the Public Investment Game.

Original: We ﬁrst show the performance against baselines after 7 iterations of training with HCMD- zero .
Revised: We ﬁrst show the performance against baselines after 7 iterations of training with HCMD- zero .

Original: Then, we explore in more detail the learning dynamics of the model of human participants, as well as the convergence of the mechanism.
Revised: Then, we explore in more detail the learning dynamics of the model of human participants, as well as the convergence of the mechanism.

Original: Finally, we provide an analysis of the mechanism’s behavior throughout training.
Revised: Finally, we provide an analysis of the mechanism’s behavior throughout training.

Original: 4. P ERFORMANCE OF HCMD- zero AT EVALUATION
Revised: 4. P ERFORMANCE OF HCMD- zero AT EVALUATION

Original: In order to validate our approach, we trained a mechanism in the Public Investment Game (PIG).
Revised: In order to validate our approach, we trained a mechanism in the Public Investment Game (PIG).

Original: Similar to Koster et al.
Revised: Similar to Koster et al.

Original: , we divided participants into one “Head” player that always received an endowment e head,t = 10 and three “tail” players that received a “tail” endowment e tail,t ∈ { 2 , 4 , 6 , 8 , 10 } .
Revised: , we divided participants into one “Head” player that always received an endowment e head,t = 10 and three “tail” players that received a “tail” endowment e tail,t ∈ { 2 , 4 , 6 , 8 , 10 } .

Original: The tail endowment was consistent within a group, across tail payers and for all mechanisms they interacted with.
Revised: The tail endowment was consistent within a group, across tail payers and for all mechanisms they interacted with.

Original: We evaluated the mechanism θ 7 by collecting new data speciﬁcally for this purpose.
Revised: We evaluated the mechanism θ 7 by collecting new data speciﬁcally for this purpose.

Original: Humans interacted with the trained mechanism and with a baseline alternative in two subsequent games (in counterbalanced order).
Revised: Humans interacted with the trained mechanism and with a baseline alternative in two subsequent games (in counterbalanced order).

Original: Our choice of baseline was Liberal Egalitarian mechanism, a redistribution scheme that disburses the public fund according to the proportion of endowment contributed by each participant, as baseline.
Revised: Our choice of baseline was Liberal Egalitarian mechanism, a redistribution scheme that disburses the public fund according to the proportion of endowment contributed by each participant, as baseline.

Original: Koster et al.
Revised: Koster et al.

Original: (2022) show that Liberal
Revised: (2022) show that Liberal

Original: Egalitarian is a strong baseline that is preferred by humans over the Strict Egalitarian, which divides the fund in equal parts.
Revised: Egalitarian is a strong baseline that is preferred by humans over the Strict Egalitarian, which divides the fund in equal parts.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: HCMD- zero was voted more often than Liberal Egalitarian, achieving an average of 54.3% of the votes ( p < 0 .
Revised: HCMD- zero was voted more often than Liberal Egalitarian, achieving an average of 54.3% of the votes ( p < 0 .

Original: 06 with a non-parametric analysis that corrects for in-group correlations).
Revised: 06 with a non-parametric analysis that corrects for in-group correlations).

Original: More speciﬁcally, HCMD- zero achieved at least half of the votes against Liberal
Revised: More speciﬁcally, HCMD- zero achieved at least half of the votes against Liberal

Original: Egalitarian (see bottom-left panel), whilst matching the contributions from players (top panel).
Revised: Egalitarian (see bottom-left panel), whilst matching the contributions from players (top panel).

Original: At the group level, HCMD- zero matched the performance of Liberal Egalitarian in trading off the productivity of the group (incentivizing the head player to contribute more) and the inequality of the group (redistributing to the tail players; bottom-right panel).
Revised: At the group level, HCMD- zero matched the performance of Liberal Egalitarian in trading off the productivity of the group (incentivizing the head player to contribute more) and the inequality of the group (redistributing to the tail players; bottom-right panel).

Original: 4. P ARTICIPANT MODELS DISPLAY EQUILIBRIUM EFFECTS
Revised: 4. P ARTICIPANT MODELS DISPLAY EQUILIBRIUM EFFECTS

Revised: al., 2019)).

Revised: Should such cycles become apparent when constructing the meta-game, one could turn to standard methods to address them (e.g. Fictitious Play or Double Oracle).

Original: We turned to look at the predictive power of the “virtual participants” model p s across iterations.
Revised: We turned to look at the predictive power of the “virtual participants” model p s across iterations.

Original: Our iterative method addresses the fact that human contribution and voting behavior depends on the mechanism.
Revised: Our iterative method addresses the fact that human contribution and voting behavior depends on the mechanism.

Original: Fig.
Revised: Fig.

Original: 3 (left two panels) shows that this effect is observed in practice.
Revised: 3 (left two panels) shows that this effect is observed in practice.

Original: We construct a contribution and vote cross-validation matrix by reporting in entry i, j the cross-entropy loss achieved by each model p i (rows) on each data-set D j (columns); recall that model p i is trained using data-sets
Revised: We construct a contribution and vote cross-validation matrix by reporting in entry i, j the cross-entropy loss achieved by each model p i (rows) on each data-set D j (columns); recall that model p i is trained using data-sets

Original: D 1 , . . .
Revised: D 1 , . . .

Original: , D i (matrix entries are normalized per-column by the corresponding diagonal entry).
Revised: , D i (matrix entries are normalized per-column by the corresponding diagonal entry).

Original: The ﬁgure clearly shows that the predictive performance of each model degrades progressively for each subsequent data-set indicating that participants contributing and voting behavior has changed.
Revised: The ﬁgure clearly shows that the predictive performance of each model degrades progressively for each subsequent data-set indicating that participants contributing and voting behavior has changed.

Original: 4. M ECHANISM IMPROVEMENT AND CONVERGENCE
Revised: 4. M ECHANISM IMPROVEMENT AND CONVERGENCE

Revised: (cid:1) (cid:2) (cid:3) (cid:4) (cid:5) (cid:6) (cid:7) (cid:8)

Original: [Table]
Revised: [Table]

Revised: (cid:10) (cid:11) (cid:12) (cid:8) (cid:10) (cid:6) (cid:7) (cid:13)(cid:3)(cid:10)(cid:14)(cid:5)(cid:6)(cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:12)(cid:8)(cid:10)(cid:6)(cid:7)(cid:15) (cid:1) (cid:2)(cid:1) (cid:1) (cid:2)(cid:1) (cid:1) (cid:2)(cid:1) (cid:1) (cid:3) (cid:1) (cid:4) (cid:1) (cid:5) (cid:1) (cid:6) (cid:16)(cid:7)(cid:10)(cid:8)(cid:10)(cid:3)(cid:14)(cid:10)(cid:17)(cid:3)(cid:8)(cid:10)(cid:6)(cid:7)(cid:18)(cid:9)(cid:3)(cid:7)(cid:4)(cid:6)(cid:19)(cid:20)

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: On every iteration s we constructed a metagame as described in the methods above, where each row and column corresponds to the mechanisms θ 0 , .. .
Revised: On every iteration s we constructed a metagame as described in the methods above [Figure] where each row and column corresponds to the mechanisms θ 0 , .

Original: , θ s and each cell corresponds to the number of votes obtained in simulation with the “virtual participants” model p s .
Revised: , θ s and each cell corresponds to the number of votes obtained in simulation with the “virtual participants” model p s .

Original: This can be found for iteration 7 on the right panel in Fig.
Revised: This can be found for iteration 7 on the right panel in Fig.

Original: With HCMD- zero , the optimization of the mechanism showed consistent improvements on every iteration, with diminishing returns until convergence on iteration 7.
Revised: With HCMD- zero , the optimization of the mechanism showed consistent improvements on every iteration, with diminishing returns until convergence on iteration 7.

Original: 4. A NALYSIS OF MECHANISM BEHAVIOR
Revised: 4. A NALYSIS OF MECHANISM BEHAVIOR

Original: We analyze the learnt mechanism policy across iterations in Fig.
Revised: We analyze the learnt mechanism policy across iterations in Fig.

Original: For each tail endowment (columns) and across iterations (rows), we illustrate the mechanism’s policy on a grid containing the contributions of the head (y-axis) and tail players (x-axis, averaged across the 3 tail players).
Revised: For each tail endowment (columns) and across iterations (rows), we illustrate the mechanism’s policy on a grid containing the contributions of the head (y-axis) and tail players (x-axis, averaged across the 3 tail players).

Original: Then, for each possible contribution pair, we computed the average redistribution weight across episodes and players.
Revised: Then, for each possible contribution pair, we computed the average redistribution weight across episodes and players.

Original: These are plotted with yellow favouring redistribution to the head player (high endowment) and blue to the tail players (low endowment).
Revised: These are plotted with yellow favouring redistribution to the head player (high endowment) and blue to the tail players (low endowment).

Original: Our mechanism player effectively learns a policy similar to the Liberal Egalitarian mechanism (see additional row at the bottom): straight lines fanning out from left to right and bottom to top; but which punishes harshly Head players that do not contribute enough: pinching at the bottom left.
Revised: Our mechanism player effectively learns a policy similar to the Liberal Egalitarian mechanism (see additional row at the bottom): straight lines fanning out from left to right and bottom to top; but which punishes harshly Head players that do not contribute enough: pinching at the bottom left.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have introduced HCMD- zero , a general purpose method to construct mechanisms that are preferred by human participants.
Revised: We have introduced HCMD- zero , a general purpose method to construct mechanisms that are preferred by human participants.

Original: Our methods require no baseline or alternative mechanisms, and no knowledge of the environment dynamics.
Revised: Our methods require no baseline or alternative mechanisms, and no knowledge of the environment dynamics.

Original: Our methods use participant modeling and self-play to minimize the amount of data that is required to train a mechanism, and they iteratively address the challenges posed by equilibrium effects, where the participants behavior changes in response to updates in the mechanism policy.
Revised: Our methods use participant modeling and self-play to minimize the amount of data that is required to train a mechanism, and they iteratively address the challenges posed by equilibrium effects, where the participants behavior changes in response to updates in the mechanism policy.

Original: Our results show that HCMD- zero produces a competent mechanism agent in the challenging Public Investment Game.
Revised: Our results show that HCMD- zero produces a competent mechanism agent in the challenging Public Investment Game.

Original: Our detailed analysis shows that our mechanism policy is consistently improved across iterations, and provides an interpretation of its ﬁnal policy.
Revised: Our detailed analysis shows that our mechanism policy is consistently improved across iterations, and provides an interpretation of its ﬁnal policy.

Original: Artiﬁcial learning agents are becoming a centerpiece of our institutions, and as such methods to ensure that mechanisms are aligned to the values of their constituents are being heavily investigated.
Revised: Artiﬁcial learning agents are becoming a centerpiece of our institutions, and as such methods to ensure that mechanisms are aligned to the values of their constituents are being heavily investigated.

Original: The ideas and results presented here indicate that integrating the most basic democratic principle of one person one vote, with modern machine learning techniques is a viable and fruitful path forward.
Revised: The ideas and results presented here indicate that integrating the most basic democratic principle of one person one vote, with modern machine learning techniques is a viable and fruitful path forward.