Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Members
Anonymous Author(s)
AffiliationAddress email
Abstract
In many multi-agent settings, participants can form teams to achieve collectiveoutcomes that may far surpass their individual capabilities.
Measuring the relative contributions of agents and allocating them shares of the reward that promotelong-lasting cooperation are difficult tasks.
Cooperative game theory offers solution concepts identifying distribution schemes, such as the Shapley value, thatfairly reflect the contribution of individuals to the performance of the team or the
Core, which reduces the incentive of agents to abandon their team.
Applicationsof such methods include identifying influential features and sharing the costs ofjoint ventures or team formation.
Unfortunately, using these solutions requirestackling a computational barrier as they are hard to compute, even in restrictedsettings.
In this work, we show how cooperative game-theoretic solutions can bedistilled into a learned model that generalizes to previously unobserved games bytraining neural networks to propose fair and stable payoff allocations.
We showthat our approach creates models that can generalize to games far from the training distribution and can predict solutions for more players than observed duringtraining.
Finally, we show how this approach can be used as an approximate precomputation for speeding up explainable AI methods.
Introduction
The ability of individuals to form teams and collaborate is crucial to their performance in manyenvironments.
Moreover, the success of humans as a species hinges on our capability to cooperateat scale [1].
Similarly, cooperation between learning agents is necessary to achieve high performancein many environments [2, 3, 4, 5] and is a fundamental problem in artificial intelligence [6, 7].
Often,individual agents are not incentivized by the joint reward achieved by a team but rather by theirshare of the spoils.
Hence, teams are only likely to be formed when the overall gains obtained bythe team are appropriately distributed between its members.
However, understanding how collectiveoutcomes arise from subsets of locally interacting parts or fairly measuring the impact of individualson the team’s performance, remain open problems.
Direct applications exist in multiple domains.
One example is identifying the most influential features that drive a model to make a certain prediction [8, 9, 10, 11]; one of the cornerstones ofexplainable AI [12].
Another example is sharing the costs of data acquisition or a joint venture ina fair way between participants [13, 14], or sharing gains between cooperating agents [15, 16].
Inmany legislative bodies individual participants have different weights, and passing a decision requires support from a set of participants holding the majority of the weight; different states in the
US electoral college have different numbers of electors, and different countries in the EU Councilof Ministers vary in their voting weight.
In such situations we’d like to quantify the true politicalpower held by each participant, or allocate a common budget between them [17, 18].
Cooperative game theory can provide strong theoretical foundations underpinning such applicationsand defines solution concepts that determine the relative impact of individuals on team performance,or the individual rewards agents are entitled to.
Power indices such as the Shapley value [19] or Banzhaf index [20] attempt to divide the joint reward in a way that is fair , and have recently beenused to compute feature importance [9].
In contrast, other solutions such as the Core [21] attemptto offer a stable allocation of payoffs, where individual agents are incentivised to continue workingwith their team, rather than attempting to break away from their team in favor of working with otheragents.
Despite their theoretical appeal, these solution concepts are difficult to apply in practicedue to computational constraints.
Calculating these solutions, even in restricted environments, istypically a hard problem [22, 23, 24].
Our contribution: We construct models that predict fair or stable payoff allocations among teammembers, combining solution concepts from cooperative game theory with the predictive power ofneural networks.
These neural “payoff machines” take in a representation of the performance orreward achievable by different subsets of agents, and output a suggested payoff vector allocatingthe total reward between the agents.
By training the neural networks based on different cooperativesolution concepts, the model can be tuned to aim for a fair distribution of the payoffs (the Shapleyvalue or Banzhaf index) or to minimize the incentives of agents to abandon the team (the LeastCore [21, 25, 24]).
Figure 1 depicts the two well-studied classes of games on which we evaluateour approach: Weighted Voting Games [17, 18, 23] and feature importance games in explainable
AI [8, 9].
Weighted voting games (WVGs) are arguably the most well-studied class of cooperative games.
Each agent is endowed with a weight and a team achieves its goal if the sum of the weights of theteam members exceeds a certain threshold (quota).
WVGs are a model of decision making bodiessuch as the US Electoral College or the EU Council of Ministers, and solving them allows estimatingtrue political power [17, 18].
We train the neural networks by generating large sets of such gamesand computing their respective game theoretic solutions.
Our empirical evaluation shows that thepredictions for the various solutions (the Shapley value, Banzhaf index and Least-Core) accuratelyreflect the true game theoretic solutions on previously unobserved games.
Furthermore, the resultingmodel can generalize even to games that are very far from the training distribution or with moreplayers than the games in the training set.
Feature Importance Games are a model for quantifying the relative influence that various featureshave on driving a machine learning model to make a certain prediction [8, 9].
Solving these gamesfor the Shapley value provides an intuitive understanding of the key factors that drove a model toreach a specific decision.
This approach is model-agnostic, thus can be applied to make any “blackbox” model more interpretable [12].
One drawback of this approach is the computational complexity of calculating the Shapley value, making such analysis slow even when using approximationalgorithms.
We show how a neural network can serve as way of pre-computing Shapley values,measuring the relative importance of features for a given model.
After training, feature importancemeasures can be obtained instantly for each new input instance through a single forward pass onthe trained neural network.
Hence, our approximate pre-computation approach can significantlyspeed-up Explainable AI analysis.
Preliminaries
We provide a brief overview of cooperative game theory (examined in details in various books [26,23]) and discuss how solution concepts in cooperative game theory have been applied in Explainable
AI [8, 9, 27, 28, 29].
2.1 Cooperative Game Theory
A (transferable utility) cooperative game consists of a set N = { 1 , 2 , . . . , n } of agents, or players,and a characteristic function v : 2 n → R which maps each team of players, or coalition C ⊆ N ,to a real number.
This number indicates the joint reward the players obtained as a team.
Games vwhere v : 2 n → { 0 , 1 } (binary range) are simple games.
Weighted voting games (WVGs) are a restricted class of simple cooperative games [23], whereeach agent i has a weight w i and a team of agents C ⊆ N wins if the sum of the weights of itsparticipants (cid:80) ni
=1 w i exceeds a quota q .
Formally, a WVG is defined as the triple ( w , q, v ) withweights w = ( w 1 , w 2 , . . .
, w n ) ∈ R n ≥ 0 and quota (threshold) q ∈ R ≥ 0 where for any C ⊆
N we decision making political influence stable have v ( C )
= 1 if (cid:80)
ni
=1 w i ≥ q and otherwise v ( C ) = 0 .
If v ( C ) =
we say C is a losing coalition,and if v ( C ) =
we say it is a winning coalitions.
WVGs have been thoroughly investigated as amodel of voting bodies, such the US Electoral College or the EU Council of Ministers [17, 18].
The characteristic function defines the joint value of a coalition, but it does not specify how thevalue should be distributed among the agents.
Solution concepts attempt to determine an allocationp =
( p 1 , . . .
, p n ) of the utility v ( N ) achieved by the grand coalition of all the agents; an allocationp is called an imputation if for any player
i we have p i ≥
and (cid:80) ni =1 p i = v ( N ) .
This allocationis meant to achieve some desiderata, such as fairly reflecting the contributions of individual agents,or achieving stability in the sense that no subset of agents is incentivized to abandon the team andform a new team.
We describe three prominent solution concepts that we use in our analysis.
The Core.
Rational players may abandon the grand coalition of all the agents if they can increasetheir individual utility by doing so.
The Core is defined as the set of all payoff vectors where nosubset of agents can generate more utility, as measured by the characteristic function, than the totalpayoff they are currently awarded by the payoff vector.
As such, the Core is viewed as the set ofstable payoff allocations.
Formally, the core [21] is defined as the set of all imputations p such that [Table]
ni =1 p i = v ( N ) and that (cid:80) i ∈
C p
i ≥ v ( C ) for any coalition C ⊆ N .
The ε -Core and Least-Core [25, 24].
Some games have empty cores, meaning that no payoffallocations achieves full stability (i.e. for any imputation p there exists at least one coalition C suchthat v ( C ) > (cid:80) i ∈
C p i ).
In such cases, researchers have proposed minimizing the instability.
Arelaxation of the core is the ε -core , consisting of imputations p where for any value coalition C wehave p ( C ) ≥ v ( C ) − ε .
Given an imputation p the difference v ( C ) − (cid:80) i ∈
C p i is called the excessof the coalition, and represents the total improvement in utility the members of C can achieve byabandoning the grand coalition and working on their own.
For an imputation in the ε -core, no agentsubset C can achieve an addition of ε in utility over the current total payoff offered to the team (i.e.no coalition has an excess of more than ε ).
The minimal ε for which the ε -core is non-empty iscalled the Least-Core Value (LCV).
The Least-Core minimizes the incentive of any agent subset toabandon the grand coalition, and the LCV thus represents the degree of instability (excess) under theimputation that best minimizes this instability.
We find the set of payoffs associated with the LVCthrough linear programming (full details in Appendix C).
We now discuss two power indices , payoff distributions reflecting the true influence a player has onthe performance of the team, that fairly allocate the total gains of the teams among the agents in it.
The Shapley Value
[19] measures the average marginal contribution of each player across allpermutations of the players.
The Shapley value is the unique solution concept that fulfills several natural fairness axioms [30], and has thus found many applications from estimating feature importance [8, 9] to pruning neural networks [31, 32, 33].
Formally, we denote a permutation of theplayers by π , where π is a bijective mapping of N to itself, and the set of all such permutationsby Π .
By C π ( i ) we denote all players appearing before i in the permutation π .
The Shapley valueϕ i ( v ) of player i is defined as:
[Equation]
Intuitively, one can consider starting with an empty coalition and adding the players’ weights in theorder of the permutation; the first player whose addition meets or exceeds the quota is considered thepivotal player in the permutation.
The Shapley value then measures the proportion of permutationswhere a player is the pivotal player.
The Banzhaf index [20] is another method for distributing payoffs according to a players’ ability tochange the outcome of the game, but it reflects slightly different fairness axioms [34].
The Banzhafindex β i of a player i is defined as the marginal contribution of a player across all subsets notcontaining that player:
[Equation]
In practice, we first compute the set of winning coalitions C win ⊆ C and count, for each player, thenumber of times it is critical or pivotal , that is, v ( { C }\ i )
= 0 .
2.2 Explainable AI By Measuring Feature Importance Through Shapley Values
We briefly introduce the use of Shapley values in Explainable AI.
For a more detailed discussionsee [8, 9, 11, 12].
Shapley values allow quantifying the relative influence of features in driving amachine learning model to reaching its decision [12].
Through a cooperative game theory lens, onemay view the features of a data instance as players, working together in a team to drive a modeltowards its current prediction.
As such, we can measure the impact of a feature on the model’sprediction by considering all the possible coalitions (combinations) of features, each time replacingthe non-coalition features by their respective dataset mean (hence eliminating the unique informationconveyed by the value that feature has in that data instance).
Consider an instance with features( x 1 , . . . , x k ) resulting in a model prediction of f θ ( x 1 , . . .
, x k ) , and denote by µ j the dataset meanof feature j . Given a subset C of features, denote by x Cj the perturbed feature where x Cj = x jif j ∈ C and
where x Cj = µ j if j / ∈ C . We can cast the “value” of a feature team C as thefollowing characteristic function v ( C ) =
f θ ( x C 1 , x C 2 , . . . , x Cj ) .
Recently, convenient frameworksthat approximate Shapley values such as SHAP [9] spurred the rapid adoption of the Shapley valuewithin the explainable AI community [12].
Methods
As computing cooperative game-theoretic concept solutions is often intractable, we use deep learning approaches to efficiently approximate them. To evaluate our methods, we consider two domainsin our experiments.
The first domain is that of Weighted Voting Games (WVGs), where we useneural networks to measure the true influence of participants or to predict a stable payoff allocation. To train our models, we creates a dataset of randomly generated WVGs (with different distributionsof their parameters). We also assess the quality of our models on the out-of-sample distributions andtheir generalization to a larger number of players. The second domain is that of feature importance measures for explainable AI.
We show how toapply our method to the Shapley-based feature importance setting discussed in Section 2.2.
Thus,we can speed-up the computations of the relative impact of features on the predictions of a machinelearning model.
We do this by training our models to approximate Shapley values of features, andexamining their performance on previously unobserved instances.
3.1 Weighted Voting Games
We consider two types of models: fixed-size models that always predict a fixed number of outputsand can handle games with a fixed number of players, and variable-size ones that predict a variablenumber of outcomes and can handle variable numbers of players.
We now describe our data generation process, models, training procedure, and evaluation metrics used in weighted voting games.
3.1.1 Data and Models: Fixed-Size
For each n player game, we generate G independent and identically distributed games.
The trainingdataset is given by D n fixed = (cid:8) X ∈ R G × n , P ∈ R G × K (cid:9) where K denotes the number of outputs ofinterest.
The features are obtained in two steps.
First, we sample a weight vector w ∼ Beta ( α =1 , β = 1) on the interval from 1 to 2 n , and a quota q ∼ N ( µ = 14 (2 n + 1) n, σ 2 = 2 n ) .
To creategames where players are dependent on each other to achieve the task at hand, the quota distributionis parameterized such that the average drawn quota is half of the sum of the players’ weights.
Wethen normalize the weights by the quota to get the feature vector x = 1 q · w .
In other words, wehave n features which are the weights normalized as the respective proportion of the quota.
Thus, avalue x i > 1 indicates that player i is a winning coalition by itself, and needs other players to meetthe quota otherwise (Figure 2 on the left).
We train models to predict the three solutions: the Least-Core, the Shapley values, and the Banzhafindices.
For the Shapey values and the Banzhaf indices, the model predicts the payoff allocation( p 1 , . .
p n ) so K ≡ n .
For the Least-Core solution, we train the model to not only predict thepayoff allocation p 1 , . . .
, p n , but also to predict the Least Core Value ε min (so in this case the modelhas K = n + 1 outputs).
For each n player game, we produce a model f θ : R n → R K , where θ arethe model parameters.
We use deep feedforward networks for f θ .
Appendix D.1 contains the fulldetails about the experimental setup.
3.1.2 Data and Models: Variable-size
For the variable-size case, we consider a maximal number of possible players M , and for gameswith less than M players we pad the inputs with zeros.
Hence, we generate a single dataset D var = 192 (cid:8)
X ∈ R G × M , P ∈ R G × K (cid:9) .
Similarly to the fixed-size dataset, the feature matrix X contains thenormalized weights and P the corresponding solutions with either K
= M +1 or K = M , with theground truth output vector again padded with zeros when there are fewer than M players.
Hence,we allow for the prediction of up to M players, and we shuffle the data so that players are located atrandom positions.
The Least Core Value ε min is not shuffled and instead stored at the last elementof each row (Figure 2 on the right).
We produce one model g η : R n → R K that can be used for different number of players n , where ηare the model parameters.
The model learns to allocate the joint payoff among at most M players.
During prediction time we pad the input with zeros when there are fewer than M players, and weredistribute the payoffs allocated to non-player entries among the players according to their originalshare of the joint payoff (i.e. we ignore the outputs in the non-player positions, and renormalize theoutputs in the player positions so they sum up to 1).
3.1.3 Training and Evaluation
During training we minimize the mean square error between the true and predicted solutions (overthe payoffs of all players).
For the variable-size models we also include the M − n padded locationsso that the model learns not to allocate value to non-player entries.
Evaluation metrics.
Given a predicted payoff vector ˆ p
= (ˆ p 1 , ˆ p 2 , . . .
, ˆ p n ) , we consider multiplemetrics for assessing the models’ performance.
First, we quantify the models’ predictive performance via the Mean Absolute Error , defined for each game as MAE = 1 n (cid:80) ni =1 | p i − ˆ p i | , where nis the number of players.
For the Least-Core there is another natural game theoretic metric.
The goalof the Least-Core is to minimize the incentives of any subset to abandon the current team and formits own sub-team.
Given a suggested payoff vector ˆ p , the maximal excess v ( C ) − (cid:80) i ∈
C ˆ p i over all possible coalitions C measures the incentive to abandon the team, and serves as a good measure forthe quality of the model.
Test data.
The vector of weights and the quota together determine the set of winning coalitions ina WVG, which in turn establish the payoff allocation.
To assess our models’ ability to generalizeto previously unobserved instances and see how well they generalize on games far outside of thetraining distribution, we sample the weights w ∼ Beta ( α, β ) with varying parameters for α and βand the distribution location (full details in Table 2). A visualization of the five standardized testdistributions is found in Figure 9.
3.2 Explainable AI: Approximately Computing Feature Importance By Predicting ShapleyValues
As discussed in Section 2.2, the Shapley value offers a principled method for identifying the relative influence that various features have on a machine learning model in making its prediction ona given instance. Despite its theoretical appeal, computing Shapley-based feature importance measures is a computationally demanding process; for each analyzed data instance, directly applyingthe Shapley formula requires considering all permutations and thus it is exponential in the numberof features. Even for a moderate number of features and even when using approximations such asthose implemented in recent packages such as SHAP [9], the process is slow.
We show that neural networks can provide a faster alternative for measuring feature importance atscale.
A neural model for predicting Shapley values can be trained on a set of instances; followingtraining this neural model can quickly predict Shapley values for many unseen instances, only requiring a single forward pass to obtain approximations of Shapley feature importance measures.
Ourprocedure is to select a sampled sub-dataset X of M instances with K features each, and exactlycompute the Shapley value for each feature, yielding a matrix Φ . Then, we train a neural networkh ψ : R K → R K that predicts a Shapley value ϕ ij for each feature x ij (full details in Appendix D.2).
A data efficient model must exhibit good performance on the test set even with a relatively smallnumber of training samples. To study the sample complexity, we perform a simple experiment. First,we create an array of T = 999 linearly spaced train-test splits t = Each element t i in t indicates the ratio of samples that are used for training the neural network, withthe remaining (1 − t i ) N samples used for evaluation.
For each train-test partition in t then, we traina feed forward neural network on t i N samples by minimizing the MSE between the predicted andactual Shapley values (full details in Table 3) for 100 epochs, and evaluate on the 1 − t i N samples.
Results
We present experimental results that allow us to assess how well neural models are able to learna representation of the various solution concepts.
We first describe the predictive performance ofneural networks in the WVG setting, and consider properties of these solutions that make themhard to learn.
We then consider the explainable AI domain, and study the performance and samplecomplexity of Shapley feature importance prediction.
4.1 Weighted Voting Games: Evaluation
For our WVG analysis we train a selection of fixed-size neural networks f N θ ∗ for each number ofplayers n ∈ { 4 , 5 , . .
. , 19 , 20 } 2 on G = 5 , 000 games each.
We also train a single variable-sizemodel g η ∗ that is trained on one dataset containing G = 17 , 500 games in total, consisting of 2 , 500games for each number of players n ∈ { 4 , 5 , . . .
, 9 , 10 } games.
The data is padded with zeros toallow for payoff allocation up to M = 20 players (see details in Section 3.1).
4.1.1 Predictive Performance
Table 1 shows that the ability to handle games of variable numbers of players comes at a the costof having a lower accuracy.
However, even for variable-size models, and even under a significantdistribution shift, the errors in predicting all solution concepts are low.
We further note that the which becomes computationally very costly when there are many players.
Hence, for games with n = 9 players or more, we use Monte-Carlo approximations for the Shapley value to obtain the ground-truth details
Appendix error in predicting Least-Core based payoffs are generally larger than for the Shapley and Banzhafpower indices.
One possible reason is that for these solutions, there are many cases where a smallperturbation in the weights or quota results in a large pretubation of the Least-Core solution [35, 36].
[Table]
We now highlight several conclusions from our analysis of the results.
Fixed-size models display stable performance across solution concepts.
Our first observationis that the fixed-size neural networks are adept at estimating solutions across the three consideredconcepts.
The average error per player ranges from 0 .
005 to 0 .
087 (Least Core), 0 .
to 0 .
069(Shapley values) and 0 .
012 to 0 .
067 (Banzhaf indices).
Table 5 provides a complete summary ofour models’ in-sample performance.
Fixed-size models are robust to shifts in the weight distribution.
As shown in Figure 3, thepredictive performance (Mean MAE) of the fixed-size models is consistent across test datasets.
Toaccount for the natural decrease in the MAE as N increases, we also display the average payoff perplayer ( 1 /N ).
As expected, the error scales approximately with the average payoff per player.
[Figure]
Variable-size models are robust to shifts in the weight distribution.
Figure 4 shows that thevariable-size models are also able to generalize outside the training distribution.
Across test sets, weobserve a stable performance that decays with the number of players, as is expected.
For the Least
Core, we see that there is an abrupt decrease in performance for the excess beyond 8 players.
[Figure]
Variable-size models generalize to a larger number of players.
Our main objective is to investigate how to leverage machine learning to perform scalable value estimation in large multi-agentsettings.
To test this, we train the variable-size models on games up n = 10 players, and evaluate on n + 1 , n + 2 , . . . , n + 10 players (full details in section 3.1.2).
Interestingly, Figure 4 demonstratesthat there is no significant decrease in performance for games with more than n = 10 players acrosssolution concepts: variable-size models are able to extrapolate beyond the number of players seenduring training. This suggests that there is be valuable information in the small player games suchthat games of larger sizes can be inferred.
Fixed-size models outperform variable-size models.
Table 1 contains the Mean MAEs acrosssolution concepts and test sets.
Corresponding error distributions are showed in Figure 5.
Fromthese, we conclude that in almost all settings the fixed-size networks outperform the variable-sizenetworks.
Variable-sized networks tend to have larger errors for all predicted variables and displaymore variance in their predictions.
[Figure]
4.1.2 Discontinuities in the Solution Space
One obstacle that makes it difficult to learn the function mapping from game parameters to theirrespective solutions is the fact that there are many discontinuities.
For various parameters, such asthe quota of the WVG, an infinitesimally small perturbation in the parameter results in a large shiftin the solution.
We now examine several examples of such discontinuities and see how the neuralnetwork models attempt to capture them.
Some solution concepts are step-wise constant functions which are difficult to capture.
Consider taking a WVG and changing the weight of a player.
This only changes the game theoreticsolutions when it restructures the subset of winning coalitions.
This means that the function outputsthe same value until the weight reaches a certain threshold where the structures of winning coalitions change, at which point the solution can change drastically.
Learning these kind of functions isdifficult, as the error around the discontinuity point is often large.
We inspect several specific WVGsto get see how the trained models capture these discontinuous jumps.
Our analysis examines an n player game with a fixed weight vector w = ( w 1 , w 2 , . . .
, w n ) andconsiders an array of quotas q = ( q 1 , q 2 , . . .
, q K ) =
( min ( w ) , min ( w ) + ϵ, . . .
, (cid:80) ni
=1 w i ) , whereϵ = 0 .
We solve the game for each combination of the fixed weights and the changing quota in qto obtain a matrix P .
Figure 6 shows the fixed-size model predictions for two selected WVGs.
Themodels capture the overall effect of the changes to the quota, but do poorly close to the discontinuitypoint (where the ground truth solution incurs a large change).
4.2 Speeding Up Feature Importance Calculations By Predicting Shapley Values
Following the experimental procedure outlined in Section 3.2, we examine the sample efficiency andperformance of neural networks trained to predict Shapley values on sets of features.
We consider two well-known datasets: the classical UCI Bank Marketing dataset [37] which has 15 differentindicators with over 11,162 observations, and the Melbourne Housing dataset [38], which contains20 features and 34,857 observations (full details in Appendix D.2).
Overall, we find that neuralnetworks achieve high performance on the test set with very few training samples.
Figure 7 displays the models’ performance (in RMSE) as a function of samples available for training.
We observe that the error decays approximately exponentially with the proportion of data usedfor training on both the Banking and Melbourne datasets.
A model trained on just 1 percent (349samples) of the Melbourne dataset has an RMSE of 19.72 on the resultant test set (34,508 samples).
Increasing the number of training samples to 3 percent (1045 samples) results in a RMSE of lessthan 0 . 07 (on 33,812 samples), a 99.6 percent decrease in error.
We observe a similar trend forthe Banking dataset: the RMSE decreases with 93.8 percent (from 4 . 10 to 0 . 25 ) when the availabletraining data increases from 0.5 to 4 percent.
For both datasets, the error stabilizes after approximately 5 percent of the data is available for training.
These results show that a very small portion ofthe instances is required for the neural model to generalize well to unseen instances.
These resultshighlight the strength of this approach: the computational cost of training the Shapley predictionnetwork is very small as compared to the speedup obtained on the vast majority of the data (and anysubsequent instances analyzed later).
Discussion and Conclusion
We considered “neural payoff machines”, models that take in a representation of a cooperative gameand solve it to produce a distribution of the joint gains of a team amongst its members.
Our analysisfocused on two concise representations of a characteristic function:
Weighted Voting Games, and Feature Influence Games for explainable AI.
Our analysis shows that neural models capture cooperative game solutions well, and can generalize well outside the training distribution, even extendingto more players than in the training set.
Given the larger MAE for the Least-Core, we believe that this is a harder solution to learn than the Shapley value and Banzhaf index.
Potentially, the fact that the Least core is complex and can onlybe computed by solving multiple Linear Programs makes a hard concept to grasp from examples.
The methods we discussed can drive better analysis of decision making bodies and for approximatingfeature importance, but such methods can also be applied to other classes of cooperative games, solong as one can generate data consisting of game samples and their solutions.
For instance, the sameapproach can be used for other cooperative representations, such as graph-based games [24, 39, 40]or set-based representations [41].
The direct applications that we have considered in this paper arealso intrinsically valuable on their own, as we allow speeding up explainable AI analysis and politicalinfluence estimation.
Some questions remain open for future work.
Can similar techniques be usedfor non-cooperative games or for other types of cooperative games?
Can this approach be appliedto other solutions such as the Kernel or Nucleolous [23]?
Finally, are there better neural networkdesigns, such as graph neural networks or transformers, that can exploit invariances or equivariancesin the game’s representation or be better equipped to deal with sharp discontinuities in the solutions?
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] The abstract and introduction provide representative
summaries of our work.
(b) Did you describe the limitations of your work?
[Yes] See section 4.1.2 and the Dis-
cussion. (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See methods and Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[Yes] See Appendix G.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
AppendixA Notation
The following notation is used throughout this paper:
• n the number of players in the game
• G the number of games
• M the maximum sequence length in variable feedforward networks• q the quota, threshold that determines when a coalition can complete the goal or task• w i the weight of a player i• x i =
w i /q , the normalized weight of a player i• N the grand coalition• C a coalition (subset of players)• C min the minimal set of winning coalitions: the coalitions where each player is pivotal• X feature matrix• P solution matrix• p i the allocated reward or payoff to a player i• ε the least core value (LCV), i.e. joint payoff to sacrifice such that there exists a feasible payoffvector in the least core
B Motivating The Use Of The Shapley Value To Analyze Influence InWeighted Voting Games
We provide an intuitive example motivating the use of the Shapley value for measuring the trueinfluence of participants in weighted voting games or political power in decision making bodies.
Consider a decision making body such as a parliament with 100 seats, and imagine that followingthe elections we have two large parties of 49 seats each, and one tiny party who has only managedto get 2 seats.
However, note that having a majority means controlling a majority of the seats, i.e.more than 50 seats.
This means that neither of the two big parties has a majority on its own (and ofcourse the small party hasn’t got a majority on its own).
Hence, to achieve a required majority and form a government or pass a bill, the parties have to workin teams.
Clearly, a coalition consisting of the two big parties has a majority, as together they have49+49=98 seats out of the total of 100.
However, a coalition consisting of any of the two big partiesand the one small party also has 49+2=51 seats, which is also a majority.
Hence, in the setting discussed above, any two parties form a winning coalition that has a majorityand is able to form a government or pass a bill.
This can be modelled as a weighted voting game,where the weights are ( w 1 = 49 , w 2 = 49 , w 3 = 2) and the quota is q = 50 .
Intuitively, as any two parties or more are a winning team and as any single party is a losing team,we might say that the weights are irrelevant.
Although the big parties each have almost 25 times theseats as the small party, they all are symmetric in their opportunities to form a winning coalition,and one could claim they actually have equal political power.
Clearly, the small party is in a verystrong negotiation position, and could demand for instance to control a significant proportion of thejoint budget.
In the above weighted voting game, the Shapley value of all agents would be equal (each getting avalue of 13 ), as it considers the marginal contributions of the parties over all permutations of players.
Due to the symmetry between the parties, they have identical marginal contributions.
The above specific example illustrates some of the principles behind the fairness axioms that
the
Shapley value exhibits, and illustrates its importance for measuring influence in decision makingbodies.
C Algorithm For Generating Least Core Solutions
The least core solutions can be generated by considering all possible coalitions.
A simple improvement, however, is to use the minimal set of winning coalitions [42] instead.
In this section, wedescribe the idea behind this approach and the algorithm used in this paper.
Let N ≡ { 1 , 2 , . .
. , n } denote the set of all N ∈ Z ≥ 0 players, let q ∈ R ≥ 0 denote the quota andlet w i ∈ R ≥ 0 denote the weights for player i , for all
i ∈ N .
The characteristic function is given byv ( C ) ∈ [0 , 1] , for coalitions C ⊆ N .
The naive approach to find the least core solutions is by considering all winning coalitions C windefined as
[Figure]
[Figure]
[Figure]
and to find a least core solution by solving LP-problem 8a.
In LP-problem (8a), all winning coalimin p 1 ,p 2 ,..., [Figure] n ,
[Figure]
551 tions are considered and there could be sub-coalitions in winning coalitions that are still winningand could deviate from the coalition by forming a smaller coalition.
This yields a higher payofffor the players in the sub-coalition and is hence a Pareto improvement for the ”decision-makers”.
Therefore, we can instead only consider the subset of minimal winning coalitions, given by
C min ≡   C ⊆ N (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
C ̸ = ∅ , (cid:88) i ∈
S w i ≥ q, and, for all j ∈ C, (cid:88) i ∈
C \{ j } w i <
q   (C.2) and solve the more efficient LP-problem 8b.
D Experimental details
In this section, we elaborate on the experimental procedures, model architectures and hyperparameters used in our experiments.
Code and notebook.
Following the guidelines, we provide python code to reproduce our results,a pedagogical notebook (called CooperativeGameTheoryPrimer.ipynb) that introduces the most important game theory formalisms and the necessary data in the accompanied zip file.
D.1 Weighted Voting Games
Optimization procedure fixed-size feedforward networks.
For each N -player dataset D N fixed weperform R =
independent runs with a maximum of E = 6000 epochs and an early-stoppingcriteria which breaks a run if, after a baseline of 500 epochs, the validation loss does not improvefor 75 consecutive epochs.
For each run, 70 percent is allocated for training and 30 percent forvalidation.
After training, we select the model with the best validation loss.
The selected modelsare stored and used for evaluation.
In all experiments we use the Adam [43] optimizer, ReLU [44]activation functions for the hidden layers, Softmax functions for output layer of the payoffs and a
Sigmoid for the output layer of epsilon.
Optimization procedure var-size feedforward network.
We perform R = 100 independent runsfor one variable dataset D var with a maximum of E = 15000 number of epochs and an early-stoppingcriteria which breaks a run if, after a baseline of 500 epochs, the validation loss does not improvefor 75 consecutive epochs.
For each run, 70 percent is allocated for training and 30 percent forvalidation.
After training, we select the model with the best validation loss, which is stored and used for evaluation.
In all experiments we use the Adam [43] optimizer, ReLU [44] activation functionsfor the hidden layers, Softmax functions for output layer of the payoffs and a Sigmoid for the outputlayer of epsilon.
Hyperparameters.
All results are generated with multi-layer perceptrons (feedforward networks)of varying hidden-layer size and width.
Further hyperparameters are the dropout rate, weight decayin the Adam optimizer, and the learning rate.
Our standard model architectures are depicted in
Figure 2 Right.
Approximated ground-truth Shapley values.
To obtain the ground-truth solutions for Shapleyvalues above nine players, we perform Monte-Carlo sampling to obtain a subset of all possible permutations.
For each of the R = 10 resamples we randomly sample P = 1000 permutations andaverage across the resamples to obtain approximations of the ground-truth.
To validate that the approximated labels are representative we compute the Mean MAE between the true and approximated
Shapley values for n = 5 up to n = 10 players with 1000 games per n -player game.
Across alln -player games the mean MAE between the true and approximated Shapley values was 0 .
0011 andnever exceeded 0 . 0014 .
Figure 10 shows the distributions per player together with the true label (reddot) for 8 and 9 player games.
Parameterizations for test datasets.
Figure 9 shows the five standardized test distributions thatare generated with the parameters in Table 2.
[Table]
Out-of-sample
[Table]
Slightly out-of-distribution
[Table]
Moderately out-of-distribution
[Table]
Considerably out-of-distribution
[Figure]
D.2 Explainable AI
To train a neural network to predict the relative importance of each feature in a dataset, our procedureis as follows:
Select a dataset.
The first step is to select a dataset of choice.
This can be a set of imagesor a set of features.
In our example, we end up with a feature matrix X N × F where each rowis an observation and each column a feature, and a target vector y ∈ R N or [0 , 1] dependingon the task.
Feature to target.
Following, we estimate a model f θ : R F
→ R that predicts one outputfor each set of features.3.
Compute Shapley values.
We use the trained model from step 2 to obtain Shapley valueswith the SHAP package [9], yielding a matrix Φ ∈ R N × F .4.
Feature to Shapley.
Finally, we train a model h ψ : R F → R F that estimates a Shapleyvalue ϕ nf for each feature x nf .
Datasets and preprocessing.
Table 4 summarizes the considered datasets.
The UCI Bankingdataset contains 15 features and a binary target (whether the client has subscribed a term deposit),whereas the Melbourne housing dataset contains 21 features and a regression target (the price ofa house).
We exclude the “duration” feature in the Banking dataset to prevent data leakage, as isrecommended in the documentation.
Moreover, we exclude the “Rooms” feature as there is a lotof overlap in this and the “Bedroom2” feature.
The numerical values in both datasets are z-scored,and categorical features are encoded as numbers.
Missing values are encoded as zeros, so that thecomplete dataset can be used for our experiment.
Feature to target.
We partition the dataset into train and test and train a Random Forest Classification model on the UCI Banking set, obtaining a final test accuracy of 72 %.
Moreover, we train adecision tree regression model to predict the price given the set of house features and obtain a final
MSE of 1.21 on the test dataset.
Compute Shapley values.
We use the trained models to obtain Shapley values for each instancein the feature dataset X using the SHAP kernel explainer object [9].
Feature to Shapley: procedure.
Finally, we can train a neural network using the datasets X , Φ .
To investigate the sample complexity required to learn a representative mapping from features to
Shapley values, we conduct the following experiment.
First, we create an array of T = 999 linearlyspaced training splits t
= (0 . 001 , 0 .
Each element in t i in t indicates theratio of samples that are used for training the neural network, leaving the remaining (1 − t i ) Nsamples for evaluation.
For each train/test partition in t then, train a feed forward neural networkon t i N samples through minimizing the MSE between the predicted and actual Shapley values(architecture and hyperparameters specified in Table 3) for 100 epochs, and evaluate on the 1 − t i Nsamples.
Figure 7 depicts the resulting RMSE for both datasets as a function of the training fraction.
[Table] [Table]
E Additional Results
In-sample predictive performance.
For completeness, we provide the MAEs for each solutionconcepts on the in sample dataset up to n = 15 players in Table 5.
[Table]
Capturing step-wise jumps: full description.
We provide a more detailed description to Figure6.
In the Figure on the left , each player is a winning coalition by itself when q = 1 .
Up to q = 3 ,player 3 is more critical than players 1 and 2 and receives a larger share of the joint payoffs.
Allplayers obtain equal payoffs when the only winning coalition is the grand coalition.
On the rightside: each player is a winning coalition by itself when q = 1 , which results in an equal payoff of0 .
2 for each player.
Increasing the quota continuously alters the set of winning coalitions, therebythe relative importance of each player.
At q = 3 players 1, 2 and 3 have become less critical andobtain a payoff of 0 .
18 each compared to 0 .
22 for player 4 and 5.
Finally, when q = (cid:80) Ni =
1 w i = 7 ,the only winning coalition that can be formed is the grand coalition: each player is required and thejoint payoff is divided equally.
F Hardware details
We trained our neural networks on an internal compute cluster with 10 Quadro RTX 6000 GPUs andtwo Intel(R)
Xeon(R)
Gold 5218 CPUs, each of which has 16 physical cores (32 logical processors).
G Asset licensing
Two main assets were used in this work:
• The UCI Banking dataset
• The Melbourne Housing dataset
The Melbourne Housing dataset was released under the license CC BY-NC-SA 4.0.
The UCI Banking dataset is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.
[-All together, this procedure takes roughly 
-]