Inverse Game Theory for Stackelberg Games: the Blessing of Bounded Rationality
Abstract
Optimizing strategic decisions (a.k.a.
computing equilibrium) is key to the success of many non-cooperative multi-agent applications.
However, in many real-world situations, we may face the exact opposite of this game-theoretic problem — instead of prescribing equilibrium of a given game, we may directly observe the agents’ equilibrium behaviors but want to infer the underlying parameters of an unknown game.
This research question, also known as inverse game theory , has been studied in multiple recent works in the context of Stackelberg games.
Unfortunately, existing works exhibit quite negative results, showing statistical hardness [27, 37] and computational hardness [24, 25, 26], assuming follower’s perfectly rational behaviors.
Our work relaxes the perfect rationality agent assumption to the classic quantal response model, a more realistic behavior model of bounded rationality.
Interestingly, we show that the smooth property brought by such bounded rationality model actually leads to provably more efﬁcient learning of the follower utility parameters in general Stackelberg games.
Systematic empirical experiments on synthesized games conﬁrm our theoretical results and further suggest its robustness beyond the strict quantal response model.
Introduction
One primary objective of game theory is to predict the behaviors of agents through equilibrium concepts in a given game.
In practice, however, we may observe some equilibrium behaviors of agents but the game itself turns out to be unknown.
For example, an online shopping platform can observe the shoppers’ purchase decisions on different sale prices, but the platform has limited knowledge of the exact utilities of the shoppers.
Similarly, while the policymaker could observe the market reactions to its policy announcement, the exact motives behind traders’ reactions are usually unclear.
In various security domains, the defender may want to understand the intentions or incentives of the attackers from their responses to different defense strategies so as to improve her future defense strategy.
As such, recovering the underlying game parameters would not only lead us to better strategic decisions, but also improve our explications of the motives and rationale in the dark.
These potentials and prospects motivate a class of research problems known as the inverse game theory [26]: given the agents’ equilibrium behaviors, what are possible utilities that induce these behaviors?
In this paper, we speciﬁcally target the sequential game setting from the perspective of the ﬁrst-moving agent (e.g., Internet platform, policymaker, or defender) whose different strategies (e.g., price, regulation, or defense scheme) would induce different equilibrium behaviors of the following agent (e.g., Internet users, traders, or attacker).
Studies of such game settings have seen broad impacts and extensive applications ranging from the principal-agent problems in contract design [21, 19], the AI Economist [43] to security games modeled for social good [16].
We formalize our problem under the normal form Stackelberg game, where a leader has the commitment power of a randomized strategy, and a follower accordingly decides his response.
It is known that the optimal commitment of the leader can be efﬁciently computed in a single linear program, given full knowledge of the game [14].
However, the inverse learning problem to determine the underlying game from the follower’s responses is more challenging: Letchford et al.
[27], Peng et al.
[37] show that learning optimal leader strategy from the follower’s best responses requires number of samples that is a high-degree polynomial in the game size and may be exponential in the worst cases.
This signiﬁcantly limits the practicality of these algorithms, as the leader usually cannot afford the time or cost to gather feedback from so many interactions.
More concerning is the inconvenient reality that we can hardly expect the agents’ optimal equilibrium responses assumed in existing work.
In fact, these shoppers, traders, or attackers themselves hardly know their exact utilities and are naturally unable to determine the expected-utility maximizing strategy.
Extensive studies of behavioral economics and psychology [23, 5, 32, 11, 10] have pinpointed the cognitive limitations that make human decisions prone to the noisy perception of their utilities.
Among various models for quantifying irrational agent behaviors, one of the most popular ones is perhaps the quantal response (QR) model [32], which adopts the well-known logit choice model to capture agents’ probabilistic selection of actions.
This will also be the bounded rationality model of our focus in this paper.
The Blessing of Bounded Rationality.
The key insight revealed from this paper is that the extra layer of behavioral complexity due to bounded rationality, while complicating the modeling and computation, provides a more informative source for us to learn the underlying utility of agents.
To understand the intuitions and motivations behind our results, consider a case where the follower has a dominated action j 1 as shown in Table 1, where the leader’s and follower’s utility of action proﬁle ( i, j ) is speciﬁed by u i,j , v i,j respectively.
Conventionally, such an instance is treated as a degenerated instance, because the leader could ignore the action j 1 that a perfectly rational follower would never play.
Then, the optimal leader strategy is clearly to always play the action i 2 .
However, when facing a
[Table]
i,j , v i,j
[Table]
boundedly rational follower, it becomes possible to observe the response j 1 and estimate the utilities regarding this dominated action.
For example, if the follower plays his action j 1 and j 2 at almost the same frequency, the follower’s expected utility on the two actions should be close.
Although such dominated action has no effect on the leader’s optimal strategy against a perfectly rational follower, it could be a potentially damaging (or beneﬁcial) action that leader want to avoid (or encourage) a bounded rational follower to play.
That is, in the above game instance, if a somewhat irrational follower plays action j 1 , it would be dangerous for the leader to play action i 2 yet rewarding to play action i 1 ; therefore, a more robust leader strategy should randomize by assigning some probability to play action i 1 .
We remark that in general, even without such extreme case of dominated actions, the extra payoff information is now available on how much worse (or better) it is to use the empirical frequency of the boundedly rational action responses (as long as some smoothness properties are exhibited), which are overlooked under the assumption of perfectly rational followers.
Our Results.
We present a set of tight analysis on the number of strategies and sample complexity sufﬁcient and necessary to learn the follower’s utility, for both situations in which the leader can observe the follower’s full mixed strategies or only the follower’s sampled pure strategies.
In the former situation of observing follower’s mixed strategies, our algorithm can recover the follower utility parameters using m follower mixed strategy responses in any general Stackelberg game where m is the number of leader actions.
Surprisingly, the required number of queries is independent of follower actions!
This is due to the fact that the randomness introduced by bounded rationality carries much more information about follower payoffs, compared to the perfect best response.
In the later (more realistic) situation of only observing follower ’s sampled pure strategy, our algorithm learns the follower utility parameters within precision (cid:15) with probability at least δ using Θ( m log( mn/δ ) ρ(cid:15)) carefully chosen queries, where n is the number of follower actions and ρ depends on agent’s bounded rationality level and is of order Θ(1 /n ) for typical boundedly rational agents.
Interestingly, the additional challenge of only observing sampled actions only deteriorates the sample complexity by a factor of log( mn ) /ρ .
These sample completexity results should be compared with that of [37, 27], which study similar learning questions but from perfectly rational follower responses.
The m log( mn ) /ρ order in our sample complexity is in sharp contrast to their complexity with exponential dependence in m or n in the worst case.
Our experimental results empirically conﬁrm the tightness of our sample complexity analysis.
At the conceptual level, our work illustrates that noises due to bounded rational behaviors could be leveraged as additional information sources to learn the follower utility.
This intuition also drives the design of our analytical tools to explain how efﬁcient and effective learning of the follower’s utility is possible in real situations, in complementing the previous negative results developed under the idealized perfect rational behavior models [27, 37].
Related Work
Learning in Stackelberg Game.
The learning problem in sequential games has been studied in several different setups.
Marecki et al.
[30], Balcan et al.
[7] consider the online learning problem in the Stackelberg security game with adversarially chosen follower types.
Bai et al.
[6] consider a bandit learning setting where one could query any entry of the followers’ utility under noise and use the estimation of utility to approximate the optimal leader strategy; however, this learning process assumes centralization , that is, the learner can control both leader’s and follower’s actions.
More similar to ours is the strategic learning setup in Stackelberg games studied by [27, 37, 9], where the leader adaptively chooses her strategies based on the observation of the follower’s best response and eventually recovers the follower’s utility up to some precision level.
Bounded Rationality.
McKelvey and Palfrey [32] introduced the quantal response equilibrium (QRE) by adopting the logit choice model [15, 31].
QRE serves as a strict generalization of Nash equilibrium (NE) — when the agents become perfectly rational, QRE converges to the NE.
The modeling success of QR model attributes to the nice mathematical and statistical properties of the logit function that can capture a variety of boundedly rational behaviors under different parameter λ .
QRE is widely adopted especially in Stackelberg (security) games [42, 35, 39, 16, 20, 12] and zero-sum games [29] and notably has been deployed in various real world application [2, 17].
Moreover, the model structure of QR has been also used in various other contexts, such as the softmax activation in neural network [18], multinomial logistic regression [8] and the multiplicative weight update algorithm for no-regret learning [3].
As an initial attempt to our general learning problem, we also adopt the QR model to capture our agent’s bounded rational behavior, for its modeling success in practice and being the most common choice of prior work [42, 35, 16, 20, 12, 29, 2, 17].
We acknowledge that there exist other models of bounded rational behaviors beyond the QR model.
For example, Kahneman [23] introduced the prospect theory to model the bounded rationality of agents in games under risk; Camerer et al. [11] proposed the cognitive hierarchy theory that classiﬁes the agents according to their degree of reasoning in forming expectations of others.
We anticipate that the message of our paper — i.e., the observation of suboptimal responses could provide additional information to learn the follower’s preferences — would apply to many of these bounded rationality models.
Inverse Game Theory.
Vorobeychik et al.
[40] considered the payoff function learning problem using the strategy proﬁles and the corresponding utilities through regression.
Kuleshov and Schrijvers [26] introduced the concept of inverse game theory, and the authors showed that the problem of computing the agents’ utilities from a set of correlated equilibrium is NP-Hard, unless the game is known to have special structures.
More recently, the inverse game theory problem is studied under the QR model and leads to a few positive results: Sinha et al. [39] considers the ofﬂine PAC-learning setup where the follower responses can be predicted with small error for a ﬁxed leader strategy distribution; Haghtalab et al.
[20] proves only three strategies are sufﬁcient to recover linear follower payoff functions in security games; Ling et al.
[29] presents an end-to-end learning framework that learns the zero-sum game payoff from its QRE.
Following their success, our paper is the ﬁrst work that provides theoretical guarantee of payoff recovery in general Stackelberg game.
Finally, inverse problems have received signiﬁcantly more attention in single-agent decision making problems; The most notable problem is the inverse reinforcement learning pioneered by Ng et al.
[34], Abbeel and Ng [1].
Problem Formulation
Game Setup We consider the Stackelberg game between a single leader (she) and follower (he).
We let U ∈ R m × n (resp.
V ∈ R m × n ) be the leader (resp.
follower’s) utility matrix, where m, n are the number of actions for the leader (resp.
follower).
We use G ( U, V ) to denote the game instance.
Each entry u i,j (resp.
v i,j ) of the utility matrix denotes the leader’s utility (resp.
follower’s utility) when leader plays action i and follower plays action j .
Without loss of generality, let u i,j , v
i,j ∈ [0 , 1] .
Let V j ∈ R m be the j th column of the matrix V .
We denote the set of the leader’s (resp.
follower’s) action set by [ m ] := { 1 , . . .
, m } (resp.
[ n ] := { 1 , . . . , n } ).
In this sequential game, the leader moves ﬁrst by committing to a (possibly randomized) strategy, x = ( x 1 , · · · , x m ) ∈
∆ m , where the simplex ∆ m = { x : (cid:80) i ∈ [ m ] x i = 1 and 0 ≤ x i ≤ 1 } and each x i represents the probability the leader playing action i .
Similarly, let ∆ n denote the follower’s strategy space.
Under perfect rationality, given the leader’s committed strategy, the follower would in turns chooses the best response action j ∗ that maximizes his utility, i.e., j ∗ = argmax j ∈ [ n ] { x (cid:62) V j } .
In our problem, we use the QR model instead to capture the follower’s bounded rational behavior.
That is, the follower would respond to the leader’s committed strategy by choosing an strategy y ∗ that maximizes his utility up to a Gibbs entropic regularizer, i.e., y ∗ = argmax y ∈ ∆ n { λ x (cid:62) V y − y ln y } .
This is shown to be equivalent to the setting where the follower is best responding according to the payoff perturbed by noises from a Gumbel distribution [22].
And we know the close form solution of follower’s optimal strategy for this convex optimization program is exactly the logit choice model on the true payoff, i.e., for each j ∈ [ n ] , y ∗ j =
[Equation]
We refer to λ as the bounded rationality constant that is given in each speciﬁc problem, as several existing work have already determined its empirical value in practice: the human behavior experiments in [38, 41] compute λ = 7 . 6 ; the experiments [28, 36, 32] show λ is in the range of 4 to 16 .
Learning Problem We consider the inverse game theory problem in sequential game with unknown follower utility and seek to quantify how much the leader can learn about a bounded rational follower’s utility.
We frame this problem under an active and strategic learning setup, where the leader can interactively choose a randomized strategy and observe follower’s strategic responses.
Speciﬁcally, at each round t ∈ [ T ] , the leader commits to a strategy x ( t ) .
The follower observes the committed x ( t ) and responds based on the QR strategy y ( t ) .
Below we will consider both feedback settings based on whether the leader is able to observe the exact distribution y ( t ) or merely its samples.
We set our primary learning objective as to recover a full characterization of the follower’s utility; our results below shall explain how it is unnecessary and almost unrealistic to expect an exact recovery of the follower’s utility.
And we show in Observation 1 and Theorem 1 that such utility characterization can be used to compute the optimal leader strategy under both perfect rationality, known as the strong Stackelberg equilibrium (SSE), and bounded rationality, known as the quantal Stackelberg equilibrium (QSE).
And besides developing the optimal (or robust) leader strategies, we believe the recovered utilities are generally useful for our better understanding and reasoning of the followers’ motives.
However, given the limited scope of the paper, we focus on the inverse game theory problems and defer the problems regarding how to strategize using the knowledge of game (i.e., the typical game-theoretical problems) to related and future work. Such learning problem has been considered in [20] speciﬁcally for Stackelberg security games, where the payoff is a strictly simpliﬁed single-dimensional linear utility function. Our paper overcomes the curse of dimensionality and answers the open question in recovering payoffs in the general Stackelberg game.
On the other hand, Sinha et al. [39] showed a case of learning the nonparametric Lipschitz function (which includes the payoff function in the general Stackelberg game as a special case) in PAC-learning setup and they obtained a sample complexity exponential to the number of actions. Notably, the PAC-learning problem is fundamentally different from our active learning problem, as its learning guarantee is tied to the given data distribution and is not guaranteed to recover the follower’s payoff.
Theoretical Results 4.1 Warm-up: Learning from Mixed Strategies
As a warm-up, we ﬁrst consider a rather ideal case where the leader can directly observe the follower’s mixed strategy y ( t ) .
In this case, it turns out that the leader would be able to perfectly recover the follower’s payoff matrix from his responses to m different strategies and thereby determine the her optimal strategy.
Despite a seemingly intuitive result, its underlying rationale is actually not as straightforward.
Speciﬁcally, many would raise the following doubt: the logit transformation is not bijective and thus its inverse mapping is not injective; in particular, it only gives us a system of at most n − 1 different linear equations w.r.t.
the follower’s utility; one can check that if we add a constant to all entries of the utility matrix, the resulting probability distribution stays the same after the logit transformation.
Thus, it should require more than m such linear equation systems to recover a utility matrix with m × n unknown parameters.
However, thanks to Observation 1, it happens that the follower’s utility matrix can be fully characterized by m × ( n − 1) parameters that is essentially the difference of each column in the utility matrix.
This somewhat coincidentally compensates the missing information on follower utility due to the logit transformation.
Knowing that m strategies is the lower bound of this learning problem in general, below we will explicitly construct a learning algorithm that have the matching upper bound.
To begin, a useful game-theoretic property of Stackelberg games is the following observation about the class of follower utilities that will induce the same leader and follower policies.
While similar observation has been made in [20, 39], we also provide a formal proof in Appendix A for completeness.
Observation 1 (Equilibrium Invariance under Payoff Transformation) .
For any (cid:101) V ∈ { V + c ⊗ 1 n | c ∈ R m } , i.e., a row-wise shifted matrix of V , the follower’s quantal response (resp.
best response) policy to leader’s committed strategy remains the same, and thus the optimal leader strategy in SSE or QSE remains the same.
Observation 1 suggests that the row-wise shifted payoff matrix is just as good as the ground-truth payoff matrix in our setting.
This essentially means that only the difference between action payoffs matters for the follower’s policy.
As such, we introduce a row-wise distance metric that accommodates such policy-invariant transformation to empirically measure the quality of the recovered follower utility.
Deﬁnition 1 (Logit Distance) .
We deﬁne a logit distance between the ground truth follower utility
V and the recovered follower utility (cid:101)
[Equation]
Whenever the distance Φ( V, (cid:101) V ) = 0 , we say that the recovered follower utility is perfect.
We next present a result that generalizes the well-known result, three strategies to success in security games , by Haghtalab et al.
Notably, we identify a simple but fundamental condition (in terms of rank) necessary to recover the game payoffs, rather than the special distance conditions tailored to the structure of the security game as in [20]. The notion of rank has a clear physical meaning and we would later follow this theoretical insights to design learning algorithm to actively select leader strategies to query.
Proposition 1 ( m Strategies to Success) .
There exists a learning algorithm that can always perfectly recover the follower strategy from m queries of the follower’s mixed strategies.
Proof Sketch.
We pick m linearly independent basis vectors for each x ( t ) in m rounds and argue that the following optimization program can perfectly recover the follower’s utility matrix (cid:101) V .
[Equation]
We can see that the objective of the optimization program is a log-sum-exp function w.r.t.
variables { z ( t ) } t ∈ [ m ] , which is convex.
This means we can determine its minimizer set of { z ( t ) } t ∈ [ m ] .
Meanwhile, the constraints of the optimization program gives a system of linear equation between { z ( t ) , x ( t ) } t ∈ [ m ] and the variable (cid:101) V .
But the solution of (cid:101) V is not unique, as the minimizer set of { z ( t ) } t ∈ [ m ] contains inﬁnitely many elements.
But it turns out that when { x ( t ) } t ∈ [ m ] forms an linearly independent basis of R m , any solution (cid:101) V to the linear system given by any minimizer { z ( t ) } t ∈ [ m ] are guaranteed to have Φ( V, (cid:101) V )
= 0 .
We defer the full proof to Appendix B.
4.2 More Realistic Situations: Learning from Realized Actions
In this section, we consider the more challenging yet realistic scenario, where the leader is able to observe a single action from follower at each round, i.e., the best response w.r.t. his perceived utility under the Gumbel noise, or equivalently the realized action of the follower’s quantal response strategy.
It turns out that the intuitions from Section 4.1 still apply, and we are able to prove a strict generalization of these results.
In particular, Theorem 1 strengthens Observation 1 in that learning the follower’s utility up to some logit distance could also lead to an approximation of the optimal leader strategy under some mild condition given by Deﬁnition 2 in general Stackelberg games.
Theorem 2 generalizes Proposition 1, as we showcase the sample complexity of our learning framework to recover the follower’s utility in face of the follower’s stochastic responses.
Deﬁnition 2 (Inducibility Gap) .
For any follower utility V , we deﬁne its inducibility gap as
[Equation] j − e j (cid:48) ] .
That is, the maximum constant σ ( V ) such that for any follower actions j ∈ [ n ] , there exists a leader strategy x
j that makes j dominate any other action j (cid:48) by a margin of at least σ ( V ) , i.e., x j V e j ≥ x j V e j (cid:48) + σ ( V ) , ∀ j (cid:48) (cid:54) = j ∈ [ n ] . If a game has small inducibility gap σ , then there must exsit two follower actions j, j (cid:48) such that the follower’s utility for action j can never be δ better than his utility for action j (cid:48) , regardless of what strategies the leader play.
In such cases, action j is essentially dominated by j (cid:48) (up to at most δ ). It is not difﬁcult to see that in such case with small δ it will be difﬁcult to recover all the payoffs in such cases since action j is expected to be played very rarely.
This intuition is also reﬂected in our following two results.
Theorem 1.
Given a follower utility (cid:101) V with inducibility gap σ ( (cid:101) V ) > 5 (cid:15) , we can construct an
O ( (cid:15)/σ ( (cid:101) V )) -optimal leader strategy for any game G ( U, V ) with logit distance Φ( (cid:101) V , V ) ≤ (cid:15)mn .
Proof Sketch.
We prove through an explicit construction.
That is, given the estimate of the follower’s utility (cid:101) V , we construct a (cid:15) -robust
strategy x = (1 − 3 (cid:15) σ ( V ) ) (cid:101) x ∗ + 3 (cid:15) σ ( V ) x (cid:101) j ∗ based on the SSE ( (cid:101) x ∗ , (cid:101) j ∗ ) of G ( U, (cid:101) V ) and the strategy x (cid:101)
j ∗ such that ( x (cid:101) j ∗ ) (cid:62) V e (cid:101)
j ∗ ≥ ( x (cid:101) j ∗ ) (cid:62) V e j (cid:48)
+ σ ( V ) , ∀ j (cid:48) (cid:54) =
(cid:101) j ∗ .
We show this strategy is guaranteed to be an ( 6 (cid:15) σ ( (cid:101) V ) − 2 (cid:15) )
-SSE of the Stackelberg game G ( U, V ) .
The proof then relies on two key observations stated in Lemma 1.1 and 1.2: First, given that Φ( (cid:101) V , V ) ≤ (cid:15)mn and σ ( V ) > 3 (cid:15) , the best response of a robust strategy x in game G ( U, V ) remains the same as that of a game G ( U, (cid:101) V ) , and so is the leader utility.
This means x gets at least (1 − 3 (cid:15) σ ( V ) ) portion of
SSE utility in G ( U, (cid:101) V ) .
Second, the difference between the SSE utility in G ( U, V ) and G ( U, (cid:101) V ) are bounded by 3 (cid:15) σ ( V ) .
Meanwhile, even though V is unknown to us, Lemma 1.3 shows that we can bound σ ( V ) ≥ σ ( (cid:101) V ) − 2 (cid:15) , so we can use σ ( (cid:101) V ) − 2 (cid:15) to substitute σ ( V ) .
And this requires σ ( (cid:101) V ) ≥ 5 (cid:15) .
Due to the space limit, we defer the full statement of the lemmas and proofs to the Appendix C. After restoring the connections between the logit distance and the leader’s optimal equilibrium utility, we now show the relationship between the logit distance and sample complexity in the learning problem. We remark that by satisfying our full rank condition, this sample complexity result does not depend on any additional parameter on the distance of queried leader strategies, such as λ, ν in [20], both of which are only guaranteed to affect the sample complexity by polynomial (not necessarily linear) factors w.r
Theorem 2.
It takes Θ( m log( mn/δ ) ρ(cid:15)) queries of the follower’s quantal response to recover the follower’s utility (cid:101) V within the logit distance Φ( V, (cid:101) V ) =
(cid:15)λ with probability at least 1 − δ , where ρ is the least non-zero measure among all of the follower’s mixed strategies induced by leader’s strategy queries during learning.
This theorem is a strict generalization of Proposition 1 and we defer the full proof to Appendix
D due to space limit.
The high level intuition comes from the fact that (1 − (cid:15) ) -multiplicative approximation guarantee is translated to (cid:15) additive error after the logarithmic transformation using the approximation that for small positive (cid:15) close to zero, we have ln( 11 − (cid:15) ) =
O ( (cid:15) ) .
And to obtain such (1 − (cid:15) ) -multiplicative approximation of an mixed strategy, we use standard concentration results for a tight sample complexity bound.
We formalize these statements and proofs in Lemma 2.1, 2.2.
Lemma 2.1.
There exists a learning algorithm that can recover the follower’s utility (cid:101) V within the logit distance Φ( V, (cid:101) V )
=
O ( (cid:15)λ ) from m queries of the (1 − (cid:15) ) -multiplicative approximation of the follower’s mixed strategies.
Lemma 2.2.
For any discrete distribution y with support size n and the least non-zero measure min i ∈ [ n ] ,y i > 0 { y i } ≥ ρ , with Θ( log( n/δ ) ρ(cid:15)) samples, the corresponding empirical distribution (cid:98) y is an (1 − (cid:15) ) -multiplicative approximation to y , with probability at least 1 − δ .
4.3 A Learning Framework of Practicality
PURE , Less is More The above results lead to a simple but provably effective method, PURE ; the name comes from the fact that it only uses the m different pure strategies in ∆ m , { x ( t ) } t ∈ [ m ] .
As speciﬁed in the proof of Theorem 2, it gathers the follower’s sampled quantal responses of these pure strategies to estimate the corresponding empirical distributions { (cid:101) y ( t ) } t ∈ [ m ] and solves for the (cid:101) V through the optimization program 4.1.
While it is a seemingly naive learning algorithm, we would like to make a few crucial points on its unique advantages from both theoretical and practical perspectives.
Theoretically, we know PURE is guaranteed to perfectly recover the follower utility in the setting of Section 4.1.
More importantly, when randomness is present, PURE guarantees that the estimation error measured by the logit distance is always bounded as O ( (cid:15)λ ) ; the Equation (D.1) in the proof of
Theorem 2 suggests that the inverse of a general row-stochastic matrix X and the error matrix β could otherwise lead to possibly unbounded estimation error.
Meanwhile, we anticipate that the simplicity of PURE would be especially valuable to its applicability in practice.
First, the randomized leader strategies in many applications are difﬁcult to be implemented precisely, because the followers may not have the perfect estimation of the leader’s distributions of randomization.
This means that observing the follower’s responses to randomized leader strategies could be more noisy in nature.
Second, it might be inappropriate and possibly forbidden for the learner (e.g., an Internet platform or policy marker) to frequently change its strategies (e.g., prices or policies).
Instead, the deployment of PURE only requires the learner to observe the responses of only a small number of pure strategies at the population level.
PURE for Structured Games We remark that the learning framework of PURE could be tailored to the special structures in Stackelberg game.
For example, let us consider a celebrated variant, known as the Stackelberg security game.
3 Namely, a leader (defender) commits to a randomized allocation of security resource to defend a set of n (= m ) targets from a follower (attacker).
In turn, the follower observes this randomized allocation and picks a target to attack.
Both the leader and the follower receive payoffs depending on the target that was attacked and the probability that it was defended.
So in this case the follower utility can be expressed as linear functions, where each entry in vector w , b ∈ R n denotes, respectively, the attacker’s cost and reward on each target.
When the leader defends each target with the randomized strategy x ∈ ∆ n , if the follower attacks the target j , he receives utility based on the cost w.r.t.
the chance target j is defended, and the reward for the attack, i.e., V ( x , j ) =
w j x j + b j .
Then, we can use the learning framework of PURE that only solves for the linear utility function parameters using the optimization program 4.2.
This not only reduces the number of parameters to be learnt but also directly gives the reward and cost parameters of each targets.
Our empirical experiments below suggest a signiﬁcantly faster error convergence rate once the structure insights is brought into the learning framework.
[Equation]
PURE-Exp for the Worst Cases In certain situations, however, the followers could be more rational and the parameter λ is larger than the standard estimation.
Then, the follower’s stochastic quantal response becomes rather deterministic, and the least non-zero measure ρ decreases.
Lemma 2.2 suggests that querying through simple pure strategies could become much less inefﬁcient in obtaining the (1 − (cid:15) ) -multiplicative approximation of the actual strategy.
Nevertheless, it turns out that we can introduce the “exploration and exploitation” principle here for the remedy, and we thus name such variant of PURE algorithm as PURE-Exp .
Speciﬁcally, we introduce an exploration procedure to search for better strategies if an empirical estimation of the follower strategy tends to concentrate on a single action.
We knew such strategy would contain more noise than information, as the error introduced by its multiplicative approximation ratio can be signiﬁcant; reversing a one-hot distribution from logit transformation provides no information about the follower utility.
In this case, we carefully replace it by a perturbed strategy from the original strategy.
This ensures that the resulting strategy set after replacement still forms a full-rank matrix that ensures the invertibility necessary for a provably more effective recovery of V in Theorem 2.
Otherwise, the algorithm would continue to exploit the leader strategies to better estimate the follower responses.
Our empirical experiments show substantial performance improvement by PURE- Exp in those extreme cases.
Experiment
In this section, we seek to further understand the empirical implications of our learnability results.
A major challenge when evaluating the learning performance is that the measures rely on the underlying ground truth utility.
While there are several real world data collected in particular to understand the human behaviors and QR model [32, 38, 41, 35], they are sensitive, proprietary datasets in security domains that we are unfortunately unable to access.
Moreover, these ofﬂine dataset only offer limited number of ofﬂine samples that can hardly be used in our active learning setup.
Therefore, our experiments have to rely on synthesized game instances, from which we can construct oracles to respond to the active learning queries and accurately evaluate for the learning performance.
As motivated in the previous section, we will use the logit distance in Deﬁnition 1 to empirically measure the quality of recovered follower utilities.
4 We start by investigating the empirical performance of
PURE in games synthesized using several sets of different parameters below.
• The number of leader and follower actions m, n : We compare the learning performance in game of varying sizes, while ﬁxing the number of query T = 10 7 .
In the left plot of Figure 1, the ﬁrst trend to notice is that the error grows almost linear to m , exactly as Theorem 2 predicts.
Meanwhile, the error also grows as n increases, as the error bound depends on 1 /ρ ≥ n .
In Appendix, we shows that 1 /ρ in average among those randomized generated game instances grows linearly with n , which justiﬁes the almost linear relation between the logit distance and n .
• The level of bounded rationality λ : We consider different λ ranging from 0 .
5 to 16 estimated in prior human behavior experiments [32, 38, 41].
In the middle plot of Figure 1, we display the convergence trend of logit distance in the number of queries.
The PURE algorithm shows consistently good performance among these different λ .
On one hand, in games with the smaller λ , the error tends to converge slower, as bounded by the 1 λ √ t convergence rate implied by Theorem 2. On the other hand, the variance of error increases especially in the initial half of the timeline in games with larger λ .
This is explained by the fact that sample complexity of learning distribution up to (1 − (cid:15) ) -multiplicative factor increases as the distribution concentrates when λ increase. • The payoff margin α : We generate the follower’s utility matrix, V = αI + (1 − α )Ξ , as a convex combination of diagonal matrix
I ∈ R m × n and Gaussian random noise Ξ normalized to [0 , 1] m × n such that the larger α , the follower are likely to have higher margin for his best response against each of the leader’s action.
In the right plot of Figure 1, we can see a consistent trend of improving estimation of the follower’s utility as query number increases across different level of α .
Interestingly, as the utility matrix becomes closer to the simple diagonal matrix, and the follower easily becomes less irrational, the convergence rate slows down; this again suggests our message on the blessing of bounded rationality that provides the stochasticity in follower’s responses used as our additional information source.
We also compare the performance of PURE and its variants introduced in Section 4.3, and the results closely match with our theoretical insights.
In the left plot of Figure 2, we compare PURE using only 10 leader strategies with the standard ofﬂine learning setup using 10 2 , 10 3 or 10 4 leader strategies with less samples in average and less accurate estimation of follower response for each leader strategy.
We can see that the PURE signiﬁcantly outperforms these ofﬂine learning setups, especially when λ is smaller such that the response of follower tends to be more irrational and thus “noisy”.
In the middle plot of Figure 2, we study the learning performance of PURE in various security games with or without using the optimization program specialized for the game structure (in dotted or straight lines).
The result suggests that the structure insights can be used for fast recovery of follower utility.
In the right plot of Figure 2, we found that PURE-Exp , with the principle of exploration and exploitation, are able to improve the learning performance in the case when the follower appears to be more rational.
However, its performance also degrades as λ further increases and the problem becomes almost the perfect rationality setting that are proved to be statistically hard to learn [27, 37].
In the limit of space, please check out Appendix E for more descriptions and analysis of our experiments.
Conclusion
Two common assumptions of a typical game theory problem are: (1) the agents know the game parameters; (2) the agents are perfectly rational.
Though these assumptions have enabled elegant mathematical models and fundamental theoretical insights, they could be limiting in some real-world scenarios.
Our paper tackles the particular problem in sequential game-theoretical interactions without these two common assumptions.
While similar inverse game theory problems under perfect rationality are shown to be statistically or computationally intractable, we made an intriguing ﬁnding in which relaxing us from these idealistic settings in turns lead us to a provably efﬁcient learning guarantee.
Therefore, we proposed the learning framework of PURE intended for fewer usage restrictions in real-world applications.
In future work, we wish to extend our analysis and insights to more general game settings and other models of bounded rationality. Acknowledgement We thank all the anonymous reviewers for their helpful comments.
Co-author Fei Fang was supported in part by NSF CAREER grant IIS-2046640.
Weiran Shen gratefully acknowledges ﬁnancial support from the National Natural Science Foundation of China (No. 62106273), the Fundamental Research
Funds for the Central Universities and the Research Funds of Renmin University of China
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 1. (c) Did you discuss any potential negative societal impacts of your work? [No] There is no foreseeable negative societal impacts of this work. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Most of them in Appendix due to space limits 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] In the supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix E (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [N/A] (b) Did you mention the license of the assets? [N/A] (c) Did you include any new assets either in the supplemental material or as a URL? [N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Proof of Observation
Proof. First, pick any leader strategy x , we show with any utility matrix V , the follower has the same action probability under quantal response model for any of its row-shifted utility matrices (cid:101) V : = V + c ⊗ 1 n 5 .
To verify this claim, we pick any of follower’s action j ∈ [ n ] , p j = [Equation] where the ﬁrst and second equality is by deﬁnition; the last equality is by separating and canceling out the same terms. Second, we show the follower’s best response remains the same for any of its row-shifted utility matrices (cid:101)
V := V + c ⊗ 1 n . Speciﬁcally, for any x , for any V and (cid:101) V := V + c ⊗ 1 n , we have, ∀ j,
[Equation] [Equation] Finally, since the follower’s quantal response (resp. best response) policy to leader’s committed strategy remains the same under any of its row-shifted utility matrices (cid:101) V :
= V + c ⊗ 1 n , pick any strategy x , the leader utility in face of quantal response (resp. best response) must remain the same. This means the leader’s equilibrium strategy in SSE or QSE also remains the same for any follower utility (cid:101) V :
=
V + c ⊗ 1 n . B Proof of Proposition Proof. Pick m linearly independent basis vectors for each x ( t ) in m rounds.
To recover the follower’s utility, we formulate an optimization program by minimizing the cross entropy loss L ( P ; Q ) = − P log Q , where P is the observed strategy y ( t ) and Q is the predicted strategy with each entry
p j ( t ) =
[Equation] for t ∈ [ m ] . [Equation] [Equation]
We now argue that the above optimization program can perfectly recover the follower’s utility matrix (cid:101) V .
Here in this program (cid:101) V is the only unknown variable, and z ( t ) serves as a proxy variable of (cid:101) V .
And we start by determining z ( t ) .
Observe that the objective of the optimization program is a log-sum-exp function w.r.t.
variables { z ( t ) } t ∈ [ m ] , which is convex.
We can compute its derivative is zero at { z i ( t )
= ln (cid:98)
y i ( t ) + c t |∀ i ∈ [ n ] , ∀ c t ∈ R , ∀ t ∈ [ m ] } , which forms the set of minimizers of this function.
Now we ﬁx any c t ∈ R for each t ∈ [ m ] , and denote the vector c :=
[ c t ] t ∈ [ m ] .
Let X := [ x ( t )] t ∈ [ m ] , Y := [ y ( t )] t ∈ [ m ] .
Then, replacing each z i ( t ) by ln (cid:98)
y i ( t ) + c t in the optimization constraint, we can formulate the linear equation λX T (cid:101) V
= ln
Y + c ⊗ 1 n , denoted as L ( c ) .
Since X T is a full rank matrix in R m × m , L ( c ) has a unique solution for (cid:101) V = λ − 1 ( X − 1 ) (cid:62) [ln Y + c ⊗ 1 n ] .
Let V :=
{ λ − 1 ( X − 1 ) (cid:62) [ln Y + c ⊗ 1 n ] |∀ c ∈ R m } denote the solutions to L ( c ) for all c .
Let V ∗ be the ground truth follower utility.
{+Following from the Observation 1, if V ⊆ { V ∗ + c (cid:48) ⊗ 1 n |∀ c (cid:
+}
) ∈ R m } , then the solution to L ( c ) of arbitrary c recovers the follower’s utility to the level that the optimal leader strategy can be exactly determined.
To see this, let c ∗ be the vector such that the unique solution to L ( c ∗ ) is the ground-truth follower’s utility V ∗ .
Since V ∗ ∈ V , such c ∗ exists.
Now for any c , we can derive that
[Equation]
[Equation]
where λ − 1 ( X − 1 ) (cid:62) [ c ∗ − c ] forms a vector in R m .
{+Hence, there exists some c (cid:
+}
) ∈ R m , (cid:101) V =
V ∗ + c (cid:48) ⊗ 1 n .
This proves that any minimizer of the above convex program, i.e., the solution to L ( c ) for any c , would allow us to solve the optimal leader strategy exactly.
C Proofs of Theorem
Theorem 1.
Given a follower utility (cid:101) V with inducibility gap σ ( (cid:101) V ) > 5 (cid:15) , we can construct an
O ( (cid:15)/σ ( (cid:101) V )) -optimal leader strategy for any game G ( U, V ) with Φ( (cid:101) V , V ) ≤ (cid:15)mn .
Proof.
We prove through an explicit construction.
Speciﬁcally, given the estimation of the follower’s utility (cid:101) V , we construct a (cid:15) -robust
strategy x = (1 − 3 (cid:15) σ ( V ) ) (cid:101) x ∗ + 3 (cid:15) σ ( V ) x (cid:101) j ∗ based on the SSE ( (cid:101) x ∗ , (cid:101) j ∗ ) of G ( U, (cid:101) V ) and the strategy x j ∗ such that ( x (cid:101) j ∗ ) (cid:62) V e j ≥ ( x (cid:101) j ∗ ) (cid:62) V e j (cid:48)
+ σ ( V ) , ∀ j (cid:48) (cid:54) =
j .
We show this strategy is guaranteed to be an ( 6 (cid:15) σ ( (cid:101) V ) − 2 (cid:15) )
-SSE of the Stackelberg game G ( U, V ) .
The proof then relies on two key observations stated in Lemma 1.1 and 1.2: First, given that Φ( (cid:101) V , V ) ≤ (cid:15)mn and σ ( V ) > 3 (cid:15) , the best response of an robust strategy x in game G ( U, V ) remain the same from that in a game G ( U, (cid:101) V ) , and so is the leader utility.
This means x gets at least (1 − 3 (cid:15) σ ( V ) ) portion of
SSE utility in G ( U, (cid:101) V ) .
Second, the difference between the SSE utility in G ( U, V ) and G ( U, (cid:101) V ) are bounded by 3 (cid:15) δ ( V ) .
Meanwhile, despite V is unknown to us, Lemma 1.3 shows that we can bound σ ( V ) ≥ σ ( (cid:101) V ) − 2 (cid:15) , so we can use σ ( (cid:101) V ) − 2 (cid:15) to substitute σ ( V ) .
And this requires σ ( (cid:101) V ) ≥ 5 (cid:15) .
Let U ∗ and (cid:101)
U ∗ be the SSE utility of G ( U, V ) and G ( U, (cid:101) V ) respectively.
Let ( x ∗ , j ∗ ) be the SSE of G ( U, V ) , and ( (cid:101) x ∗ , (cid:101) j ∗ ) be the SSE of G ( U, (cid:101) V ) .
The leader utility strategy x in G ( U, V ) can be bounded as
[Equation]
where the ﬁrst equality is by the construction of x ; The ﬁrst inequality uses the fact that ( x (cid:101) j ∗ ) (cid:62) Ue (cid:101) j ∗ ≥ 0 and the deﬁnition of (cid:101) U ∗ .
The second inequality uses the fact that (cid:101) U ∗ ≤ 1 .
The third inequality follows from Lemma 1.2.
The last inequality uses 0 <
[Equation] Lemma 1.1 (Invariance of Best Response under (cid:15) -Robust Strategy) . Let j be the follower’s best response against the leader strategy x in G ( U, V ) and there exists x j such that x j V e j ≥ x j V e j (cid:48) + σ, ∀ j (cid:48) (cid:54) =
j . If σ ≥ 3 (cid:15) , then follower’s best response against the (cid:15) -robust strategy x = (1 − 3 (cid:15)σ ) x + 3 (cid:15)σ x j remains j , and the leader’s utility of strategy x remains the same in any game G ( U, (cid:101) V ) with Φ( (cid:101) V , V ) ≤ (cid:15)mn . Proof.
To show j is the follower’s best response to the leader strategy x in G ( U, (cid:101) V ) , we directly show through the deﬁnition, x (cid:62) (cid:101) V ( e j − e j (cid:48) ) ≥ 0 , ∀ j (cid:48) (cid:54) = j . Pick any j (cid:48) (cid:54) =
j , x [Equation] V [Equation]
The ﬁrst inequality is by Lemma 1.4.
The equality is by construction of x and linearity of D .
The second inequality is by the fact that x (cid:62) V ( e j − e j (cid:48) ) ≥ 0 since j is the best response to x under follower utility V .
The last inequality is by using the fact that x j (cid:62) V ( e j − e j (cid:48) ) ≥ σ .
Lemma 1.2 (Bounded SSE Utility Difference) .
Let U ∗ and (cid:101)
U ∗ be the SSE utility of G ( U, V ) and
G ( U, (cid:101) V ) respectively.
If Φ( (cid:101) V , V ) ≤ (cid:15)mn , then we have (cid:101)
[Equation]
Proof.
Let ( x ∗ , j ∗ ) be the SSE of G ( U, V ) , and ( (cid:101) x ∗ , (cid:101) j ∗ ) be the SSE of G ( U, (cid:101) V ) .
We construct an (cid:15) -robust
strategy x = (1 − 3 (cid:15)σ ) x ∗ + 3 (cid:15)σ x j ∗ from SSE of G ( U, (cid:101) V ) and the strategy x j ∗ V e j ∗ ≥ x j ∗ V e j (cid:48)
+ σ, ∀ j (cid:48) (cid:54) =
j .
Hence, by Lemma 1.1, we know the follower’s best response to x remains j ∗ under utility V or (cid:101) V , and so the leader utility of strategy x remains the same in G ( U, (cid:101) V ) and G ( U, V ) .
Then, we show the following inequalities hold:
[Equation]
where the ﬁrst inequality is by the fact that ( (cid:101) x ∗ , (cid:101) j ∗ ) is the SSE of G ( U, (cid:101) V ) whose leader utility must be no smaller than strategy proﬁle
( x , x j ∗ ) .
The second inequality is by construction of x and linearity of D .
The last inequality is by the fact that ( x ∗ ) (cid:62) Ue j ∗ =
U ∗ ≤ 1 and ( x j ∗ ) (cid:62) Ue j ∗ ≥
Lemma 1.3 (Bounded Inducibility Gap Difference) .
For any (cid:101) V , V such that Φ( (cid:101) V , V ) ≤ (cid:15)mn , we have σ ( (cid:101) V ) ≥ σ ( V ) − 2 (cid:15) .
Proof.
We prove directly by deﬁnition of the inducibility gap,
[Equation]
The ﬁrst and last equalities are by deﬁnition, the inequality is by Lemma 1.4.
Lemma 1.4 (Bounded Utility Gap Difference) . Given Φ( (cid:101) V , V ) ≤ (cid:15) mn , for any x ∈ ∆ m , y 1 , y 2 ∈ ∆ n , [Equation] Proof.
Let (cid:101) V = V + c ⊗ 1 n + Ξ for constant vector c ∈ R m and || Ξ
|| 1 ≤ (cid:15) .
By linearity, we decompose x (cid:62) ( V − (cid:101) V )( y 1 − y 2 ) as x (cid:62) ( c ⊗ 1 n )( y 1 − y 2 ) + x (cid:62) Ξ( y 1 − y 2 ) .
We have already seen in Observation 1 that for any x ∈ ∆ m , y 1 , y 2 ∈ ∆ n , x (cid:62) (
[Equation]
⊗
[Equation]
n
[Equation]
− y
[Equation]
) =
[Equation]
So it only remains to argue that (cid:12)(cid:12) x (cid:62) Ξ( y
[Equation]
− y 2 ) (cid:12)(cid:12) ≤ 2 (cid:15) .
By triangle inequality, we have (cid:12)(cid:12)
[Equation] [Equation] y [Equation]
(cid:12)(cid:12) + (cid:12)(cid:12) x (cid:62)
[Equation]
y [Equation] (cid:12)(cid:12) . Meanwhile, given || x || 1 = 1 and || y || 1 = 1 , by Holder’s inequality, we have (cid:12)(cid:12) x (cid:62) Ξ y (cid:12)(cid:12) ≤
[Equation] = (cid:15) . D Proofs of Theorem Theorem 2.
It takes Θ( m log( mn/δ ) ρ(cid:15)) queries of the follower’s quantal response to recover the follower’s utility (cid:101) V within the logit distance Φ( V, (cid:101) V ) =
(cid:15)λ with probability at least 1 − δ , where ρ is the least non-zero measure in the induced follower’s strategies.
Proof.
We prove by combining the results of Lemma 2.1 and Lemma 2.2.
Following from Lemma 2.2, we can obtain m queries of the (1 − (cid:15) )
-multiplicative approximation of the follower’s mixed strategies with Θ( m log( mn/δ ) ρ(cid:15)) samples, with probability at least 1 − δ . Using Lemma 2.1, these m queries can recover the follower utility of distance (cid:101) V within the logit distance Φ( V, (cid:101) V ) = (cid:15)λ .
Lemma 2.1.
There exists a learning algorithm that can recover the follower’s utility (cid:101) V within the logit distance Φ( V, (cid:101) V ) = O ( (cid:15)λ ) from m queries of the (1 − (cid:15) ) -multiplicative approximation of the follower’s mixed strategies.
Proof.
Pick m linearly independent basis vectors for each x ( t ) in m rounds.
We can observe the data { ( x ( t ) , (cid:101) y ( t )) } t ∈ [ m ] , where (cid:15) -multiplicative approximation guarantee ensures that the observed distribution (cid:101) y j ( t ) (cid:98) y j ( t ) ∈ [1 − (cid:15), 11 − (cid:15) ] , ∀ j ∈ [ n ] , t ∈ [ T ] w.r.t.
the ground-truth strategy (cid:98) y ( t ) .
To recover the follower’s utility, we formulate an optimization program by minimizing the cross entropy loss L ( P ; Q ) = − P log Q , where P is the observed strategy (cid:101) y t and Q is the predicted strategy with each entry p j ( t ) = [Equation] minimize (cid:80) t ∈ [ m ] (cid:104) log (cid:80)
[Equation] n
[Equation]
exp z j ( t ) − (cid:101) y
( t ) · z ( t ) (cid:105) z ( t ) = λ x ( t ) (cid:62) (cid:101) V , for t ∈ [ m ] .
We now argue that the above optimization program can recover the follower’s utility matrix (cid:101) V such that Φ( V, (cid:101) V )
= O ( αλ ) . Here in this program (cid:101) V is the only unknown variable, and z ( t ) serves as a proxy variable of (cid:101) V . And we start by determining z ( t ) .
Observe that the objective of the optimization program is a log-sum-exp function w.r.t. variables { z ( t ) } t ∈ [ m ] , which is convex. We can compute its derivative is zero at { z i ( t ) = ln (cid:101)
y i ( t ) + c t |∀ i ∈ [ n ] , ∀ c t ∈ R , ∀ t ∈ [ m ] } , which forms the set of minimizers of this function. Now we ﬁx any c t ∈ R for each t ∈ [ m ] , and denote the vector c := [ c t ] t ∈ [ m ] . Let X :=
[ x ( t )] t ∈ [ m ] , (cid:101) Y := [ (cid:101) y ( t )] t ∈ [ m ] .
Then, replacing each z i ( t ) by ln (cid:98) y i ( t ) + c t in the optimization constraint, we can formulate the linear equation λX T (cid:101) V = ln (cid:101)
Y + c ⊗ 1 n , denoted as L ( c ) . Since X T is a full rank matrix in R m × m , L ( c ) has a unique solution for (cid:101) V = λ − 1 ( X − 1 ) (cid:62) [ln (cid:101) Y + c ⊗ 1 n ] . Let c ∗ be the vector such that the unique solution to L ( c ∗ , (cid:101) Y ) is the ground-truth follower’s utility V .
Let (cid:101) Y = (cid:98) Y ◦ β , where each entry in β is in [1 − (cid:15), 11 − (cid:15) ] . By construction, such c ∗ must exist. Now for any c, (cid:101) Y , we can derive that
[Equation]
where λ − 1 ( X − 1 ) (cid:62) [ c ∗ − c ] forms a vector in R m . Hence, by deﬁnition, we can normalize out the λ − 1 ( X − 1 ) (cid:62) [ c ∗ − c ] ⊗ 1 n in the logit distance, and thus,
[Equation]
where we use the approximation that for small positive (cid:15) close to zero, we have ln( 11 − (cid:15) ) = O ( (cid:15) ) , and we pick X to be an identity matrix such that || X − 1 ln β || 1 ≤ mn(cid:15) . Lemma 2.2.
For any discrete distribution y with support size n and the least non-zero measure min i ∈ [ n ] ,y i > 0 { y i } ≥ ρ , with Θ( log( n/δ ) ρ(cid:15)) samples, the corresponding empirical distribution (cid:98) y is an (1 − (cid:15) ) -multiplicative approximation to y , with probability at least 1 − δ . Proof. We start with the sample complexity upper bound:
Given T
=
O ( log n +log1 /δ ρ(cid:15)) number of i.i.d.
samples { y ( t ) ∈ [ n ] } t ∈ [ T ] from distribution y , we use the standard mean estimator to construct the empirical distribution (cid:98) y with each entry (cid:98) y i = (cid:80) t ∈ T 1 [ y ( t )= i ] T .
By deﬁnition, if ∀ i ∈ [ n ] , (cid:98) y i y i ∈ [1 − (cid:15), 11 − (cid:15) ] , then (cid:98) y is a (1 − (cid:15) ) -multiplicative approximation of y . We know for any y i = 0 , the empirical estimation of (cid:98) y
i must be perfect. Otherwise, for all y i > 0 , we can use the Chernoff multiplicative bound [13], as E [ (cid:98) y i ] = y i taking expectation over randomness of the samples. That is, with probability at least 1 − ρ/n , with O ( log n +log1 /δ ρ(cid:15)) number of samples, we get (cid:98)
y
i y i ∈ [1 − (cid:15), 1 + (cid:15) ] ⊂ [1 − (cid:15), 11 − (cid:15) ] . Therefore, by union bound, (cid:98) y is an (1 − (cid:15) ) -multiplicative approximation to y , with probability at least 1 − δ . We now show the sample complexity lower bound: there exists some distribution y with support size n and the least non-zero measure ρ that requires at least Ω( log( n/δ ) ρ(cid:15)) to learn an (1 − (cid:15) )
-multiplicative approximation of y .
We prove by constructing n − 1 probability distributions that are hard to distinguish and reducing the estimation error into such a testing problem.
In the lower bound instance, we let n ≥ 3 and ρ =
o (1 /n ) .
Speciﬁcally, consider the following n − 1 distributions, where for each i ∈ [ n − 1] , we let the j th entry of distribution y i be y ij =
  ρ + 3 ξ, i =
j ρ, i (cid:54) =
j 1 − ( n − 1) ρ − 3 ξ, j = n with ξ < ρ < 1 − 3 ξ n .
By such construction, each y i have support size n and the least non-zero measure ρ , and the TV distance between any two of these distributions, d TV ( y i , y j ) = 3 ξ .
If we let ξ = (cid:15)ρ , we can reduce the learning problem of (1 − (cid:15) ) -multiplicative approximation of y to the test problem of distinguishing the n − 1 probability distributions.
That is, pick any y i ∗ , if we have enough samples to learn (1 − (cid:15) )
-multiplicative approximation of y i ∗ , we obtain an empirical estimation (cid:98) y that has TV distance d TV ( (cid:98) y , y i ∗ ) ≤ (cid:15)ρ = ξ . With such (cid:98) y , we can tell apart y i ∗ from y i according to the triangle inequality that ∀ i (cid:54) = i ∗ , d TV ( (cid:98) y , y j ) ≥ d TV ( y i ∗ , y j ) − d TV ( (cid:98) y , y i ∗ ) ≥ 2 ξ . So we now determine the lower bound of the testing problem using the Assouad’s Lemma [4]: it takes Ω( log( n/δ ) d H ( y i , y j ) )
samples to distinguish any two distribution y i , y j with probability at least 1 − δ/n . In this case, the squared Hellinger distance of d H ( y i , y j ) can be computed as ξ < ρ ,
[Equation]
By union bound, using at least Ω( ρ(cid:15) 2 ) samples, we can distinguish any two distributions y i , y j with probability at least 1 − δ . Then, the reduction implies it requires at least Ω( log( n/δ ) ρ(cid:15)) to learn an (1 − (cid:15) ) -multiplicative approximation of y .
E Additional Experiment Details and Results
In this section we provide the detailed experiment descriptions and some additional empirical results to further understand the learning performance of the PURE framework. For the ease of reproduction, we also include the implementation details in the our released code in supplementary materials. • The number of leader and follower actions m, n : In compliment to the experiment on the change of logit distance of recovered utility w.r.t. the number of leader and follower actions m, n , we also investigate the change of 1 /ρ in the randomly generated instances here.
This helps us to further understand the effect of game size on the learning performance through ρ , since our Theorem 1 predicts a linear relation between 1 /ρ and the logit distance. In the left plot of Figure 3, it shows that in expectation 1 /ρ scales almost linearly with n . This justiﬁes the almost linear relation between the logit distance and n in the left plot of Figure 1. In addition, 1 /ρ is almost independent of m , while its variance decreases as m increases — it becomes less likely to have extremely small non-zeros measures in the follower’s QR strategies.
• The rank of follower utility: We also investigate the inﬂuence from the rank of the follower utility on the learning performance. We randomly generate matrices of follower utility with m = n =that respectively contains { 2 , 4 , 8 , 16 , 20 } linearly independent rows. In the middle plot of Figure 3, it suggests that the rank of the follower utility has very little effect on the performance of PURE . This result does match with our expectation, as the structure insight on rank of follower utility is not incorporated into the learning framework.
We also anticipate a performance boost if the model were to utilize the prior knowledge on linear independence of certain rows in the utility matrix. However, this appears to be a rather unrealistic assumption, and the rank constrained optimization in general is known to be an NP-hard problem. • Active over ofﬂine learning: In the left plot of Figure 3, we showcase more results comparing the learning performance between our framework PURE and ofﬂine learning from randomly generated data points.
Speciﬁcally, the ofﬂine learning data is generated from K ∈ { 10 2 , 10 3 , 10 4 , 10 5 , 10 6 } number of strategies with T = 10 6 samples in total. We estimate the empirical distributions induced by these K strategies, using T/K ∈ { 10 4 , 10 3 , 10 2 , 10 , 1 } samples of each. So the more strategies, the less accurate estimation we have for each of the corresponding follower QR strategy.
In the case when K ∈ { 10 2 , 10 3 , 10 4 } , we directly solve for the (cid:101) V through the optimization program 4.1. Due to the computation and memory bottleneck of the optimization solver as the number of terms in the optimization objective grows, we have to resort to gradient descent in the case when K ∈ { 10 5 , 10 6 } , where the gradient is determined at each iteration using ∂(cid:96) t /∂ (cid:101) V t and (cid:96) t is the cross entropy loss computed between the empirical (from data) and the predicted (from (cid:101) V t ) follower QR strategy w.r.t. the t th sample of follower response. This gives us an iterative form of the optimization program 4.1, but the gradient descent method does not guarantee the convergence to optimality as the optimization solver does.
We plot the results using the Adam optimizer with learning rate 10 − 3 in the last two columns. In contrast, we plot the error of PURE that only use the K = 10 different strategies in the ﬁrst column. We can see that the PURE signiﬁcantly outperforms these ofﬂine learning setups, especially when λ is smaller such that the response of follower tends to be more irrational and thus “noisy”. And the gradient descent method gives much worse optimization results than the optimizer solver.
This showcases that our learning framework is indeed able to leverage the noise in follower responses as valuable feedback for payoff recovery.