Strategy-Aware Contextual Bandits
Abstract
Algorithmic tools are often used to make decisions about people in high-stakes domains.
In the presence of such automated decision making, there is incentive for strategic agents to modify their input to the algorithm in order to receive a more desirable outcome.
While previous work on strategic classification attempts to capture this phenomenon, these models fail to take into account the multiple actions a decision maker usually has at their disposal, and the fact that they often have access only to bandit feedback .
In contrast, we capture this setting as a contextual bandit problem, in which a decision maker must take actions based on a sequence of strategically modified contexts.
We provide a low-strategic-regret algorithm for the two action setting, and prove that sublinear strategic regret is generally not possible for settings in which the number of actions is greater than two.
Along the way, we obtain impossibility results for multi-class strategic classification which may be of independent interest.
Introduction
When subjugated to algorithmic decision making, decision subjects may strategically modify their input to the algorithm in order to receive a better outcome.
This may be viewed as a form of distribution shift in which the data distribution the machine learning (ML) algorithm observes has been strategically altered.
It is therefore desirable to design reliable ML algorithms in the presence of such strategic interactions .
While this area of study has received much recent interest in the literature under the name of strategic classification (see Appendix A for a list of related work), current work in this area fails to capture several salient features of reality, as most models only consider binary classification or linear regression settings under full feedback (i e., the decision maker sees the agent’s true label after making a prediction).
In contrast, we study a model in which a decision maker must assign a strategic agent one of several actions under bandit feedback (i e., the decision maker only observes some reward for the action they assign).
Our model captures several settings of practical interest including (1) personalized shopping, in which an online platform (decision maker) must decide which level of discount (action) to give to give to customers (strategic agents) given their previous buying patterns and (2) lending, in which a bank (decision maker) needs to decide which type of loan (e.g., high-interest, low-interest, no loan) to offer to applicants (strategic agents) given their credit history.
Our main contributions are as follows: (1) We formalize the strategy-aware contextual bandit setting , in which a decision maker must assign a sequence of strategic agents one of n actions, and observes their outcomes under bandit feedback.
(2) For n = 2 actions, we show that it is possible to minimize strategic regret , a strong notion of hindsight rationality.
Finally, we show that in sharp contrast to the two-action setting, when n ≥ 3 there exist cases where achieving sublinear strategic regret is not possible.
Along the way, we obtain impossibility results for multi-class strategic classification which may be of independent interest.
In particular, we show
For each round t ∈ [ T ] :
Decision maker publicly commits to a mapping π t : X → A .
Agent Γ t arrives with private type x t . 2.
Agent t strategically selects context x ′ t according to Assumption 2.3. 3.
Decision maker observes context x ′ t and plays action a t =
π t ( x ′ t ) .
Decision maker receives reward r Dt ( a t ) and agent receives reward r A ( a t ) .
that there exist situations in which perfect classification under strategic responses is not possible with three or more labels, even if agents are linearly separable before strategic modification.
Model and Preliminaries
We consider a setting in which a decision maker interacts with a sequence of T strategic agents .
At each time t ∈ [ T ] , a new agent Γ t arrives with a private type x t ∈ X ⊂ [ − L, L ] d for L ∈ R + , and presents a context x ′ t ∈ R
d to the decision maker.
Note that it is possible for x ′ t
= x t , but the two do not necessarily coincide.
Given a context x ′ t , the decision maker takes an action a t from some set of actions A , where A = { 1 , . . . , n } .
For each action a ∈ A , we assume that the decision maker’s expected reward E [ r Dt ( a )] ∈ [ − 1 , 1] is a linear function of the agent’s private type.
In particular, we make the following assumption on the decision maker’s reward.
Assumption 2.1 (Linearity of Rewards) .
The decision maker reward of each arm a ∈ A at time t is given by
[Equation]
for some θ a ∈ R d which is unknown to the decision maker, where ε t is zero-mean sub-Gaussian noise.
Note that if x ′ t = x t ∀ t ∈
[ T ] , our setting reduces to that of linear contextual bandits (e.g, [19, 2]).
In contrast to the decision maker’s reward, we assume that each agent’s reward r At ( a ) is a function of the action alone, i.e., r At ( a ) =
r A ( a ) , ∀ t ∈ [ T ] , and is known to the decision maker.
In the personalized shopping example, a consumer’s private type can be thought of as the type of customer they are (e.g., loyal customer, new customer, thrifty customer, etc.).
If customers knew the online platform was going to offer them a discount based on their buying habits, they may strategically alter their short-term behavior on the platform (i.e., their context) in order to obtain the highest discount possible.
The decision maker’s policy π t : X → A at time t is a mapping from agent contexts to actions.
In particular, the goal of the decision maker is to deploy a sequence of policies π 1 , . . .
π T in order to minimize strategic regret , which is defined as follows:
Definition 2.2 (Strategic Regret) .
The expected strategic regret of a sequence of policies { π t } Tt = is defined as the cumulative expected reward of π ∗ ( x ) :
= arg max a ∈A (cid:80) t ∈ [ T ] E [ r Dt ( a )
| x ] , the optimum-in-hindsight policy given the agents’ private types, minus the cumulative expected reward of the deployed sequence of policies.
Formally,
[Equation]
A decision maker with sublinear strategic regret will approach the performance of the best-inhindsight policy if agents are not strategic as T → ∞ , thereby learning to “account” for the strategic behavior of agents in the long run.
Given the decision maker policy π t , it is natural for an agent to choose their context in a way which maximizes their reward.
Specifically, we assume that agent Γ t determines their context by strategically modifying their private type based on the decision maker’s policy, subject to a constraint on the amount of modification which is possible.
In addition to being a common assumption in the literature (e.g., [6, 17, 13]), this budget constraint on the amount an agent can modify their private type reflects the fact that agents have inherent constraints on the amount of time and resources they can spend on modification.
Assumption 2.3 (Agent Best Response) .
We assume that agent Γ t best-responds to the decision maker’s policy π t in order to maximize their expected reward, subject to the constraint that their context x ′ t is within a δ -radius of their private type x t .
Formally, we assume that the agent solves the following optimization to determine their context:
[Equation]
Furthermore, we assume that if an agent is indifferent between modifying their context and not modifying, they choose not to modify.
See Figure 1 for a summary of the setting we consider.
Finally, we conclude this section by introducing the idea of an equivalence region , which will be useful when discussing our theoretical contributions.
Roughly speaking, an equivalence region corresponds to all contexts for which a policy π ( · ) assigns action a . Definition 2.4 (Equivalence region) . An equivalence region for an action a ∈ A under policy π is defined as the set of all x ∈ X such that π ( x ) = a .
For the policies we consider, each equivalence region will be a closed, convex set.
Sublinear strategic regret for two actions
We now outline an algorithm which achieves sublinear strategic regret in the two-action setting.
W.l.o.g.
, we assume that r A (2) >
r A (1) for the remainder of the subsection.
While our algorithm is similar to previous algorithms from the “strategic classification” literature (e.g., [11, 3]), we view it as an important structural result which (1) illustrates that sublinear strategic regret is possible in the two action setting and (2) provides intuition as to why this is generally not possible in the case of 3+ actions.
We begin with a result for the offline, non-strategic setting.
Proposition 3.1.
In the two-action setting in which agents are not strategic (i.e., x ′ t = x t , ∀ t ∈ [ T ] ) and θ 1 , θ 2 are known to the decision maker, the optimal policy assigns action a t
= 1 if ⟨ β 12 , x ′ t ⟩ ≥and action a t = 2 otherwise, where β 12 := θ 1 − θ 2 .
Proposition 3.1 follows from Assumption 2.1.
While the decision boundary of Proposition 3. is optimal in the non-strategic setting, its performance may suffer in the strategic setting (i.e., if x ′ t ̸ = x t ).
In order to account for potential strategic manipulations, the optimal decision boundary needs to be shifted by an amount proportional to the agents’ ability to manipulate their private type.
Intuitively, this shift disincentives strategic manipulations from the agents who would have manipulated without such shift.
Proposition 3.2.
In the two-action setting in which agents strategically reveal their contexts according to Assumption 2.3, the optimal policy assigns action a t = 1 if ⟨ β 12 , x ′ t ⟩ ≥ − δ ∥ β 12 ∥ 2 and action a t = 2 otherwise, where β 12 := θ 1 − θ 2 .
Moreover, the performance of such a policy when agents strategically reveal their contexts according to Assumption 2.3 matches that of Proposition 3.1 in the non-strategic setting ( i.e., when x ′ t
= x t , ∀ t ∈ [ T ] ).
See Figure 2 for a visual depiction of the policies of Propositions 3.1 and 3.2.
Given the result of Proposition 3.2, it is rather straightforward to achieve sublinear strategic regret in the online setting by using the standard “explore-first” technique.
In particular, under minor distributional assumptions on the population of agents (e.g., the covariance of the agent population has full rank), the decision maker can apply Algorithm 1 to achieve O ( T 2 / 3 ) strategic regret. t = 1 , . . , T 0 do if t ≤ T 0 / 2 then Assign action a 1 else
Assign action a 2 end end
Estimate θ 1 , θ 2 via OLS as ˆ θ 1 , ˆ θ 2 . Let ˆ β 12 : = ˆ θ 1 − ˆ θ 2 .
for t = T 0 + 1 , . . , T do Assign action a t = (cid:40)if ⟨ ˆ β 12 , x t ⟩ − δ ∥
ˆ β 12 ∥ 2 ≥ o.w. end
Theorem 3.3 (Abridged; see Appendix B.2 for formal statement) . For sufficiently large T , if T 0 = ˜ O ( T 2 / 3 )
E [ R [Equation] = [Equation]
( d [Equation]
[Equation] T 2 / 3 ) , where ˜ O ( · ) suppresses terms which do not depend on T, d , as well as logarithmic terms. The idea of Algorithm 1 is as follows: the first T 0 rounds are the exploration phase, i.e., contexts are not used to make decisions about which action to assign. Since the agents have no incentive to modify their private type, x ′ t = x t , ∀ t ∈ [ T 0 ] .
The decision maker then uses data from the explore phase to estimate β 12 , and acts greedily according to ˆ β 12 for the remaining T − T 0 rounds.
Ω( T ) strategic regret for more than two actions
In the previous subsection, we showed that it is generally possible to achieve sublinear strategic regret in the two-action strategy-aware contextual bandit setting using a linear separating hyperplane, shifted by an appropriate amount.
However, we will now show that this intuition does not carry over to the 3+ action setting.
Furthermore, we show that there exist situations for which no linear policy (e.g., set of linear hyperplanes) can achieve sublinear strategic regret if the decision maker has three or more actions.
1 As was the case in Section 3, we begin with a result for the offline setting.
Note that this result immediately implies that similar “bad examples” exist for settings in which n > 3 and d > 2 .
Theorem 4.1.
Suppose X = R 2 , n = 3 , and r A (2) >
r A (1) =
r A (3) .
For θ 1 = [1 0 . 5] ⊤ , θ 2 =
[0 1] ⊤ , θ
=
[ − 1 0 .
5] ⊤ , there exists a distribution over agent private types such that no linear policy can achieve perfect classification if agents choose their contexts according to Assumption 2.3.
Proof Sketch.
The proof of Theorem 4.1 proceeds by constructing a “bad” distribution over agents (shown in orange in Figure 3) in which it is necessary to shift the separating hyperplanes β 12 and β (defined analogously to β 12 in Proposition 3.2) in order to achieve perfect classification on a certain subpopulation of agents (far-left and far-right orange lines in Figure 3).
Next, we show that shifting β 12 and β 23 by such an amount will make it impossible to classify another agent subpopulation (orange dot in Figure 3).
Finally, we conclude that perfect classification of this distribution of strategic agents is not possible.
Note that this is in contrast to the non-strategic setting, in which this agent population is trivially linearly separable.
Proof.
Recall that assigning a t = arg max a ∈A ⟨ θ a , x t ⟩ will achieve perfect classification in the non-strategic setting.
Define β 12 := θ 1 − θ 2 = [ − 0 .
5] ⊤ , β 13 :
= θ 1 − θ 3 = [2 0] ⊤ , and β 23 : = θ 2 − θ 3 = [1 0 . 5] ⊤ Assigning actions according to the following set of decision boundaries is equivalent to assigning a t = arg max a ∈A ⟨ θ a , x t ⟩ : [Equation]
See Figure 3 for a visual depiction of the decision space (blue lines).
Consider the following agent private types: x 1 = [0 α 1 ] ⊤ , X 2 = { x : ⟨ β 12 , x ⟩ = α 2 , x [1] > 0 } ,
X ′ 2 = { x : ⟨ β 12 , x ⟩ = β 12 [2] · α 1 , x [1] > 0 } , X 3 = { x : ⟨ β 23 , x ⟩ = − α 2 , x [1] < 0 } ,
X ′ 3 = { x : ⟨ β 23 , x ⟩ = β 23 [2] · α 1 , x [1] < 0 } , where α 1 , α 2 > 0 and the corresponding distribution over X : P ( x t = x 1 ) = 15 , P ( x t ∈ X 2 ) = 15 , P ( x t ∈ X ′ 2 )
= 15 , P ( x t ∈ X 3 )
= 15 , P ( x t ∈ X ′ 3 )
= 15 .
Suppose that r A (2) >
r A (1) =
r A (3) . Under this setting, we make use of the following two lemmas to identify necessary conditions on the hyperplane shift ε in order to achieve perfect classification in the strategic setting: Lemma 4.2. To prevent gaming, the decision maker must assign action a 2 only if ⟨ β 12 , x t ⟩ + ε 12 < 0 and ⟨ β 23 , x t ⟩ − ε 23 ≥ 0 , where ε 12 ≥ δ ∥ β 12 ∥ 2 − α 2 and ε 23 > δ ∥ β 23 ∥ 2 − α 2 .
Lemma 4.3.
To correctly classify agents with x t = x 1 , the decision maker must assign action a 2 only if ⟨ β 12 , x t ⟩ + ε 12 < 0 and ⟨ β 23 , x t ⟩ − ε 23 ≥ 0 , where ε 12 ≤ − β 12 [2]( α 1 + δ ) and ε 23 ≤ β 23 [2]( α 1 + δ ) .
An immediate consequence of the above two lemmas is that unless δ ∥ β 12 ∥ 2 − α 2 ≤ − β 12 [2]( α 1 + δ ) and δ ∥ β 23 ∥ 2 − α 2 ≤ β 23 [2]( α 1 + δ ) , perfect classification is not possible.
In the above setting, this implies that perfect classification is not possible unless 12 α 1 + α 2 ≥ δ ( √ 1 . 25 − 0 . 5) , which does not hold for sufficiently small α 1 , α 2 . The following corollary follows immediately from Theorem 4.1 and Definition 2.2.
Corollary 4.4.
In the setting of Theorem 4.1, no online learning algorithm can achieve sublinear strategic regret.
A Related Work
Our work is closely related to two lines of work: learning in the presence of strategic behavior (e.g., [3, 4, 8, 11, 13, 14, 17, 12, 5, 10, 16, 18, 6]) and contextual bandits (see e.g. [15, 20, 21, 19, 7, 1, 9, 2] for a highly incomplete list).
Strategic learning.
The problem of strategic classification was first introduced in Hardt et al.
The original formulation considers a sequential game between a “jury” (decision maker), who publishes a classifier, and a “contestant” (strategic agent), who best responds by strategically modifying their observable features.
In the years since, there have been extensions to online learning (Dong et al. [8], Chen et al.
[6], Ahmadi et al.
[3]), repeated interactions (Harris et al.
[13]), and social learning settings (Bechavod et al.
[4]), although to the best of our knowledge, we are the first to consider multi-class strategic classification under bandit feedback.
Contextual bandits.
The contextual bandit setting can be thought of a multi-armed bandit problem, in which the bandit algorithm has access to extra information about the reward of each action (i.e., the context ) at every round.
Various relationships between contexts and rewards have been studied; several of the most common are Lipschitz contextual bandits (e.g., Hazan and Megiddo [15], Lu et al.
[20], Slivkins [21]), in which the expected reward of each arm is Lipschitz in the context, linear contextual bandits (e.g., Li et al.
[19], Chu et al.
[7]), where the expected reward is linear in the context, and agnostic algorithms (e.g., Agarwal et al.
[2], Foster and Rakhlin [9]), which make no assumptions on the underlying reward distribution, but assume access to various classification/regression oracles.
In particular, our setting is most similar to that of linear contextual bandits, although with the notable exception that in our setting, the expected reward of each action is linear in the agent’s private type .
Since each agent’s private type is not observed by the decision maker, this introduces significant additional difficulties in the decision maker’s learning problem and leads to our impossibility results in Section 4.
B Proofs from SectionB.1 Proof of Proposition 3.
Proposition B.1.
In the two-action setting in which agents strategically reveal their contexts according to Assumption 2.3, the optimal policy assigns action a t = 1 if ⟨ β 12 , x t ⟩ ≥ − δ ∥ β 12 ∥ 2 and action a t = 2 otherwise, where β 12 := θ 1 − θ 2 .
Proof.
Denote the unit vector along the direction of β 12 as n = β 12 ∥ β 12 ∥ 2 .
We proceed on a case-bycase basis.
Case 1 : The policy of Proposition 3.1 assigns action a t = 1 to agent Γ t :
Since a t = 1 , ⟨ β 12 , x t ⟩ ≥ 0 .
In order to move across the boundary under as little modification as possible, the agent should modify their context in the direction of − n .
However, the most they can manipulate their context by is δ , and ⟨ β 12 , x t − δ n ⟩ ≥ − δ ∥ β 12 ∥ 2 since ⟨ β 12 , x t ⟩ ≥ 0 .
Therefore no valid manipulations exist for which the policy of Proposition 3.2 assigns action a = 2 to agent Γ t .
Case 2: The policy of Proposition 3.1 assigns action a t = 2 to agent Γ t :
Since a t = 2 , ⟨ β 12 , x t ⟩ < 0 .
If agent Γ t modifies their private from x t to context x t − δ n , ⟨ β 12 , x ′ t ⟩ = ⟨ θ 1 − θ 2 , x t ⟩ − δ ⟨ θ 1 − θ 2 , n ⟩ < − δ ∥ β 12 ∥ 2 .
Therefore, they can receive action a =under the policy of Proposition 3.2 by strategically modifying their private type from x t to context x t − δ n .
[-Consider the following set over agent private types: x 1 = [0 α 1 ] ⊤ , X 2 = { x : ⟨ β 12 , x ⟩ =α 2 , x [1] > 0 } , X ′ 
-]
[-= { x : ⟨ β 12 , x ⟩ = β 12 [2] · α 1 , x [1] > 0 } , X 3 = { x : ⟨ β 23 , x ⟩ =− α 2 , x [1] < 0 } , X ′ 
-]
B.2 Proof of sublinear strategic regret for two actions
Lemma B.2.
[-To correctly classify agents with x t = x 1 , the decision maker must assign actiona 
-]
If T 0 ≥
[Equation]
(cid:112)
[Equation]
with probability at least 1 − δ .
An identical bound holds for |⟨ ˆ θ 2 − θ 2 , x ⟩| .
Proof.
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Bounding (cid:13)(cid:13)(cid:13)(cid:80)
[Equation]
Since x t ( j ) ∈ [ − L, L ] and ε t is a sub-Guassian random variable with variance σ 2 ε , x t ( j ) ε t is a zero-mean sub-Gaussian random variable with variance at most L 2 σ 2 ε .
Therefore, we can use the following result to bound (cid:13)(cid:13)(cid:13)(cid:80) T 0 / 2 t =1 x t ε t (cid:13)(cid:13)(cid:13) 2 with high probability.
Theorem B.3 (High probability bound on the sum of unbounded sub-Gaussian random variables) .
Let x t ∼ subG (0 , σ 2 ) .
For any δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
Using Theorem B.3,
[Equation]
with probability at least 1 − δ , where the last line follows from a union bound where δ j = δ/d for all j ∈ [ d ] .
Bounding λ min (cid:16)(cid:80) T 0 / 2 t =1 x t x ⊤ t (cid:17)
Let Y =
(cid:80) T 0 /
[Equation]
= [Equation] t x ⊤
[Equation]
[Equation]
λ min ( σ 2 x I d × d ) and λ min ( E [ x 1 ] E [ x ⊤ 1 ]) commute, so
[Equation]
We use the following result to bound λ min (cid:16)(cid:80) T 0 / 2 t =1 x t x ⊤ t (cid:17) with high probability:
Theorem B.4 (Matrix Chernoff) .
Consider a finite sequence { X t } Tt =1 of independent, random,
Hermitian matrices with common dimension d . Assume that 0 ≤ λ min ( X t ) and λ max ( X t ) ≤ L ′ for each index t
Introduce the random matrix
[Equation]
Define the minimum eigenvalue µ min of the expectation E [ Y ] :
[Equation]
Then,
[Equation]
for ε ∈ [0 , 1) . [Equation] ′ =
dL .
Pick ε = 12 .
Applying Theorem B.4 to λ min ( (cid:80) T 0 / 2 t =1 x t x ⊤ t ) ,
[Equation]
After inverting the bound, we see that λ min ( (cid:80) T 0 / 2 t =1 x t x ⊤ t ) ≥ 14 T 0 σ 2 x with probability at least 1 − δ if T 0 ≥
[Equation]
Combining the bounds for (cid:13)(cid:13)(cid:13)(cid:80) T 0 / 2 t =1 x t ε t (cid:13)(cid:13)(cid:13) 2 and λ min (cid:16)(cid:80) T 0 / 2 t =1 x t x ⊤ t (cid:17) via a union bound, we obtain our stated result.
Theorem B.5.
If T 0 ≥
[Equation]
log
[Equation]
log(2 dT 4 ) , Algorithm Explore-First achieves expected strategic regret
[Equation]
Proof.
[Equation]
If T 0 ≥
[Equation]
, the following two events hold simultaneously with probability at least 1 − δ :
[Equation]
If a t is the suboptimal action to assign to agent t
[Equation]
then with probability at least 1 − δ ,
[Equation]
Therefore, (cid:10) θ a ∗ t
[Equation]
with probability at least 1 − δ
[Equation]
By an argument similar to the proof of Proposition 3.2, the Explore-First ’s strategic regret is the same as that of making decisions according to ˆ β 12 on non-strategic agents.
Using this observation, we can set δ = 1 T 4 and upper-bound expected strategic regret by
[Equation]
Corollary B.6.
If
[Equation]
T ≥
[Equation]
then
[Equation]
The proof of Corollary B.6 follows straightforwardly from Theorem B.5 and the inequality log x ≤ αx − log α − 1 for any α, x > 0 .
C Proofs from SectionC.1 Proof of Lemma 4.
Lemma C.1.
To prevent gaming, the decision maker must assign action a 2 only if ⟨ β 12 , x t ⟩ + ε 12 < 0 and ⟨ β 23 , x t ⟩ − ε 23 ≥ 0 , where ε 12 ≥ δ ∥ β 12 ∥ 2 − α 2 and ε 23 > δ ∥ β 23 ∥ 2 − α 2 .
Proof. Since r A (1) = r A (3) , agents are indifferent between receiving actions 1 and 3 , and so βis the optimal decision boundary between the a = 1 and a = 3 equivalence regions. Due to the structure of the distribution over agents, it is w.l.o.g. to consider linear decision boundaries which are parallel to β 12 and β 23 .
Since r A (2) >
r A (1) , an agent with private type x t ∈ X 2 has incentive to choose their context such that ⟨ β 12 , x ′ t ⟩ + ε 12 < 0 , where x ′ t is agent t ’s context.
It is w.l.o.g. to assume the agent modifies their private type in the direction perpendicular to β 12 .
The most the agent can modify in this direction is − δ β 12 ∥ β 12 ∥ 2 .
Therefore if we want ⟨ β 12 , x ′ t ⟩ + ε 12 ≥ 0 ,
[Equation]
Similarly for an agent x t ∈ X 3 ,
[Equation]
C.2 Proof of Lemma 4.
Lemma C.2.
To correctly classify agents with x t = x 1 , the decision maker must assign action a 2 only if ⟨ β 12 , x t ⟩ + ε 12 < 0 and ⟨ β 23 , x t ⟩ − ε 23 ≥ 0 , where ε 12 ≤ − β 12 [2]( α 1 + δ ) and ε 23 ≤ β 23 [2]( α 1 + δ ) .
Proof.
It is w.l.o.g. to assume that x ′ t = [0 α 1 + δ ] ⊤ . If we want ⟨ β 12 , x ′ t ⟩ + ε 12 < 0 , then β 12 [2]( α 1 + δ ) + ε 12 < Similarly, if we want ⟨ β 23 , x ′ β
( α 1 + δ ) − ε 23 ≥