Learn to Match with No Regret: Reinforcement Learning in Markov Matching Markets
Abstract
We study a Markov matching market involving a planner and a set of strategic agents on the two sides of the market.
At each step, the agents are presented with a dynamical context, where the contexts determine the utilities.
The planner controls the transition of the contexts to maximize the cumulative social welfare, while the agents aim to ﬁnd a myopic stable matching at each step.
Such a setting captures a range of applications including ridesharing platforms.
We formalize the problem by proposing a reinforcement learning framework that integrates optimistic value iteration with maximum weight matching.
The proposed algorithm addresses the coupled challenges of sequential exploration, matching stability and function approximation.
We prove that the algorithm achieves sublinear regret
Introduction
Large-scale digital markets play a crucial role in modern economies.
Increasingly, the understanding and design of such markets require tools from both economics and machine learning.
Indeed, dynamically changing market environments require mechanisms to be adaptive, scalable and incentive compatible in the face of signiﬁcant nonstationarity.
The data streams that arise from digital markets provide opportunities to cope with such challenges, via learning-based mechanism design.
Recent work (Jagadeesan et al., 2021;
Liu et al., 2021; Sankararaman et al., 2021;
Basu et al., 2021; Liu et al., 2022) has begun to apply modern machine learning tools to problems in adaptive mechanism design.
One particular area of focus in learning-aware market design has been matching, a class of 36th Conference on Neural Information Processing Systems (NeurIPS 2022). problems central to microeconomics (Mas-Colell et al., 1995).
Existing work has focused on static matching markets, however, and the more challenging yet critically important setting of dynamic matching markets has been neglected.
Such markets are our focus in the current paper, where we provide reinforcement learning (RL) techniques to address dynamic matching market problems.
Speciﬁcally, we propose a Markov matching market model involving a planner and a set of two-sided agents.
Given the time horizon H ∈ Z + , a set of agents I h ∪ J h enter the market in context C h at step
h ∈
[ H ] , with unknown utility functions u h and v h for agents in I h and J h respectively.
The agents’ utilities u h and v h depend on the context C h .
Given the utility functions, the agents seek to achieve a myopic stable matching with transferable utilities (Shapley and Shubik, 1971).
In particular, the contexts C h are subject to a Markov transition kernel which is controlled by the planner’s policy.
The goal of the planner is to select an optimal policy, which together with the stable matching in each step maximizes the expected accumulated social welfare.
As an illustration, consider a simpliﬁed abstraction of ride-hailing platforms (Qin et al., 2020; Özkan and Ward, 2020; Hu and Zhou, 2021), where the horizon is set as the time span of a day.
In this case, the platform is the planner, and the two-sided agents
I h and J h are the drivers and the riders.
The contexts may include information such as GPS location, car types, and pricing.
The platform is required to ensure the stability of the matching at each step so that the agents do not prefer alternative outcomes, otherwise they may leave the platform.
In addition, the platform needs to adjust the policy to maximize the accumulated social welfare as a measure of the satisfaction of both drivers and riders.
As illustrated in this example, there are several key challenges in Markov matching markets.
First, the agents’ utilities and the transition of contexts are unknown, so we need to perform efﬁcient exploration to collect the required information.
Second, we need to ensure the stability of the matching in each step, for which a consideration of only the difference in total utilities is insufﬁcient, as discussed in
Jagadeesan et al.
To this end, we adopt a notion of Subset Instability (SI) from Jagadeesan et al.
(2021) as a metric that quantiﬁes the distance between a proposed matching and the optimal stable matching.
This metric has the ﬂavor of Shapley value, comparing the discrepancy between the total utilities over all subsets of the participating agents, while accounting for transfers.
For efﬁcient estimation, we need to take into account the function class of the utilities as well as the Markov transition kernel, thus demanding a systematic usage of function approximation.
To tackle these challenges, we develop a novel algorithm called Sequential Optimistic Matching (SOM), which features a combination of optimistic value iteration and max-weight matching.
Note that the planner’s problem becomes a standard Markov decision process (MDP) if we regard the value of the max-weight matching over the true utilities as the reward.
Inspired by this observation, on the agents side, our algorithm applies the optimism principle to construct UCB estimates of the utilities.
Based on these UCB estimates, the algorithm computes the corresponding max-weight matching, and the values of the resulting matching serve as the surrogate rewards.
Although we do not have access to unbiased estimate of the rewards, the key property here is that the surrogate rewards upper bound the true social welfare, thus justifying optimistic planning by the planner.
Interestingly, our framework can readily incorporate any online RL algorithm based on optimism.
On the theoretical side, we show that the suboptimality of the accumulated social welfare for our algorithm consists of two parts: (1) the planner’s regret in terms of the suboptimality of the policy, and (2) the agents’ regret in terms of SI.
Jagadeesan et al.
(2021) proved that SI can be bounded by the sum of the optimistic bonuses, and we further show that the planner’s regret can also be bounded by the bonus sum.
In this way, we reconcile the seemingly independent learning goals of the planner and the agents, and thereby provide a uniﬁed approach to controlling the suboptimality of the total social welfare.
In particular, based on the above decomposition, we further show that in the case of linear function approximation, our algorithm enjoys a sublinear regret independent of the size of the context space.
Compared with existing methods on online RL such as LSVI-UCB (Jin et al., 2020), our framework incorporates a matching problem in each time step.
Compared with the setup of matching bandits (Jagadeesan et al., 2021), our model provides an extension to a dynamic setup with transitions between contexts.
More importantly, our work lies beyond a straightforward combination of online
RL with matching bandits due to the unique technical challenges that we will discuss in Sectionand further elaborate in Section C.
The main contributions are summarized as follows:
(i)
We propose a novel Markov matching market model that captures a range of instances of centralized matching problems.
(ii) We develop a novel algorithm that combines optimistic value iteration with max-weight matching, such that any online RL algorithm based on optimism can be readily incorporated into the framework.
(iii)
We provide a general analysis framework and show that our proposed algorithms achieve sublinear regret under proper structural assumptions on the underlying model.
Related Work
There is an emerging line of research on learning stable matchings with bandit feedback (Das and
Kamenica, 2005; Liu et al., 2020, 2021; Sankararaman et al., 2021; Cen and Shah, 2021; Basu et al., 2021) using the mature tools from the bandit literature.
Most of them focus on matchings with non-transferable utilities (Gale and Shapley, 1962), which fails to capture real-world markets with monetary transfers between agents, e.g., payments from passengers to drivers on ride-hailing platforms.
The study of learning for matchings with transferable utilities is comparably limited, and our work extends Jagadeesan et al.
(2021) to dynamic scenarios in this regime.
Broadly speaking, our work is also related to the research on learning economic models via RL.
In particular, Kandasamy et al.
; Rasouli and Jordan (2021) studied VCG mechanisms, and Guo et al.
(2021) studied exchange economies.
Although similar in spirit, these models differ from our Markov matching markets in their mathematical structure, and thus have different solution concepts and planning methods.
Our model of Markov matching markets is related to the topic of dynamic matching in the economics literature (Taylor, 1995; Satterthwaite and Shneyerov, 2007; Niederle and Yariv, 2009; Ünver, 2010;
Anderson et al., 2014; Lauermann and Nöldeke, 2014; Leshno, 2019; Akbarpour et al., 2020; Baccara et al., 2020;
Loertscher et al., 2018; Doval and Szentes, 2019).
Instead of studying the learning problem in matching markets, the goal therein is mainly focused on the problem of optimal mechanism design (Akbarpour et al., 2014) with known utilities or explicit modelling of agents’ arrivals via queuing models (Zenios, 1999; Gurvich and Ward, 2015).
In this work we focus on the notion of (static) stability of myopic matchings (Shapley and Shubik, 1971).
There is also a line of literature on the notion of dynamic stability for matching markets (Damiano and Lam, 2005; Doval, 2014; Kadam and Kotowski, 2018; Doval, 2019; Kotowski, 2019; Kurino, 2020; Liu, 2020), and it is an interesting open problem to study learning for dynamically stable matchings.
Our methodology builds upon recent progress in online RL, where the “optimism in face of uncertainty” principle has engendered efﬁcient algorithms that are either model-based (Jaksch et al., 2010;
Osband et al., 2016; Azar et al., 2017; Dann et al., 2017) or model-free (Strehl et al., 2006; Jin et al., 2018; Fei and Xu, 2022; Fei et al., 2021a), and can be combined with function approximation techniques (Yang and Wang, 2019; Jin et al., 2020; Zanette et al., 2020; Ayoub et al., 2020; Wang et al., 2020; Fei et al., 2021b;
Yang et al., 2020; Zhou et al., 2021;
Min et al., 2021a,b; Du et al., 2021; Jin et al., 2021).
We note that these approaches can be incorporated into our framework with proper modiﬁcations on structural assumptions and correspondingly the algorithm.
We do not pursue these extensions here, however, as our regret analysis is already sufﬁciently challenging given the lack of an unbiased estimate of the reward, and the additional constraints imposed by the requirements of matching stability.
Preliminaries 3.1 Markov Matching Markets
We ﬁrst review basic concepts for matching with transferable utilities (Shapley and Shubik, 1971).
Consider a two-sided matching market where I denotes the set of all side-1 agents (e.g., buyers) and J denotes the set of all side-2 agents (e.g., sellers).
Given any set of participating agents
I × J ∈ 2 I × 2 J , a matching X is a set of pairs ( i, j ) indicating i
∈
I is matched with j ∈ J , and each agent can be matched at most once.
For any pair of agents ( i, j ) ∈ I × J , we denote by u ( i, j ) the utility of agent i and v ( i, j ) the utility of agent j when they are matched.
In addition to the matching X , we also allow transfers between agents, summarized by the transfer function τ : I ∪ J → R .
For each agent i ∈
I ∪ J , τ ( i ) is the transfer that it receives.
We assume that the transfers are within agents, so (cid:80) i ∈ I ∪ J τ
( i )
= 0 .
The overall market outcome is denoted by a tuple ( X, τ ) , where X represents the matching and τ represents transfers.
For any ( i, j ) ∈ X , the total utilities for agent i and j are u ( i, j )
+ τ ( i ) and v ( i, j )
+ τ ( j ) respectively.
Moreover, if no agents prefer any alternate outcome, then we say ( X, τ ) is stable (see Deﬁnition B.1 for details).
The stable matching can be found by solving the corresponding max-weight matching, as will be explained in Section 4.1.
Based on these (classical) deﬁnitions, we formulate the notion of a Markov matching market involving a planner and a set of two-sided agents.
Throughout the paper, we focus on matchings with transferable utilities between two-sided agents, and we may omit such descriptions for convenience.
Deﬁnition 3.1 (Markov matching markets) .
A Markov matching market is denoted by a tuple
M = ( C , Υ , { I h }
Hh =1 , { J h } Hh =1 , { P h } Hh =1 , { u h } Hh =1 , { v h } Hh =1 ) .
Here C is the set of contexts and Υ denotes the set of planner’s actions.
At each step h ∈
[ H ] , I h ∪ J h
⊂ I × J is the set of participating agents, and u h : C × Υ
× I × J
→ R and v h : C × Υ
× I × J
→ R are utility functions for two sides of agents respectively.
For each h ∈ [ H ] , P h ( C (cid:48) | C, e ) is the transition probability for context
C to transit to C (cid:48) given action e .
In such Markov matching markets, the learning goal is two-fold: (1) to learn the stable matching in each step and (2) to maximize the accumulated social welfare.
3.2 A Reinforcement Learning Approach to Markov Matching Markets
We now translate the problem into the language of RL.
Given Deﬁnition 3.1, we consider an episodic setting with K episodes where each episode consists of H steps of sequential matchings.
Each episode proceeds in the following way:
at each step h ∈ [ H ] under context C h , a set of agents I h ∪ J h enter the market.
The planner takes action e h , implements the matching ( X h , τ h ) , and observes the noisy feedback of utilities
u h ( C h , e h , i, j ) and v h ( C h , e h , i, j ) for all ( i, j
) ∈
X h .
Then the context transits according to P h ( · | C h , e h ) , and the market proceeds to the next step.
Note that here the implemented matching in each step is myopic and we seek for stability for each of these matchings.
The maximization of the accumulated social welfare is achieved through the planner’s actions { e h } Hh =1 that control the transitions of contexts, which together with planner’s actions determine the optimal values of the matchings.
To apply RL techniques to maximize the accumulated social welfare, let us specify the ingredients of the corresponding RL problem.
States and Actions.
The state space is S =
C ×
I × 2 J .
The action space is A = Υ
× X × T where X denotes the set of all matchings and T is the set of all possible transfers among the agents.
For each step h ∈
[ H ] , the state s h
= ( C h , I h , J h ) contains the context and participating agents, and the action a h
= ( e h , X h , τ h ) contains the planner’s action e h and the matching outcome ( X h , τ h ) for the agents.
Rewards.
At each step h ∈
[ H ] , given the state s h
= ( C h , I h , J h ) and the action a h
= ( e h , X h , τ h ) , the immediate reward is the social welfare (i.e. sum of utilities):
[Equation]
Note that the transfer τ h ∈ T does not appear in the reward since the total transfer sums to zero.
Transition of States.
The state consists of the context in C and the sets of agents in 2 I × 2 J .
The transition of context at step h follows the heterogeneous transition function P h ( C h +1 | C h , e h ) , which only depends on the planner’s action e h and is independent of the matching ( X h , τ h ) .
We assume that the sequence of agent sets { I h , J h } Hh =1 is generated independently from other components in this matching market.
We also assume the same sequence through all K episodes for the sake of clarity.
Note that I h and J h can also be handled as part of the context C h and covered by our current argument with more involved transition dynamics, which is often task-speciﬁc.
Our minor simpliﬁcation serves to build a generic framework and avoid detailed modeling of the agent sets.
Policies and Value Functions.
A policy π is deﬁned as π = { π h } Hh =1 , where for each s ∈ S , π h ( ·| s ) is a distribution on A .
The policy consists of two parts: the planner’s part (i.e., choosing e ∈ Υ to inﬂuence the market context C ), and the agents’ part (i.e., determining the matching-transfer ( X, τ ) ).
We use Π to denote the set of all such policies.
For any policy π , we deﬁne each value function V πh ( · ) as
[Equation]
Maximizing the accumulated social welfare is equivalent to maximizing V π 1 over π ∈ Π , so the overall performance over K episodes is evaluated through the regret
[Equation]
where π k denotes the policy in episode k .
See Appendix A for detailed deﬁnitions of our notation.
Method: An Optimistic Meta Algorithm
The dynamic matching problem introduced in Section 3 faces several coupled challenges.
Speciﬁcally: 1) we study the dynamic setting and therefore the algorithm proposed by (Jagadeesan et al., 2021) for the bandit setting (i.e., H = 1 ) does not apply; 2) the dynamic regime also requires us to handle the propagation of error through H steps in the analysis; 3) compared with standard MDPs in RL, here we aim to maximize the social welfare and thus the regret involves a new notion called Subset Instability, which is a nonlinear functional of the rewards of the agents.
Regret decomposition involving such a complex quantity was never considered in previous works (Jaksch et al., 2010; Osband et al., 2016;
Azar et al., 2017; Jin et al., 2020; Ayoub et al., 2020; Wang et al., 2020; Fei et al., 2021b;
Yang et al., 2020; Zhou et al., 2021).
We now introduce an algorithm for the Markov matching market that addresses all these challenges.
This algorithm involves solving a sequence of combinatorial optimization problems to obtain a UCB estimate (Section 4.1).
Moreover, for the purpose of the theoretical analysis, as the regret is in terms of Subset Instability, we need to connect the suboptimality measured by this metric with the uncertainty of the value function estimation.
We thus incorporate a novel decomposition of the planner and the agents in the algorithm design (Section 4.2).
In Section 4.3 we summarize our proposed Algorithm 1 which serves as a meta stereotype and can readily incorporate various existing RL methods.
4.1 Optimistic Estimation of Rewards
Note that we do not directly observe the rewards deﬁned in (1) and have no unbiased estimates of them, so we cannot explicitly construct their optimistic estimates.
However, thanks to the nature of the stable matching being a max-weight matching, we show that we can still obtain useful optimistic rewards estimates based on the optimistic estimates of the utilities.
To see this, recall the deﬁnition of reward in (1).
We know that there exists some stable matching ( X h , τ h ) that maximizes r h and can be obtained by solving a linear program and its dual program (Shapley and Shubik, 1971).
Deﬁne the following linear program LP ( I, J, u, v ) :
[Equation]
and its dual program DP ( I, J, u, v ) :
[Equation]
Shapley and Shubik (1971) proved that the stable matching ( X, τ ) corresponds to the solution to the linear program (4) (for X ) and its dual program (5) (for τ ).
It is clear from (4) that the optimal value of the objective function is equal to the total social welfare of the stable matching.
Now, suppose we have some optimistic estimates of the utilities, i.e., (cid:98) u and (cid:98) v such that (cid:98) u ( · , · ) ≥ u ( · , · ) and (cid:98) v ( · , · ) ≥ v ( · , · ) .
It is easy to see that when substituting ( (cid:98) u, (cid:98) v ) into the linear program (4), the resulting optimal value will be an upper bound of the original optimal value (see Lemma C.2 and its proof).
Based on this observation, let us return to the reward in (1).
The previous argument implies that as long as we have optimistic estimates of the utilities u h and v h , we can get optimistic estimates of the reward by solving the max-weight matching over the optimistic utilities.
Moreover, it is further an upper bound of the following pseudo-reward:
[Equation]
where M h :=
M ( I h , J h , u h , v h , C h , e h ) denotes the set of all myopic stable matching on ( I h , J h ) with utility functions
u h ( C h , e h , · , · ) and u h ( C h , e h , · , · ) .
Finally, the optimistic estimates of the utilities can be constructed from noisy observations of agents’ utilities via any standard approach in the online learning literature.
The deﬁnition of the pseudo-reward in (6) provides a way to decompose the total regret into the planner’s regret and the agents’ regret, as will be clear in the next subsection.
4.2 Decomposition of The Planner and The Agents
Recall that we require the matching in each step to be stable, which is an additional constraint apart from maximizing the social welfare.
We need to separate these two entangled goals from each other.
Indeed, we will show that the total regret consists of two parts: 1) the suboptimality of the planner’s policy over the entire episode , and 2) the distance between the proposed matching and the optimal myopic stable matching at each step .
We identify the former as the planner’s problem and the latter the agents’ problem.
The Planner’s Problem.
The planner’s problem focuses on the transition of the contexts, so we need to partial out the effects from the actual matching.
This has been done in the deﬁnition of the pseudo-reward in (6), and the corresponding pseudo-value function V πh for h ∈
[ H ] is deﬁned as
[Equation]
where we slightly abuse the notation e l ∼ π l ( ·| s l ) .
Also note that we can write s h +1 ∼ P ( ·| s h , e h ) instead of the more general s h +1 ∼ P ( ·| s h , a h ) since we condition on ( I h , J h ) and the transition of
C h only depends on the planner’s action e h as C h +1 ∼ P h ( ·| C h , e h ) .
Clearly, V πh is an upper bound of V πh and does not depend on the actual matching { X h , τ h } h ∈
[ H ] since it has been maximized out.
Now, we specify the planner’s problem as trying to maximize the pseudo-value V π 1 , and deﬁne the planner’s regret given the initial state s 1 as
[Equation]
From a control-theoretic perspective, the planner’s problem can be viewed as learning an MDP with the same state space S , the action space reduced to Υ , and reward being the value of the myopic max-weight matching at each step.
The reward cannot be observed, nor do we have an unbiased estimator.
From an economic perspective, the planner’s problem captures only the market context and not the speciﬁc market outcome (i.e. matching).
Now, note that
[Equation]
where the planner’s regret has been captured in (8), and it remains to control the utility difference on the agents’ side.
Agents’ Problem.
The agents’ problem amounts to controlling the suboptimality of each implemented matching, which boils down to SI proposed by Jagadeesan et al.
Deﬁnition 4.1 (Subset Instability, Jagadeesan et al. 2021) .
Given any agent sets I, J and utility functions u, v : I × J → R , the Subset Instability SI ( X, τ ; I, J, u, v ) of the matching and transfer ( X, τ ) is deﬁned as
[Equation]
where X ( · ) and X (cid:48) ( · ) denotes the matched agent in matching X and X (cid:48) respectively.
Subset Instability has several key properties for learning.
Importantly, it can be shown that given ( I, J, u, v ) , the utility difference between the optimal matching-transfer pair and ( X, τ ) is upper bounded by SI ( X, τ ; I, J, u, v ) .
With a slight abuse of notation, for s h
=
( C h ,
I h , J h ) and a h
= ( e h , X h , τ h ) , we denote by SI ( s h , a h , u h , v h ) the SI of ( X h , τ h ) given I h , J h and u h ( C h , e h , · , · ) and u h ( C h , e h , · , · ) .
We deﬁne the regret of the agents as
[Equation]
Moreover, SI itself can be bounded by the sum of optimistic bonuses.
Therefore, quite surprisingly, we can control the planner’s regret and the agents’ regret at the same time by bounding the bonus sums.
In this way, the total regret can be controlled due to the following proposition.
Proposition 4.2 (Proof in Appendix D.1) .
For R ( K ) , R P ( K ) and R M ( K ) deﬁned by (3) , (8) , (9) , it holds that R ( K ) ≤ R P ( K ) + R M ( K ) .
4.3 A Meta Algorithm
Now, we are ready to present our meta algorithm as displayed in Algorithm 1.
As is clear from the previous derivations, it sufﬁces to construct optimistic estimates of the utilities, which then induces 1) matchings of agents and 2) optimistic estimates of the value functions.
The latter then enables the optimistic planning for the planner.
In particular, for the estimation part, Algorithm 1 ﬁrst constructs the Q-function estimates in a backward fashion.
Using these estimates, Algorithm 1 computes optimistic estimates of utilities by calling the subroutine UE (utility estimation), which then leads to estimates of the pseudo-reward using the subroutine RE (reward estimation).
Next, optimistic estimates of the Q functions are obtained via the subroutine QE (Q-function estimation).
Next, for the planning part, Algorithm 1 chooses action in Υ greedily and the matching-transfer pair by calling OM (optimal matching).
Finally, OM ﬁnds a matching-transfer pair which is stable with respect to the estimated utility functions.
As discussed in Section 4.1, the optimistic estimates of the rewards come from solving the linear program in (4) and its dual program in (5), which produce the optimal matching given the set of participating agents and utilities.
Therefore, the RE oracle is deﬁned as Algorithm 2, and the OM oracle is deﬁned as Algorithm 3.
The remaining subroutines ( UE and QE ) are ﬂexible and can be carefully calibrated for different model assumptions.
The modular nature of Algorithm 1 facilitates incorporation of existing RL algorithms.
In particular, we study a special case of linear function approximation in the next section, where we provide explicit oracles for these subroutines, and show that the corresponding algorithm enjoys a sublinear regret.
Case Study: Markov Matching Markets with Linear Features
In this section, we illustrate the power of our framework under linear function approximation, which is the simplest case of function approximation, yet still a rich enough model.
1: Require: λ, β u , β V 2: Initialize: u 1 h ≡ 1 , v
h ≡ 1 and D 0 h = ∅ , ∀ h ∈
[ H ] 3: for episode k = 1 , 2 , . . .
, K do 4:
Receive the initial state s k 1 =
( C k 1 , I 1 , J 1 ) 5:
Set Q kH +1 ≡6: for stage h = H, H − 1 , . .
, 1 do 7:
Estimate utilities ( u kh , v kh ) ← UE ( D k − 1 h , β u , λ ) 8:
Estimate pseudo-reward r kh ← RE ( u kh , v
kh , I h , J h ) 9:
Estimate Q-function Q kh ← QE ( D k − 1 h , r kh , Q kh +1 , β V , λ ) 10: end for 11: for stage h = 1 . .
, H do 12:
Planner takes action e
kh ← argmax e ∈ Υ Q kh
( s kh , e ) 13:
Compute the optimal matching ( X kh , τ kh ) ← OM ( u kh , v
kh , I h , J h , C kh , e kh ) 14:
Implement matching ( X kh , τ kh ) 15:
Observe utilities u kh ( i, j ) , v kh
( i, j ) for ( i, j ) ∈
X kh 16:
Receive next state s kh +
=
( C kh +1 ,
I h
+1 , J h +1 ) 17:
Update utility dataset D kh
= D k − 1 h ∪ { C kh , e kh } ∪ { u kh ( i, j ) , v kh
( i, j ) }
( i,j
) ∈
X kh 18: end for 19: end for
5.1 Model Assumptions
Utility Model.
We assume there are known feature mappings ψ : C × Υ → R d and φ : I ×J → R d , such that for any h ∈ [ H ] and ( C h , e h , i, j ) , the utility functions are
[Equation]
Here { θ h , γ h } Hh =1 are unknown parameters in R d 2 .
We further deﬁne the vectorized feature vector:
[Equation]
Then the immediate reward deﬁned in (1) can be written as
[Equation]
Transition Model.
Conditioning on the agents’ sets { I h , J h } H h
=1 , the state transition reduces to that of the contexts.
We assume a linear transition model (Jin et al., 2020):
[Equation]
for all h ∈ [ H ] , where µ h : C → R d is some unknown measure.
Next, we introduce some standard assumptions for matching and linear MDPs (Jin et al., 2020;
Jagadeesan et al., 2021).
Assumption 5.1.
We assume WLOG that for any h ∈ [ H ] , (cid:107) u h ( · ) (cid:107) , (cid:107) v h ( · ) (cid:107) ≤ 1 .
Assume that for any ( I h , J h ) ∈ 2 I × 2 J , there exists W h > 0 such that for any context C ∈ C and action e ∈ Υ , the max-weight (possibly unstable) matching on ( I h , J h ) with utility functions u h ( C, e, · , · ) and v h ( C, e, · , · ) has total utility upper bounded by W h .
The quantities { W h } Hh =1 can be viewed as a measure of complexity of the matching problem, and indeed (cid:80) Hh =1 W h determines the magnitude of V π (cid:63) 1 .
Assumption 5.1 implies a trivial upper bound that W h ≤ min {
| I h | , | J h |} , but when the max-weight matching involves only a subset of the agents,
W h can be much smaller.
Therefore, we regard { W h } Hh =1 as instance-dependent parameters.
Input: u, v, I, J
Output: (cid:98) r
( C, I, J, e ) as the solution to the LP ( I, J, u, v ) , for any ( C, e ) ∈ C × Υ
Input: u, v, I, J, C, e
Output: ( X, τ ) as the solution of the primal-dual program deﬁned by (4): LP ( I, J, u ( C, e, · , · ) , v ( C, e, I, J )) , and (5): DP ( I, J, u ( C, e, · , · ) , v ( C, e, I, J )) .
Assumption 5.2.
We assume WLOG that (cid:107) ψ
( C, e ) (cid:107) 2 ≤ 1 and (cid:107) φ ( i, j ) (cid:107) 2 ≤ 1 , implying (cid:107) Φ ( C, e, i, j ) (cid:107) 2 ≤ 1 , for any ( C, e, i, j ) .
We assume that for any h ∈ [ H ] , (cid:107) θ h (cid:107) 2 ≤ d , (cid:107) γ h (cid:107) 2 ≤ d and (cid:107) µ h ( · ) (cid:107) 2 ≤ √ d .
Moreover, assume that max h ∈
[ H ] (cid:107) (cid:82) C f ( C )d µ h ( C ) (cid:107) 2 ≤ √ d for any function f : C
→ R such that sup | f | ≤ 1 .
Assumption 5.3.
We assume that the observed utilities of matched pairs are the true utilities plus independent 1-subgaussian noise.
5.2 Algorithms
Based on previous model assumptions, we now present explicit computation oracles for Algorithm 1.
Utility Estimation.
At the beginning of episode k , for any h ∈ [ H ] , denote the available data by
D k − 1 h which consists of { u th ( i, j ) }
( i,j ) ∈
X th t ∈ [ k − 1] and { v th ( i, j ) } ( i,j ) ∈
X th t ∈ [ k − 1] , where by default D 0 h = ∅ .
For the linear case, each u th ( i, j ) =
(cid:104) Φ( C th , e th , i, j ) , θ h (cid:105) + noise , and similar for v th ( i, j ) .
So we can estimate θ h and γ h by ridge regression:
[Equation]
We then add a bonus to ensure optimism in the utility function estimates and apply the truncation:
[Equation]
The UE for the linear case is summarized in Algorithm 4.
Q-function Estimation.
By the assumption on the context transition, for any function f , the function P h f is linear in features ψ , which induces the commonly used LSVI-type algorithms (Jin et al., 2020).
Together with the reward estimates, we can estimate the Q-function using Bellman equation via backward ridge regression.
For each ( k, h ) , denote the estimate of the Q-function by Q kh and the value function by V kh .
Given Q kh +1 , maximizing over e ∈ Υ yields V kh +1 , then we solve the following ridge regression:
[Equation]
which further yields the estimated expectation of V k h
+1 :
[Equation]
where Λ kh = (cid:80) k − 1 t =1 ψ ( C th , e th ) ψ ( C th , e th ) (cid:62) + λ I d .
Finally, we estimate Q kh using the Bellman equation.
These steps are summarized in Algorithm 5.
Input: D k − 1 h , β u , λ if D k − 1 h is empty then
Output : u kh ≡ 1 and v kh ≡ 1 , ∀ h ∈
[ H ] end if
Compute θ kh , γ kh by (11)
Estimate utility functions with u kh and v kh by (12)
Output: the functions u kh and v kh
Input: D k − 1 h , r kh , Q kh +1 , β V , λ
V kh +
( C, I h , J h ) =
max e Q kh +
( C, e, I h , J h )
Compute (cid:98) P
h
V
kh
+1 by (13)
Q kh ( C, I h , J h , e ) =
( r kh ( C, I h , J h , e ) + (cid:98) P
h V
kh +
( C, e ))
[ 0 , (cid:80) Hl = h W l ]
Output: function Q kh
5.3 Theoretical Results
In this section, we present our theoretical results, with the proofs deferred to Appendix D.
We start with the main theorems on the agents’ regret and the planner’s regret.
Theorem 5.4 (Agents’ Regret) .
With probability at least 1 − 2 δ , the agents’ regret can be bounded as R M ( K ) ≤ O ( d 2 ( (cid:80)
H h
=1 min {| I h | , | J h |} ) κ √ K ) where κ = log ( dK min( |I| , |J | ) /δ ) .
Comparing this result with Theorem 5.3 in Jagadeesan et al.
(2021), which proves the regret of their MatchLinUCB algorithm designed for the linear utility class under H = 1 , and ignoring the logarithmic term, we see that both have a linear dependence on the cardinality of the agents’ set, while we have an extra summation over the horizon H due to the sequential setting.
It might seem that their d -dependence is O ( d ) while ours is O ( d 2 ) , but this is because their feature is in R d while our feature Φ is d 2 -dimensional.
Therefore, the dominant term in our regret bound matches that of
Jagadeesan et al.
and our result can be viewed as an extension.
Theorem 5.5 (Planner’s Regret) .
Under Assumption 5.1, 5.2, assuming KH > 32 , there exists a problem-independent constant η > 0 , such that for any δ > 0 , setting λ = 1 , β V = ηd 2 (cid:0) (cid:80)
Hh =1 W h (cid:1) · √ ι where ι = log( dKH min {|I| , |J |} /δ ) and β u as given in Lemma C.1, the planner’s regret is bounded by R P ( K ) ≤ O ( ηd 5 / 2 H ( (cid:80)
Hh =1 W h ) ι √ K ) with probability at least 1 − δ .
Combining the above two theorems, we have shown that we ﬁnd the optimal policy for both the planner and agents at sublinear rates, and accordingly for the total regret.
Notably, the regret upper bound only depends on the size of the market through { W h } Hh =1 , which is instance-dependent.
Due to the sublinear regret, our algorithm can be further adapted to a PAC algorithm (cf.
Jin et al., 2018).
We also remark that our result with LSVI-type estimation can be naturally extended to Eluder dimension (Wang et al., 2020; Ayoub et al., 2020).
The key step in our analysis is to show that the estimated pseudo-reward function r kh satisﬁes optimism, i.e., r kh ≥ r h , and we need to ensure that r kh is not too far away from r h (Lemma C.2).
Due to space limit, we refer interested readers to Appendix C for a proof sketch of the main theory and an introduction of all the key technical lemmas.
The full proof is presented in Appendix D.
Conclusion
We propose a novel Markov matching market model and a general framework that incorporates max-weight matching and RL algorithms for efﬁcient online learning.
We prove that our proposed algorithms achieve sublinear regret under proper structural assumptions on the structural model.
Overall, our algorithm addresses the coupled challenges of sequential exploration, matching stability, and function approximation.
Acknowledgments and Disclosure of Funding
Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two
Sigma for their supports.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work is seeking to develop a mathematical understanding of dynamic matching.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use existing theorems and cite them appropriately.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Clariﬁcation of Notation
In this section, we provide a comprehensive clariﬁcation on the use of notation in this paper.
Throughout the paper, we use O ( · ) to hide problem-independent constants and use ( · ) [ a,b ] to denote the truncation into the range [ a, b ] .
I and J denote the sets of all side-1 and side-2 agents respectively.
Further, we use I ∈
I and J ∈ 2 J to denote any set of participating agents, which are the subsets of
I and J .
For any given stage h ∈ [ H ] , we use s h and ( C h , I h , J h ) interchangeably when describing any state s h ∈ S where S = C
× 2 I × 2 J .
Analogously, we also use a h and ( e h , X h , τ h ) interchangeably for the action a h ∈ A
where A = Υ
× X × T
We use π = { π h } Hh =1 to denote a policy, where each π h is deﬁned to be a mapping from S to a distribution ∆ A on A .
Therefore, for any h ∈ [ H ] and s h ∈ S , π h ( ·| s h ) denotes a probability distribution on A .
Note that because A = Υ × X × T , the policy π is a joint policy.
We may slightly abuse the notation in the paper and refer to π as the policy restricted on Υ only, whenever it is clear from the context.
In such case, we refer to π h ( ·| s h ) as a distribution on Υ only.
We also present the following table of notations.
The π in the superscript can be replaced by π k or π (cid:63) , where the former refers to the policy in episode k , and the latter refers to the optimal policy.
[Table]
value
[Table]
Q functions under π , optimal value functions
[Table]
the transition functions { P h } Hh =1 and reward functions { r h } Hh = V πh , Q πh , V (cid:63)h pseudo-value, pseudo-Q functions under π , optimal pseudo-value functions w.r.t.
the transition functions { P h } Hh =1 and reward functions { r h } Hh = V kh
Q kh estimated value, Q functions for stage h in episode k in Algorithmπ k
the policy followed by Algorithm 1 in episode k , where π k = { π k,h } Hh = π k,h the policy followed by Algorithm 1 at stage h in episode k
B Supplementary Information on Matching and Stability
In this section, we review some basics on the matching problem.
We ﬁrst introduce the classic problem of (static) matching with transfers and the notion of stability.
We then recap the primal-dual formulation that provides an efﬁcient way to solve a stable matching (Shapley and Shubik, 1971).
Finally, we give more details about Subset Instability and its properties.
B.1 Matching with Transferable Utilities
This section is a supplementary to Section 3.1 in the main text.
We introduce the two-sided static matching with transferable utilities.
Denote the sets of participating agents by I and J for two sides respectively.
A matching X ⊆
I × J is a set of pairs of agents, and ( i,
j ) ∈
X means i ∈
I is matched to j ∈ J .
Each agent is matched at most once.
We denote by X ( i ) =
j and X ( j )
=
i for any matched pair ( i, j ) ∈ X , while for any unmatched agent a ∈
I ∪ J , we write X ( a )
= a .
Matched agents receive utilities, denoted by u : I × J
→ R for agents in I and v : I × J
→ R for agents in J .
Speciﬁcally, if ( i, j ) ∈ X , then agent i receives an utility u ( i, j ) and agent j receives an utility v ( i, j ) .
Remaining unmatched agents receive zero utility.
With a slight abuse of notation, we overwrite u
( i, i ) = 0 and v ( j, j ) =
for any i ∈ I and j ∈ J .
In addition, there are utility transfers between (and only between) agents.
We denote the transfer function by τ : I × J
→ R such that for any agent a ∈
I ∪ J , τ ( a ) is the transfer received by agent a .
Since the transfers are within agents, we have (cid:88) a ∈
I ∪ J τ ( a ) = 0 .
We denote the market outcome by ( X, τ ) , under which the net utility received by an agent i ∈
I is u ( i, j ) + τ ( i ) if ( i, j ) ∈ X , and similarly for agents in J .
The notion of stable matching is as follows.
Deﬁnition B.1 (Stable matching) .
A matching-transfer pair ( X, τ ) on I, J is stable if:
The net utility of of any agent is non-negative, i.e.
[Equation]
for all i ∈ I and j ∈ J . 2.
There are no blocking pairs, i.e.
[Equation]
for all pairs ( i, j ) ∈ I × J .
Stable matching implies that no matched agents would rather be unmatched and no pair of agents can ﬁnd a transfer between themselves so that both would rather match with each other than follow ( X, τ ) .
The following proposition provides a fundamental and important max-weight interpretation for stable matchings.
Proposition B.2 (Shapley and Shubik 1971) .
For the matching with transfer problem, if ( X, τ ) is a stable matching under Deﬁnition B.1, then X must be the max-weight matching, i.e.,
[Equation]
where the maximum is over all matchings on I × J .
Therefore, by Proposition B.2, to maximize the total social welfare (i.e. sum of utilities), it sufﬁces to ﬁnd a stable matching.
But how?
This is answered in the next subsection.
B.2 The Linear Program and Dual Program
In this subsection, we explain how to ﬁnd a stable matching ( X, τ ) given input I, J, u, v , which gives rise to the algorithm OM (i.e. Algorithm 3) in the main text.
Shapley and Shubik (1971) showed that, assuming the utility functions are known, the stable ( X, τ ) can be found by solving the following linear program and its dual program (recapped from Section 4.1 in the main text):
[Equation]
and its dual program:
[Equation]
Denote the solution pair to the primal-dual problems by ( w, p ) .
Shapley and Shubik (1971) proved that ( w, p ) leads to a max-weight stable matching-transfer pair ( X, τ ) .
Speciﬁcally, it is proved that the vector w must have integer entries, i.e., w i,j = 0 or 1 , which naturally induces a matching X such that
( i, j
) ∈
X if and only if w i,j = 1 .
Correspondingly, the transfers are τ ( i ) =
p ( i ) − u ( i, X ( i )) for i ∈ I , and similarly for j ∈ J .
The above procedure constitutes the subroutine oracle OM as displayed in Algorithm 3.
It takes as input the sets of participating agents and estimated utility functions, then outputs the stable matching ( X, τ ) by solving the primal-dual linear program described above.
Note that the matching is only stable with respect to the estimated utility functions.
B.3 Details about Subset Instability
Next, we review the notion of Subset Instability and its properties.
We refer the interested reader to (Jagadeesan et al., 2021) for the full details.
Given a matching-pair ( X, τ ) , deﬁne its utility difference as
[Equation]
Recall the deﬁnition of Subset Instability:
Deﬁnition 4.1 (Subset Instability, Jagadeesan et al. 2021) .
Given any agent sets I, J and utility functions u, v : I × J → R , the Subset Instability SI ( X, τ ; I, J, u, v ) of the matching and transfer ( X, τ ) is deﬁned as
[Equation]
where X ( · ) and X (cid:48) ( · ) denotes the matched agent in matching X and X (cid:48) respectively.
By deﬁnition, Subset Instability indicates whether there exists any subset I (cid:48)
× J (cid:48) of agents who can achieve a higher total utility by taking some alternative matching X (cid:48) among themselves other than the current matching-transfer pair ( X, τ ) .
Thus, Subset Instability is an upper bound of the total utility difference, and quantiﬁes the distance from a proposed matching to the optimal stable matching, as summarized in the following proposition.
Proposition B.3 (Proposition 4.4 in Jagadeesan et al. 2021) .
The following holds for Subset Instability
Subset Instability is always nonnegative and is zero if and only if ( X, τ ) is stable matching.
Subset Instability is Lipschitz continuous with respect to the (cid:96) ∞ norm of the utility functions.
[Equation]
, (cid:101) v ) | ≤(cid:18) (cid:88)
i ∈
I (cid:107) u ( i, · ) − (cid:101) u ( i, · ) (cid:107) ∞ − (cid:88) j ∈ J (cid:107) v ( · , j ) − (cid:101) v ( · , j ) (cid:107) ∞ (cid:19) .
Subset Instability is always at least the utility difference (16) .
In our problem, this allows us to bound the total regret of the agents by the sum of Subset Instability of matchings across all episodes, as reﬂected in Proposition 4.2.
C Proof Sketch C.1 Optimistic Utility and Reward Estimates
The key step in our analysis is to show that the estimated pseudo-reward function r kh satisﬁes optimism, i.e., r kh ≥ r h , and we need to ensure that r kh is not too far away from r h .
The lemma below justiﬁes the optimism of utility estimates.
Lemma C.1 (UCB for Utility Estimates; proof in E.1) .
For any 0 < δ < 1 , set β u as β u = (cid:112) d 2 log[2(1 + d 2 K max h min {
| I h | , | J h |} ) / ( λδ )] + √ λd .
Then with probability at least 1 − δ , u kh and v kh in Algorithm 1 satisfy u kh ≥ u h and v kh ≥ v h .
Furthermore, | u kh ( · ) − u h ( · ) | and | v kh ( · ) − v h ( · ) | are bounded by 2 β u (cid:107) Φ ( · ) (cid:107) ( Σ kh ) − 1 .
Next, we explain why the optimism of utility estimates implies that of reward estimates.
By Lemma
C.1, we can write u kh = u h
+ b u,h and v kh = v h
+ b v,h where b u,h and b v,h are bonus functions satisfying b u,
[Equation]
[Equation] [Equation]
[Equation] [Equation]
) , b v,
[Equation]
[Equation] [Equation]
, j )
[Equation]
(cid:104)
[Equation] [Equation]
u (cid:107)
[Equation] [Equation]
[Equation] [Equation]
[Equation]
) (cid:107) ( Σ
[Equation]
) − 1 (cid:105) .
Lemma C.2 (Planner’s Optimism; proof in E.2) .
Under the event of Lemma C.1, it holds that for any ( C, e ) ∈
C × Υ ,
[Equation]
, i, j )) .
In the sequel, we denote by π k the policy whose market making part is greedy w.r.t.
Q kh and whose matching part chooses the max-weight stable matching given u kh and v kh .
C.2 Proof Sketch of Theorem 5.
By deﬁnition (9), the agents’ regret can be interpreted as the expected sum of total SI across all time steps, where the expectation is over the trajectory induced by π k for k ∈ [ K ] .
To bound the regret, we relate the expected SI in (9) with the realized SI via a martingale difference sequence.
Speciﬁcally, writing SI h
= SI ( s h , a h , u h , v h ) and SI kh =
SI ( s kh , a kh , u h , v h ) , we deﬁne the sum of differences as
[Equation]
We bound the difference (17) and the sum of realized SI (cid:80)
Kk =1 (cid:80)
Hh =1 SI kh separately, where the former is a sum of martingale difference sequences that concentrates and the latter can be bounded using the following lemma.
Lemma C.3 (Lemma 5.4 in Jagadeesan et al. 2021; proof in E.3) .
Under the event of Lemma C.1, we have
[Equation]
Remark C.4 .
Note that each implemented matching induces several utility observations at a time, so bounding the bonus sum for utilities has a resemblance to lazy policy updates in the online learning literature (Abbasi-Yadkori et al., 2011).
It particularly is similar to techniques used in the low switching cost problem in RL (Bai et al., 2019; Wang et al., 2021; Gao et al., 2021).
This will be clear in the proof of Theorem 5.4 in Appendix D.3.
C.3 Proof Sketch of Theorem 5.
Deﬁne the following functions δ kh and terms ζ 1 k,h , ζ 2 k,h :
[Equation]
To simplify the notation, in the following, we omit I h , J h from the arguments of the functions since we are conditioning on { I h , J h } Hh =1 being ﬁxed.
Lemma C.5 (Regret Decomposition of Planner; proof in E.4) .
The planner’s regret deﬁned by (8) satisﬁes
[Equation]
where the expectation E π (cid:63) [ ·| C k 1 ] is with respect to the trajectory { C h , e h } Hh =1 induced by the policy π (cid:63) conditioning on C
= C k 1 and (cid:104)· , ·(cid:105) Υ means sum over all e ∈ Υ .
In decomposition (19), term E 1 is controlled using a standard martingale concentration.
Next, to bound E 2 , we show that δ kh ≤ 0 with high probability, which implies that E 1 ≤ (cid:80)
Kk =1 (cid:80)
Hh =1 (cid:12)(cid:12) δ kh
( C kh , e kh ) (cid:12)(cid:12) .
Bounding each | δ kh ( C kh , e kh ) | by the corresponding optimistic bonus (cid:107) ψ ( C kh , e kh ) (cid:107) ( Λ k +1 h ) − 1 , we then apply Elliptical Potential lemma to get
[Equation]
Finally, for E 3 , note that by Algorithm 1, the market-making part of the policy π k,h is the greedy policy with respect to Q kh .
Based on this observation, it follows that (cid:80)
e ∈ Υ Q
kh
( C h , e )( π (cid:63)h ( e | C h )
− π k,
[Equation]
=
[Equation]
, so E
[Equation]
Combining yields the bound in Theorem 5.5.
Full details are presented in Appendix C.4.
Note that our notion of meta algorithm for Algorithm 1 is different from that of meta learning (Finn et al., 2017; Xu et al., 2021).
C.4 Proof of Theorem 5.
To prove Theorem 5.5, we need the following two lemmas which is helpful for bounding E 1 and E 2 .
Lemma C.6 (Proof in Section E.5) .
Under the setting of Theorem 5.5, with probability at least 1 − δ , for all ( h, k ) ∈ [ H ] × [ K ] and ( C, e ) ∈ C
×
Υ , it holds that
[Equation]
Lemma C.7 (Proof in Section E.6) .
For any δ > 0 , with probability at least 1 − δ , it holds that
[Equation]
We are now ready to prove Theorem 5.5.
Proof of Theorem 5.5.
By Lemma C.5, we bound the three terms separately.
Bound on E 3 .
According to Algorithm 1, the planner’s policy π k,h is the greedy policy with respect to Q kh .
It follows that
[Equation]
Therefore, we have E 3 ≤ 0 .
Bound on E 1 .
We apply Lemma C.6.
The E 1 term can be bounded as
[Equation]
Note that Λ kh
= λ
I + (cid:80) k − 1 t =1 ψ ( C th , e th ) ψ ( C th , e th ) (cid:62) by deﬁnition, where (cid:107) ψ (cid:107) 2 ≤ 1 by Assumption 5.2 and λ = 1 by our choice.
Thus we have Λ k +1 h
[Equation]
equivalently, ( Λ kh ) − 1 (cid:52) 2( Λ k +1 h ) − 1 , for all k .
Applying this to the above inequality, we get the ﬁnal bound for E 1 :
[Equation]
where the second step is by the Elliptical Potential Lemma (Lemma G.2).
Bound on E 2 .
By Lemma C.7, we have
[Equation]
Combining Lemma C.5, (20), (21) and E 3 ≤ 0 , we get that, with probability at least 1 − 3 δ ,
[Equation]
where the 1 − 3 δ probability is from the union bound on the events of Lemma C.1, Lemma C.6 and
Lemma C.7.
Since dKH min {|I| , |J |} /δ > 3 , replacing δ with δ/ 3 and absorbing the constant into the big-O notation, we ﬁnish the proof.
D Proof of the Main Theory D.1 Proof of Proposition 4.
Proof of Proposition 4.2.
By (3), the total regret can be written as
[Equation]
where V π k 1 is the pseudo-value function deﬁned by (7) corresponding to the pseudo-reward r h (deﬁned by (6)) and induced by the policy π k .
Note that here we only care about the Υ part of π k since matching-transfer has been maximized out by the deﬁnition of r h .
Now for any policy π ∈ Π where π = { π h } h ∈
[ H ] , there exists a counterpart π (cid:48)
= { π (cid:48) h } h ∈
[ H ] , such that π h ( C | s ) =
π (cid:48) h
( C | s ) for all C ∈ Υ and s ∈ S , whereas for the matching part π (cid:48) always chooses the stable matching w.r.t.
the true and unknown utility functions u h ( C h , e h , · , · ) and v h ( C h , e h , · , · ) .
Since π (cid:48) can be viewed as a function of π , we write π (cid:48)
=
π (cid:48) ( π ) .
Since the matching does not affect the context transition by assumption, and the stable matching maximizes total utility by Proposition
B.2, we then have that
[Equation]
where the fourth step is by deﬁnition of r h .
It follows that
R
[Equation] [Equation]
The second term in the R.H.S. can be written as
K
[Equation]
Note that r h ( s h , e h ) − r h
( s h , a h ) is exactly the utility difference deﬁned by (16).
Since Subset
Instability is at least the utility difference by Proposition B.
[Equation]
it follows that
[Equation]
h ∈ [ H ] (cid:21) =
R M ( K ) , where the last step is by the deﬁnition of R M ( K ) in (9).
Plugging into (22), we get
R ( K ) ≤ R P ( K ) + R M
( K ) .
This completes the proof.
D.2 Bounds for Regression Estimators
We ﬁrst bound the norm of the regression estimators in the algorithm.
Lemma D.1.
The regression estimators w kh in Algorithm 1 satisfy
[Equation]
Proof of Lemma D.1.
Consider w kh for arbitrary h, k .
For any vector v ∈ R d ,
[Equation]
where the ﬁrst inequality holds because of the truncation of Q kh and V kh ( C, I, J )
=
max e Q kh ( C, e, I, J ) , the second inequality is by the Cauchy-Schwarz inequality, and the last inequality is by Lemma G.1.
Since above holds for any v , we conclude that
[Equation]
For θ kh
[Equation]
let v ∈ R d × d .
Then by the same analysis we have
[Equation]
which implies
[Equation]
|} λ .
The same holds for γ kh .
D.3 Proof of Theorem 5.
We present the complete proof of Theorem 5.4.
Proof of Theorem 5.4.
Recall from Proposition B.3 that Subset Instability is at least the utility difference.
We thus have
[Equation]
For each k ∈ [ K ] , let F k − 1 denote all the history until the beginning of Episode k .
Then π k ∼ F k −and R Mk ∼ F k − 1 .
Furthermore, we have R Mk =
E
[ (cid:101) R Mk | F k − 1 ] , i.e., R Mk is the conditional expectation of the realized quantity (cid:101) R M k .
We can view (17) as the sum of a martingale difference sequence.
From Deﬁnition 4.1, it holds almost surely that
[Equation]
By Lemma G.3, we have that, for any 0 < δ < 1 , with probability at least 1 − δ ,
[Equation]
To bound the second term (cid:80) Kk =1 (cid:101) R Mk , note that by Lemma C.1 and Lemma C.3, we have
[Equation]
Note that by deﬁnition, we have
[Equation]
By picking λ = 1 and the assumption that (cid:107) Φ (cid:107) ≤ 1 , we have
[Equation]
X kh
[Equation]
It follows that Σ k +1 h (cid:52) (1 + min( | I h | , | J h | )) · Σ kh , and thus ( Σ kh ) − 1 (cid:52) (1 + min( | I h | , | J h | )) ·
( Σ k +1 h ) − 1 .
Combining with (24), we get that
K
[Equation]
where the second step is by min( | I h | , | J h | ) ≥ 1 .
To bound the summation in the bracket, we seek to use the elliptical potential lemma.
However, note tat the telescoping sum involves several utility observations (i.e. all ( i, j ) ∈ X kh ) at one time, instead of a single observation.
To address this issue, for any ( k, h ) , let’s index all the pairs ( i, j
) ∈
X kh by an index m = 1 , · · · , | X kh | .
Here the order of the indexing does not matter.
With a slight abuse of notation, we denote Φ kh ( m ) =
Φ ( C kh , e kh , i, j ) if ( i, j ) has index m in our indexing.
We also deﬁne, for n = 1 , ·
· · , | X kh | , that
[Equation]
By the above deﬁnition, we have Σ kh ( n 1 ) (cid:52) Σ kh ( n 2 ) for all 1 ≤ n 1 < n 2 ≤ | X kh | , and Σ k +1 h
= Σ k +1 h ( | X kh | ) .
It follows that
[Equation]
where the ﬁrst inequality is by Σ kh ( | X kh | ) − 1 (cid:52) Σ kh ( m ) − 1 , and the second inequality is by Lemma
G.2 with λ = 1 , Φ ∈ R d 2 , and | X kh | ≤ min( | I h , J h | ) .
Combining with (25), we get that
[Equation]
The ﬁnal bound follows by plugging in the expression of β u , and using the fact that W h ≤ min {| I h | , | J h |} .
The 1 − 2 δ probability comes from the union bound of the event of Lemma
C.1 and the event of (23).
E Proof of Lemmas E.1 Proof of Lemma C.
Proof of Lemma C.1.
Fix arbitrary h .
Since by Assumption 5.2, the noises in the observed utilities are independent and 1-sub-Gaussian
[Equation]
we can apply Theorem 2 in (Abbasi-Yadkori et al., 2011).
We then have that, with probability at least 1 − δ/ 2 , for any k ∈ [ K ] ,
[Equation]
We then have that for any C, e, i, j , u
[Equation]
where the ﬁrst inequality follows from the Cauchy-Schwarz inequality and the second in equality is my the choice of β u .
Similarly, we have
[Equation]
The same argument holds for v kh − v h with probability at least 1 − δ/ 2 .
Finally, we take a union bound and conclude that the event holds with probability at least 1 − δ .
E.2 Proof of Lemma C.
Proof of Lemma C.2.
Note that r h is the maximum value of the linear program (4) with ( I h , J h , u h , v h ) , and r kh is the maximum value of (4) with ( I h , J h , u kh , v kh ) .
Since u h ≤ u kh and v h ≤ v kh by Lemma C.1 and the weights in (4) are restricted to be, it immediately holds that r h ( C, e, I h , J h ) ≤ r kh ( C, e, I h , J h ) .
On the other hand, denote by X h the matching corresponding to r h .
It follows that
[Equation]
where the inequality is due to the sub-optimality of X kh under ( u h , v h ) .
The result follows by using
[Equation]
and rearranging the terms.
This completes the proof.
E.3 Proof of Lemma C.
Lemma C.3 is a restatement of Lemma 5.4 in Jagadeesan et al.
For completeness, we present the proof here.
Speciﬁcally, we prove a general version of Lemma C.3, which is Lemma E.1 below.
Lemma E.1.
Let ( u, v ) and ( (cid:98) u, (cid:98) v ) be two pairs of utility functions on the agents set I, J , such that each of u , v , (cid:98) u , (cid:98) v maps from I × J to R .
Let ( (cid:98) X, (cid:98) τ ) be a stable matching on ( I, J ) w.r.t.
the utility functions ( (cid:98) u, (cid:98) v ) .
Suppose u ≤ (cid:98) u , v ≤ (cid:98) v .
Then the Subset Instability of ( (cid:98) X, (cid:98) τ ) w.r.t.
the utility ( u, v ) satisﬁes
[Equation]
Proof of Lemma E.1.
Deﬁne the function
[Equation]
j
[Equation]
( v ( X (
[Equation] [Equation]
+
[Equation]
( j )) ,
where I (cid:48) × J (cid:48)
⊂ I × J .
Then by Deﬁnition 4.1 of Subset Instability, SI ( X, τ ; I, J, u, v ) =
max I (cid:48)
× J (cid:48)
⊂ I × J f ( I (cid:48) , J (cid:48) , X, τ ; u, v ) , and thus
SI ( (cid:98) X, (cid:98) τ ; I, J, u, v ) − SI ( (cid:98) X, (cid:98) τ ; I, J, (cid:98) u, (cid:98) v ) ≤
[Equation]
u, (cid:98)
[Equation]
(cid:105) .
To bound f ( I (cid:48) , J (cid:48) , (cid:98) X, (cid:98) τ ; u, v ) − f
( I (cid:48) , J (cid:48) , (cid:98) X, (cid:98) τ ; (cid:98) u, (cid:98) v ) , we decompose f
( I (cid:48) , J (cid:48) , (cid:98) X, (cid:98) τ ; u, v ) − f
( I (cid:48) , J (cid:48) , (cid:98) X, (cid:98) τ ; (cid:98) u, (cid:98) v ) =
[Equation]
i
[Equation]
I
[Equation]
u ( i, X (cid:48)
( i ))
+ (cid:88)
j
[Equation]
J (cid:48) v ( X (cid:48)
( j ) , j ) (cid:19) − max X (cid:48) (cid:18) (cid:88) i ∈ I (cid:48)
(cid:98) u ( i, X (cid:48)
( i ))
+ (cid:88) j ∈ J (cid:48) (cid:98) v ( X (cid:48)
( j ) , j ) (cid:19)
+ (cid:88)
i
[Equation]
I (cid:48)
(cid:16)(cid:98)
[Equation]
( i, (cid:98)
[Equation]
( i ))
[Equation]
(cid:98)
[Equation]
( i ) (cid:17)
[Equation]
(cid:88)
j
[Equation]
(cid:48) (cid:16)(cid:98)
[Equation]
( (cid:98)
[Equation]
( j )
[Equation] [Equation]
(cid:98)
[Equation]
( i ) (cid:17)
[Equation]
(cid:88) i ∈ I (cid:48)
(cid:16)
[Equation]
( i, (cid:98) X ( i ))
[Equation]
(cid:98) τ
( i ) (cid:17)
[Equation]
(cid:88)
[Equation]
(cid:48) (cid:16)
[Equation]
[Equation] [Equation]
(cid:98)
[Equation]
[Equation] [Equation]
= max
[Equation]
(cid:18)
[Equation]
i
[Equation] [Equation]
(cid:98)
[Equation] [Equation]
(cid:48) ( j )
[Equation]
) (cid:19)
[Equation]
(cid:123)(cid:122)
[Equation]
Term I is nonpositive.
To see this, note that by assumption u ≤ (cid:98) u and v ≤ (cid:98) v .
Thus the max-weight matching on I (cid:48)
× J (cid:48) w.r.t.
the utility functions ( u, v ) cannot exceed the max-weight matching on
I (cid:48)
× J (cid:48) w.r.t.
( (cid:98) u, (cid:98) v ) .
To bound term II , note that all the transfers (cid:98) τ ( i ) and (cid:98) τ ( j ) in the expression cancel out, and it follows that
[Equation]
where the inequality follows from the assumption that u ≤ (cid:98) u and v ≤ (cid:98) v
and I (cid:48)
× J (cid:48) ⊆
I × J .
This ﬁnishes the proof.
Proof of Lemma C.3.
For any ﬁxed k ∈ [ K ] and h ∈ [ H ] , replace I, J in Lemma E.1 with I h , J h , and replace u , v , (cid:98) u , (cid:98) v in with u h ( C kh , e kh , · , · ) , v h ( C kh , e kh , · , · ) , u kh ( C kh , e kh , · , · ) , v kh ( C kh , e kh , · , · ) .
Since
X kh is the stable matching w.r.t.
u kh ( C kh , e kh , · , · ) , v kh ( C kh , e kh , · , · ) , we can replace (cid:98) X with X kh .
It then follows from Lemma E.1 that
SI kh
[Equation]
where the second step holds because the true utility and the estimated utility are zero for unmatched agents under X kh , and the last step is by the deﬁnition of the bonus function
b u,h and b v,h .
E.4 Proof of Planner’s Regret Decomposition
We ﬁrst restate the lemma in its complete form.
Lemma C.5.
The planner’s regret deﬁned by (8) can be decomposed as
[Equation]
where the expectation is over the trajectory { C h ,
e h } h ∈ [ H ] induced by executing the policy π (cid:63) (on the choice of e ∈ Υ only), and conditioning on { I h , J h } h ∈
[ H ] being ﬁxed.
Proof of Lemma C.5.
Recall the deﬁnition of the planner’s regret from (8).
We write
[Equation]
where s
[Equation]
J h ) by our notation.
Term I. We deﬁne two operators J (cid:63)h and J k,h as
[Equation]
for all ( k, h ) ∈
[ H ] ×
[ K ] , s ∈ C × 2 I × 2 J , and function Q :
C × I × J × Υ
→ R .
Then by deﬁnition, we have V kh =
J k,h Q kh , and V (cid:63)h = J (cid:63)h Q (cid:63)h .
It follows that
[Equation]
kh
[Equation]
Q kh (cid:17) = (cid:16) J (cid:63)h Q (cid:63)h − J (cid:63)h Q kh (cid:17) + ξ kh , (26) where ξ kh :=
J (cid:63)h Q kh − J k,h Q kh .
Also, by the deﬁnition of δ kh in (18), we have
Q (cid:63)h
[Equation]
Q kh =
[Equation]
+ P
[Equation]
(cid:63)h +1 − (cid:16)
[Equation]
h + P
[Equation]
V kh +1 (cid:17) +
δ kh =
P h (cid:16)
[Equation]
(cid:63)h +1 − V kh +1 (cid:17) + δ kh .
Combining with (26), we get
[Equation]
P h (cid:16) V (cid:63)h +1 − V kh +1 (cid:17) + J (cid:63)h δ kh + ξ kh .
Applying the above equation recursively, we have that for any C 1 , I 1 , J 1 ,
[Equation]
where the second step holds because V (cid:63)H +
=
V kH
+1 = 0 .
By deﬁnition of the operators, it is clear that here the expectation E π (cid:63) is over the trajectory { ( C h , e h ) }
h ∈
[ H ] induced by the planner executing the policy π k to choose actions in Υ .
Term II.
First note that by (18), the function δ kh : C × 2 I × 2 J × Υ
→ R can be written as
[Equation]
where the last step is by Q π k h
= r h
+ P
h V π k h
+1 .
Then for any h
[Equation]
we can write
[Equation]
where the second step is by (28).
Applying the above equation recursively [Equation] we get
[Equation]
where we use V (cid:63)H +
=
V kH
+1 =
again.
Combining (27) and (29), we get
K (cid:88)
[Equation]
which ﬁnishes the proof.
E.5 Proof of Lemma C.
Proof of Lemma C.6.
By the deﬁnition of Q kh in Algorithm 1, the function
δ kh satisﬁes
[Equation]
+1 ( C, I, J, e ) .
In the sequel, we will show that r kh and (cid:98) P
h V kh +1 upper bound r h and P
h V kh +
respectively.
We ﬁrst consider the term r h ( C, e, I h , J h ) − r kh
( C, e, I h , J h ) .
It immediately follows from Lemma
C.2 that, under the event of Lemma C.1,
[Equation]
We now consider (cid:98) P
h V
kh +1 .
Since we are conditioning on { I h , J h } which is independent of anything else, we can essentially treat it as a deterministic sequence.
By (10), for any V kh +1 , there exists w kh ∈ R d such that for any C, e , P h
V
kh +
( C, e, I h , J h ) =
ψ ( C, e ) (cid:62) w kh .
This is because
[Equation]
, I h + [Equation] J h +1 )
d µ h ( C (cid:48) ) (cid:105) .
Then we can write
[Equation]
where the ﬁrst step uses the construction of w kh in Algorithm 1 and the last step uses the construction of Λ kh .
For convenience, we write s th =
( C th , I h , J h ) and ( s th , e th ) =
( C th , e th , I h , J h ) .
It follows from the Cauchy-Schwarz inequality that the ﬁrst part on the R.H
[Equation]
S. of (31) satisﬁes
[Equation]
[Equation]
In the following, to bound (31)
[Equation]
we ﬁrst bound the self-normalized stochastic process using tools from self-normalized martingale.
The issue is that, according to Algorithm 1, the function V kh + depends on the ﬁrst ( k − 1) episodes and thus depends on the trajectory { ( C th , e th , C th +1 ) } t ∈
[ k − 1] .
We thus adopt a common approach to solve this issue by considering the function class containing each value function estimator V kh +1 .
The covering trick is a commonly used technique (Ling et al., 2019), and we will discuss the detail of the construction of the function class and its covering in Section F.
The covering trick allows us to get the following lemma.
Lemma E.2.
Under the setting of Theorem 5.5, with probability at least 1 − δ , for any ( h, k ) ∈ [ H ] × [ K ] , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] log
[Equation]
Proof of Lemma E.2.
See Appendix E.7 for the proof.
For the term (cid:13)(cid:13) w kh (cid:13)(cid:13) ( Λ kh ) − 1 , by Assumption 5.2 and | V kh | ≤ H min {|I| , |J |} , we have [Equation] w
[Equation]
=
[Equation]
C V kh
[Equation]
Combine the above inequality with Lemma E.2 and (32)
[Equation]
and we get that
[Equation]
where
[Equation]
It remains to show that there exists choice of β V (or equivalently, the constant η in the description of
Theorem [Equation] 5) such that
[Equation]
Speciﬁcally, we show that we can pick some constant η and set
[Equation]
Indeed, plug in the expression of β V and β u and we get
[Equation]
where ι = log(3 dKH min {|I| , |J |} /δ )
Since ι > log 3 , it sufﬁces to pick η such that
[Equation]
which ﬁnishes the proof.
E.6 Proof of Lemma C.
Proof of Lemma C.7.
The lemma can be proven by standard martingale concentration, similar to the analysis in (Cai et al., 2020;
Yang et al., 2020).
Speciﬁcally, we deﬁne the σ -ﬁelds as
[Equation]
By deﬁnition, it is clear that these σ -ﬁelds form a ﬁltration under the dictionary order on the index tuple ( k, h, o ) where o ∈ { 0 , 1 } .
Note that for any ( k, h ) ∈
[ K ] ×
[ H ] , since V kh , Q kh and the policy π k are all functions of the ﬁrst ( k − 1) episodes, they are all F k, 1 , 1 -measurable.
As a result, ζ 1 k,h is F k,h, 1 -measurable and ζ 2 k,h is
F k,h, 2 -measurable, for all ( k, h ) .
According to Algorithm 1, the planner’s action e kh ∼ π k ( ·| C kh ) .
This indicates that condition on C kh ,
V π k h ( C kh , I h , J h )
− Q π k h ( C kh , e kh , I h , J h )
= 0 .
Also, since e kh ← argmax e ∈ Υ Q kh ( s kh , e, I h , J h ) , and
V kh ( C, I, J ) =
max e ∈ Υ Q kh ( C, e, I, J ) for all ( C, I, J ) by the algorithm
, we have V kh
( C kh , I h , J h ) −
Q kh ( C kh , e kh , I h , J h )
= 0 .
Altogether we have ζ 1 k,h = 0 for all ( k, h ) ∈
[ K ] ×
[ H ] .
For ζ 2 k,h , ﬁrst note that there is no dependence issue between the value functions and the trajectory since V kh +1 and V π k h +1 are functions of the ﬁrst ( k − 1) episodes.
Then since C kh +1 ∼ P h ( ·| C kh , e kh ) , we have
[Equation]
[Equation]
Thus we conclude that { ( ζ 1 k,h , ζ 2 k,h ) }
( k,h ) ∈
[ K ] × [ H ] is a martingale difference sequence.
Since V kh ,
Q kh , V π k h , Q π k h are all bounded by W h for all h, k , we apply the Azuma-Hoeffding inequality (Lemma
G.3) and get that,
[Equation]
Equivalently, with probability at least 1 − δ , we have
[Equation]
which ﬁnishes the proof.
E.7 Proof of Lemma E.
Proof of Lemma E.2.
By the analysis in Section F, there exists a function class V containing all V kh , and the (cid:15) -covering number of V is given by Lemma F.1.
Also note that by the truncation, we have | V kh | ≤ (cid:80)
Hl = h W l .
Then we apply Lemma G.5 with R =
(cid:80)
Hl = h W l and combine with Lemma F.1, and get that, ﬁx any 0 < (cid:15) < 1 , with probability at least 1 − δ/H , for all k ∈
[ K ] , (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88) t =
[Equation]
(cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ( Λ kh ) −
[Equation]
Let λ
= 1 , pick (cid:15) =
d 2 (cid:16)(cid:80)
Hl = h W l (cid:17) /K
and then take a union bound over h ∈
[ H ] , we get (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k
[Equation]
[Equation]
[-V kh +
-]
F Covering Number of Function Classes
In this section, we will construct a function class V that provably contains V kh for all ( k, h ) ∈
[ K ] ×
[ H ] .
And we will compute the covering number of V .
The result is summarized by Lemma F. below.
Lemma F.1.
Assume KH > 32 .
For any (cid:15) < 1 , the (cid:15) -covering number of V is upper bounded by log N V (cid:15) ≤ 6 d 2 log (cid:18) 1 + dKH min {|I| , |J |} · β V min { λ [Equation] [Equation] (cid:19)
[Equation]
[Equation]
[Equation]
1 + d min {|I| , |J |} ·
[Equation]
u min { λ, 1 } · [Equation] (cid:19) . To prove Lemma F.1, we will ﬁrst construct a function class G that contains all (cid:98) P
h V
kh +1 , R that contains all r kh , and Q that contains all Q kh .
The formal deﬁnition of these classes will be given in the following.
We also introduce the following technical lemma.
Lemma F.2 (Covering Number of (cid:96) 2 Ball) .
For any (cid:15) > 0 , the (cid:15) -covering number of the (cid:96) 2 ball in R d with radius L is upper bound by (1 + 2 L/(cid:15) )
d .
The proof of this classical result can be found in, for example, Chapter 5 in (Vershynin, 2010).
Now we prove Lemma F.1.
F.1 Proof of Lemma F.
Covering of (cid:98) P
h V
kh +1 .
The next lemma is helpful to bound the covering number of the function class containing the function (cid:98) P h V k h
+1 .
The proof is the same as that of Lemma D.6.
in (Jin et al., 2020).
Lemma F.3.
Let G
= G ( L, B ) denote the function class with functions mapping from C × Υ to R and of the following parametric form
[Equation]
where (cid:107) w (cid:107) ≤ L , β ∈ [0 , B ] , and λ min ( Λ ) ≥ λ > 0 .
Assume (cid:107) ψ ( · , · ) (cid:107) 2 ≤ 1 .
Let N (cid:15) denote the (cid:15) -covering number of G with respect to the (cid:96) ∞ distance.
Then we have
[Equation]
Suppose for now that there exists L w > 0 such that (cid:107) w kh (cid:107) 2 ≤ L w for all ( k, h ) .
The value of L w will be determined later.
By applying Lemma F.3 with L =
L w and B = β V , we get the following upper bound on the (cid:15) -covering number of the function class G ( l w , β V ) which contains all (cid:98) P
h V
kh +1 :
[Equation]
Covering of r kh .
We now deﬁne a function class R which provably contains all the pseudo-reward estimates r kh .
Formally speaking, by Algorithm 2, the functions in R are parametrized by the utility function estimates u kh and v kh .
Denote the functions class containing all these utility function estimates by U .
Then according to Algorithm 4, any function u : C × Υ
× I × J
→ R in U can be written as
[Equation]
where Φ ∈ R d 2 satisfying (cid:107) Φ (cid:107) 1 ≤ 1 by Assumption 5.2, (cid:107) θ (cid:107) ≤ L u for some L u > 0 to be determined, and Σ ∈ R d × R d such that λ min ( Σ ) ≥ λ .
Since the truncation is a contraction mapping, by Lemma F.3, the (cid:15) -covering number of U is upper bounded by
[Equation]
We now consider the function class R .
We formally deﬁne R to be the function class such that any function r ∈ R can be represented by
[Equation]
for some u, v ∈ U .
Let functions r 1 , r 2 ∈ R be parametrized by u 1 , v 1 and u 2 , v 2 respectively,
such that
[Equation]
for all C ∈ C , e ∈ υ , I ⊂ I and J ⊂ J .
According to the linear program 4, there exist some weights w 1 = { w 1 ,i,j } ( i,j ) ∈
I × J and w 2 = { w 2 ,i,j } ( i,j ) ∈
I × J , such that
[Equation]
It follows that
[Equation]
where the second step holds because w 1 is the optimal weight given u 1 and v 1 and w 2 satisﬁes the constraint of the linear program with u 1 and v 1 .
The same upper bound holds for the difference ( r 2 − r 1 ) .
Therefore, for any I, J , we have
[Equation]
Since I ⊂ I and J ⊂ J , in order for (cid:107) r 1 − r 2 (cid:107) ∞ ≤ (cid:15) to hold, it sufﬁces to have (cid:107) u 1 − u 2 (cid:107) ∞ ≤ (cid:15) (cid:48) and (cid:107) v 1 − v 2 (cid:107) ∞ ≤ (cid:15) (cid:48) where (cid:15) (cid:48) =
(cid:15)/ (2 min {|I| , |J |} ) .
Therefore, by (36), the (cid:15) -covering number of
R satisﬁes
[Equation]
In the above we have shown that the function class R contains all r kh and G contains all (cid:98) P
h V
kh +1 .
We now deﬁne the function class Q :=
R + G as
[Equation]
Then it immediately follows from the algorithm that Q contains all Q kh functions.
By (34) and (37), the (cid:15) -covering number of the function class Q can be upper bounded by
[Equation]
Since by construction, V kh ( C, I, J ) = max e Q kh ( C, e, I, J ) and taking the maximum is a contraction mapping, the upper bound in (38) also holds for log N V (cid:15) .
By Lemma D.1, we can pick
[Equation]
λ and
L u =
[Equation]
|I| , |J |} λ .
From the above analysis, we can simplify the R.H.S. of (38) and get the desired bound for the covering number of V .
This ﬁnishes the proof of Lemma F.1.
G Auxiliary Lemmas
Lemma G.1 (Lemma D.1 in Jin et al. 2020) .
For arbitrary d , let Λ k
= λ I d + (cid:80) k − 1 t =1 x t x (cid:62) t where x t ∈ R d and λ > 0 .
Then k
[Equation]
Proof of Lemma G.1.
We can write
[Equation]
Denote the eigenvalue of (cid:80) k − 1 t =1 x (cid:62) t x t as { λ 1 , · · · , λ d } , and decompose (cid:80) k − 1 t =1 x (cid:62) t x =
U diag ( λ 1 , · · · , λ d ) U (cid:62) .
Then we have Λ k = U diag ( λ 1 + λ, · · · , λ d + λ ) U (cid:62) .
It follows that tr (( Λ k ) − 1 (cid:80) k − 1 t =1 x (cid:62) t x t ) =
(cid:80) dj
=
λ j / ( λ j + λ ) ≤ d .
The next is the well-known Elliptical Potential Lemma (Cesa-Bianchi and Lugosi, 2006; AbbasiYadkori et al., 2011; Lattimore and Szepesvári, 2020).
Lemma G.2 (Elliptical Potential Lemma) .
For arbitrary d , let Λ k
= λ I d + (cid:80) k − 1 t =1 x t x (cid:62) t where x t ∈ R d and λ > 0 .
Then
[Equation]
Lemma G.3 (Azuma-Hoeffding inequality (Azuma, 1967)) .
Let { X t } ∞ t =0 be a real-valued martingale such that for every t ≥ 1 , it holds that | X t − X t − 1 | ≤ B t for some B t ≥ 0 .
Then
[Equation]
G.1 Concentration Inequalities for Self-normalized Martingales
Theorem G.4 (Hoeffding inequality for Self-normalized martingales (Abbasi-Yadkori et al., 2011)) .
Let { η t } ∞ t =1 be a real-valued stochastic process.
Let {F t } ∞ t
=
be a ﬁltration, such that η t is F tmeasurable.
Assume η t | F t − 1 is zero-mean and R -subgaussian for some R > 0 , i.e.,
[Equation]
Let { x t } ∞ t =1 be an R d -valued stochastic process where x t is F t − 1 -measurable.
Assume Λ 0 is a d × d positive deﬁnite matrix, and let Λ t = Λ 0 + (cid:80) ts =
1 x s x (cid:62) s .
Then, for any δ > 0 , with probability at least 1 − δ , for all t > 0 ,
[Equation]
Lemma G.5 (Lemma D.4 in Jin et al. 2020) .
Let V be a function class such that any V ∈ V maps from S → R and (cid:107) V (cid:107) ∞ ≤ R .
Let {F t } ∞ t
=
be a ﬁltration.
Let { s t } ∞ t =1 be a stochastic process in the space S such that s t is F t -measurable.
Let { x } ∞ t =0 be an R d -valued stochastic process such that x t is F t − 1 -measurable
and (cid:107) x (cid:107) 2 ≤ 1 .
Let Λ k
= λ
I + (cid:80) k − 1 t =1 x t x (cid:62) t .
Then for any δ > 0 , with probability at least 1 − δ , for any k , and any V ∈ V , we have
[Equation]
, where N V (cid:15) is the (cid:15) -covering number of V with respect to the (cid:96) ∞ distance.
Proof of Lemma G.5.
For any V ∈ V , there exists V (cid:48) in the (cid:15) -covering such that
[Equation]
and (cid:107) ∆ V (cid:107) ∞ ≤ (cid:15).
Then we have
[Equation]
( Λ k
[Equation]
For the ﬁrst term on the R.H.S., we apply Theorem G.4 and a union bound to the (cid:15) -covering.
The second term can be bound by 8 k 2 (cid:15) 2 /λ by using (cid:107) x t (cid:107) 2 ≤ 1 , λ min ( Λ k ) ≥ λ and (cid:107) ∆ V (cid:107) ∞ ≤ (cid:15) .