Online Minimax Multiobjective Optimization: Multicalibeating and Other Applications
Abstract
We introduce a simple but general online learning framework in which a learner plays against an adversary in a vector-valued game that changes every round.
Even though the learner’s objective is not convex-concave (and so the minimax theorem does not apply), we give a simple algorithm that can compete with the setting in which the adversary must announce their action ﬁrst, with optimally diminishing regret.
We demonstrate the power of our framework by using it to (re)derive optimal bounds and efﬁcient algorithms across a variety of domains, ranging from multicalibration to a large set of no regret algorithms, to a variant of Blackwell’s approachability theorem for polytopes with fast convergence rates.
As a new application, we show how to “(multi)calibeat” an arbitrary collection of forecasters — achieving an exponentially improved dependence on the number of models we are competing against, compared to prior work.
Introduction
We introduce and study a simple but powerful framework for online adversarial multiobjective minimax optimization.
At each round t , an adaptive adversary chooses an environment for the learner to play in, deﬁned by a convex compact action set X t for the learner, a convex compact action set Y t for the adversary and a d -dimensional continuous loss function (cid:96) t : X t × Y t → [ − 1 , 1] d that, in each coordinate, is convex in the learner’s action and concave in the adversary’s action.
The learner then chooses an action or distribution over actions, x t , and the adversary responds with an action y t .
This results in a loss vector (cid:96) t ( x t , y t ) , which accumulates over time.
The learner’s goal is to minimize the maximum accumulated loss over each of the d
[Equation]
One may view the environment chosen at each round t as deﬁning a zero-sum game in which the learner wishes to minimize the maximum coordinate of the resulting loss vector.
The objective of the learner in the stage game in isolation can be written as:
[Equation]
Unfortunately, although (cid:96) tj is convex-concave in each coordinate, the maximum over coordinates does not preserve concavity for the adversary.
Thus the minimax theorem does not hold, and the value of the game in which the learner moves ﬁrst (deﬁned above) is larger than the value of the game in which the adversary moves ﬁrst— that is, w t L > w T A , where w t A is deﬁned as:
[Equation]
Nevertheless, ﬁxing a series of T environments chosen by the adversary, this deﬁnes in hindsight an aspirational quantity W TA = (cid:80) Tt =1 w tA , summing the adversary-moves-ﬁrst value of the constituent zero sum games.
Despite the fact that these values are not individually obtainable in the stage games, we show that they are approachable on average over a sequence of rounds, i.e., there is an algorithm for the learner that guarantees that against any adversary,
[Equation]
Our derivation is elementary and based on a minimax argument, and is a development of a gametheoretic argument from the calibration literature due to Hart [2020] and Fudenberg and Levine [1999].
The generic algorithm plays actions at every round t according to a minimax equilibrium strategy in a surrogate game that is derived both from the environment chosen by the adversary at round t , as well as from the history of play so far on previous rounds t (cid:48) < t .
The loss in the surrogate game is convex-concave (and so we may apply minimax arguments), and can be used to upper bound the loss in the original games.
We then show that this simple framework can be instantiated to derive a wide array of optimal bounds, and that the corresponding algorithms can be derived in closed form by solving for the minimax equilibrium of the corresponding surrogate game.
Despite its simplicity, our framework has a number of applications to online learning— we sketch these below.
“Multi-Calibeating”: Foster and Hart [2021] recently introduced the notion of “calibeating” an arbitrary online forecaster: making online calibrated predictions about an adversarially chosen sequence of inputs that are guaranteed to have lower squared error than an arbitrary predictor f , where the improvement in error approaches f ’s calibration error in hindsight.
Foster and Hart give two methods for calibeating an arbitrary collection of predictors F simultaneously, but these methods have an exponential and polynomial dependence in their convergence bounds on |F| , respectively.
Using our framework, we can derive optimal online bounds for online multicalibration [HébertJohnson et al., 2018, Gupta et al., 2022], and as an application, obtain bounds for calibeating arbitrary collection of models with only a logarithmic dependence on |F| .
Our algorithm naturally extends to the more general problem of online “multi-calibeating” — i.e. combining the goals of online multicalibration and calibeating.
Namely, we give an algorithm for making real-valued predictions given contexts from some space Θ .
The algorithm is parameterized by (i) a collection G ⊆ 2 Θ of (arbitrary, potentially intersecting) subsets of Θ that we might envision to represent e.g. different demographic groups in a setting in which we are making predictions about people; and (ii) an arbitrary collection of predictors F .
We promise that our predictions are calibrated not just overall, but simultaneously within each group g ∈ G — and moreover, that we calibeat each predictor f ∈ F not just overall, but simultaneously within each group g ∈ G .
We do this by proving an online analogue of what Hébert-Johnson et al.
[2018] call a “do no harm” property in the batch setting using a similar technique: multicalibrating with respect to the level sets of the predictors.
Fast Polytope Blackwell Approachability: We give a variant of Blackwell’s Approachability
Theorem [Blackwell, 1956] for approaching a polytope.
Standard methods approach a set in Euclidean distance, at a rate polynomial in the payoff dimension.
In contrast, we give a dimension-independent approachability guarantee: we approximately satisfy all halfspace constraints deﬁning the polytope, after logarithmically many rounds in the number of constraints, a signiﬁcant improvement over a polynomial dimensional dependence in many settings.
It is equivalent to the results of Perchet [2015], which show that the negative orthant R d ≤ 0 is approachable in the (cid:96) ∞ metric with a log( d ) dependence in the convergence rate.
This result follows immediately from a specialization of our framework that does not require changing the environment at each round, highlighting the connection between our framework and approachability.
We remark that approachability has been extended in a number of ways in recent years
[Mannor et al., 2014a,b, Perchet and Mannor, 2013].
However most of our other applications take advantage of the ﬂexibility of our framework to play a different game at each round (which can be deﬁned by context) with potentially different action sets, and so do not directly follow from Blackwell approachability.
Therefore, while many of our regret bounds could be derived from approachability to the negative orthant by enlarging the action space exponentially to simulate aspects of our framework, this approach would not easily lead to efﬁcient algorithms.
Recovering Expert Learning Bounds: Algorithms and optimal bounds for various expert learning problems fall naturally out of our framework as corollaries.
This includes external regret [Vovk, 1990, Littlestone and Warmuth, 1994], internal and swap regret [Foster and Vohra, 1998, Hart and
Mas-Colell, 2000, Blum and Mansour, 2007], adaptive regret [Littlestone and Warmuth, 1994, Hazan and Seshadhri, 2009, Adamskiy et al., 2012], sleeping experts [Freund et al., 1997, Blum, 1997, Blum and Mansour, 2007, Kleinberg et al., 2010], and the recently introduced multi-group regret [Blum and Lykouris, 2020, Rothblum and Yona, 2021].
Multi-group regret refers to a contextual prediction problem in which the learner gets contexts from Θ before each round.
It is parameterized by a collection of groups G ⊆ 2 Θ : e.g., if the predictions concern people, G may represent an arbitrary, intersecting set of demographic groups.
Here the “experts” are different models that make predictions on each instance; the goal is to attain no-regret not just overall, but also on the subset of rounds corresponding to contexts from each g ∈ G .
Multi-group regret, like multicalibration, is one of the few solution concepts in the algorithmic fairness literature known not to involve tradeoffs with overall accuracy [Globus-Harris et al., 2022].
Blum and Lykouris [2020] derived their algorithm for online multigroup regret via a reduction to sleeping experts, and Gupta et al.
[2022] derived their algorithm for online multicalibration via a direct argument.
Here we derive online algorithms for both multicalibration and multigroup regret as corollaries of the same fundamental framework.
General Framework 2.1 The Setting
A Learner (she) plays against an Adversary (he) over
rounds t ∈ [ T ] := { 1 , . .
, T } .
Over these rounds, she accumulates a d -dimensional loss vector ( d ≥ 1 ), where each round’s loss vector lies in [ − C, C ] d for some C > 0 .
At each round t , the Learner and the Adversary interact as follows:
Before round
t
, the Adversary selects and reveals to
the Learner
an
environment
comprising: (a) The Learner’s and Adversary’s respective convex compact action sets X t , Y t embedded into a ﬁnite-dimensional Euclidean space; (b) A continuous vector loss function (cid:96) t ( · , · ) : X t × Y t → [ − C, C ] d , with each (cid:96) tj ( · , · ) : X t × Y t → [ − C, C ] (for j ∈ [ d ] ) convex in the 1st and concave in the 2nd argument.
The Learner selects some x t ∈ X t . 3.
The Adversary observes the Learner’s selection x t , and responds with some y t ∈ Y t . 4.
The Learner suffers (and observes) the loss vector (cid:96) t ( x t , y t ) .
The Learner’s objective is to minimize the value of the maximum dimension of the accumulated loss vector after T rounds—in other words, to minimize: max j ∈ [ d ] (cid:80) t ∈
[ T ] (cid:96) tj ( x t , y t ) .
To benchmark the Learner’s performance, we consider the following quantity at each round t :
Deﬁnition 2.1 (The Adversary-Moves-First (AMF) Value at Round t ) .
The Adversary-Moves-First value of the game deﬁned by the environment ( X t , Y t , (cid:96) t ) at round t is:
[Equation]
If the Adversary had to reveal y t ﬁrst and the Learner could best respond, w tA would be the smallest value of the maximum coordinate of (cid:96) t she could guarantee.
However, the function max j ∈ [ d ] (cid:96) tj ( x t , y t ) is not convex-concave (as the max does not preserve concavity); hence the minimax theorem does not apply, making this value unobtainable for the Learner, who is in fact obligated to reveal x t ﬁrst.
However, we can deﬁne regret to a benchmark given by the cumulative AMF values of the games:
Deﬁnition 2.2 (Adversary-Moves-First (AMF) Regret) .
On transcript π t = { ( X s , Y s , (cid:96) s ) , x s , y s } ts =1 , we deﬁne the Learner’s Adversary Moves First (AMF) Regret for the j th dimension at time t to be:
R tj ( π t ) :
= t (cid:88) s =
(cid:96) sj ( x s , y s ) − t (cid:88) s = w sA .
The overall AMF Regret is then deﬁned as follows: R t ( π t )
=
max j ∈
[ d ] R tj .
Again, the game played at each round is not convex-concave, so we cannot get R T ≤ 0 .
Instead, we will aim to obtain sublinear AMF regret, worst-case over adaptive adversaries: R T =
o ( T ) .
2.2 General Algorithm
Our algorithmic framework will be based on a natural idea: instead of directly grappling with the maximum coordinate of the cumulative vector valued loss, we upper bound the AMF regret with a one-dimensional “soft-max” surrogate loss function, which the algorithm will then aim to minimize.
Deﬁnition 2.3 (Surrogate loss) .
Fixing a parameter η ∈ (0 , 1) , we deﬁne our surrogate loss function (that implicitly depends on the transcript π t through the respective round t ) as:
L
[Equation]
= (cid:88)
[Equation]
for t ∈ [ T ] , and L 0 :=
d.
This surrogate loss tightly bounds the AMF regret R T = max j ∈ [ d ] R Tj :
Lemma 2.1.
The Learner’s AMF Regret is upper bounded using the surrogate loss as: R T ≤ ln L T η .
Next we observe a simple but important bound on the per-round increase in the surrogate loss.
Lemma 2.2.
For any t , any transcript through round t , and any η ≤ 12 C , it holds that:
[Equation]
The proof is very simple (see Appendix B.1): we write out the quantity L t − L t − 1 , use the deﬁnition of AMF regret R t , and then bound L t − L t − 1 via the inequality e x ≤ 1 + x + x 2 for | x | ≤ 1 .
We now exploit Lemma 2.2 to bound the ﬁnal surrogate loss L T and obtain a game-theoretic algorithm for the Learner that attains this bound.
While the above steps should remind the reader of a standard derivation of the celebrated Exponential Weights algorithm via bounding a log-sum-exp potential function, the next lemma is the novel ingredient that makes our framework signiﬁcantly more general by relying on Sion’s powerful generalization of the Minimax Theorem to convex-concave games.
Lemma 2.3.
For any η ≤ 12 C , the Learner can ensure that the ﬁnal surrogate loss is bounded as:
[Equation]
Proof sketch; see Appendix B.1.
Deﬁne, for t ∈ [ T ] , continuous convex-concave functions u t :
X t × Y t → R by: u t ( x, y ) := (cid:80)
j ∈ [ d ] exp (cid:0) ηR t − 1 j
[Equation]
(cid:0) (cid:96) tj ( x, y ) − w tA (cid:1) .
If the Learner can ensure u t ( x t , y t ) ≤ 0 on all rounds t ∈
[ T ] regardless of the Adversary’s play, then Lemma 2.2 implies L t ≤ (cid:0) 4 η 2 C 2 + 1 (cid:1)
L t − 1 for all t ∈
[ T ] , leading to the desired bound on L T .
Due to the continuous convexconcave nature of each u t (inherited from the loss coordinates (cid:96) tj ), we can apply Sion’s Minimax
Theorem to conclude that: min x t ∈X t max y t ∈Y t u t ( x t , y t ) = max y t ∈Y t min x t ∈X t u t ( x t , y t ) .
In words, the Learner has a so-called minimax-optimal strategy x t , that achieves (worst-case over all y t ∈ Y t ) value u t ( x t , y t ) as low as if the Adversary moved ﬁrst and the Learner could bestrespond.
But in the latter counterfactual scenario, using the deﬁnitions of u t and the Adversarymoves-ﬁrst value w t A , we can easily see that by best-responding to the Adversary, the Learner would always guarantee herself value ≤ 0 : that is, max y t ∈Y t min x t ∈X t u t ( x t , y t ) ≤ 0 .
Thus, min x t ∈X t max y t ∈Y t u t ( x t , y t ) , and so by playing minimax-optimally at every round t ∈ [ T ] , the
Learner will guarantee u t ( x t , y t ) ≤ 0 for all t , leading to the desired regret bound.
In fact, via a simple algebraic transformation (see Appendix B.1) taking advantage of the values w tA being independent of the actions x t , y t , we can explicitly express the Learner’s minimax optimal strategies at all rounds as: argmin
[Equation]
Together with the proof of Lemma 2.3, this immediately gives the following algorithm for the Learner that achieves the desired bound on L T (and thus, as we will show, on the AMF regret R T ).
Algorithm 1: General Algorithm for the Learner that Achieves Sublinear AMF Regret for rounds t = 1 , . .
, T do
Learn adversarially chosen X t , Y t , and loss function (cid:96) t ( · , · ) .
Let
[Equation]
for j ∈ [ d
[Equation]
Play
[Equation]
argmin
[Equation]
Observe the Adversary’s selection of y t ∈ Y t .
Theorem 2.1 (AMF Regret guarantee of Algorithm 1) .
For any T ≥ ln d , Algorithm 1 with learning rate η = (cid:113) ln d 4 TC 2 obtains, against any Adversary, AMF regret bounded by: R T ≤ 4 C √ T ln d.
Indeed, using Lemma 2.1, then Lemma 2.3, then 1 + x ≤ e x , and ﬁnally setting η = (cid:113) ln d 4 TC 2 , we get:
R T ≤ ln L T η ≤
[Equation]
d
[Equation]
η
[Equation]
C 2 +
[Equation]
T
[Equation]
Remark 2.1.
Our framework is easy to adapt to the setting where the Learner randomizes, at each round, amongst a ﬁnite set of actions A t (i.e. X t = ∆ A t ), and wishes to obtain in-expectation and high-probability AMF regret bounds.
This is useful in all our applications below.
Additionally, our AMF regret bounds are robust to the Learner playing only an approximate (rather than exact) minimax strategy at each round: we use this to derive our simple multicalibration algorithm below.
See Appendix B.2 for both these extensions.
Deriving No-X-Regret Algorithms from Our Framework
The core of our framework — the Adversary-Moves-First regret — is strictly more general than a very large variety of known regret notions including: external, internal, swap, adaptive, sleeping-experts, multigroup, and wide-range ( Φ ) regret .
Speciﬁcally, in Appendix C, we use our framework to derive simple O ( √ T ) -regret algorithms for what we call subsequence regret , which encapsulates all these regret forms.
In each of these cases, our generic algorithm is efﬁcient, and often specializes (by computing a minimax equilibrium strategy in closed form) to simple combinatorial algorithms that had been derived from ﬁrst principles in prior work.
We note that in any problem that involves context or changing action spaces (as the sleeping experts problem does), we are taking advantage of the ﬂexibility of our framework to present a different environment at every round, which distinguishes our framework from more standard Blackwell approachability arguments.
In fact, as we will see in
Section 5 below, our framework recovers
fast Blackwell approachability as a special case.
For our general subsequence regret algorithms, please see Appendix C. Now, as a warm-up application of our framework, we directly instantiate it for the simplest case of obtaining O ( √ T ) external regret.
Simple Learning From Expert Advice: External Regret In the classical experts learning setting
Littlestone and Warmuth [1994], the Learner has a set of pure actions (“experts”) A .
At the outset of each round t ∈ [ T ] , the Learner chooses a distribution over experts x t ∈ ∆ A .
The Adversary then comes up with a vector of losses r t =
( r ta ) a ∈A ∈ [0 , 1] A corresponding to each expert.
Next, the Learner samples a t ∼ x t , and experiences loss corresponding to the expert she chose: r ta t .
The Learner also gets to observe the entire vector of losses r t for that round.
The goal of the
Learner is to achieve sublinear external regret — that is, to ensure that the difference between her cumulative loss and the loss of the best ﬁxed expert in hindsight grows sublinearly with T :
[Equation]
Theorem 3.1.
Fix a ﬁnite pure action set A for the Learner and a time horizon T ≥ ln |A| .
Then, an instantiation of our framework’s Algorithm B.1 lets the Learner achieve the following regret bounds:
[Equation]
with prob.
[Equation]
Proof.
We instantiate (the probabilistic version of) our framework (see Section B.2.1).
At all rounds, the Learner’s pure action set is A , and the Adversary’s strategy space is the convex and compact set [0 , 1] |A| , from which each round’s collection ( r ta ) a ∈A of all actions’ losses is selected.
Next, we deﬁne a |A| -dimensional loss function (cid:96) t = ( (cid:96) tj ) j ∈A , where each coordinate loss (cid:96) tj expresses the regret of the Learner’s chosen action a relative to action j ∈ A : (cid:96) tj ( a, r t ) =
r ta − r tj , for a ∈ A , r t ∈ [0 , 1] |A| .
By Theorem B.1, E (cid:104) max j ∈A (cid:80) t ∈
[ T ] (cid:96) tj ( a t , r t ) − (cid:80)
t ∈ [ T ] w t A (cid:105) ≤ 4 (cid:112) T ln |A| , where w t A is the
AMF value at round t .
Using this AMF regret bound, we can bound the Learner’s external regret as:
[Equation]
It thus remains to show that the AMF value w tA ≤ 0 for all t .
This holds, since if the Learner knew the Adversary’s choice of losses ( r ta ) a ∈A before round t , then picking the action a ∈ A with the smallest loss r ta would get her 0 regret in that round.
This gives the in-expectation regret bound; the high-probability bound follows in the same way from Theorem B.2.
A bound of (cid:112) T ln |A| is optimal for external regret in the experts learning setting, and so serves to witness the optimality of our framework’s general AMF regret bound in Theorem 2.1.
In fact, the above instantiation of Algorithm B.1 yields the classical Exponential Weights algorithm
Littlestone and Warmuth [1994]: at each round t , the action a t is sampled with Pr[ a t =
j ] ∼ exp (cid:16) − η (cid:80) t − 1 s =1 r sj (cid:17) , for j ∈ A .
We denote this distribution by EW η ( π t − 1 )
∈ ∆( A ) .
Indeed, given the above deﬁned loss (cid:96) t , the Learner solves the following problem at each round:
[Equation]
where χ tj =
[Equation]
si ) . That is, the per-coordinate weights ( χ tj ) j ∈A themselves form the Exponential Weights distribution with rate η .
For any choice of r t by the Adversary, the quantity inside the expectation, (cid:96) tj ( a, r t ) =
r ta − r tj , is antisymmetric in a and j : that is, (cid:96) tj ( a, r t ) =
− (cid:96) ta ( j, r t ) .
Due to this antisymmetry, no matter which r t gets selected by the Adversary, by playing a ∼ EW η ( π t − 1 ) the Learner obtains
E a,j ∼ EW η ( π t − 1 ) (cid:2) r ta − r tj (cid:3) = 0 , thus achieving the value of the game.
It is also easy to see that x t = EW
η ( π t − 1 ) is the unique choice of x t that guarantees nonnegative value, hence Algorithm B.1, when specialized to the external regret setting, is equivalent to the Exponential Weights Algorithm C.1.
[Equation]
Hence, the AMF value is indeed nonpositive at each round: w tA =
[Equation]
, r t ) ≤ 0 .
Multicalibration and Multicalibeating
We now apply our framework to derive an online contextual prediction algorithm which simultaneously satisﬁes a (potentially very large) family of strong adversarial accuracy and calibration conditions.
Namely, given an arbitrarily complex family G of subsets of the context space (we call them “groups”, a term from the fairness literature), the predictor will be both calibrated and accurate on each group g ∈ G (that is, over those online rounds when the context belongs to g ).
The accuracy benchmark that we aim to satisfy was recently proposed by Foster and Hart [2021], who called it calibeating : given any collection F of online forecasters, the goal is (intuitively) to “beat” the (squared) error of each f ∈ F by at least the calibration score of f .
In Section 4.1, we use our framework to rederive the online multigroup calibration (known as multicalibration ) algorithm of Gupta et al.
In Section 4.2, we show that by appropriately augmenting the original collection of groups G , this algorithm will, in addition to multicalibration, calibeat any family of predictors f ∈ F on every group g ∈ G , which we call multicalibeating .
4.1 Multicalibration
Setting There is a feature (or context ) space
Θ encoding the set of possible feature vectors representing individuals θ ∈ Θ .
There is also a label space [0 , 1] .
At every round t ∈ [ T ] :
The Adversary announces a particular individual θ t ∈ Θ , whose label is to be predicted; 2.
The Learner predicts a label distribution x t over [0 , 1] ; 3.
The Adversary observes x t , and ﬁxes the true label distribution y t over [0 , 1] ; 4.
The (pure) guessed label a t ∼ x t and the (pure) true label
b t ∼ y t are sampled.
Objective: Multicalibration The Learner is initially given an arbitrary collection G ⊆ 2 Θ of protected population groups .
Her goal, multicalibration , is empirical calibration not just marginally over the whole population, but also conditionally on individual membership in each g ∈ G .
Formally, for any n ≥ 1 we let the n -bucketing of the label interval [0 , 1] be its partition into subintervals [0 , 1 /n ) , . . .
, [1 − 2 /n, 1 − 1 /n ) , [1 − 1 /n, 1] .
The i th of these intervals (buckets) is denoted B in .
Deﬁnition 4.1 ( ( α, n ) -Multicalibration with respect to G ) .
Fix a real α > 0 and an integer n ≥ 1 .
Given the transcript of the interaction { ( a t , b t ) } t ∈ [ T ] , the Learner’s sequence of guessed labels { a t } t ∈ [ T ] is ( α, n ) -multicalibrated with respect to the collection of groups G if:
T (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) T (cid:88) t = 1 θ t ∈ g · 1 a t ∈ B in · ( b t − a t ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ α, for every group g ∈ G and every bucket B in (for i ∈ [ n ] ).
Using our framework, we now derive the guarantee on α that matches that of Gupta et al.
Theorem 4.1 (Multicalibration) .
Fix a family of groups G , a time horizon T ≥ ln(2 |G| n ) , and any natural n, r ≥ 1 .
Then, our framework’s Algorithm B.1 can be instantiated as Algorithm D.1 to produce ( α, n ) -multicalibrated predictions w.r.t.
G , where α satisﬁes (over transcript randomness): E
[Equation]
[Equation]
G
[Equation]
n
[Equation]
T and
[Equation]
Proof sketch; see Appendix D for details.
The adversary’s strategy space will be Y =
The learner will randomize over A r = { 0 , 1 / ( rn ) , 2 / ( rn ) , . .
, 1 } for an integer r ≥ 1 (to ensure continuity of the loss functions that we are about to deﬁne), i.e., her strategy space is X = ∆
A r .
Loss functions: The deﬁnition of multicalibration consists of 2 |G| n constraints (one for each ± sign, group g , and bucket i ) of the following form: ± 1 T (cid:80) Tt =1 1 θ t ∈ g · 1 a t ∈ B in · ( b t − a t ) ≤ α .
Thus, we deﬁne (for each t ∈
[ T ] , σ = ± 1 , g , and i ) a loss function over ( a t , b t ) ∈
A r × Y as: (cid:96) t i,g,σ ( a t , b t ) :
= σ · 1 θ t ∈ g · 1 a t ∈ B in · ( b t − a t ) .
Now, deﬁning a 2 |G| n dimensional loss vector (cid:96) t := (cid:0) (cid:96) ti,g,σ (cid:1) i ∈ [ n ] ,g ∈G ,σ ∈{− 1 , 1 } for each t ∈ [ T ] recasts multicalibration in our framework as requiring that max
[Equation]
Bounding the AMF regret: To bound the Adversary-Moves-First value with these loss functions, suppose the Adversary announces b t ∈ [0 , 1] .
Then, we easily see that by (deterministically) responding with a t = argmin a ∈A r
| b t − a | , for all σ, g, i , (cid:96) ti,g,σ ( a t , b t ) ≤ 12 rn .
Hence,
[Equation]
for every t ∈ [ T ] .
Now, for T ≥ ln(2 |G| n ) , the AMF regret R T = max i ∈ [ n ] ,g ∈G ,σ ∈{− 1 , 1 } (cid:80) Tt =1 (cid:96) ti,g,σ ( a t , b t ) − (cid:80) Tt =1 w tA , by our framework’s guarantees, satisﬁes E [ R T ] ≤ 4 (cid:112) T ln(2 |G| n ) over the Learner’s randomness.
Since (cid:80) Tt =1 w tA ≤
T 2 rn , we get E [max i ∈
[ n ] ,g ∈G ,σ ∈{− 1 , 1 } (cid:80) Tt =1 (cid:96) ti,g,σ ( a t , b t )] ≤ T 2 rn + 4 (cid:112) T ln(2 |G| n ) .
This gives ( α, n ) -multicalibration with E [ α ] ≤ 1 T (cid:16) T 2 rn + 4 (cid:112) T ln(2 |G| n ) (cid:17) = 12 rn + 4 (cid:113) ln(2 |G| n ) T .
The high-probability bound on α is obtained similarly.
Simplifying Learner’s algorithm:
To attain the AMF value w tA = 12 rn at each round, our framework has the Learner solve a linear program (that encodes her minimax strategy).
However, she can obtain the almost optimal value 1 rn without solving an LP: this observation gives Algorithm D.1 (see
Appendix D).
The guarantees on α only differ from optimal ones by replacing 12 rn → 1 rn .
4.2 Multicalibeating
We now give an approach to “beating” arbitrary collections of online forecasters via online multicalibration.
The goal, called calibeating by Foster and Hart [2021] who introduce the problem, is to make calibrated forecasts that are more accurate than each of an arbitrary set of forecasters, by exactly the calibration error in hindsight of that forecaster.
They achieve optimal calibeating bounds for a single forecaster, but their extension to calibeating multiple forecasters incurs at least a polynomial dependence on the number of forecasters.
We achieve a logarithmic dependence on the number of forecasters.
Additionally, we are able to simultaneously calibeat forecasters on all (big enough) subgroups in some set G , with still only a logarithmic dependence on |G| and the number of forecasters in the group-wise convergence bound.
We call this multicalibeating.
We now give an overview of our setting, results, and techniques.
For full details, see Appendix E.
Setting The Learner (predictor a = { a t } t ∈ [ T ] ) and the Adversary (true labels b = { b t } t ∈ [ T ] ) interact in the same way as in Section 4.1, but the Adversary additionally reveals to the Learner a ﬁnite set of forecasters F , where each f ∈ F is a function f : Θ → D f .
Here D f ⊂ [0 , 1] is assumed to be a ﬁnite set of all possible forecasts that f makes: it will characterize the level sets of f .
We often suppress the dependence on the transcript, denoting f t ∈ D
f the forecast at time t .
The Learner’s goal is to “improve on” the forecasts of all f ∈ F , for some suitable scoring of the predictions.
We measure the Learner’s and the forecasters’ accuracy via the squared error, alternatively known as the Brier score.
Deﬁnition 4.2 (Brier Score) .
The Brier score of a forecaster f over all rounds t ∈
[ T ] is deﬁned as:
[Equation]
The Brier score can be decomposed into so-called calibration and reﬁnement parts.
The former quantiﬁes the extent to which the predictor is calibrated, while the latter expresses the average amount of variance in predictions within every calibration bucket.
To deﬁne this decomposition, we need some extra notation.
We denote by S i the subsequence of days on which the Learner’s prediction is in bucket i .
Similarly, S d ( f ) (eliding ( f ) when clear from context) denotes days on which forecaster f predicts d .
We let S di ( f ) =
S i ∩ S d ( f ) .
Finally, we use bars to indicate average predictions over given subsequences.
For instance, ¯ a ( S ) is the Learner’s average prediction over a given subsequence S .
Deﬁnition 4.3 (Calibration and Reﬁnement) .
The calibration score K and reﬁnement score R of a forecaster f over the full transcript π T are deﬁned as:
[Equation]
Fact 1 (Calibration-Reﬁnement Decomposition of Brier Score
[DeGroot and Fienberg, 1983]) .
[Equation]
The goal of calibeating is to beat the forecaster’s Brier score by an amount equal to its calibration score.
Or equivalently, to attain a Brier score (almost) equal to the reﬁnement score of the forecaster.
Deﬁnition 4.4 (Calibeating) .
The Learner’s predictor a is said to τ -calibeat a forecaster f if:
B a ( π T ) ≤ R f ( π T ) + τ.
We will now extend the deﬁnition of calibeating simultaneously along two natural directions.
First, we will want to calibeat multiple forecasters at once.
The second extension is that we will want to calibeat the forecasters not just overall, but also on each of the subsequences corresponding to each “population group” g ∈ G in a given family of subpopulations G ⊆ 2 Θ .
Deﬁnition 4.5 (Multicalibeating) .
Given a family of forecasters F , groups G ⊆ 2 Θ , and a mapping β : F × G → R ≥ 0 , the Learner’s predictor a is an ( F , G , β ) -multicalibeater
if for every g ∈ G :
[Equation]
Note that ( { f } , { Θ } , β ( f, Θ) := τ ) -multicalibeating is equivalent to τ -calibeating a forecaster f .
We ﬁrst show how to calibeat a single forecaster (Deﬁnition 4.4).
The modularity of multicalibration will then let us easily extend this result to multiple forecasters and population subgroups.
The idea is to show that if our predictor is multicalibrated with respect to the level sets of f , then we achieve calibeating.
Hébert-Johnson et al.
[2018] give a similar bound in the batch setting.
We denote the collection of level sets of f as: S ( f ) := { θ ∈ Θ : f ( θ ) =
d } d ∈ D f .
Theorem 4.2 (Calibeating One Forecaster) .
Suppose that the Learner’s predictions a are ( α, n ) - multicalibrated on the collection of groups S ( f ) ∪ { Θ } .
Then the Learner is ( α, n ) -calibrated on Θ , and she ( αn ( | D f | + 2) + 2 n ) -calibeats forecaster f .
Proof sketch.
We show that a has small calibration score, and reﬁnement score close to that of f .
Step 1: Replace B a with a surrogate Brier score B an .
Consider a (pseudo-)predictor ˜ a given by ˜ a t = ¯ a ( S i at ) for t ∈ [ T ] (where i a t is the bucket of a t ).
That is, whenever a t ∈ B in , ˜ a t predicts the average of a over all such rounds s ∈ [ T ] that a s ∈ B in .
This is a pseudo-predictor, as the bucket averages of a are unknown until after round T .
Thus, ˜ a has precisely n level sets, unlike a .
Now, we deﬁne B an , K an , R an to be the Brier, calibration, and reﬁnement scores of ˜ a .
We can show
B a ≤ B an + 1 /n , allowing us to switch to bounding the more manageable Brier loss B an = K an + R an .
Step 2:
Bound the surrogate calibration score K an .
Since the Learner is ( α, n ) -calibrated on the domain Θ , the calibration error per level set is at most α .
There are n level sets, so K an ≤ αn.
Step 3:
Bound the surrogate reﬁnement score R an .
We connect R f and R an via a joint reﬁnement score: R f × a , which measures the average variance of the partition generated by all intersections of the level sets of a and f .
The ﬁner the partition, the smaller the reﬁnement score, so R f ≥ R f × a .
Next, informally, multicalibration ensures that a has already “captured” most of the variance explained by f .
Therefore, reﬁning a ’s level sets by f does little to reduce variance.
More precisely, we show that R an ≤ R f × a + αn ( | D f | + 1) + 1 n .
Combining with our previous inequality, we have:
R an ≤ R f + αn
( | D f | + 1) + 1 n .
Combining the above, we get: B a ≤ R an + K an + 1 n ≤ ( R f + αn
( | D f | + 1) + 1 n ) + αn + 1 n .
Calibeating many forecasters Generalizing the above construction, we can easily calibeat any collection of forecasters F on the entire context space Θ : it sufﬁces to ask for multicalibration with respect to the level sets of all forecasters, i.e. (cid:16)(cid:83)
f ∈F S ( f ) (cid:17) ∪ { Θ } .
Theorem 4.2 applies separately to each f ; the only degradation in the guarantees will come in the form of a larger α , since we are asking for multicalibration with respect to more groups than before.
But this effect will be small, since α depends on the number of required groups |G (cid:48) | as O ( (cid:112) ln |G (cid:48) | ) .
See Corollary E.2.
However, to fully satisfy Deﬁnition 4.5 of multicalibeating, we need to calibeat all f ∈ F on all groups g ∈ G in a given collection G ⊆ 2 Θ .
For that, we simply extend the above construction by requiring multicalibration with respect to all pairwise intersections of the forecasters’ level sets with the groups g ∈ G .
By further augmenting this collection with the protected groups G themselves, we ﬁnally achieve our ultimate goal: simultaneous multicalibeating and multicalibration.
Theorem 4.3 (Multicalibeating + Multicalibration) .
Let G ⊆ 2 Θ , and F some set of forecasters f : Θ → D f .
The multicalibration algorithm on G (cid:48) :
= (cid:16)(cid:83) f ∈F { g ∩ S : ( g, S ) ∈
G × S ( f ) } (cid:17) ∪ G with parameters r, n ≥ 1 , after T rounds, attains expected ( F , G , β ) -multicalibeating, where:
E [ β ( f, g )] ≤ 2 n + | D f | +2 r ·| S ( g ) | /T +4 n ( | D f | +2) (cid:114) 1 | S ( g ) | 2 /T ln (cid:16) 2 n |G| (1 + (cid:80) f
| D f | ) (cid:17) ∀ f ∈ F , g ∈ G , while maintaining ( α, n ) -multicalibration on G , with: E [ α ] ≤ 1 rn + (cid:114)
T ln (cid:16) 2 n |G| (1 + (cid:80) f | D f | ) (cid:17) .
In particular, for any group g occurring more than T − 1 / 2 of the time, we asymptotically converge ton -calibeating as T → ∞ , thus combining the goals of online multicalibration and multigroup regret.
Polytope Blackwell Approachability
Consider a setting where the Learner and the Adversary are playing a repeated game with vectorvalued payoffs, in which the Learner always goes ﬁrst and aims to force the average payoff over the entire interaction to approach a given convex set.
Blackwell’s Theorem [1956] states that a convex set is approachable if and only if it is response-satisﬁable (roughly, for any choice of the
Adversary, the Learner has a response forcing the one-round payoff inside the convex set).
The rate of approachability typically depends on the dimension of the payoff vectors.
This is a specialization of our framework to the case of a ﬁxed environment across rounds.
Thus our framework can be used to obtain a dimension-independent rate bound in the fundamental case where the approachable set is a convex polytope .
Our bound is only logarithmic in the polytope’s number of facets, and is achievable via an efﬁcient convex-programming based algorithm.
Let us formalize our setting.
In rounds t = 1 , 2 , . . . , the Learner and the Adversary play a repeated game.
Their respective pure strategy sets are A and Y , where A is a ﬁnite set and Y ⊆
R m (for some integer m ≥ 1 ) is convex and compact.
The game’s utility function is λ -dimensional (for some integer λ ≥ 1 ), continuous, concave in the second argument, and is denoted by u : A × Y → R λ .
At each round t , the Learner plays a mixed strategy x t ∈ ∆
A , the Adversary responds with some y t ∈ Y , and the Learner then samples a pure action a t ∼ x t .
This gives rise to the utility vector u ( a t , y t ) .
The average play up to any round t ≥ 1 is then deﬁned as ¯ u t = 1 t (cid:80) ts =1 u ( a s , y s ) .
The target convex set that the Learner wants to approach is
a polytope P ( H ) ⊆
R λ , deﬁned as the intersection of a ﬁnite collection of halfspaces H =
( h α,β ) , where for any given α ∈ R λ , β ∈ R we denote h α,β = { x ∈ R λ : (cid:104) α, x (cid:105) − β ≤ 0 } .
Finally, by way of normalization, consider any two dual norms || · || p and || · || q .We require, ﬁrst, that || α || p ≤ 1 and | β | ≤ 1 for each halfspace
h α,β ∈ H ; and second, that the payoffs be in the || · || q -unit ball: || u ( a, y ) ||
q ≤ 1 for a ∈ A , y ∈ Y .
Theorem 5.1 (Polytope Blackwell Approachability) .
Suppose the target convex polytope P ( H ) is response-satisﬁable , in the sense that for any Adversary’s action y ∈ Y , the Learner has a mixed response x ∈ ∆
A that places the expected payoff inside P ( H ) : that is, E a ∼ x [ u ( a, y )]
∈ P ( H ) .
Then, P ( H ) is approachable , both in expectation and with high probability with respect to the transcript of the interaction.
Namely, the Learner has an efﬁcient convex programming based algorithm which simultaneously offers both following guarantees (see Appendix F for the proof):
For any margin (cid:15) > 0 , the average play ¯ u t up to any round t ≥ 64ln |H| (cid:15)will satisfy E (cid:2) max h α,β ∈H ( (cid:104) α, ¯ u t (cid:105) − β ) (cid:3) ≤ (cid:15).
For any δ ∈ (0 , 1) , the average play ¯ u t up to any round t ≥ ln |H| will satisfy max h α,β ∈H ( (cid:104) α, ¯ u t (cid:105) − β ) ≤(cid:114)T ln (cid:16) |H| δ (cid:17) with probability at least 1 − δ.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This concern is not directly applicable here: this is a theory paper whose aim is to present a unifying rigorous framework encompassing many disparate, both novel and classical, theoretical results.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Full proofs of all results are in the Appendix, and the main part contains sketches of some of them.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]