A Theory of Learning with Competing Objectives and User Feedback
Abstract
Large-scale deployed learning systems are often evaluated along multiple objectives or criteria.
But, how can we learn or optimize such complex systems, with potentially conﬂicting or even incompatible objectives?
How can we improve the system when user feedback becomes available, feedback possibly alerting to issues not previously optimized for by the system?
We present a new theoretical model for learning and optimizing such complex systems.
Rather than committing to a static or pre-deﬁned tradeoff for the multiple objectives, our model is guided by the feedback received, which is used to update its internal state.
Our model supports multiple objectives that can be of very general form and takes into account their potential incompatibilities.
We consider both a stochastic and an adversarial setting.
In the stochastic setting, we show that our framework can be naturally cast as a Markov Decision Process with stochastic losses, for which we give efﬁcient vanishing regret algorithmic solutions.
In the adversarial setting, we design efﬁcient algorithms with competitive ratio guarantees.
We also report the results of experiments with our stochastic algorithms validating their effectiveness.
Introduction
Learning algorithms trained on large amounts of data are increasingly adopted in a variety of applications and form the engine driving complex large-scale systems such as e-commerce platforms, online advertising auctions and recommender systems.
Their system designer must take into account multiple metrics when optimizing them [Kaminskas and Bridge, 2016, Masthoff, 2011, Lin et al., 2019].
As an example, consider the case of a recommendation system for recipes, videos or fashion.
There is no single metric that deﬁnes what a good recommendation engine should do.
One needs to carefully take into consideration metrics measuring the quality of recommendations provided to end-users, their relevance and utility, the long-term growth of the content creators, and the overall revenue generated for the hosting platform.
Furthermore, it is crucial to consider the risk of bias in these systems [Speicher et al., 2018, Xiao et al., 2017, Holstein et al., 2019].
Hence, additional metrics may need to be incorporated, such as performance across demographic groups, geographical locations or other identity terms.
This can easily lead to hundreds of metrics that need to be simultaneously optimized for user satisfaction.
Further complicating the above scenario is the fact that often the multiple metrics considered are incompatible and inherently in conﬂict with each other [Kleinberg et al., 2017, Sener and Koltun, 2018, Jin, 2006].
For instance, in the context of a recommendation system, there is a tension between maximizing revenue via ad placements and maximizing end-user “happiness”.
Another tension may be between maximizing quality versus diversity of recommendations.
In many cases, resolving such conﬂicts may force the designer to make hard choices among notions that seem perfectly reasonable in isolation, weighing in current use-patterns, wins and losses.
An illuminating example is the analysis of the COMPAS tool for predicting recidivism by Angwin et al.
The authors showed that, among black defendants who do not recidivate, the tool predicted incorrectly at twice the rate than it did for white defendants who did not recidivate, i.e., the tool was unfair according to the false positive rate metric.
The creator of the tool, Northpointe, responded by demonstrating that the tool was fair according to other natural measures such as AUC (Area Under the ROC Curve), for which each group had similar values.
Later work showed that this tension is inherent and that it is often impossible to simultaneously satisfy multiple seemingly natural criteria [Kleinberg et al., 2017]
(see also [Feller et al., 2016]).
The above discussion raises the question of how one should deﬁne the optimal trade-off among multiple conﬂicting metrics to optimize for user satisfaction.
A natural approach is to deﬁne the trade-offs in a static manner, either by using domain knowledge and human expertise, or by analyzing past historical data.
Another line of work studies optimization in the presence of multiple objectives by designing algorithms that compete with any linear combination of the objectives [Mohri et al., 2019, Cortes et al., 2020] or by designing pareto-optimal solutions [Sener and Koltun, 2018, Shah and Ghahramani, 2016].
However, these solutions may be sub-optimal for the richer situation where user feedback is available.
While algorithms tailored to a speciﬁc metric or a combination of metrics would be effective at ﬁrst, experience shows that they become non pertinent over time: once a system is deployed and it interacts with its end-users, inefﬁciencies in the system design emerge, as evident via the user feedback, which in turn could lead one to prefer metrics originally not accounted for [Liu et al., 2018].
In the context of the COMPAS tool discussed above, this would correspond to the situation where user complaints make the system designers change loss function to ensure equal false-positive rates.
The important aspect is that the underlying data distribution on which the tool has been trained does not change, new user feedback simply alerts the designers to short-comings of the system.
Motivated by the above, in this work, we present a theoretical data-driven model for optimizing multiple conﬂicting metrics by taking into account the user feedback.
Our proposed framework allows for the design of algorithmic solutions with strong theoretical guarantees.
In the context of a recommender system, user initiated feedback may be a "dislike", "too spicy", or "age inappropriate" [Chen and Pu, 2012], but feedback may also be indirectly observed by e.g. high abandonment rates or low click-through rates.
Going from complaints to actionable solutions involves many steps.
First, the complaints are analyzed, typically by human specialists, and attributed to a set of predeﬁned criteria, such as low accuracy of classiﬁers, false positive rates or AUC scores.
Each complaint could trigger several criteria and a human specialist can monitor the aggregate performance on each criterion.
Since criteria are often incompatible, based on the analysis of the complaints and their affect on the criteria, a decision is made to allocate resources to improve a subset of the criteria and this process repeats [Yu et al., 2020].
While human involvement is crucial in the above process for both analyzing complaints and trading off metrics, a large portion of the above process could be made algorithmic and automated.
In practice, the problem of multiple conﬂicting metrics may emerge, even when a single ﬁxed criterion is adopted [Klinkman et al., 1998, Buolamwini and Gebru, 2018].
As an example, consider again a recommendation system for videos.
Let us assume that a system designer has opted for the false positive rate and the false negative rate to measure the performance of the system.
The overall false positive (FP) rate or the false negative (FN) rate is rarely a good indicator of performance, particularly from an algorithmic bias point of view.
Instead, the system designer would wish to monitor and optimize the FP/FN rates across different slices of the data, such as “sports”, “religion”, “LGBTQ issues” videos, or videos originating from different geographic locations, or a combination of them.
This could easily result in hundreds of relevant slices of the data, where each can be viewed as a separate metric.
As discussed before, these slices will often admit mutual incompatibilities [Kleinberg et al., 2017, Feller et al., 2016].
Thus, a user feedback data-driven method is needed to make the optimal choice.
Our main contribution is precisely a data-driven model and algorithms for that
� � � � � � � � � � ��� � � � � � � � � � �
[Figure] purpose. Not only is our proposed framework grounded in theory, it can also be effectively realized in practice as we will show later.
Our model assumes predetermined costs for user complaints along the multiple metrics.
The difﬁculty in optimizing for user happiness arises from the fact that the nature and volume of the complaints depend on the state of the model.
Of course if no complaints is received, an optimal state has been reached, but most often complaints will arise.
Fixing the model to optimize for this set of complaints will most likely spur a different set of complaints, etc.
Only by visiting all incompatible states of the model and observing the associated complaint set would one be able to fully optimize the model.
Such an exhaustive search is prohibitive from both a time and development perspective.
This paper presents a model that effectively reaches a beneﬁcial state and provides performance guarantees.
The rest of the paper is organized as follows.
In Section 2, we deﬁne our model.
In the stochastic setting (Section 3), we show that our framework can be cast as a Markov Decision Process with stochastic losses, for which we give efﬁcient vanishing regret algorithmic solutions.
We also further discuss our modeling assumptions and extensions.
In the adversarial setting (Section 4), we give algorithms with competitive ratio guarantees.
Section 5 demonstrates how our framework can be realized in practice and reports the results of experiments with our algorithms in the stochastic setting that demonstrate their effectiveness and the applicability of our model.
We defer the related work discussion as well as many of the proofs to the appendix.
Conﬂict resolution model
We consider optimization in the presence of multiple criteria, where not all criteria can be satisﬁed simultaneously.
The constraints are speciﬁed by an undirected graph G =
( V, E ) , where each vertex represents a criterion and where an edge between vertices v i and v j indicates that criteria v i and v j cannot be simultaneously satisﬁed.
We denote by V = { v 1 , . .
. , v k } the set of k criteria considered.
Figure 1 illustrates these deﬁnitions.
Note that vertices may represent joint criteria as in Figure 1(b).
We consider a machine learning system that evolves over a sequence of time steps in the presence of the criteria represented by the graph G .
At each time step t , the system is in some state s t characterized by its performance on all criteria in V .
Note, a state is distinct from a vertex of G .
The system then receives a new batch of feedback that depend on its current state and incurs a loss.
The objective of the algorithm is to minimize the total cost incurred over a period of time, which includes the total loss accrued, as well as the total cost of ﬁxing various criteria over that period.
We envision that the algorithm is solving a constraint optimization problem with the criteria as constraints.
The assignment of a complaint to one or more criteria can be achieved by human analysis or via a multi-class multi-label classiﬁer trained on past data and making use of known classiﬁers for speciﬁc criteria.
Even when a complaint is related to a single criterion, we do not simply advocate taking that raw feedback as the ground truth.
We discuss the risks associated with doing so in Sectionand Appendix F, in the context of the COMPAS example.
To further improve and maintain the accuracy of this multi-class multi-label classiﬁer, in practice, there may be ongoing data labeling and assistance by expert auditors analyzing complaints.
Note that not all complaints received by the system are relevant and the classiﬁer, or a human in the loop, may decide to not assign a complaint to any criterion.
This also helps protect the system against potential attacks by coordinated users.
Recent work on interactive models for ML fairness has studied this for speciﬁc metrics and auditor behavior [Bechavod et al., 2020].
Loss .
As a result of the complaints, the system incurs a loss and responds by changing its state.
The deﬁnition of the loss, which depends on the criteria affected by the complaints is critical, a poor choice can yield a so-called loudest voice effect (see discussion in Section 3).
The notion of complaints and the associated loss may seem abstract at this point.
In Section 5, we demonstrate how our model can be applied in practice.
Graph and criteria .
The assumption that the graph G is known a priori may seem restrictive.
However, in many settings, graph G can be derived from analyzing past complaints and by measuring how ﬁxing one criterion affects the performance on others.
For instance, in the recommendation system example discussed above, where each metric corresponds to the false positive rate on a different slice of the data, one can easily use past data to see how optimizing the false positive rate on one slice affects the other and get the graph of incompatibilities.
See the experiments in Section 5 for a more concrete example.
Our model also provides the ﬂexibility of accounting for incompatibilities among criteria such as those discussed by Kleinberg et al.
[2017] and Feller et al.
This can be achieved by augmenting the graph with vertices representing joint criteria as in Figure 1(b).
The graph stipulates in particular that v 1 , v 2 and v 3 cannot be all simultaneously satisﬁed.
States .
For our theoretical and algorithmic analysis, we will adopt the following simplifying assumptions and will later discuss their extensions or relaxation in Section 3.
We assume that each criterion can only be in one of two states: ﬁxed , meaning that criterion v i is met or is not violated, or unﬁxed , meaning the opposite.
Hence, the overall state of the system can be described by a k -dimensional
Boolean vector.
An action corresponds to ﬁxing a particular criterion, or set of criteria, and moving to a different vertex v i in the graph.
Fixing the criteria associated to v i entails an algorithmic and resource allocation cost that we denote by c i .
Initially, all criteria are unﬁxed.
At each time step, a conﬂict resolution system or algorithm selects some action, which may be to ﬁx an unﬁxed vertex v i , thereby incurring the cost c i and unﬁxing any vertex adjacent to v i , or the algorithm may select the null action, not to ﬁx or unﬁx any vertex and wait to collect more data.
Note that the incompatibilities in our model deﬁned via edges in the graph are data agnostic.
In practice, it is possible that two generally incompatible criteria can be simultaneously satisﬁed for a given dataset, say via incorporating a slack.
This is a direction for future work.
Fixing costs .
The ﬁxing cost can be estimated from past experience.
In the absence of any prior information, one could assume a unit ﬁxing cost for all criteria.
We deliberately avoid making speciﬁc choices.
This gives us ﬂexibility in dealing with different types of metrics in a uniﬁed manner.
While the focus of our study is theoretical, let us emphasize that our model is easily applicable and implementable in practice.
We further discuss this in the end of Section 3 and illustrate it in Section 5.
Stochastic setting
We ﬁrst detail a stochastic setting of our model that can be described in terms of a Markov Decision
Process (MDP).
Next, we present algorithms with strong regret guarantees.
Description.
The distribution of complaints received by the system is a function of its current state , that is the current set of ﬁxed or unﬁxed criteria v i .
Thus, we consider an MDP with a state space S ⊆ { 0 , 1 } k representing the set of bit vectors for criteria: a state s ∈ { 0 , 1 } k is deﬁned by s ( i )
=when criterion v i is unﬁxed and s ( i ) =
1 when it is ﬁxed.
By deﬁnition of the incompatibility graph G , s is a valid state if and only if the set of ﬁxed criteria at s is an independent set of G .
When in state s ∈ S , the system incurs a loss (cid:96) si due to complaints related to criterion i ∈ [ k ] .
Loss (cid:96) si is a random variable assumed to take values in [0 , B ] with mean µ si .
We do not assume independence across criteria, i.e., (cid:96) s i and (cid:96) s j may be dependent for a given state s .
The action set is
A = { 0 , 1 , . . .
, k } .
A non-zero action i corresponds to ﬁxing criterion i .
Action 0 is the null action, that is no criterion is ﬁxed.
Transitions are deterministic: given state s and action i ∈ A , the next state is s if i = 0 since the ﬁxed-unﬁxed bits for criteria are unchanged; otherwise, for i (cid:54) =
the next state is the state s (cid:48) that only differs from s by s (cid:48)
( i ) =
1 and (possibly) s (cid:48)
( j ) =
for all j ∈ N ( i ) , where N ( i ) is the neighbors of v i in G , since neighbors of i must be unﬁxed once i is ﬁxed.
Each action a
=
i admits a ﬁxing cost c i .
The cost for unﬁxing, as well as the null action, is zero.
The loss incurred when taking action a at state s is the sum of the ﬁxing cost c a and the complaint losses at the (possibly) next state s (cid:48) :
λ ( s, a ) =
c a + (cid:80) ki =1 (cid:96) s (cid:48)
i .
The expected loss of transition ( s, a, s (cid:48) ) is:
E
[Equation]
Note, c a and the losses (cid:96) s (cid:48)
i are observed by the algorithm, but the mean values µ s (cid:48)
i are unknown.
To keep the formalism simple we assume that the cost c a of taking an action a is independent of the current state s .
Figure 1 (c) illustrates our stochastic model for three mutually incompatible criteria.
The notion of each metric in a binary state is a simplifying modeling assumption for our theoretical investigation.
We discuss this more at the end of the Section.
Correlation sets.
In practice, the distribution of complaints related to a criterion v i at two different states may be related.
To capture these correlations in a general way, we assume that a collection C
= { C 1 , C 2 , . .
. , C n } of correlation sets is given, where each C j is a subset of the k criteria and has size at most m .
By allowing correlation sets of varying sizes, we can capture a range of dependencies that may exist between different criteria.
These dependencies affect the loss observed by the algorithm at each time.
for a graph with four criteria. We assume that at a given state s , each set C j generates losses with mean value θ sj per vertex, and that if two states s and s (cid:48) admit the same conﬁguration for the vertices in C j , then they share the same parameter θ sj =
θ s (cid:48)
j .
Given a criterion i and a state s , we assume that the loss incurred by criterion i equals the sum of the individual losses due to each correlation set C j that contains i .
Thus, µ si can be expressed as follows: µ si = (cid:80) nj =1 θ sj 1 ( i ∈ C j ) .
If a criteria is not correlated with any other vertex, we add to C a correlation set of size one for that criterion.
See Figure 2 for an illustration.
For each j ∈ [ n ] , there are at most 2 m conﬁgurations for the vertices of C j in a state s , hence there are at most 2 m n distinct parameters θ sj .
Let θ denote the vector of all distinct parameters θ sj .
Our MDP model can then be denoted MDP ( S , A , C , θ ) .
Algorithm.
We consider an online algorithm that at time t takes action a t from state s t and reaches state s t +1 , starting from the initial state (0 , . . . , 0) .
The objective of an algorithm can be formulated as that of learning a policy, that is a mapping π : S
→ A , with a value close to that of the optimal.
We are mainly interested in the cumulative loss of the algorithm over the course of T interactions with the environment.
The goal is to minimize the pseudo-regret:
[Equation]
where λ t ( s, a ) is the total loss incurred by taking action a at state s at time t , s
= (0 , . .
. , 0) and π ∗ is the optimal policy.
Note, λ t is only a function of the current state and the action taken.
The expectation is over the random generation of the complaint losses.
Given the correlation sets and the parameter θ , the optimal policy π ∗ corresponds to moving from the initial state (0 , . . . , 0) to the state s ∗ ∈ S with the most favorable distribution and remaining at s ∗ forever.
We deﬁne by g ( s ) the expected (per time step) loss incurred by staying in state s , that is, g ( s ) :
= (cid:80) ki
=1 µ si .
The optimal state s ∗ is then deﬁned as follows:
[Equation]
Note, in this deﬁnition of s ∗ , we disregard the one-time cost of moving to a state from the initial state, since in the long run the expected cost incurred by staying at a given state governs the choice of the optimal state.
We will assume that we have access to an oracle that can solve the above ofﬂine optimization problem.
This is a standard assumption in the theory of online learning and MDPs.
Since our problem can be seen as that of learning with a deterministic MDP with stochastic losses, we could adopt an existing algorithm for that problem [Jaksch et al., 2010].
However, the running-time of such algorithms would directly depend on the size of the state space S , which here is exponential in k , and that of the action set A .
Furthermore, the regret guarantees of these algorithms would also depend on | S
|| A | .
Instead, by exploiting the structure of the MDP, we can design vanishing regret algorithms with a computational complexity that is only polynomial in k and the number of parameters.
We will assume access to an oracle that, given θ , can optimize (3).
In Appendix B, we show how to approximately solve (3) for the case of m = 1 , i.e., singleton correlation sets.
In that case, the true parameters θ can also be estimated efﬁciently (see Theorem 5).
There are certain important distinctions between our proposed stochastic model and the standard online learning setting.
Our model has a notion of a state that evolves as a result of the stochastic losses suffered.
These losses in turn depend on the distribution of complaints received.
This distribution should not be confused with the distribution of data over which classiﬁers may be trained to ﬁx a criterion.
The latter is assumed ﬁxed over time.
Via correlation sets we can model complex correlations among the criteria when deﬁning the stochastic losses.
Case m = 2 .
To illustrate the ideas behind our general algorithm, we ﬁrst consider a simpler setting where correlation sets are deﬁned on subsets of size at most two.
This setting also captures an important case where ﬁxing a particular criterion affects the complaints of its neighbors.
The algorithmic challenge we face here is to avoid exploring the exponentially many states in the MDP.
Instead, we will design an algorithm that spends an initial exploration phase by visiting a speciﬁc subset of states of size at most 4 n .
This subset denoted by K , that we call a cover of C will help the algorithm estimate the expected loss of any state in the MDP given the estimates of losses for states in the cover.
After the exploration phase, the algorithm creates an estimate ˆ θ of the true parameter vector θ , uses the optimization oracle for solving Eq.
(3) to ﬁnd a near optimal state ˆ s and selects to stay at state ˆ s for the remaining time steps.
We next deﬁne the cover.
For two criteria i, j and b ∈ { 0 , 1 } , we say that ( i, j, b ) is a dichotomy if there exist two states s, s (cid:48)
∈ S such that: (1) s ( j ) =
and s (cid:48)
( j ) = 1 , and (2) s ( i )
=
s (cid:48)
( i ) =
b .
We call the two states s, s (cid:48) an ( i, j, b ) -pair.
Note that if an edge ( v i , v j ) is present in G , then ( i, j, 1) cannot be a dichotomy, since criteria i and j cannot be ﬁxed simultaneously.
A cover K of C is simply a subset of the states in the MDP that contains an ( i, j, b ) -pair for every { i, j } ∈ C and valid dichotomy ( i, j, b ) .
Furthermore, for every singleton set { i } in C , K contains states s, s (cid:48) such that s ( i ) = 0 , s (cid:48)
( i ) =
and s ( j ) =
s (cid:48)
( j ) for all j (cid:54)
=
i .
Note that we only need the cover to contain an ( i, j, b ) -pair if { i, j } is a correlation set.
Hence, it is easy to see that when m = 2 , there is always a cover of size at most 4 n .
Next, we state our key result that estimating the loss values for the states in a cover is sufﬁcient.
Theorem 1.
Let K be a cover for C .
For any state s ∈ S and any i ∈ [ k ] with s ( i )
=
b , we have:
[Equation]
where s (cid:48) is any state in K with s (cid:48)
( i ) =
b , and for { i, j } ∈ C , X i,jb := µ s 1 i − µ s 2 i where ( s 1 , s 2 ) is some ( i, j, b ) pair.
If { i, j } / ∈ C , we deﬁne X i,jb to be zero.
From the above theorem we have the following guarantee.
Theorem 2.
Consider an MDP ( S , A , C , θ ) with losses in [0 , B ] , maximum ﬁxing cost c , and correlations sets of size at most m = 2 .
Let K be a cover of C of size r ≤ 4 n , then, the algorithm of Figureachieves a pseudo-regret bounded by O ( kr 1 / 3 ( c + B )(log rkT ) 1 / 3 T 2 / 3 ) .
Furthermore, given access to an oracle for (3) , the algorithm runs in time polynomial in k and n = |C| .
oracle
solving Eq.
with the
estimates as in
to get a
to state s .
in state s for t
There is a natural extension to arbitrary correlation sets via extending the notion of a dichotomy and a cover (Algorithm in Figure 8, Appendix B). Our algorithms are also scalable. During step 1 they only explore the states in the cover K that could be much smaller than the full state space.
Beyond T 23 regret. Next, we present algorithms that achieve ˜ O ( √ T ) regret, ﬁrst in the case m = 1 , next for any m , under the assumption that each criterion does not participate in too many correlations sets.
Although our problem can be cast as an instance of the stochastic multi-armed bandit problem with switching costs, and arms corresponding to the states in the MDP, existing algorithms achieving ˜ O ( √ T ) have time complexity that depends on the number of arms which in our case is exponential ( 2 k )
[Cesa-Bianchi et al., 2013, Simchi-Levi and Xu, 2019].
We will show here that, in most realistic instances of our model, we can achieve ˜ O ( √ T ) regret efﬁciently.
When correlation sets are of size one, the parameter vector θ can be described using the following 2 k parameters: for each i ∈
[ k ] , let γ
i denote the expected loss incurred by criterion i when it is unﬁxed and γ 1 i its expected loss when it is ﬁxed.
Our proposed algorithm is similar to the UCB algorithm
[Auer et al., 2002].
For every vertex i , let τ
i,t be the total number of time steps up to t (including t ) during which v i is in an unﬁxed position and let τ
1 i,t be the number of times steps up to t during which v i is in a ﬁxed position.
Fix δ ∈ (0 , 1) and let ˆ γ bi,t be the empirical average loss observed when vertex v i is in state b , for b ∈ { 0 , 1 } .
Our algorithm maintains optimistic estimates
[Equation]
The algorithm divides the T time steps into consecutive intervals that we call as episodes .
In episode h , the algorithm moves to and stays at a ﬁxed state for t ( h ) time steps.
At the end of the episode it makes a query to the optimization oracle (using the current optimistic estimates) to decide on the state to go to for the next episode.
The algorithm carefully chooses t ( h ) to maintain low regret.
The algorithm is described in Figure 4.
We will prove that it beneﬁts from the following regret guarantee.
Theorem 3.
Consider MDP ( S , A , C , θ ) with losses in [0 , B ] and maximum ﬁxing cost c .
Given correlations sets C of size one, the algorithm of Figure 4 achieves a pseudo-regret bounded by
O ( k 2 ( c + B ) 2 √ T log T ) .
Furthermore, given access to an oracle for (3) , the algorithm runs in time polynomial in k .
The algorithm of Figure 4 can be extended to higher m values (see Figure 10 in Appendix C).
Modeling assumptions and extensions .
Here, we brieﬂy discuss the assumptions we adopted and various natural extensions.
i) Scalability .
The running time of our algorithms depends linearly on the size of the cover K .
While in the worst case the size of the cover could be exponential in n, m , in practice, we expect it to be very small, in which case our algorithms are quite efﬁcient.
ii) Loss function .
The choice of the loss function is critical in the effectiveness of our model.
We made the simplifying assumption that the loss at each time step is additive in the losses incurred by correlation sets.
A careless choice of what the additive losses correspond to may result in a sub-optimal overall.
For example, a poor choice is one that uses the volume of complaints, i.e., how many complaints have triggered a criterion.
This will make us vulnerable to the loudest voices in the system.
In Section 5, we discuss how our framework can be implemented in practice and present reasonable choices for the loss function.
We further discuss the choice of the loss function in the case of the COMPAS example in Appendix F. (iii) Adversarial manipulation.
Our model may be vulnerable to strategic coordination.
A malicious group of users can form a sub-community generating a large number of complaints to press the system to include a new criterion in the graph.
The presence of such poor criteria may result in an overall suboptimal system.
Modeling this scenario is beyond the scope of the current work.
(iv) Continuous states .
While this is a direction for future work, our method offers a simple way to achieve this by simply adding, for each criterion i , additional criteria to the graph with different levels or thresholds τ 1 , τ 2 , . . .
for satisfying the criterion.
Adversarial setting
Motivated by the above discussion on adversarial manipulation, we next study a setting with no distributional assumptions about the arrival of complaints.
We consider an adversarial model where, at each time step, multiple complaints arrive for the vertices in G .
Initially all the vertices in G are in an unﬁxed state and each vertex has a ﬁxing cost of c i .
Each time, the algorithm can decide to ﬁx a particular vertex, and as a result its neighbors get unﬁxed.
At time step t , if criterion v i is unﬁxed, then the algorithm incurs a loss of (cid:96) i ( t ) (which depends on the current state of the system), otherwise the algorithm incurs no loss.
For an algorithm A , during T time steps, the total loss is
[Equation]
Let OPT be the algorithm that, given the entire loss sequence in advance, makes the decisions to ﬁx vertices.
We deﬁne the competitive ratio [Borodin and El-Yaniv, 1998] of A to be the maximum of
Loss ( A ) / Loss ( OPT ) over all possible complaint sequences.
Our main result is stated below.
Theorem 4.
Let G be a graph with ﬁxing costs at least one.
There is a polynomial-time algorithm with a competitive ratio of at most 2 B + 4 on any sequence of complaints with loss values in [0 , B ] .
Our algorithm for this setting is provided in Figure 11 in Appendix D.
The intuition behind the algorithm is the following.
For each criteria we consider it ﬁxing cost, the accumulated loss since the last time it was ﬁxed plus a measure of the cost of ﬁxing its neighbors.
We ﬁx a criteria (and implicitly unﬁx its neighbors) once its accumulated loss is larger than both its ﬁxing cost and a measure of the cost of ﬁxing its neighbors (the decision in the algorithm is more reﬁned).
This allows us to "charge" the cost of ﬁxing a criteria to its accumulated loss.
In addition, we make sure that (future) ﬁxing of the neighbors of the criteria can also be charged appropriately.
For details for the algorithm and the analysis see Appendix D.
Experiments
Simulated Data .
We ﬁrst evaluated the performance of our stochastic setting algorithms (Section 3) on simulated data.
We generated the graph G from the Erd ˝ os-Rényi model: G ( k, p ) and set p = 2 log k k , to ensure connectivity.
We generated correlation sets ( C ) of size two by picking αk pairs of vertices at random and adding them to C , where α is a parameter.
Hence, on average, each vertex is in α correlation sets.
We also added to C singleton sets for each vertex in G .
The ﬁxing cost of a vertex was sampled uniformly in [1 , 5] .
Parameters governing the loss distribution were drawn from a
Beta distribution.
We approximated the oracle for the optimization in (3) via a linear programming relaxation.
Our algorithms of Figure 3 and Figure 10 admit complementary guarantees.
The former admits a higher regret as a function of T , but only a polynomial dependence on α , i.e., the average number of correlation sets a vertex participates in.
The latter incurs a smaller regret of ˜ O ( √ T ) as a function of T at the expense of an exponential dependence on α .
Figure 5 shows an empirical illustration of this: for smaller values of α , the ˜ O ( √ T ) regret algorithm performs signiﬁcantly better, while for larger values of α the ˜ O
( T 2 / 3 ) regret algorithm is more desirable.
We choose to compare the performance of our two proposed stochastic algorithms as we are not aware of any existing baselines for simultaneously optimizing multiple diverse metrics.
Additionally, we focus on experiments with the stochastic model as it is hard to approximate the best ofﬂine algorithm in the adversarial setting of Section 4.
See Appendix E for more details and experimental results.
Real-world dataset.
We next illustrate how our framework can be applied to real-world data.
Due to space constraints, we provide a brief description here and refer the reader to Appendix E for details and more results.
We studied the UCI Adult dataset [Kohavi, 1996] which includes 48 , 852 examples, each represented by 124 features, after processing.
Each data point corresponds to a person and the label is a 0 / 1 -value representing whether the income of the person is more or less than $50 , 000 .
The dataset contains information about sensitive attributes such as race and gender.
We simulated an online scenario where a classiﬁer is making predictions on the income of individuals.
At each time step, a batch of complaints arrive, the system incurs a loss and responds by transitioning to a different state (and updating the classiﬁer).
We now describe the various components.
Graph G : We used race in { black , white } as an attribute to obtain two sub-populations and considered two natural criteria, namely the true positive rate and the AUC score, equivalent to the criteria of the COMPAS tool.
This leads to four vertices tpr w , tpr b , auc w , auc b .
Furthermore, we added the overall classiﬁer accuracy as a ﬁfth vertex.
We consider a unit ﬁxing cost for all criteria.
Losses and Correlation Sets: We consider correlation sets of size one, and the total loss of a state is the sum of the losses of each criterion.
For the accuracy vertex, we deﬁne the loss to be the error of the classiﬁer.
For a vertex, say tpr w , if the overall tpr of the classiﬁer and the tpr on the white population deviated by more than τ , we penalized the classiﬁer linearly: the loss for tpr w was deﬁned as: max(0 , | tpr overall − tpr w | − τ ) .
Other losses are deﬁned similarly.
We set τ = 0 .
Incompatibilities and State Transitions: We solved, for each state s ∈ { 0 , 1 } 5 , an optimization via the tensorﬂow constrained optimization toolkit [Cotter et al., 2018a,b] to get a classiﬁer.
by the online algorithm, as a function of the time steps.
We evaluated the classiﬁer on a test set and if the loss of any criterion was more than a speciﬁc threshold ( 0 . 01 ), we considered the state invalid.
As an example, in the instance corresponding to Figure 6, we obtained four valid states.
We obtained the state transitions as follows.
If the algorithm asked to ﬁx v i in state s , we set s ( i ) = 1 to go to the next state s (cid:48) .
While s (cid:48) is invalid, we unﬁxed the criterion (not including v i ) with the highest loss in the state s (cid:48) to reach another state.
Simulating Complaints: We divided the dataset into 16 , 000 examples that we used to update our classiﬁer at each time step and the remaining test set to simulate the arrival of complaints.
At each step, we randomly selected a batch of examples from the test set to generate complaints.
This batch was used to compute the loss at the given time step.
Benchmark and Results: We compared our Algorithm of Figure 4 with an ofﬂine optimal solution computed by ﬁnding the state with the minimum average loss over the sequence of complaints.
The results are in Figure 6.
We plot the loss of the algorithm as compared to the benchmark, as well as the states chosen by the algorithm, as a function of the time steps.
Our algorithm quickly converges to the ofﬂine optimal solution after an initial exploration phase.
Note that the choice of the loss functions was important in this case and that we did not weight each criterion by the volume of the complaints.
This demonstrates that our algorithms, when combined with a good choice of the loss function, can be useful in practice.
See Appendix E for details.
Conclusion
We presented a new data-driven model of online optimization from user feedback in the presence of multiple criteria, with algorithms beneﬁting from theoretical guarantees both in the stochastic and the adversarial setting.
We provided empirical evidence that our model can be effectively realized in practice.
Several extensions are worth exploring in future work.
These include ﬁxing costs that can vary with time to capture varying algorithmic price and human effort cost.
Similarly, the expected losses in our stochastic model could be time-dependent to express the growing cost of a criterion not being addressed.
Reviews and changes from NeurIPS 2022 main conference
Based on the reviewer comments from the NeurIPS 2022 main conference (printed from openreview in the next page), we have made the following changes to the manuscript.
• Clariﬁed reviewer comment about the choice of the baselines in the experiments.
Clariﬁed the connection to multi-task learning in the related work.
• Added clariﬁcation about adaptive thresholds in the details of the experiments.
• Added more details about the generality of our model and the algorithms.
• Added a note about access to the ofﬂine optimization oracle.
• Further clariﬁed the discussion of the COMPAS example.
9/29/22, 8:34 PM
Learning with Competing Objectives and User Feedback | OpenReview https://openreview.net/forum?id=wFG4ywRbNlR&noteId=ZIq45h5Z5to&referrer=%5BAuthor Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConfer… 2/ 14 Sept 2022 NeurIPS 2022 Conference Paper8299 Decision Readers: Program Chairs,
Paper
Senior Area Chairs, Paper8299 Area Chairs, Paper8299 Reviewers, Paper
Authors
Paper Decision
[Table] [Table]
Submitted, Program Chairs
Meta Review of Paper8299 by Area Chair CF2w
[Table]
Metareview: This paper denitely raised interest among the reviewers.
It tackles a very important problem and proposes an approach that draws on tools from many dierent elds of ML/Mathematics (e.g. online optimization, reinforcement learning, combinatorial optimization).
While this can be great for innovation, it also requires very diverse background knowledge.
Therefore, in order to remain understandable (and thus impactful) for readers who are external to any of these elds, the paper requires to be very well organized and clearly written.
Possibly due to a lack of clarity, reviewers had a hard time seeing how the proposed approach could be of any practical interest with all the requirements brought along by the dierent parts of the solution (i.e. compatibility graph for the objectives, binarization of target values for each objective).
Reviewers were open to believing that there exist realistic scenarios where such information is available.
Picking a problem perfectly suited to the proposed method from the beginning to the end of the paper, with some numerical results on it, could be a strong improvement for clarifying both the approach and the potential applications, thus strongly strengthening the contributions.
Overall, this paper contains many technical innovations that could benet the community in an improved version.
Award: No [–] 08 Aug 2022 NeurIPS
Conference Paper8299 Reviewers
Author Rebuttal Acknowledgement Readers: Program Chairs, Paper
Senior Area Chairs, Paper Area Chairs, Paper
Authors, Paper
Reviewer T1Df
Author Rebuttal Acknowledgement by Paper
Reviewer T1Df
NeurIPS 2022 Conference Paper
Reviewer T1Df
Author Rebuttal Acknowledgement: Yes [–] 07 Aug 2022 NeurIPS
Conference Paper8299 Reviewers
Author Rebuttal Acknowledgement Readers: Program Chairs, Paper
Senior Area Chairs, Paper Area Chairs, Paper
Authors, Paper
Reviewer a6LN
Author Rebuttal Acknowledgement by Paper
Reviewer a6LN
NeurIPS 2022 Conference Paper
Reviewer a6LN
Author Rebuttal Acknowledgement: Yes [–]
Author Rebuttal Acknowledgement by Paper
Reviewer VJXj
NeurIPS 2022 Conference Paper
Reviewer VJXj [–]
9/29/22, 8:34 PM
Learning with Competing Objectives and User Feedback | OpenReview https://openreview.net/forum?id=wFG4ywRbNlR&noteId=ZIq45h5Z5to&referrer=%5BAuthor Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConfer… 3/ 07 Aug 2022 NeurIPS
Conference Paper8299 Reviewers
Author Rebuttal Acknowledgement Readers: Program Chairs, Paper
Senior Area Chairs, Paper
Area Chairs, Paper8299 Authors, Paper
Reviewer VJXj Author Rebuttal Acknowledgement: Yes 13 Jul 2022 NeurIPS 2022 Conference Paper8299 Ocial Review Readers: Program Chairs, Paper
Senior Area Chairs, Paper8299 Area Chairs, Paper8299 Reviewers
Submitted, Paper8299 Ethics Reviewers, Paper
Authors
Ocial Review of Paper8299 by Reviewer T1Df
[Table]
Conference Paper
Reviewer T1Df
Summary:
The paper is proposing a new multi-constrained optimization problem, where many competing targets are available in a context where high variance performance feedback is obtained regarding the satisfaction of these constraints.
This is presented as a signicant problem in recommender systems, where many targets can be dened (possible through specic segmentations of the data according to specic attributes), and where user feedback is used to guide recommendations (hence high variance).
The problem is modeled as a Markov decision process (MDP), with many criteria dened in order to attain good performance over the various targets.
The assumption of the approach is that all criteria cannot be satised simultaneously, but a subset of them can be.
The goal is to identify the subset of criteria that would minimize the number of complaints received overall (the loss used for this problem).
The approach assumes that not all criteria are compatible, and a graph is provided to express criteria that cannot be satised simultaneously.
The proposed approach proceeds by activating/deactivating satisfaction of the criteria one by one.
A graph should be provided as prior knowledge to dene which criteria cannot be satised simultaneously.
The activation/deactivation of criteria satisfaction is modeled into the MDP, and the optimization proceeds by executing the MDP for a long enough period to reach a stable state minimizing the loss received.
The loss can be the negative feedback on the recommendations provided the users, hence can be highly stochastic.
Strengths And Weaknesses
[Table] Strengths
Presents a problem that appears to be novel in its formulation (to the best of my knowledge) of some practical importance.
Aims at doing some formalization of the problem and presents with some theoretical support.
Weaknesses
The solution to the problem is quite specic, relatively complex and not necessarily well justied.
The MDP resolution of it is not very well justied and imposes a way to solve the problem that imposes some structure that are not well justied (i.e. add/remore one criterion at the time and reach a stable state).
The requirement for providing a graph of compatibility of the dierent criteria to satisfy is quite strong, even if the authors argue otherwise.
Once such a graph can be properly articulated, most of the job is done in terms of optimization.
Nevertheless, I even doubt that the relation between criteria can be simplied as a graph, as reaching satisfaction of many criteria may be dicult and not properly modeled with the type of graph proposed.
Results are reported on a purely synthetic and a toy real-world problem (UCI Adult dataset).
No comparison to any baselines is provided and the results are not quite convincing.
The code is not provided as part of it is stated as proprietary.
However, considering the experiments made, I think that a completely open version of the code could have been given to allow results on the experiments presented to be reproduced.
General comment on the soundness of the approach I have diculty making sense of the problem tackled.
What is proposed here is to identify the subset of targets / criteria that can be satised in order to minimize negative users of the recommender system.
I get that the feedback is highly stochastic, but I am quite puzzled by the fact that we are assuming that the incompatibility between criteria is set in a graph.
First, building up that graph appears not an easy job, I would rather argue that being able to build that model is the hardest problem to solve here.
Second, the relations are more complex than pairwise relation, I think that we can safely consider that we can see situations where criteria A, [–]
[Table] [Table]
in pairs
(i
[Table]
e,.
[Table]
simultaneous satisfaction of A-B, A-C or B-C), but
[Table]
satisfaction of A, B, and C simultaneously can be much harder.
I think that getting to model this over all combinations can be achieved in the general case only by going through all possible subsets, which is not tractable.
Thirdly, the fact that each criteria is binarized is quite limitative, that target value for each criteria should be dened in a rather arbitrary way.
Also, it removes subtleties in the fact that one solution may barely satisfy a criteria while another may oer superior performance in a given case.
In brief, I have a hard time to buy the assumptions on which the paper is building, and as such I am not convinced by the usefulness of the solutions presented.
Questions: Can some baseline results be provided?
Even if nothing similar has been proposed in the literature, I think that simple heuristics, random search or exhaustive enumeration (brute force) approach in searching over the criteria subset can be devised and used as a comparison point with the proposed approach in the experiments provided.
From my perspective, I see the problem tackled boiling down to some constrained combinatorial optimization setting with stochastic performance evaluation.
It is not presented that way, but I think that looking at it from that perspective may lead to simpler explanations and sounder resolution approach.
Limitations: The paper aims for an approach that would minimize user complaints on recommendation made by a system, using a set of constraints to be satised, with all constraints cannot be satised simultaneously.
It puts all constraints at the same level, but in practice I think it can be much more complex than that, especially when considering constraints with signicant social impacts.
If such important constraints are not satised, we can expect to get more negative feedback.
But that’s quite a bad practice to wait until we
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
Ocial Comment by Paper8299 Area Chair CF2w • Discussion with authors [+] Ocial Comment by Paper8299 Authors • Response to reviewer T1Df
[+
11 Jul 2022 NeurIPS 2022 Conference Paper8299 Ocial Review Readers: Program Chairs, Paper
Senior Area Chairs, Paper8299 Area Chairs, Paper8299 Reviewers
Submitted, Paper8299 Ethics Reviewers, Paper
Authors
Ocial Review of Paper8299 by Reviewer a6LN
NeurIPS 2022 Conference Paper
Reviewer a6LN
[Table] [Table]
PM
[Table]
https://openreview.net/forum?id=wFG4ywRbNlR&noteId=ZIq45h5Z5to&referrer=%5BAuthor Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConfer… 5/
Strengths
The problem of competing objectives in learning a data-driven model is a problem of great interest to the community.
The authors clearly describe the limitations of oine methods and provide a comparison in the experiments.
The technical parts of this paper seem to be sound.
The paper focuses on a unique problem of large-scale deployed learning systems.
However, I have the following concerns for which I wish the authors can clarify:
Weakness
It seems that the paper does not cover related work in rich literature on multi-task and online learning.
I believe works in these elds are very related.
It would also be valuable to add a more in-depth discussion regarding the settings where we can expect the method to be particularly good (e.g., user feedback) and where can we expect it to fail or not provide further benets.
Questions: How would the authors distinguish this paper from works in the rich literature on multi-task and online learning?
How can the proposed methods generalize to other online learning settings beyond user feedback?
[Table]
authors
[Table]
discussion
[Table] [Table]
not
[Table] [Table]
Ocial Comment by Paper8299 Area Chair CF2w • Discussion with authors [
+]
Ocial
Comment by Paper8299 Authors
Response to reviewer a6LN
[+] 11 Jul
[Table]
2022 Conference Paper8299 Ocial Review
[Table]
: Program
[Table] [Table]
, Paper8299 Area Chairs, Paper8299 Reviewers
Submitted, Paper8299 Ethics Reviewers, Paper
Authors
Ocial Review of Paper8299 by Reviewer VJXj
NeurIPS 2022 Conference Paper
Reviewer VJXj
Summary: This paper proposes a general theoretical framework of learning complex models based on user feedback/complaints.
The problem is formulated as MDP and solved with low regrets with the help of an oracle.
Both the application and underlying theory are interesting.
Overall, I think this is a good paper, both empirically and theoretically.
Strengths And Weaknesses: Strenghts:
Learning complex models based on user feedback is very interesting and promising.
The theoretical framework, albeit building on top of MDP, captures key aspects of the practice and is interesting on its own.
[Table]
https://openreview.net/forum?id=wFG4ywRbNlR&noteId=ZIq45h5Z5to&referrer
[Table]
Ocial Comment by Paper8299 Authors • Response to reviewer VJXj [+]
[Table]
Ocial Review of Paper8299 by Reviewer qEve
[Table]
This
[Table]
studies
[Table]
problem of deploying a learned
[Table]
required to satisfy a number of potentially conicting objectives.
For example, the classier might be required to minimize multiple metrics quantifying unfairness as well as minimize the training error.
Suppose each objective is quantied via a loss function .
Then, this problem could be viewed as a multi-objective optimization problem.
Prior work in this area has considered nding a Pareto-optimal solution, e.g. (Sener & Koltun, 2018), minimizing a linear combination of the , or minimizing the worst-case convex combinations of the (Cortes et al, 2020).
This paper proposes a new, data-driven approach.
Specically, they suppose the entity responsible for maintaining the classier periodically receives batches of complaints from users of said classier.
Moreover, they suppose these complaints are labeled according to which metric/objective's violation they pertain too.
Using these complaints, a dierent combination of the is targeted, and the classier retrained.
Importantly, incompatibility between the is encoded as a graph, and only compatible combinations of the can be targeted.
The process repeats.
After formalizing this idea into the problem of determining a suitable policy for a Markov Decision Process, the author(s) propose several algorithms drawing on both the bandit- and the combinatorial- optimization literatures.
They prove these algorithms achieve low regret.
Theory is then complemented with two experiments, one using synthetic data on a random compatibility graph, and one on real data.
[Figure]
π
PM
Learning with Competing Objectives and User Feedback | OpenReview https://openreview.net/forum?id=wFG4ywRbNlR&noteId=ZIq45h5Z5to&referrer=%5BAuthor Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConfer… 7/
Strengths And Weaknesses:
Strengths
Studies a highly relevant problem.
Provides an interesting and novel mathematical formulation of said problem.
Provides theoretically grounded algorithms.
The experiments, though limited, are enough to provide a proof of concept.
It is clear the authors have put thought into how their work could be used in practice and/or generalized, and there are numerous interesting remarks throughout the paper to this eect.
Weaknesses.
See questions.
Questions:
Major Questions
I thoroughly enjoyed reading this paper, and found it well-written and thought-provoking!
That being said, I found it challenging to see how all the pieces t together.
Below I will try and esh out some of these misgivings.
In case my confusions are overly naive, you can take them as pointers towards improving the exposition for non-experts!
I would be happy to adjust my score upwards based on your responses.
I was puzzled by
the notion
of
xing a criterion
In
the
context
of the COMPAS model, would an example of this be retraining the model such that the false positivity rate for sub-population is constrained to be less than a threshold ?
This
seems to be what is suggested by Appendix E.
If yes, how would
one
choose the threshold in a data-driven way?
Also, why then would one choose your approach over minimizing the classication error over the training set, subject to in an
oine
manner
(Here
denotes a
loss
associated
to the -th criterion).
Perhaps you can elaborate on this in the
context
of the COMPAS model within the States paragraph in Section 2.
Related to the above, there seemed to me to be some blurring between constraints and losses.
Specically, suppose we transition from state to state by xing .
Suppose further this corresponds to retraining the model such that , for some threshold .
Most likely, this constraint will be tight: .
Unless the data distribution changes radically, it seems reasonable to assume , so why include it in the loss function?
I agree that your model can be described by an MDP, but perhaps this is overkill.
In my experience with MDPs (e.g, the rst few chapters of Puterman), they are equipped with a scalar reward function.
Here, you get much richer feedback, namely a tuple of losses .
Thus, it feels more like the simpler multi-armed bandit formalism would suce.
You mention this at the bottom of page 6, and claim that the number of arms is .
But theorem 1 seems to suggest you only need consider arms (here is a cover) at least in the case.
Also, the algorithms presented in the main text (Fig.3 and 4) don't (at least to me) have a MDP/RL avor.
Specically, there is no Bellman equation and no value function.
So, could you say a few words on why y'all think the MDP machinery is necessary here?
Can you say more about the optimization oracle for eq. 3?
For the case covered in the appendix, it appears that eq.3 reduces to a weighted max independent set problem, which I believe to be NP-hard.
So, is it realistic to assume such an oracle exists, especially for large , and are your results still valid if one uses an approximate oracle?
Sidenote: I like the interplay between combinatorial and online optimization running through this paper.
Perhaps you can say more about this.
For example, are the papers in the combinatorial optimization literature studying the problem expressed in eq (3)?
In eq 3 you do not consider the xing cost when determining the best state.
However in the proof of Theorem 5 in Appendix B you do.
The synthetic experimental example is OK, but could be better.
Specically, there is no classier to train here.
It would be better if you cooked up some sort of binary classication problem coupled to the randomly generated ER graph.
Minor Comments
There are some hints scattered through the paper about adding new metrics/ vertices to the graph (e.g. line 55 and line 294--295).
This is interesting, but not addressed by your proposed model.
I would recommend removing these references to new metrics.
i
[Table]
L
i ≤ τ i
L i
i s s ′ v i ℓ si ≤ τ i τ i ℓ si = τ i ℓ s ′
i =
τ i (ℓ s 1 ,…,ℓ sk )
O (2 k ) | K |
K m =m =m
[Table]
Ocial
Comment
by
Paper8299 Area Chair CF2w
[Table]
Discussion with authors
[Table]
Ocial
Comment
by Paper
Authors
[Table]
to
[Table]
qEve
[Table]
Feedback
[Table]
Privacy Policy (/legal/privacy)
OpenReview (/about) is a long-term project to advance science through improved peer review, with legal nonprot status through Code for Science & Society (https://codeforscience.org/).
We gratefully acknowledge the support of the OpenReview
Sponsors (/sponsors).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section 3 and Section(b)
Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See 3 and Section(b)
Did you include complete proofs of all theoretical results?
[Yes] See the appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Part of the code is proprietary (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] see Section E
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] see Section E (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section E (b) Did you mention the license of the assets?
[Yes] See Section E (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Table of Contents – Appendix
Related work[-
-]
B[-Stochastic setting 
-]
[-B.1 Case m = 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.2 General case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C Beyond T 23 regret 
-]B
D Adversarial setting[-
-]
E Experiments[-
-]
E.1 Experiments with simulated data . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Experiments with a real-world dataset . . . . . . . . . . . . . . . . . . . . . . . .
F Further discussion on the COMPAS example[-
-]
A Related work
There is extensive literature on optimizing multiple metrics or objectives under speciﬁc criteria.
The recent works of Mohri et al.
[2019], Cortes et al.
[2020] consider optimizing in the presence of multiple base objectives.
Given objectives L 1 , . . .
, L i these works aim to design “agnostic” algorithms that can simultaneously compete with any linear or convex combination of the objectives.
Another line of work considers design algorithms that can achieve the Pareto optimal solution [Jin and Sendhoff, 2008, Sener and Koltun, 2018, Shah and Ghahramani, 2016, Marler and Arora, 2004].
Another line of work considers optimizing multiple constraints (inspired by group fairness metrics) via constrained non-convex optimization [Agarwal et al., 2018, Cotter et al., 2018a, Thomas et al., 2019].
These publications either reduce the problem to that of cost-sensitive classiﬁcation [Agarwal et al., 2018, Dwork et al., 2018] or replace the non-convex constraints by convex proxies and next optimize them via external or swap regret minimization algorithms [Cotter et al., 2018b,a].
There have also been studies of the inherent tension between satisfying multiple metrics.
Kleinberg et al.
[2017] and Feller et al.
[2016] demonstrate that it is impossible to satisfy equal opportunity and calibration at the same time.
Inspired from fairness applications the work of Menon and Williamson [2018] studies the tradeoff between accuracy and other metrics of interest such as false positive and false negative rates.
Since we are concerned with optimizing multiple metrics, it is natural to consider whether the problem can be framed via multi-task learning.
However, there are certain crucial differences.
In multi-task learning, the learner has access to data from multiple tasks and the goal is to jointly learn these tasks, which are assumed to be somewhat related or similar, to achieve a better generalization across all tasks.
The online version of the problem admits many variants and with the aim of learning both a task similarity and predictors or only predictors when a task similarity is already supplied.
The literature is indeed very rich.
In our setting, there is typically only one task (same label), but different loss functions.
In lieu of a similarity between tasks, we have an incompatibility graph between losses.
We consider user feedback which does not seem to have a direct counterpart in the multi-task setting.
Furthermore, a different predictor is typically learned for each task, while this is not our setting.
In most settings of multi-task online learning, the objective is in terms of an adversarial regret, while our MDP scenario is for a stochastic scenario.
Hence, while there are some aspects that seem reminiscent of our scenario, the traditional multi-task learning scenario seems to be quite different from our considered setting.
Recent works have also studied the long-term impact of optimizing multiple conﬂicting criteria in settings with feedback mechanisms [Liu et al., 2018, Hashimoto et al., 2018, Mouzannar et al., 2019, Kannan et al., 2019].
Liu et al.
[2018] show that, in certain situations, constrained loss minimization to equalize certain criteria could lead to further disparate impact on the end users in the long run.
Hashimoto et al.
[2018] proposed algorithms for minimizing such disparate impact in settings involving repeated loss minimization.
More recently, Jabbari et al.
[2017], Wen et al.
[2019] study the problem of satisfying multiple constraints in reinforcement learning settings involving a
Markov Decision Process.
The authors in Jabbari et al.
[2017] consider learning in an MDP where the criteria to be optimized require that the algorithm never takes an action a over action a (cid:48) if the long-term reward is higher.
It is clear to see that the optimal policy for the MDP indeed satisﬁes this property.
Hence, there does exist a policy that satisﬁes the required criterion.
However, the authors show that ﬁnding a near optimal policy while satisfying the criterion requires time exponential in the size of the state space.
Wen et al.
[2019] consider other metrics such as demographic parity in the context of learning in
MDPs.
Doroudi et al.
[2017] show that existing importance sampling methods for off-policy policy selection in reinforcement learning can lead to bad outcomes according to other natural criteria and present algorithms to mitigate this effect.
While our work also involves learning in a Markov Decision Process (MDP) and optimizing multiple criteria in the long term, the setup and the motivation are different.
Unlike all the previous work mentioned, we do not commit to a ﬁxed deﬁnition of quality or a metric, and allow for arbitrary criteria.
Hence, states in our MDP correspond to the current conﬁgurations of different criteria.
Rather than studying each metric in isolation, the objective of our work is to propose a data-driven model that can learn from feedback, a near-optimal conﬁguration of the metrics to impose on the system.
To the best of our knowledge, ours is the ﬁrst work to incorporate optimizing metrics of arbitrary types in an online setting.
In this context, inspired by fairness applications, the recent work of Kearns et al.
[2019] studies a speciﬁc combination of group and individual fairness metrics.
The authors consider a setting where there is a distribution over individuals as well as a distribution over classiﬁcation tasks.
They consider algorithms for achieving average individual fairness, that is in expectation over classiﬁcation tasks, the performance of the algorithm on a group fairness metric such as demographic parity should be the same for each individual.
An important aspect of our stochastic MDP-based model requires the ability to observe the losses associated with different criteria at each time.
This relates to the problem of evaluating and monitoring the performance of the system according to different metrics from data.
There has been work in recent years on developing auditing and monitoring approaches Bastani et al.
[2019], Ghosh et al.
Bellamy et al.
Furthermore, many metrics require access to both labeled data and to certain sensitive attribute information such as race or gender, for accurate evaluation.
A recent line of work has studied this estimation problem when one has limited and/or noisy access to sensitive attribute information Gupta et al.
[2018], Coston et al.
[2019], Lamy et al.
[2019], Wang et al.
Finally, we note that our model learns from feedback received as a form of complaints.
These complaints are a result of a (potentially incorrect) decision made by an ML system.
There has been recent work in developing counterfactual based explanations Tsirtsis and Gomez-Rodriguez [2020] for such decisions and exploring recourse strategies Gupta et al.
B Stochastic setting
We ﬁrst show that in the stochastic model, if correlation sets are of size one then one can efﬁciently approximate the cost of the optimal state up to a factor of two.
Theorem 5.
If correlations sets are of size one ( m = 1 ), then, for any (cid:15), δ > 0 , the true parameter vector for MDP ( S , A , C , θ ) can be approximated to (cid:15) -accuracy in (cid:96) ∞ -norm with probability at least 1 − δ , in at most O
( B 2 k (cid:15) 2 log( kδ ))
time steps and exploring at most k + 1 speciﬁc states in S .
Furthermore, given a parameter vector θ , there is an algorithm that runs in time polynomial in k and ﬁnds an approximately optimal state s (cid:48)
such that g ( s (cid:48) ) ≤ 2 min s ∈ S g ( s ) .
Proof.
Notice that when correlation sets are of size one, the expected loss incurred for criterion v i at any given state s solely depends on whether s ( i ) = 0 or s ( i ) = 1 .
Hence in this case the MDP consists of 2 k parameters where we use γ 1 i and γ
i to denote the expected losses incurred by vertex i when it is in ﬁxed and unﬁxed position respectively.
For any δ > 0 , by Hoeffding’s inequality, we have that if we stay in state s = (0 , 0 , . . . , 0) for N = B 2 (cid:15) 2 log(2 k/δ ) time steps then with probability at least 1 − δ 2 , we have each γ
i estimated up to (cid:15) accuracy.
Let e i ∈ { 0 , 1 } k denote the indicator vector for i .
If we stay in state s = e i for B 2 (cid:15) 2 log(2 k/δ ) time steps, then with probability at least 1 − δ
we have γ 1 i estimated up to (cid:15) accuracy.
Hence, overall after O ( B 2 k (cid:15) 2 log( k δ )) time steps, we have each parameter estimated up to (cid:15) accuracy.
Notice that in total we observe at most k + 1 states.
Next we show how to efﬁciently approximate the loss of the best state.
Given the parameters of the
MDP each vertex has two costs Λ (1) i = γ
i , denoting the cost incurred if the vertex is unﬁxed and Λ (2) i
=
c i + γ 1 i , denoting the cost incurred if the vertex is ﬁxed.
Without loss of generality assume that Λ (1) i > Λ (2) i (any vertex that does not satisfy this can be safely left unﬁxed).
For each i , deﬁne y i = 1 if vertex i is unﬁxed otherwise deﬁne y i = 0 .
Then the ofﬂine problem of ﬁnding the best state can be written as
[Equation]
Here γ i = Λ (1) i − Λ (2) i > 0 .
By relaxing y i to be in [0 , 1] and solving the corresponding linear programming relaxation, we get a solution y ∗ 1 , y ∗ 2 , . . .
, y ∗ k .
Let LPval denote the linear programming objective value achieved by y ∗ 1 , y ∗ 2 , . . .
, y ∗ k .
Since the linear programming formulation is a valid relaxation of the problem of ﬁnding the best state, we have LPval ≤ min s ∈ S g ( s ) .
We output the state s (cid:48) in which a vertex i if and only if y ∗ i < 1 / 2 .
Let S be the set of ﬁxed vertices.
We have
[Equation]
B.1 Case m =
To illustrate the ideas behind our general algorithm, we ﬁrst consider a simpler setting where correlation sets are deﬁned on subsets of size at most two.
This setting also captures an important case where ﬁxing a particular criterion affects the rate of complaints of its neighbors.
Our algorithm consists of an exploration phase where it observes the losses for a speciﬁc subset of at most 4 n states.
We will show that after the exploration phase, the algorithm can accurately estimate the expected loss for any other state s ∈ S .
Notice that the number of states in S is in general exponential in k .
Thus, the subset of states to observe must be carefully chosen and must take into account the structure of the graph G .
After the exploration phase, the algorithm creates an estimate ˆ θ of the true parameter vector θ , uses the optimization oracle for solving Eq.
(3) to ﬁnd a near optimal state ˆ s and selects to stay at state ˆ s for the remaining time steps.
Let c denote the maximum ﬁxing cost: c = max i ∈ [ k ] c i .
We will show that the pseudo-regret of our algorithm is bounded by O ( k log k ( c + B ) 1 / 3 T 2 / 3 log( kT )) .
We ﬁrst describe how we select the subset of states to observe in the exploration phase.
We say that ( i, j, b ) is a dichotomy if for two criteria
i and j and for b ∈ { 0 , 1 } , there exist two states s, s (cid:48) ∈ S such that: (1) s ( j ) =
and s (cid:48)
( j ) = 1 , and (2) s ( i )
=
s (cid:48)
( i ) =
b .
Note that if an edge ( v i , v j ) is present in G , then ( i, j, 1) cannot be a dichotomy, since criteria i and j cannot be ﬁxed simultaneously.
Deﬁnition 1.
Consider a subset K ⊂ S .
We will say that K is a cover for C if for any dichotomy ( i, j, b ) , where { i, j } is a correlation set ( { i, j } ∈ C )
there exist two states s, s (
cid:48) ∈ K such that
: (1) they agree in all criteria except criterion j : s ( l ) =
s (cid:48)
( l ) for all l (cid:54) =
j ; (2) criteria i is in state b in both: s ( i ) =
s (cid:48)
( i )
=
b ; (3) we have that s ( j ) =
and s (cid:48)
( j )
= 1 .
We call such a pair ( s, s (cid:48) )
an ( i, j, b ) -pair.
Furthermore, for every singleton set { i } in C , the cover K contains states s, s (cid:48) such that s ( i ) = 0 , s (cid:48)
( i ) =
1 and s ( j ) =
s (cid:48)
( j ) for all j (cid:54)
=
i .
We can always ﬁnd a cover K of size at most 4 n by picking for each { i, j } ∈ C , at most four states corresponding to different bit conﬁgurations for i and j , with all other bits set to zero.
For any valid dichotomy ( i, j, b ) we deﬁne X i,jb as
[Equation]
where s, s (cid:48) ∈ K is an ( i, j, b ) pair.
If { i, j } / ∈ C we deﬁne X i,jb to be zero.
Notice that the values
X i,jb can be approximated from estimating the loss values of states in the cover.
Next, we state our key result showing that, given the loss values for the states in a cover, we can accurately estimate the loss values for any vertex in any other state.
Theorem 1.
Let K be a cover for C .
Then, for any state s ∈ S and any i ∈ [ k ] with s ( i ) =
b , we have:
[Equation]
where s (cid:48)(cid:48) is any state in K with s (cid:48)(cid:48)
( i ) =
b .
Proof.
Consider a correlation set { i, j } .
The expected loss incurred by vertex v i or v j due to this set in any given state depends solely on the conﬁguration of v i and v j in that state.
Hence there are four parameters in the θ vector corresponding to the correlation set { i, j } and we denote them using γ a,b i,j , where a, b ∈ { 0 , 1 } .
Let s, s (cid:48) be an ( i, j, b ) pair.
When we switch from s to s (cid:48) the only difference in the expected losses for vertex i comes from the pair ( i, j ) .
Hence we have
[Equation]
Input graph G , correlation sets
, ﬁxing costs c i .
• Move from current state to s in at most k time steps.
• Play action a = 0 in state s for the next N time steps to obtain an estimate (cid:98) µ si for all i ∈ [ k ] .
for the optimization 3) to obtain an approximately
state ˆ s
Hence, given the loss estimates for states in K we can estimate X i,jb for each i, j ∈ [ k ] and b ∈ { 0 , 1 } .
Next, given an arbitrary state s with s ( i ) =
b let s (cid:48)(cid:48) ∈
K such that s (cid:48)(cid:48)
( i ) =
b .
We have
[Equation]
From the above theorem we have the following guarantee.
Theorem 2.
Consider an MDP ( S , A , C , θ ) with losses in [0 , B ] , a maximum ﬁxing cost c , and correlations sets of size at most m = 2 .
Let K be a cover of C of size r ≤ 4 n , then, the algorithm of Figure(same as Figure 3) achieves a pseudo-regret bounded by O ( kr 1 / 3 ( c + B )(log rkT ) 1 / 3 T 2 / 3 ) .
Furthermore, given access to the optimization oracle for Eq.
(3) , the algorithm runs in time polynomial in k and n = |C| .
Proof.
In each time step the maximum loss incurred by any criterion is bounded by c + B .
Let { s 1 , s 2 , . . .
, s r } be the states in K .
During the exploration phase the algorithm stays in each state for N time steps and incurs a total loss bounded by kNr ( c + B ) .
During the exploration phase the algorithm moves from one state to another in at most k steps and incurs a total additional loss of at most rk 2 ( c + B ) .
At any given state s ∈ K and vertex v i , after N time steps we will, with probability at least 1 − δ , an estimate of µ si up to an accuracy of 2 B (cid:113) log1 /δN .
Setting δ = 1 / ( rkT 4 ) and using union bound, we have that at the end of the exploration phase, with probability at least 1 − 1 T 4 , the algorithm will have estimate ˆ µ si for all s ∈ K and i ∈
[ k ] such that
[Equation]
Hence during the exploitation phase, with high probability, the algorithm will have the estimate for the expected loss of each state in S , i.e., (cid:80) i µ si up to an error of 4 kB (cid:113) log rkT N .
Combining the above we get that the total pseudo-regret of the algorithm is bounded by
[Equation]
Setting N =
[Equation]
we get that
[Equation]
B.2 General case
The algorithm for the case of m = 2 naturally extends to arbitrary correlation set sizes.
Overall the structure of the algorithm remains the same where we pick a cover of C and estimate the losses incurred in states that belong to the cover.
Using the estimated losses we are able to approximately estimate the loss of any vertex at any other state.
In order to do this we extend the deﬁnition of the cover as follows.
Given correlation sets of arbitrary size in C , a vertex v i may participate in many of them.
We say that vertices v i and v j share a correlation set, if they appear together in a set in C .
Consider the set of indices of all the vertices that v i shares a correlation set with.
We partition this set into disjoint subsets such that no two vertices in different subsets share a correlation set.
For a given vertex v i , we denote this collection of disjoint subsets by I i .
For example, if C contains sets { 1 , 2 } , { 1 , 3 } , and { 1 , 4 } , then, I 1 consists of the set { 2 , 3 , 4 } .
On the other hand if C contains sets { 1 , 2 , 3 } , { 1 , 3 , 4 } , and { 1 , 6 , 7 } then, I 1 consists of sets { 2 , 3 , 4 } and { 6 , 7 } .
For a given state s and
J ∈
I i we denote by s ( J ) the vector s restricted to indices in J .
Notice that, in the worst case, I i will consist of a single set of size at most min( k − 1 , nm ) .
However, for more structured cases (e.g, m = 2 ) we expect I i to consist of subsets of small sizes.
Given i ∈ [ k ] , J ∈
I i , b ∈ { 0 , 1 } and vectors u 1 , u 2 , we say that ( i, b, J, u 1 , u 2 ) is a dichotomy, if there exist two states s, s {+(cid:
 )+} ∈ S such that: (1) s ( J ) =
u 1 , s (cid:48)
( J ) =
u 2 , (2) s ( i ) =
b = s (cid:48)
( i ) , and (3) s, s (cid:48) agree in all other criteria.
We call such a pair of states s, s (cid:48) an ( i, b, J, u 1 , u 2 ) pair.
We next extend the deﬁnition of a cover as follows.
A subset K ⊆ S is called a cover of C if for any valid dichotomy ( i, b, J, u 1 , u 2 ) , there exists an ( i, b, J, u 1 , u 2 ) pair s, s (cid:48) ∈ K .
In general, we will always have a cover of size at most n 2 mn .
Similar to (7), for a valid dichotomy ( i, b, J, u 1 , u 2 ) , we deﬁne
X i,u 1 ,u 2 b,J as
[Equation]
where s, s (cid:48) ∈ K is an ( i, b, J, u 1 , u 2 ) pair.
Given the loss values in the states present in K , we can estimate the loss of any other state using Theorem 6 stated below.
Theorem 6.
Let K be a cover for C .
Then, for any state s ∈ S and any i ∈ [ k ] with s ( i ) =
b , we have:
[Equation]
Here s (cid:48)(cid:48) is any state in K with s (cid:48)(cid:48)
( i ) =
b .
Proof.
Let s, s (cid:
) ∈ K be an ( i, b, J, u 1 , u 2 ) pair.
When we move from state s to s (cid:48) , the only difference between the expected losses incurred by vertex v i comes from the conﬁguration of the vertices in J .
Hence there at at most 2 | J | +1 distinct parameters governing the expected loss incurred by vertex i in a given state s due to the conﬁguration of the vertices in J .
Denoting these parameters by γ b,s ( J ) i,J we have
[Equation]
Given the loss values for the states in the cover K , we can estimate the quantities X i,s ( J ) ,s [Equation]
( J ) b,J .
Next, for an arbitrary state s such that s ( i )
=
b , let s (cid:48)(cid:48) ∈
K be such that s (cid:48)(cid:48)
( i ) =
b .
We have
[Equation]
• Move from current state to s in at most k time steps.
• Play action a = 0 in state s for the next N time steps to obtain an estimate (cid:98) µ si for all i ∈ [ k ] .
Using the estimated losses for the states in K and Equation (11), run the oracle for the optimization (3) to obtain an approximately optimal state ˆ s . 5.
Move from current state to ˆ s and play action a = 0 from ˆ s for the remaining time steps.
For general correlation sets with each vertex participating in at most n sets, we use (11) instead of (8) to estimate losses in step 4 of the algorithm in Figure 7.
The algorithm for general m is described in Figure 8 and has the following associated regret guarantee.
The proof is identical to the proof of
Theorem 2.
Theorem 7.
Consider an MDP ( S , A , C , θ ) with losses bounded in [0 , B ] and maximum cost of ﬁxing a vertex being c .
Given correlations sets C of size at most m , and a cover K of C of size r ≤ n 2 mn , the algorithm in Figure 8 achieves a pseudo-regret bounded by O ( kr 1 / 3 ( c + B )(log rkT ) 1 / 3 T 2 / 3 ) .
Furthermore, given access to the optimization oracle for Eq.
(3) the algorithm runs in time polynomial in k , n = |C| and r =
| K | .
C Beyond T 23 regret
In this section, we present algorithms for our problem that achieve ˜ O ( √ T ) regret, ﬁrst in the case m = 1 , next for any m , under the natural assumption that each criterion does not participate in too many correlations sets.
Let us ﬁrst point out that our problem can be cast as an instance of the stochastic multi-armed bandit problem with switching costs, where each state s is viewed as an arm and where the cost of transitions from state s to state s (cid:48) is the switching cost between s and s (cid:48) .
For the instance of this problem with identical switching costs, Cesa-Bianchi et al.
[2013][Appendix A] gave an algorithm achieving expected regret ˜ O ( √ T ) , via an arm-elimination technique with at most O (log log T ) switches.
However, naturally, the regret guarantee and the time complexity of that algorithm depend on the number of arms, which in our case is exponential ( 2 k ).
We will show here that, in most realistic instances of our model, we can achieve ˜ O ( √ T ) regret efﬁciently.
We ﬁrst consider the case where the correlations sets in C are of size one ( m = 1 ).
In this case, the parameter vector θ can be described using the following 2 k parameters: for each i ∈
[ k ] , let γ {+
 i denote+} the expected loss incurred by criterion i when it is unﬁxed and γ 1 i its expected loss when it is ﬁxed.
In this case, the cover K is of size k + 1 and includes the all-zero state, as well as k states corresponding to the indicator vectors of the k vertices.
Our algorithm is similar to the UCB algorithm for multi-armed bandits Auer et al.
and maintains optimistic estimates of the parameters.
For every vertex i , we denote by τ 0 i,t the total number of time steps up to t (including t ) during which the vertex v i is in an unﬁxed position and by τ 1 i,t the total number of times steps up to t during which vertex v i is in a ﬁxed position.
Fix δ ∈ (0 , 1) and let ˆ γ bi,t be the empirical expected loss observed when vertex v i is in state b , for b ∈ { 0 , 1 } .
Our algorithm maintains the following optimistic estimates at each time step t ,
[Equation]
, correlation sets C , ﬁxing costs c i .
Let K be the cover of size k + 1 that includes the all zeros state and the states corresponding to indicator vectors of the k vertices.
Move to each state in the cover once and update the optimistic estimates according to (12).
For episodes h = 1 , 2 , . . .
do: • Run the optimization oracle for solving Eq.
(3) with the optimistic estimates as in (12) to get a state s .
• Move from current state to state s .
Stay in state s for t ( h ) time steps and update the corresponding estimates using (12).
Here t ( h ) =
min i τ s ( i ) i,t h and t h is the total number of time steps before episode h starts.
To minimize the ﬁxing cost incurred when transitioning from one state to another, our algorithm works in episodes.
In each episode h , the algorithm ﬁrst uses the current optimistic estimates to query the optimization oracle and determine the current best state s .
Next, it remains at state s for t ( h ) time steps before querying the oracle again.
The number of time steps t ( h ) will be chosen carefully to avoid incurring the ﬁxing costs too often.
The algorithm is described in Figure 9 (same as Figure 4 in main body ).
We will prove that it beneﬁts from the following regret guarantee.
Theorem 3.
Consider an MDP ( S , A , C , θ ) with losses bounded in [0 , B ] and maximum cost of ﬁxing a vertex being c .
Given correlations sets C of size one, the algorithm of Figure 9 (same as Figure 4) achieves a pseudo-regret bounded by O ( k 2 ( c + B ) 2 √ T log T ) .
Furthermore, given access to an oracle for (3) , the algorithm runs in time polynomial in k .
Proof.
We ﬁrst bound the total number of different states visited by the algorithm.
Initially the algorithm visits k + 1 states in the cover.
After that, each time the optimization oracle returns a new state s , by the deﬁnition of t ( h ) , the number of time steps where some vertex is in a 0 or 1 position is doubled.
Hence, at most O ( k log T ) calls are made to the optimization oracle.
Noticing that one can move from one state to another in at most k time steps, the total loss incurred during the switching of the states is bounded by O ( k 2 ( c + B ) log T ) .
For (cid:15) > 0 to be chosen later, we consider the episodes where the algorithm plays a state s with expected loss at most (cid:15) more than that of the best state s ∗ .
The total expected regret accumulated in these good episodes is at most (cid:15)T .
We next bound the expected regret accumulated during the bad episodes.
From Hoeffding’s inequality we have that for any time t , with probability at least 1 − δT 3 , for all i ∈ [ k ] , b ∈ { 0 , 1 } ,
[Equation]
Let G be the good event that (13) holds for all t ∈ [1 , T ] .
Conditioned on G we also have that for any state s and vertex
i
[Equation]
where ˜ µ si is the estimated loss using the optimistic estimates.
We will bound the expected regret accumulated in the bad episodes conditioned on the event G above.
In order to do this we deﬁne certain key quantities.
Consider a particular trajectory T of T time steps executed by the algorithm.
Furthermore, let T be such that the good event in (13) holds during the T time steps.
We associate the following random variables with the trajectory.
Let N (cid:15) be the total number of time steps spent in bad episodes.
Furthermore, let Reg (cid:15) be the total accumulated regret during these time steps.
Then it is easy to see that E [ Reg (cid:15) | G ] > (cid:15)N (cid:15) .
For each vertex v i and b ∈ { 0 , 1 } we deﬁne τ (cid:15) ( i, b ) to be the total number of time steps that vertex v i spends in bad episodes in position b and τ (cid:15) ( i, b, t ) to
Notice that
[Equation]
[Equation]
Consider a particular bad episode h and let s be the state returned by the optimization oracle during that episode.
Then conditioned on the good event G , the total regret Reg h accumulated during episode h satisﬁes
[Equation]
since s is best state according to the optimistic losses (cid:1)
[Equation]
[Equation]
In the above inequality [Equation] the expectation is taken over the loss distribution for each vertex during states visited in the trajectory T [Equation]
Since τ bi [Equation] t h ≥ τ (cid:15) ( i, b, t h ) we have we have that
[Equation]
Summing over bad episodes, the total expected regret in bad episodes can be bounded by
[Equation]
Notice that τ (cid:15) ( i, b, t h ) = (cid:80) h (cid:48) <
h : h (cid:48) is bad t ( h (cid:48) ) .
Furthermore, we know that (Jaksch et al.
for any sequence z 1 , z 2 , . . .
, z h of non-negative numbers such that
z
i ≥ 1 , h
[Equation]
i
[Equation]
From (17) we get:
[Equation]
Substituting into (16) we get that
[Equation]
Using (15) we have that the above expected regret is maximized when τ (cid:15) ( i, b ) are equal, thereby implying
[Equation]
Using the fact that E [ Reg (cid:15) | G ] > (cid:15)N (cid:15) we get that conditioned on G ,
[Equation]
Combining trajectories T where the good event G holds, we get that the total expected regret accumulated in the bad episodes satisﬁes
[Equation]
Combining the above with the total expected regret accumulated in the good episodes, the loss of moving to different states, and the probability of good event G not holding, we get
[Equation]
Setting (cid:15) = 1 √ T and δ = 1 T 4 , we have the ﬁnal bound
[Equation]
The above result extends to higher m values, assuming that each vertex does not participate in too many correlation sets.
If a vertex v i appears in at most O (log k ) correlation sets, then the total loss incurred by vertex v i in any state depends on the position of v i and every other vertex that it is correlated with.
Hence the total loss incurred by vertex
v i depends on an O ( m log k ) -dimensional vector.
For every conﬁguration b of this vector, we associate with each vertex v i , parameters γ b i .
Notice that there are at most O ( k m ) such parameters.
Each parameter is in turn a sum of a subset of the parameters in θ .
Notice that in this case the size of the cover K is upper bounded by O ( k m +1 ) .
Our algorithm for higher m values is similar to the one for m = 1 , but instead maintains optimistic estimates of the parameters γ b i via
[Equation]
Here τ b i,t is the total time spent up to and including t where the vertex i and the vertices that it is correlated with are in conﬁguration b .
Similarly, for a given state s , we will denote by s ( i ) , the conﬁguration of the vertex i and the vertices that it is correlated with.
The algorithm is sketched below
Input: The graph G , correlation sets C , ﬁxing costs c i .
Let K be the cover of size O ( k m +1 ) .
Move to each state in the cover once and update the optimistic estimates according to (18).
For episodes h = 1 , 2 , . . .
do: • Run the optimization oracle (3) with the optimistic estimates as in (18) to get a state s .
• Move from current state to state s .
Stay in state s for t ( h ) time steps and update the corresponding estimates using (18).
Here t ( h ) =
min i τ s ( i ) i,t h and t h is the total number of time steps before episode h starts.
For m ≥ 1 , we obtain the following guarantee.
Theorem 8.
Consider an MDP ( S , A , C , θ ) with losses bounded in [0 , B ] and maximum cost of ﬁxing a vertex being c .
Given correlations sets C of size at most m such that each vertex participates in at most O (log k ) sets, the the algorithm in Figure 10 achieves a pseudo-regret bounded by
O ( mk 2 m +2 ( c + B ) 2 √ T log T ) .
Furthermore, given access to an oracle for (3) , the algorithm runs in time polynomial in O ( k m +1 ) .
Proof.
The proof is very similar to the proof of Theorem 3.
Since each time the optimization oracle is called the time spent in some conﬁguration s ( i ) is doubled, we get that the total number of calls to the optimization oracle are bounded by O ( k m log T ) .
Hence the total loss incurred during the exploration phase can be bounded by O ( k m ( c + B ) log T ) .
Let G be the good event that (18) holds for all t ∈ [1 , T ] .
As before, the loss incurred during good episodes is bounded by (cid:15)T .
Deﬁne τ (cid:15) ( i, b ) to be the total time that vertex i and vertices that it is correlated with spend in conﬁguration b during bad episodes.
Then analogous to (15) we have
[Equation]
For a trajectory T where the good event G holds, the total expected regret in bad episodes can be bounded as
[Equation]
Using the fact that E [ Reg (cid:15) |T ] > [Equation] (cid:15) we get that for a trajectory where the event G holds,
[Equation]
Hence we get that conditioned on the good event G , the total expected regret accumulated in the bad episodes is at most
[Equation]
Combining the above with the total expected regret accumulated in the good episodes, the loss of moving to different states, and the probability of the event G not holding we get
[Equation]
Setting (cid:15) = 1 √ T and δ = 1 T 4 , we have the ﬁnal bound
[Equation]
An important corollary of the above is the following
Corollary 1.
If G is a constant degree graph with correlation sets consisting of subsets of edges in G , then there is a polynomial time algorithm that achieves a pseudo-regret bounded by O ( k 6 ( c +
B ) 2 √ T log T ) .
D Adversarial setting
In the previous section, we studied a stochastic model for arrival of complaints and designed no regret algorithms.
In this section, we study the setting when we cannot make assumptions about the arrival of complaints.
In particular, we study an adversarial model where at each time step multiple complaints arrive for the vertices in G via the choice made by an oblivious adversary.
For a given vertex v i and time step t , we denote by (cid:96) i ( t ) the loss incurred if criterion v i is unﬁxed at time t .
Similar to the setting from the previous section, initially all the vertices in G are in unﬁxed state and each vertex has a ﬁxing cost of c i .
At each time step the algorithm can decide to ﬁx a particular vertex.
As a result all its neighbors get unﬁxed.
At time step t , if criterion v i is unﬁxed then the the algorithm incurs a loss of (cid:96) i ( t ) .
If v i is ﬁxed at time step t then algorithm incurs no loss.
The overall loss incurred by the algorithm is the total ﬁxing cost and the total loss incurred over the arrival complaints.
As before, we will denote a conﬁguration of the vertices in G using a vector s ∈ { 0 , 1 } k with s ( i ) = 0 representing an unﬁxed vertex.
For an algorithm A processing the request sequence,
During the course of T time steps, the total loss of processing the complaints is
[Equation]
Deﬁne OPT to be the algorithm that given the entire loss sequence in advance, makes the optimal choice of decisions to ﬁx vertices.
Following standard terminology we deﬁne the competitive ratio of an algorithm A to be the maximum of Loss ( A ) / Loss ( OPT ) over all possible complaint sequences.
We will design efﬁcient online algorithms for processing the complaints that achieve a constant competitive ratio.
Notice that in this setting, in order for the competitive ratio to be ﬁnite, we need to bound the range of the losses and the ﬁxing costs of the vertices.
We will assume that the cost of ﬁxing each vertex is at least one and as before assume that the losses are bounded in the range [0 , B ] .
For ease of exposition, in the rest of the discussion we will assume that at each time step complaints arrive for one of the vertices in G .
A simple reduction shows that an algorithm that is competitive with OPT in this setting remains so in the general setting with the same competitive ratio.
We discuss this at the end of the section.
Via this reduction we can consider the loss sequence to be of the form (( i 1 , (cid:96) i 1 ) , . . .
, ( i T , (cid:96) i T )) where i t is the index of the criterion for which the t th complaint arrives and (cid:96) i t is the associated loss.
To get a better understanding of the above adversarial setting, consider the case when the graph G over the criteria has no edges, i.e., there are no conﬂicts.
In this case, given a sequence of complaints, each with unit loss value, the optimal ofﬂine algorithm that has the entire loss sequence in advance can independently make a decision for each vertex.
In particular, if the total loss of the complaints incurred at vertex v i exceeds the ﬁxing cost
c i then the optimal decision is to ﬁx the vertex
v i , and otherwise simply incur the loss from the arriving complaints.
In this case the online algorithm can also simply process each vertex independently.
At each vertex the algorithm is faced with the classical ski-rental problem for which there exists a deterministic algorithm that is 2 -competitive with optimal algorithm Karlin et al.
For each vertex i , the online algorithm simply waits till a total loss of c i or more has been incurred on vertex i and then decides to ﬁx it.
It is easy to see that the total cost incurred by this strategy is at most twice the cost incurred by OPT.
However, the above algorithm will fail miserably in the presence of conﬂicts in the graph G .
As an example consider a graph with two vertices v i and v j that are connected by an edge.
Let the ﬁxing cost of v i be 1 and the ﬁxing cost of v j be C (cid:29) 1 .
Consider a sequence of complaints, each of unit loss, consisting of C complaints for v j followed one complaint for v i .
If this sequence is repeated T times the optimal ofﬂine algorithm OPT incurs a loss of C + T by ﬁxing v j and incurring losses due to v i .
However, the algorithm above will incur a cost of (2 C + 2) T thereby leading to an unbounded competitive ratio.
Hence, in order to achieve a good competitive ratio one must make decisions not only based on the loss incurred at the given vertex v i , but also the status of the vertices in the neighborhood of v i .
Our main result in this section is the algorithm in Figure 11 that achieves a constant factor competitive ratio.
The algorithm described in Figure 11 makes decisions based on local neighborhood information of a vertex.
Intuitively, if a vertex is ﬁxed only once or a few times in the optimal algorithm one would like to avoid ﬁxing it too many times.
In order to achieve this, each time a vertex v i is ﬁxed, it adds a barrier of κ i =
c i to the loss any of its neighbors need to incur before getting ﬁxed.
Hence, if a vertex is connected to a lot of ﬁxed vertices then it has a high barrier to cross before getting ﬁxed.
During the course of the algorithm each unﬁxed vertex is in one of the two phases.
In phase one, the vertex is accumulating losses to pay for the barrier introduced by its neighbors (step 2(b) of the algorithm).
In phase two, once the barrier has been crossed the vertex follows the standard ski-rental strategy independent of other vertices for making a decision as to ﬁx or not.
Notice that via step 2(b) of the algorithm, multiple neighbors of a vertex v i can help bring down the barrier of c i introduced by the action of ﬁxing vertex v i .
This is necessary to ensure the online algorithm does not incur a large loss on a vertex by waiting too long to ﬁx it.
As an example consider a graph G with k vertices and k − 1 edges, where vertex v 0 is the central vertex connected to every other vertex.
Let the ﬁxing cost of vertex v 0 be a large value C , and the ﬁxing cost of other vertices be one.
We consider a sequence of C complaints, each with unit loss arriving for vertex v 0 , followed by a sequence of C complaints for vertex v 1 and so on.
In this case the optimal ofﬂine solution incurs a loss of C + k by deciding to ﬁx every vertex except v 0 .
After processing C complaints for v 0 , the online algorithm will ﬁx v 0 and incur a loss of 2 C .
Next, during the course of processing C complaints for v 1 , the algorithm ﬁxes v 1 and incurs an additional loss of
C + 1 .
More importantly, due to step 2(b), the barrier κ 0 introduced by vertex v 0 has been reduced to zero and hence the algorithm only incurs a loss of 2 per vertex for the remaining sequence for a total loss of 3 C + 2 k − 1 .
Without the presence of step 2(b) each vertex will incur a loss of C leading to a large competitive ratio.
Notice that our algorithm in Figure 11 is designed for a setting where in each time step complaints arrive for a single vertex in G .
If multiple vertices accumulate complaints in a time step, we can simply order them arbitrarily and run the algorithm on the new sequence.
Let OPT be the optimal ofﬂine algorithm according to the chosen ordering of the complaints.
Let OPT’ be the optimal ofﬂine algorithm when processing multiple complaints per time step.
Notice that for each time step, the loss of OPT cannot be larger than that of OPT’ since any choice available to OPT’ is available to OPT as well.
Hence it is enough to design an algorithm that is competitive with OPT.
In particular, we have the following theorem.
Theorem 4.
Let G be a graph with ﬁxing costs at least one.
Then, the algorithm of Figure 11 achieves a competitive ratio of at most 2 B + 4 on any sequence of complaints with loss values in [0 , B ] .
Proof.
Recall that (cid:96) i ( t ) denotes the loss incurred by vertex v i at time t .
We divide this loss into the amount that was used to reduce the κ j value of one its neighbors and the rest.
Formally, for every edge ( i, j ) we deﬁne δ ti → j as follows.
If in time step t , the complaint arrived for vertex i and step 2(b) was executed to reduce κ j by ∆ , then we deﬁne δ ti → j = ∆ .
Otherwise we deﬁne δ ti → j to be zero.
We also deﬁne
[Equation]
If vertex v i is ﬁxed f i times during the course of the algorithm then we have that the total loss incurred by the algorithm can be written as
[Equation]
Next we notice that each time a vertex v i is ﬁxed it accumulates a value of κ i
=
c i .
Furthermore, the total loss incurred by vertices as a result of executing step 2(b) is upper bounded by the total κ value accumulated.
Hence we have
[Equation]
Substituting into (24) we have
[Equation]
Next we bound the above loss for each vertex separately.
For a given vertex v i that is ﬁxed f i times by the algorithm, we can divide the time steps into f i +1 intervals consisting of an interval I 0 starting from t =
up to (and including) the ﬁrst time v i is ﬁxed.
The next f i intervals correspond to the time spent by v i between two successive ﬁxes.
Denoting these intervals as I 0 , I 1 , . . .
we have that
[Equation]
Next we compare the above to the loss incurred by OPT for vertex v i .
Let (cid:96) ∗ i ( t ) be the loss incurred by
OPT for vertex v i at time t .
We will denote by s ∗ t the state of the vertices at time t according to OPT.
We instead redeﬁne the loss incurred by OPT for vertex v i at time t to be
[Equation]
Notice that
[Equation]
Hence we get that
[Equation] [Equation]
Next we consider each interval in (26) separately.
For any interval I r we have that
[Equation]
This is because after incurring a loss of more than c i , any additional loss incurred by v i is due to step 2(b), since otherwise step 2(c) will be executed and v i will be ﬁxed.
Next consider interval I 0 .
The loss incurred by the algorithm on vertex v i equals (cid:80) t ∈
I
δ ti → i ≤
Bc i .
Either OPT ﬁxes v i at least once during this interval or incurs the total loss.
Either way we have that the loss incurred by OPT is at least
[Equation]
Next consider an interval I r between two successive ﬁxes.
The loss incurred by the algorithm for vertex v i during this interval is at most
[Equation]
If OPT ﬁxes v i at least once during this interval then it incurs a cost of c i .
If v i remains unﬁxed in
OPT during the course of the interval then OPT incurs a loss of at least c i .
This is because vertex v i went from being unﬁxed to ﬁxed during the second half of the interval and hence a total loss of at least c i must have arrived for the vertex v i during this interval.
Finally, suppose vertex v i is ﬁxed in OPT before the start of the interval and remains so throughout.
Since v i goes from being ﬁxed to unﬁxed during the ﬁrst half of the interval, we must have (cid:80) t ∈
I r [Equation] j ∈ N ( i ) δ tj → i ≥ c i .
Furthermore, since v i is ﬁxed by OPT during this interval, OPT must incur a loss on all neighbors of j .
In particular, from (28) we have
[Equation]
j ∈ N ( i ) δ tj
[Equation]
i
[Equation]
[Equation]
∗ t ( j ) [Equation] 0) (33) ≥ c i . In either of the three cases we have that the loss (cid:80) t ∈ I r ˜ (cid:96) i ( t ) incurred by OPT is at least a 1 / ( B + 2) fraction of the loss incurred by the algorithm.
Summing over all the vertices and the corresponding intervals, we get that the total loss incurred by the algorithm can be bounded by
Loss ( A ) ≤ ( B + 2)
T (cid:88) t = k (cid:88) i = ˜ (cid:96) i ( t ) ≤ 2( B + 2) Loss ( OPT ) .
E Experiments
In this appendix we provide more detail of the experiments discussed in Section 5 and also report additional results with the algorithm for the stochastic setting.
We view our work as primarily theoretical and of course a more extensive empirical evaluation is a direction for future work.
Regarding the choice of baselines, we are not aware of any efﬁcient algorithms that directly apply to our setting.
Existing general algorithms for solving MDPs will not scale to our setting since their complexity is proportional to the number of states.
Note that in our experiments we will demonstrate that our proposed algorithms can compete with the ofﬂine optimal (the best solution in hindsight) which is a strong comparison point.
E.1 Experiments with simulated data
We evaluate the performance of our algorithms developed in the stochastic setting of Section 3.
We ﬁrst detail experiments on simulated data.
We consider a simulated environment where the conﬂict graph G is generated from the Erd ˝ os-Renyi model: G ( k, p ) where we set p = 2 log k k .
This ensures that with high probability G is connected.
Next we generate correlation sets C consisting of pairs of vertices in G sampled uniformly at random.
For a parameter α > 0 that we vary, we choose αk pairs of vertices at random and add them as correlation sets in C .
Hence on average, each vertex participates in α correlation sets.
We also add to C singleton sets for each vertex in G .
The ﬁxing cost of each vertex is samples uniformly at random in the range [1 , 5] .
Next we describe the choice of parameters governing the loss distribution of the different states in the MDP.
For a correlation set { i } of size one corresponding to vertex v i , we sample a parameter γ 1 i from the beta distribution Beta (0 . 5 , 0 . 5) .
For a given state s with s ( i ) = 1 , the loss generated due to { i } is drawn from an exponential distribution with mean γ 1 i .
For a given state s with s ( i ) = 0 , the loss generated due to { i } is drawn from an exponential distribution with mean λγ 1 i , where λ > 1 is a parameter that we vary.
For a correlation set { i, j } of size two, we generate two parameters γ 1 , 1 i,j and γ 1 , 0 i,j from the beta distribution Beta (0 . 5 , 0 . 5) such that γ 1 , 0 i,j > γ 1 , 1 i,j .
For a given state s with s ( i ) = 1 and s ( j )
= 1 , the loss generated due to { i, j } is drawn from an exponential distribution with mean γ 1 , 1 i,j .
For states where s ( i ) =
and s ( j ) = 1 or vice-versa, the loss is generated from an exponential distribution with mean γ 1 , 0 i,j .
Finally, for states where both s ( i ) = 0 and s ( j ) = 0 , the loss is generated from an exponential distribution with mean λγ 1 , 0 i,j .
In general, computation of the optimal state in (3) requires time exponential in k .
In our experiments we approximate the optimal state by a linear programming relaxation of the optimization in (3) and use the appropriately rounded linear programming relaxation solution as a proxy for the optimal state.
For general m , our proposed algorithms in Figure 3 and Figure 10 have complementary strengths.
While the algorithm in Figure 3 incurs a higher regret as a function of the number of time steps T , its running time has a polynomial dependence on the parameter α , i.e., the number of correlation sets that a vertex participates in, on average.
The algorithm in Figure 10 incurs a smaller regret of ˜ O ( √ T ) as a function of T at the expense of an exponential dependence on α .
In Figures 12 andwe empirically demonstrate this behavior where for small values of α , the ˜ O ( √ T ) -regret algorithm is much better, whereas for higher values of α the ˜ O
( T 2 / 3 ) -regret algorithm is more desirable.
For the case of m = 1 however, i.e., singleton correlation sets, the algorithm in Figure 10 achieves a smaller regret and runs in polynomial time and hence is expected to outperform the explore-exploit Figure 12: shows the accumulated loss incurred by the Algorithms in Figure 3 and Figure 10 on with α , we add based algorithm from Figure 3.
As can be seen from Figure 14 this is indeed the case and the ˜ O ( √ T ) regret algorithm signiﬁcantly outperforms the ˜ O
( T 2 / 3 ) regret algorithm.
E.2 Experiments with a real-world dataset
In this section we demonstrate via experiments how our framework and algorithms can be applied to real world data.
In order to do this we study the UCI Adult dataset [Kohavi, 1996].
The dataset comprises of 48852 examples each represented using 124 features, after binarizing categorical features.
Each data point corresponds to a person and the label is a 0 / 1 value representing whether the income of the person is more or less than $50,000.
The dataset contains information about sensitive attributes such as race and gender.
We will simulate an online scenario where a classiﬁer is making predictions on the income of individuals.
At each time step a batch of complaints arrive, the system incurs a loss and responds by transitioning to a different state (and updating the classiﬁer).
We next describe how we instantiate various components of our stochastic model from Section 3.
Graph G : We take race as a sensitive attribute that takes values in { black , white } , to obtain two sub-populations and consider two natural criteria namely the true positive rate and the AUC score.
This leads to four vertices tpr w , tpr b , auc w , auc b .
Furthermore, we add the classiﬁer accuracy as another criterion.
This leads to total 5 vertices in the graph.
Losses and Correlation Sets: We consider correlation sets of size one, and hence the total loss incurred at any state is the sum of the losses incurred by each criterion.
For the accuracy criterion we simply deﬁne the loss to be the error of the system (the classiﬁer).
We next describe how we deﬁne the loss for the tpr w criterion.
We ﬁrst compute the overall true positive rate of the classiﬁer and also the true positive rate on the white population.
If the two deviate by more than a threshold τ , then we penalize the classiﬁer linearly in the violation.
Therefore the loss for tpr w is deﬁned as: max(0 , | tpr overall − tpr w | − τ ) .
The loss for all other criteria is deﬁned the same way.
In our experiments we choose τ = 0 .
Note that while we ﬁx the threshold apriori, our method does indeed offer a way to choose the thresholds themselves in a data-driven manner.
This can be achieved by simply adding, for each metric i , additional metrics to the graph with different thresholds τ i, 1 , τ i, 2 , . . .
and so on.
Incompatibilities and State Transitions: To generate incompatibilities among criteria we compute a set of valid and invalid states as follows.
For each state s ∈ { 0 , 1 } 5 , we solve a constrained optimization problem on a training set to compute a classiﬁer.
We then evaluate the classiﬁer on the test set to compute the loss of each criterion.
If the loss of any criterion is more than a speciﬁc threshold then we consider the state as an invalid state, otherwise the state is valid.
In our experiments we set a threshold of 0 . 4 for the accuracy criterion.
For the considered criteria we present results for two thresholds, 2 τ and 6 τ , the ﬁrst one resulting in 4 valid states and other second one resulting in 7 valid states.
To solve a constrained optimization problem we use the tensorﬂow constrained optimization toolkit [Cotter et al., 2018a,b].
We use the default parameter settings provided by the toolkit.
The toolkit is released under Apache license 2.0.
If a state s has accuracy criterion set to 1 , then we optimize for model accuracy subject to constraints for the other criteria that are set to 1 in s .
If the accuracy criterion is set to 0 then we optimize for a constant loss function subject to constraints.
Recall that our proposed algorithms function by ﬁxing a criterion and as a result moving to another state.
We obtain these state transitions as follows.
If the algorithm asks to ﬁx criterion v i in state s , we set s ( i ) = 1 to go to the next state s (cid:48) .
While s (cid:48) is invalid, we unﬁx the criterion (not including v i ) with the highest loss in the state s (cid:48) to reach another state.
Fixing Cost: We simply take the ﬁxing cost of each criterion to be 1 .
Simulating Complaints: We divide the dataset into a set of 16000 examples that we use to update our classiﬁer at each time step and the remaining test set to simulate the arrival of complaints.
At each time step, we randomly select a batch of examples from the test set to generate complaints.
This set of complaints is used to compute the loss of a given state at a given time step.
Benchmark and Results: We compare our Algorithm from Figure 4 with an ofﬂine optimal solution that has been computed to ﬁnd the state with the minimum average loss over the arrival sequence of complaints.
The results are averaged over 10 independent runs.
The results are shown in Figure 15 and Figure 16.
We show results for two values of the threshold parameters and in each case plot the loss of the algorithm as compared to the benchmark, as well as the states chosen by the algorithm, as a function of the number of time steps.
As can be seen from
Figure 15 our algorithm quickly converges to the ofﬂine optimal solution after an initial exploration phase.
To get a better understanding of the performance of the algorithm in the initial phases, in
Figure 16 we plot the same setting as in the case of Figure 15, but with x -axis on a log-scale.
For the case of threshold being 0 .
01 , one can see that the state 0 results in much higher loss and, during exploration, the algorithm alternates in a periodic pattern between states 1 and 3 that have similar loss.
A similar pattern holds for the case of the threshold being 0 .
It is important to note that the choice of the loss functions was important in this case and that we did not weight each criterion by the volume of the complaints.
This demonstrates that our algorithms, when combined with a good choice of the loss function, can be useful in practice.
We present results for two threshold values, and in each case plot the loss of the ofﬂine solution and the online algorithm as well as the states chosen by the online algorithm, as a function of the time steps.
Compute Resources.
All our experiments were performed on a machine containing a Tesla [-P
-]P
GPU with 80 GB of RAM and four CPUs.
Hyperparameters.
For the case of simulated data the hyperparameters have been mentioned in
Section E.1.
For the case of real data, apart from the hyperparameters mentioned in Section E.2, we used the default learning rates and optimizers provided by the tensorﬂow constrained optimization toolkit [Cotter et al., 2018a,b].
We performed a random train/test split as detailed in Section E.2.
Assets.
We used publicly available code from the tensorﬂow constrained optimization toolkit 1 and the publicly available UCI Adult Dataset 2 .
F Further discussion on the COMPAS example
Throughout the main sections, we have mentioned that the choice of the loss function is important in the effectiveness of our model.
We brieﬂy discussed this in Section 3.
Below, we present a more detailed discussion of the effect of the loss function on our model, by using the COMPAS scenario from Section 1 as an example.
Loss function – COMPAS illustration .
Consider the COMPAS example with a graph G with four criteria namely, false positive rate on population A , false positive rate on population B , AUC score for population A and AUC score for population B .
We want to understand what kinds of loss functions will result in an overall suboptimal system when our model and algorithms from Section 3.
Suppose our algorithm take an action to ﬁx a criterion and reach a state where the true positive rates and the
AUC scores associated with the four criteria are: [0 . 1 , 0 . 8 , 0 . 5 , 0 . 5] .
Then a poor choice of the loss function would be f 1 · 0 .
+ f 2 · 0 .
+ f 3 · 0 .
+ f 4 · 0 . 5 , where f i represents the fraction of complaints that trigger criterion i .
Such a choice of the loss function will make our system vulnerable to the loudest voices in the system and as a result might not lead to a good solution at all.
A more reasonable choice of the loss is 0 .
1 + 0 .
8 + 0 .
5 + 0 .
5 , that weighs each criteria equally and does not take into account the underlying size of the population.
Another alternative is λ 1 | 0 .
1 − 0 .
8 | + λ 2 ( | 0 .
5 − 0 .
5 | ) , that aims at keeping both the discrepancy in the false positive rate and the AUC scores small.
Finally, the choice we make in our experiments of penalizing each criterion for the deviation from the value over the entire population, i.e., max(0 , | tpr overall − tpr w | − τ ) , also leads to good solutions empirically.
Another case where additive losses are a poor choice is if the criteria in G is not chosen carefully.
For instance, consider a scenario in the COMPAS example where all except one of the criteria correspond to the performance of the system on population A .
An additive loss would then naturally force the system to disproportionately favor population A over a period of time.
While the above discussion used the COMPAS scenario as a speciﬁc example, we would like to re-iterate that our model and algorithms are much more general and can be motivated from different applications.
As another motivating scenario for our work, consider a large organization that is building a classiﬁer to detect harmful content that the users of their platform may be exposed to.
The organization wants to build a classiﬁer that has a good overall performance, say measured in terms of false positive rates (FPRs) and false negative rates (FNRs) (these in general could be arbitrary metrics).
Furthermore, the organization also wants to ensure good FPRs and FNRs on users sliced by different attributes such as race, gender, geographic location, education level etc.
While the overall classiﬁer performance is still of paramount importance, the organization’s policy team may have given them guidelines to try and enforce that FPRs and FNRs on different slices are less than a certain threshold.
However, not all such constraints may be satisﬁable and the organization wants to ﬁgure out the optimal tradeoffs between these metrics via end user feedback.
Our model and algorithms address this question