Online Decision Mediation from Scratch
Abstract
Consider learning a decision support assistant to serve as an intermediary between(oracle) expert behavior and (imperfect) human behavior: At each time, the algorithm observes an action chosen by a fallible agent, and decides whether to acceptthat agent’s decision, intervene with an alternative, or request the expert’s opinion.
For instance, in clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances, thus real-world decision support is often limited to monitoring and forecasting.
Instead, such an intermediary would strike a prudent balancebetween the former (purely prescriptive) and latter (purely descriptive) approaches,while providing an efﬁcient interface between human mistakes and expert feedback.
In this work, we ﬁrst formalize the sequential problem of online decision mediation—that is, of simultaneously learning and evaluating mediator policies from scratchwith abstentive feedback :
In each round, deferring to the oracle obviates the risk oferror, but incurs an upfront penalty, and reveals the otherwise hidden expert actionas a new training data point.
Second, we motivate and propose a solution that seeksto trade off (immediate) loss terms against (future) improvements in generalizationerror; in doing so, we identify why conventional bandit algorithms may fail.
Finally,through experiments and sensitivities on a variety of datasets, we illustrate consistent gains over applicable benchmarks on performance measures with respect tothe mediator policy, the learned model, and the decision-making system as a whole.
Introduction
Research in data-driven decision support has burgeoned in recent years, with proposed applicationsin a wide variety of domains such as ﬁnance [1], psychology [2] and medicine [3].
Most work onmachine learning for decision support falls into two categories:
On one hand, descriptive approachesdeal with monitoring, forecasting and learning interpretable parameterizations of observed humanbehavior [4–8].
While such tools can help audit and debug decision-making, they play a limited rolein directly guiding human behavior.
On the other hand, prescriptive approaches deal with systemsthat behave autonomously, optimally, and with minimal manual control [9–12].
While such tools canreduce the need for expert input, they are often at odds with ethical considerations—especially in highstakes settings such as healthcare [13–18].
Instead, we argue for a third: We believe machine learningdecision support has a viable role as an intermediary between (oracle) expert behavior and (imperfect)human behavior—that is, as an “assistant”, which would strike a prudent balance between the abovetwo approaches, while providing an efﬁcient interface between human mistakes and expert feedback.
Online Decision Mediation
In this paper, we consider learning and evaluating mediator policiesfor online decision support from scratch:
At each time step, upon observing the context vector of anincoming instance, the mediator policy decides whether to accept the human’s action, intervene withits own output, or request the expert’s opinion—and this determines which action is ultimately taken.
Deferring to the oracle obviates the risk of error, but incurs an upfront penalty, and reveals the otherwise hidden expert action as a new training data point.
As our running example, consider the task of early diagnosis in Alzheimer’s disease, where the action is to diagnose each incoming patient as cognitively normal, mildly impaired, or at risk of dementia [19,20].
Our problem setting is distinguishedprimarily by three key characteristics: (1) Instances—i.e. patients—arrive in a streaming process,and actions must be taken immediately and sequentially.
(2) Feedback—i.e. true diagnoses—is onlyavailable in an abstentive manner, meaning ground truths are only revealed if the oracle is deferred to.(3) Evaluation—i.e. cumulative regret—is computed in an online fashion, without convenient separation into “training” versus “testing” phases.
This setting is challenging but general, and applicablewherever domain experts are resource-constrained, e.g. if the costs of deﬁnitive examinations are high.
Contributions In the sequel, we ﬁrst formalize this sequential problem of ooooooooooooooooonline dddddddddddddddddecision mmmmmmmmmmmmmmmmmediation( “ODM” ), and establish its unique challenges versus more conventional problem settings (Section 2).
Second, we identify why conventional bandit algorithms may fail, and describe our proposed solution, for iiiiiiiiiiiiiiiiintervention and rererererererererererererererererequisition ( “UMPIRE” ), which seeks to trade off(immediate) loss terms against expected (future) improvements in generalization error (Section 3).
Finally, through experiments and sensitivities on a variety of real-world datasets, we illustrate consistent gains over applicable benchmarks on a comprehensive set of performance measures with respectto the mediator policy, the learned model, and the entire decision-making system as a unit (Section 4).
Implications Humans are heterogeneous, and mistakes require timely correction.
Machines are alsofallible, and models require timely learning.
The implications are clear: Rather than pitting computers
, an efﬁcient mediator should augment clinician capabilities by leveraging costly butinformative expert resources.
By focusing on the (human-expert-mediator) decision-making systemas a whole, we take a ﬁrst step towards more methodical integration of machines “into the loop”.
Moreover, the technical problem itself combines diverse challenges from sequential decision-making,learning with rejection, and active learning, thus opening the door to multiple avenues of further work.
Online Decision Mediation2.1 Problem Formulation
We use uppercase for random variables, and lowercase for speciﬁc values.
Let X denote the inputvariable, taking on values x 2 X , and let Y denote the target variable, taking on values y 2 Y .
In line 65 with related ﬁelds, X may synonymously be referred to as “contexts”, “features”, and “states” depending on the underlying task, and Y may likewise be referred to as “actions”, “decisions”, and “labels”.
Per our motivating example, we shall adopt the “context-action” terminology for consistency, althoughour framework applies to any decision task that requires mapping inputs to speciﬁc outputs.
Followingmost related settings, we focus on discrete action spaces, and leave continuous actions for future work.
Human, Expert, and Mediator Let ⇢ ( X ) denote an exogenous distribution from which a streamingsequence of contexts { X t } t is drawn and indexed by time step.
We consider three decision-makers: ahuman, an expert, and a mediator.
In each round, a human action is drawn as ˜ Y t ⇠ ˜ ⇡ ( ·| X t ) from an(unknown and possibly stochastic) human policy ˜ ⇡ 2 ⇧ .
For instance, this is the noisy diagnosis issued by an apprentice clinician.
If prompted, an expert action may likewise be drawn as Y t ⇠ ⇡ ⇤ ( ·| X t ) 75 from an (unknown and possibly stochastic) expert policy ⇡
⇤ 2 ⇧ .
For instance, this is the ﬁnal diagno- 76 sis issued by a senior doctor—that is, should they indeed be appointed to conduct a full examination ofthe patient.
Finally, a “mediator” is identiﬁed by the tuple (ˆ ⇡ ,  ) , consisting of a (learned) model policyˆ ⇡ 2 ⇧ —from which model actions ˆ Y t ⇠ ˆ ⇡ ( ·| X t ) may be drawn—as well as a mediator policy  2  :
Deﬁnition 1 (Decision System) Let S :=(˜ ⇡ , ˆ ⇡ , ⇡ ⇤ ,  ) denote the decision system as a whole.
Givenan incoming ( X, ˜ Y ) , the mediator policy deﬁnes a distribution  ( ·| X, ˜ Y ) over the space of mediatoractions Z := { 0 , 1 , 2 } , consisting of options accept ( z =0 ), intervene ( z =1 ), and request ( z =2 ).
Let ( Y  y ) be the Dirac delta centered at y ; drawing Z ⇠  ( ·| X, ˜ Y ) induces the overall system policy :
[Equation]
Intervening incurs some cost k int (e.g. inconveniencing the apprentice clinician to reconsider/alter theirdecision).
Requesting incurs some cost k req (e.g. appointing the senior doctor to provide their opinion),but also reveals the otherwise hidden ground-truth action: Let D t :
= { ( X ⌧ , Y ⌧ ) : Z ⌧ = 2 } t ⌧ =1 denotethe cumulative dataset of requested points, taking on values d t 2 D t :=
[ t ( X ⇥ Y ) t , and constitutes 87 the training set with which the model policy ˆ ⇡ is deﬁned.
Thus feedback (for learning) is “abstentive”in that it is only observable when the system “abstains” in favor of deferring the decision to the expert.
Risk and Evaluation Among other aspects, our objective of interest differs from supervised learningin two important ways: First, we are chieﬂy interested in the performance of the decision system S ,instead of the model ˆ ⇡ per se.
Second, learning and evaluation are both conducted online.
By way ofcontrast, consider ﬁrst the familiar supervised learning objective, which is simply concerned with minimizing the generalization error of the model over the underlying data distribution, or the “model risk”:
[Equation]
where ` : Y ⇥  ( Y ) !
R is some choice of loss function.
In decision problems, this most commonly 95 takes the form of the zero-one loss ` ( Y, ˆ ⇡ ( ·| x )):= ` 01 ( Y, argmax y ˆ ⇡ ( y | x )) , or in some cases—if asurrogate loss is required—the cross-entropy ` ( Y, ˆ ⇡ ( ·| x )):=log ˆ ⇡ ( Y | x ) ; we shall use the former to beconsistent with comparable literature.
Now, our main focus is not the model risk, but the “system risk”:
Deﬁnition 2 (System Risk) Let M :=( X, Y, ˜ ⇡ , ⇡ ⇤ , ⇢ , k int , k req ) denote the mediation setting .
Givena mediator (ˆ ⇡ ,  ) , the system risk in each round t is the expected error of the induced system policy(i.e.
having selected from human, model, and expert actions), plus the upfront cost of mediator actions:
[Equation]
Then the online decision mediation (“ ODM ”) problem is to select a mediator (ˆ ⇡ ,  ) to minimize (cumulative) regret over a possibly-unspeciﬁed horizon.
Importantly, note that this is a more challenging objective than simply minimizing the generalization error of the model, system, or some asymptotic complexity thereof: Here we have no separation between “training” versus “testing”, since losses beginaccumulating from the very ﬁrst step of the sequential process.
The regret at any round n is given by:
[Equation]
where we assume realizability such that the best-in-class mediator is deﬁned as the tuple consistingof the expert policy ⇡ ⇤ and the greedy mediator policy  ⇤ (i.e. always choosing Z to minimize eachround’s immediate system risk).
Note the above notation makes it explicit that both the model policyand mediator policy evolve as sequences ˆ ⇡ := { ˆ ⇡ t } t and  := {  t } t that—in general—depend on D t ).
Remark 1 (Assumptions) For ease of exposition, we assume all mistakes are equally important, thatk int , k req are constants, and expert action classes are more or less balanced over the input distribution;allowing relaxations is straightforward and left for future work.
To eliminate the more trivial cases,we assume 0 <k int <k req , and operate in the common rejection regime where
k req =
mm  1   for somesmall  > 0 , with m being the number of actions in Y ; this induces the most interesting tradeoff settingwhere abstention is neither excessive nor immediately ruled out by the greedy policy.
(In our experiments, we shall empirically consider a range of sensitivities).
We assume nothing about ˜ ⇡ , for instanceif it is even stationary.
Lastly, we assume D 0 is randomly seeded with one example per action class.
2.2 Context and Challenges
The ODM problem lies at the conﬂuence of three classes of learning problems while being distinct fromall: (i) learning with rejection, (ii) stream-based active learning, and (iii) stochastic contextual bandits.
As before, we employ generic notation and make note of synonymous terminology as appropriate.
Learning with Rejection Compared to standard supervised learning, learning with rejection is aproblem setting that endows the algorithm—during test time—with the option to “reject” their ownprediction in favor of expert advice [21–26].
This is variously referred to as the option to “abstain”from a decision, or to “defer” to an oracle.
Exercising it incurs an abstention cost k abs , but enablesavoiding misclassiﬁcation when the model is uncertain.
Typically, a solution consists of a modelpolicy ˆ ⇡ and a rejection policy deﬁning a distribution ( U | X ) over the space of actions U :=
{ 1 , 2 } , 128 consisting of the options not abstain ( u =1 ) and abstain ( u =2 ).
While the rejection option is similarto that in ODM , learning proceeds from a static dataset, evaluation focuses on model risk, and—likesupervised learning—labels in the batch are always available.
An online variant of this setting sharesmore similarity with the ODM problem by focusing on minimizing the cumulative loss over thecourse of learning, instead of simply minimizing the held-out performance of the algorithm [27–30].
However, a key distinction from us is that feedback is not an active choice, and expert labels are alwaysstreamed (or when the model does not defer to the expert, which is exactly the opposite of our setting).
[Table]
⇤ ))
(always)  (Stream-based)ActiveLearning[31–36]   ⇡
⇤ , ˆ ⇡ ,
R = E X ⇠ ⇢ Y ⇠ ⇡ ⇤ ( ·| X ) ` ( Y, ˆ ⇡ ( ·| X ))
R (ˆ ⇡ ) a.s.c.
Z =
ActiveLearningwithAbstention[37–40]  3 ⇡ ⇤ , ˆ ⇡ , ,
R = E X ⇠ ⇢ Y ⇠ ⇡ ⇤ ( ·| X ) [ (1 | X ) ` ( Y, ˆ ⇡ ( ·| X ))
+ (2 | X ) k abs ]
R (ˆ ⇡ , ) a.s.c.
Z =
DualPurposeLearning[41]
[Table]
3 ⇡ ⇤ , ˆ ⇡ ,  =
R = E X ⇠ ⇢ Y ⇠ ⇡ ⇤ ( ·| X ) [ ` ( Y, ˆ ⇡ ( ·| X ))
| Z
= 1]
R (ˆ ⇡ ,  ) a.s.c.
Z =
(StochasticContex-tual)
[Table] [Table]
,
[Table]
R t =
E X t ⇠ ⇢ ˆ Y t ⇠ ˆ ⇡ ( ·| X t )  ( X t , ˆ Y t )
P t (
[Table]
t (ˆ ⇡ )
[Table]
t ( ⇡ ⇤ ))
(always)
[Table]
BanditswithActiveLearning[49–52]
[Table]
[Table] [Table]
=
E
[Table]
t ⇠ ⇢ ˆ Y t ⇠ ˆ ⇡ ( ·| X t )
[  (2 |
[Table]
t ) k
[Table] [Table]
t ,
[Table]
Y t )]
P t (
[Table] [Table]
[Table]
( ⇡
[Table]
Z =
[Table]
AppleTastingwithContext[53–55]
[Table] [Table]
=
[Table]
t =
[Table]
t
[Table] [Table]
|
[Table]
t
[Table]
k
[Table]
P t (
[Table]
Z =
[Table]
ReinforcedActiveLearning[56–58]   ⇡ ⇤ , ˆ ⇡ ,
R t =
E X t ⇠ ⇢ Y t ⇠ ˆ ⇡ ( ·| X t )
[  (2 | X t ) ` ( Y, ˆ ⇡ ( ·| X ))]
P t ( R t (  )  R t (  ⇤ ))
(always)
OnlineDecisionMediation  3 ⇡ ⇤ , ˜ ⇡ , ˆ ⇡ ,  =
R t =
E X t ⇠ ⇢ ˜ Y t ⇠ ˜ ⇡ ( ·| X t )
Y t ⇠ ⇡ ⇤ ( ·| X t )
[  (0 | X t , ˜ Y t ) ` ( Y t ,  ( Y  ˜
Y t ))
+  (1 | X t , ˜ Y t )( ` ( Y t , ˆ ⇡ ( ·| X t ))
+ k int )
+  (2 | X t , ˜ Y t ) k
[Table]
] P t ( R t (ˆ ⇡ ,  )
[Table]
t ( ⇡ ⇤ ,  ⇤ ))
Z =
Stream-based Active Learning
In contrast with standard incremental learning, ground truths instream-based active learning are unobserved unless actively “acquired” by the algorithm duringtraining [31–36].
This is variously referred to as the option to “request” or “query” the oracle for itsdecision.
Like supervised learning, the goal is to minimize test-time model risk, but with emphasison reducing labeled and/or unlabeled asymptotic sample complexity.
Typically, a solution consistsof a model ˆ ⇡ and an acquisition policy  deﬁning a distribution  ( Z | X ) over the space of actions
Z := { 1 , 2 } , consisting of the options not request ( z =1 ) and request ( z =2 ).
While the active requestaspect is similar to that in ODM , evaluation focuses on model risk, so  is not evaluated in the objectiveitself; moreover, the model has no ability to abstain from a prediction.
One variant includes abstentionto enable the algorithm—during test time—to reject their own prediction [37–40].
However, theobjective remains to minimize test-time loss and its asymptotic complexity, which contrasts with ourfocus on evaluating cumulative losses over the entire process.
A second variant called dual purposelearning is perhaps more similar in that the model abstains from a decision just when the expert isqueried for its decision [41] (i.e. the acquisition policy  coincides with the rejection policy , and
Z = U ).
But the risk function of interest is still like the usual test-time model risk, but now conditionedon Z =1 , so  only enters the objective as a conditioning term to omit points where the model abstains.
Stochastic Contextual Bandits Lastly, ODM bears resemblance to stochastic contextual bandits, aclass of sequential decision problems with the goal of minimizing (cumulative) regret deﬁned in termsof an arbitrary “reward” function  : X ⇥ Y !
R [42–48].
A main difference from ODM is that bandit 154 rewards and are always observed as feedback for learning after each round, but no “expert” actions (viz.best-in-class policy) are available to be queried.
One variant incorporates elements of active learningby stipulating that feedback must be requested at some cost k req [49–52]; in a special case dubbedapple tasting , model decisions are tied to acquisition decisions (i.e. the acquisition policy
coincideswith the model policy ˆ ⇡ , and Z = Y ) [53–55].
But unlike ODM , the model has no option to abstain,and there is no tradeoff between requesting information and making predictions.
A second variantturns around and treats active learning itself as a bandit problem [56–58]; in the streaming, contextualcase [56], the model policy ˆ ⇡ itself is actually not evaluated at all: Instead, the acquisition policy  isrewarded positively just when querying the expert turns out to be “useful” (i.e. Y
= ˆ Y ), and punishedjust when querying the expert turns out to be “redundant” (i.e. Y =
ˆ Y )
; moreover, feedback (for  , inthis setting) is always observed.
While these works are similar to ODM in focusing on the online evaluation objective of
Table 1 contextualizes ODM versus related work: Our setting combines the challenges from each, andis uniquely characterized by the three key aspects alluded to in Section 1: streaming instances, abstentive feedback, and online evaluation.
In Section 3, we argue that a good algorithm must appropriatelyaccount for these challenges simultaneously.
In Section 4, we verify empirically that neglecting anyof them results in poor performance.
(Additionally, since we are motivated from the perspective ofdecision support as an intermediary between humans, models, and experts, note that—as anotherpractical component— ODM also extends Z with the option to accept human decisions
˜ Y ⇠ ˜ ⇡ ( ·| X ) ).
Mediator Policies
In light of the preceding discussion, it is clear a good mediator should satisfy the following criteria.
The ﬁrst deals with immediate loss, the second with future loss, and the third with trading off the two:
• It should accept or intervene only when errors are thereby unlikely (cf. learning with rejection).
• It should acquire ground truths when uncertainty may thereby be reduced (cf.
active learning).
• It should balance such exploration and exploitation adaptively over time (cf.
contextual bandits).
Greedy Mediator
It is instructive to ﬁrst examine the greedy policy.
Given any model policy ˆ ⇡ , thegreedy mediator policy  ⇤ simply chooses Z to minimize the immediate (i.e. one-step) system risk,which balances immediate probabilities of error with immediate costs of intervention/requisition:
[Equation]
(5) where  ( Z  z ) denotes the Dirac delta centered at z , and ˆ Y :=
argmax y ˆ ⇡ ( y | X ) .
It is clear that such amediator policy is optimal in terms of regret if the model policy were already perfect (i.e. if ˆ ⇡ = ⇡ ⇤ ), orif the model policy were otherwise ﬁxed (e.g. if Z =2 no longer provided any feedback for learning).
Passive Exploration But what if the model is not perfect, and must incorporate new data points forlearning?
Actually, the greedy policy already “inadvertently” performs a sort of passive exploration:
Whenever the target probabilities [ˆ ⇡ ( y =1 | X ) , ..., ˆ ⇡ ( y = m | X )] for a context X are not sufﬁcientlyconcentrated,  ⇤ would request from the expert, which may reduce uncertainty for points similar to X .
However,  ⇤ may learn too slowly: If—at any point—the model is even slightly erroneously conﬁdent(e.g.
ˆ ⇡ ( y 0 | X )  1  k req + " for any y 0
= y , for any " > 0 ), then it would simply not query the expert,and may commit similar mistakes again later.
This is because it fails to distinguish between aleatoricand epistemic uncertainty:
Not only do we wish to defer (viz.
abstain) when the former is high at X ,but we also wish to defer (viz.
learn) when the latter may be reduced by knowing the ground truth at X .
So the question is: Can we explore in a manner that better balances these (present vs. future) demands?
3.1 Bandit Mediator Policies
Prima facie, this resembles a bandit tradeoff, so the immediate question becomes: Can we simplyformulate this as a speciﬁc instance of contextual bandits?
Consider an ODM problem with m actionsin Y :
This gives m + 2 “arms” in total (i.e. an accept , a request , and an intervene arm for each of them underlying actions).
However, precisely due to the nature of abstentive feedback, the answer is no :
Loss vs. Feedback
In conventional bandit problems, there is no distinction between the notions “loss”and “feedback”.
In each round, when an arm y t is pulled in response to a context x t , the negativereward   ( x t , y t ) serves dual purposes: It is always incurred into the performance measure (viz. regret), and it is always observed as a new data point for learning (viz.
reinforcement).
In ODM , however,“loss” and “feedback” are distinct quantities:
In each round, when a mediator action z t is chosen inresponse to a context-action pair ( x t , ˜ y t ) , the resulting loss is always incurred (viz.
Deﬁnition 2), butno information whatsoever (i.e. not even the loss) is observed as feedback for learning unless z t =2 .
Exploring without Learning It should now be apparent why bandit algorithms may suffer in ODM .
The crux of the issue here is that only one arm can actually provide any new information (i.e. therequest arm).
So the role of “exploration” here is very different: Bandit strategies would “explore”different arms pointlessly with no learning occurring most of the time.
It is easy to see that this appliesto all manner of algorithms such as ✏ -greedy policies, posterior sampling, and strategies that rely onoptimism in the face of uncertainty (the latter actually leading to strictly fewer request arms beingpulled!).
In Appendix C, we formally show that ODM is actually a distinct and concretely “harder”problem than contextual bandits (Theorem 4)—precisely due to the nature of abstentive feedback.
3.2 UMPIRE Mediator Policy
Given the previous discussion, a simple “ ✏ -request” mediator policy may seem promising: It executesthe greedy policy  , but with probability ✏ opts to request from the expert.
Learning thus occurs morefrequently, and there are no exploratory actions that yield no learning.
But an important problem is thatnot all exploratory actions are equally useful :
The value of any requested information surely dependson the context x t ; by randomizing “indiscriminately”, an ✏ -request strategy does not account for this.
We propose a more principled basis for interpolating between exploration and exploitation that we termuncertainty-modulated policy for intervention and requisition (“ UMPIRE ”).
The main idea is that wemight be willing to pay more to request an oracle action now, if it means we can more conﬁdently relyon model predictions in the future (i.e. by accepting or intervening autonomously).
So our motivationis to explicitly trade off (immediate) system risk with expected improvements in the (future) model risk .
To do so, we need a method that allows us to estimate the latter.
Operating in the probabilistic setting,let W denote the parameter variable, taking on values in w 2 W , such that we can write ˆ ⇡ w ( Y | X ):= 229 p
( Y | X, w ) , and can also speak of the marginal p ( Y | D, X )=
E W ⇠ p ( ·| D ) p
( Y | X, W ) .
Now, denote 230 the expected model risk with ¯
R ( D ) :=
E W ⇠ p ( ·| D ) R
(ˆ ⇡ W ) ; note that this is itself a random variable 231 due to its dependence on D .
Consider the t -th round of play: If the mediator chooses z 2 { 0 , 1 } , then 232 d t = d t  1 so we have ¯ R ( d t )
=
¯ R ( d t  1 ) .
But in deciding whether to choose z =2 , we wish to capture ameasure of how much this risk might possibly improve—that is, if we were to reveal (and learn from)the ground-truth label Y t .
So we are interested in how far ¯ R ( D t ) can end up relative to ¯ R ( d t  1 ) ,where Y t ⇠ p ( ·
| d t  1 , x t ) .
The following result gives such an upper bound on expected improvement:
Theorem 1 (Expected Improvement) Let R be bounded as [  b, b ] —for instance, by centering ` 01 .
Let I [ W ; Y t | d t  1 , x t ] denote the mutual information between W and Y t conditioned on d t  1 and x t , 238 and let W 0 denote the principal branch of the product logarithm function.
Then (proof in Appendix C):
[Equation]
This motivates a straightforward technique: Deﬁne g : v 7!
g ( v ) =
2 b ( e W 0 ( 1 e ( v  1))+1  1) , and let denote some tradeoff coefﬁcient.
Then we can designate ¯ k req :
=(1   g
( I [ W ; Y t | d t  1 , x t ]))
k req , andsimply use ¯ k req in place of k req wherever it appears—but keeping the greedy mediator otherwise intact.
UMPIRE thus has one hyperparameter  ; in our experiments we simply set its value as the normalizingconstant  0 :=(2 b ( e W 0 ((log m  1) /e )
+1  1))
1 , which has the effect of keeping all costs non-negative.
Interpretation Since g is monotonically increasing, Theorem 1 is naturally interpreted as translatingan information-theoretic criterion (i.e. the mutual information) into a decision-theoretic criterion (i.e.the expected improvement in posterior risk)—which is what we require.
In particular, the argumentto g expands as I [ W ; Y t | d t  1 , x t ]= H
[ W | d t  1 ]  E Y t ⇠ p ( ·
| d t  1 ,x t ) H
[ W | d t  1 , x t , Y t ] , which has the 248 interpretation of how much the (epistemic) uncertainty in the model policy is expected to decrease if
Y t ⇠ p ( ·
| d t  1 , x t ) is revealed; this view is reminiscent of entropy-based approaches to active learning 250 [59,60].
Observe that when deployed, G t :=
g ( I [ W ; Y t | D t  1 , X t ]) is large in the beginning, so ¯ k reqis small and UMPIRE behaves like standard incremental learning.
In the limit of a perfect model, G tgoes to zero, so ¯ k req =
k req and UMPIRE behaves the same as the (optimal) greedy mediator ( ⇡ ⇤ ,  ⇤ ) .
3.3 Practical Implementation
Some practical remarks deserve mention.
First, UMPIRE is compatible with any choice of probabilisticmodeling technique, such as Gaussian processes, Bayesian neural networks, and dropout-based approximations.
Second, since integration over parameter posteriors is generally intractable, we use standard Monte-Carlo sampling to compute expectations: Let s denote the number of samples taken fromthe posterior, and let { w i,t } si =1 indicate the set of samples drawn from p ( W | d t ) .
In computing the 259 value of g t , observe that the inner expression E Y t ⇠ p ( ·
| d t  1 ,x t ) H
[ W | d t  1 , x t , Y t ] requires retraining 260 the model policy on every possible value of Y t .
Instead, we can rely on the symmetry of mutual information and expand I
[ Y t ; W | d t  1 , x t ]= H [ Y t | d t  1 , x t ]  E W ⇠ p ( ·| d t  1 ) H
[ Y t | x t , W ] , so we can write:
[Equation]
where we deﬁne H [ p ( Y )]
: =
P y 2 Y p ( y ) log p ( y ) , giving us a more efﬁcient way to compute the expression without such retraining (see Appendix C for detail).
Lastly, to guarantee consistency we caneasily still request from the expert with some small probability ✏ (i.e. in the same way the “ ✏ -request”policy above does so over the greedy policy).
Algorithm 1 summarizes UMPIRE as applied to ODM .
[Table]
Empirical Results
Three aspects of UMPIRE deserve investigation: (a) Performance :
Does it work?
Section 4.1 comparesit to existing methods, validating its role in decision support by most consistently improving decisions.(b) Source of Gain : Why does it work?
Section 4.2 deconstructs the key characteristics of UMPIRE ,verifying the importance of each.
(c) Sensitivity Analysis : Finally, Section 4.3 assesses the sensitivityof UMPIRE and benchmarks to the expert’s stochasticity, costs of request, and number of samples used.
Datasets We experiment with six environments.
In GaussSine , synthetic points are generated inthree categories by rounding a sinusoidal latent function on 2D Gaussian input [61].
In HighEnergy ,the task is to identify signals in high energy particles registered in a Cherenkov gamma telescope [62].
In MotionCapture , the task is to recognize hand postures from data recorded by glove markers onusers [63].
In LunarLander , the task is to perform actions in the OpenAI gym [64] Atari environment,with the expert deﬁned as a PPO2 agent [65,66] trained on the true reward.
In Alzheimers , the task isto perform early diagnosis of patients in the Alzheimer’s Disease Neuroimaging Initiative study [67]as cognitively normal, mildly impaired, or at risk of dementia [19,20].
Lastly, in CysticFibrosis ,the task is to perform diagnosis of patients enrolled in the UK Cystic Fibrosis registry [68] as to their
GOLD grading in chronic obstructive pulmonary disease [69].
See Appendix B for additional detail.
Benchmarks We consider adaptations of algorithms from related work.
First as our minimal baseline,
Human always accepts z =0 , thus constituting the starting point for performance comparison.
Randomdraws z at random.
Supervised picks z 2 { 0 , 1 } based solely on ˆ ⇡ ’s output, and z =2 w.p. ✏ , andthus resembles supervised learning.
Cost-Sensitive is the greedy (ˆ ⇡ ,  ⇤ ) from Section 3, whichadditionally accounts for costs k int , k req .
Thompson Sampling [46] draws from the posterior p ( W | d )and selects z greedily using (ˆ ⇡ W ,  ⇤ ) ; the Full version also uses that sampled model when predicting.
Epsilon-Greedy [47] is greedy but draws z at random w.p. ✏ ; the Request version is the smarter “ ✏ -request” from Section 3.2.
Pessimistic Bayesian Sampling adapts OBS [44] to ODM by reversing the direction of optimism such that the tendency to request actually increases .
Bayesian Active
Request adapts Bayesian active learning [60] to ODM by requesting w.p. ⇠ expected reduction in entropy.
To further highlight the advantage of our proposed criterion, Matched Decaying Request isan artiﬁcially boosted benchmark that is similar to ✏ -request—but where ✏ is a decay function that hasthe beneﬁt of matching the effective request rate of UMPIRE :
This is done post-hoc by searching for apolynomial function that best models UMPIRE ’s request pattern.
See Appendix B for additional detail.
Experiment Setup Each experiment run consists of n =2000 rounds of interactions (except for thesynthetic GaussSine , for which n =500 ), and this is repeated for a total of 10 runs with random seeds.
For all algorithms, the underlying model policy is implemented identically using Dirichlet-based
Gaussian process classiﬁers [61,70–72].
We simulate fallible human decisions as random perturbations of the ground truth with some probability ↵ .
As mentioned in Section 2.1, we let k req = m m  1  for some small  > 0 :
To do so, we simply set k req to mm  1 rounded down to the nearest decimal point.
However, we shall perform additional sensitivities on this below.
In all experiments, we set k int =0 .
1 ,↵ = 12 , ✏ = 10% where applicable, and  =  0 as noted in Section 3.2.
Regret is deﬁned with respect tothe oracle mediator ( ⇡ ⇤ ,  ⇤ ) , for which ⇡
⇤ is approximated by training on the full dataset in advance.
Performance metrics for each benchmark are reported as means and standard deviations across all runs.
[Figure]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Recall from Section 1 we motivated ODM from the view of decision support : To best improvedecision-making by the (human-expert-mediator) system as a whole.
Primarily, then, we evaluate theperformance of the decision system —that is, in terms of system regret (Equation 4).
Figure 1 showsresults for all algorithms, with Human (i.e. without support) also shown for comparison: UMPIRE consistently accumulates lower regret.
This is our objective function, and this is the main takeaway.
Secondarily, we can also assess the (heldout) performance of the model policy —that is, if it were taskedwith acting autonomously at any point.
Figure 2 shows the rate of heldout mistakes: UMPIRE appearsto induce more efﬁcient learning.
As another auxiliary, we can also consider how the mediator policybehaves—that is, if it accepts erroneously ( z =
but ˜ y 6 = y ), intervenes excessively ( z =1 but oraclez ⇤ 2 { 0 , 2 } ), or abstains not conservatively enough ( z 2 { 0 , 1 } while ˜ y
=
y
and ˆ y 6 = y ).
Table 2 shows 317 the results, likewise with UMPIRE as best.
(Note that this is not simply due to more frequent requests,since Matched Decaying Request selects z =2 just as often as UMPIRE using a dynamic ✏ t tunedpost-hoc).
For more comprehensive measures of the system (loss, regret, mistakes) and model (heldoutcross entropy, mistakes, AUROC , and AUPRC ), see Figures 8, 9, 10, 11, 12, 13, and 14 in Appendix A.
4.2 Source of Gain
[Figure]
Recall from Section 2 that ODM combines the challenges from allthree related settings, and from Section 3 that UMPIRE is designedwith three corresponding desiderata in mind.
We now examine eachaspect’s contribution to ﬁnal performance.
Table 3 enumerates somesettings to isolate the following: (i) Does it act exploit ˆ ⇡ in a costsensitive manner (“ CS ”), like in learning with rejection?
(ii) Doesit attempt to explore deliberately in addition (“ DE ”), like in banditalgorithms?
(iii) Does it leverage uncertainty awareness in decision-making (“ UA ”), similar to activelearning?
(Abbreviations: Dynamic-Request w.p.KG requests w.p.  G t , and Dynamic-Requestwith ME requests with matched ✏ t , i.e. same as Matched Decaying Request from above).
Figure 3shows results for the decision system in terms of system regret, and secondarily Figure 4 shows resultsfor the learned model in terms of heldout mistakes:
It is apparent that all three aspects are crucial forperformance, and cannot be neglected.
For more comprehensive source-of-gain evaluation metrics forthe system, model, and mediator, see Figures 15, 16, 17, 18, 19, 20, and 21, and Table 5 in Appendix A.
4.3 Sensitivity Analysis
Lastly, we examine UMPIRE ’s sensitivity to several factors.
First, the expert might be more or lessstochastic depending on the environment.
We simulate this by progressively injecting additional noiseto the latent function in GaussSine .
Figure 5 shows the results: The advantage of UMPIRE is mostpronounced at lower noise levels, and decreases as noise levels rise; this makes sense, as our estimates
[Figure] [Figure]
[Figure] [Figure]
[Figure]
of uncertainty become more entangled with noise.
For more comprehensive results, see Figures 22, 23,24, and 25 in Appendix A. Second, we examine the consequence of different costs of request.
Recallthat we have been operating in the regime
where
k req =
mm  1   for some small  > 0 [22,25,37,40].
We now allow the cost of request to vary from k req
=
to mm  1 as sensitivities.
(For completeness, weactually go slightly beyond the standard threshold and go as high as  =  0 .
05 < 0 ).
Figure 6 showsthe results: The advantage of UMPIRE appears most pronounced at higher costs, and decreases in theopposite direction; this makes sense, as cheaper costs mean abstention becomes a more trivial decision.
For more comprehensive results, see Figures 26, 27, 28, and 29 in Appendix A. Finally, recall that
Equation 7 requires estimating the mutual information by Monte-Carlo samples.
Figure 7 showsthe sensitivity of performance to the practical choice of sample size s : Observe that performanceappears to stabilize with reasonable choices of s ⇡ 64 and above.
See Appendix A for additional detail.
Conclusion
Research in machine learning for decision-making is proliferating, such as in data-driven clinical decision support—but much focus is exclusively placed on comparing computers versus clinicians [73]:
Less explored is how machines can serve as adjuncts to make decision systems more efﬁcient as a unit .
In this work, we take a ﬁrst step by formalizing ODM as a sequential problem, proposing UMPIRE as apotential solution, and demonstrating the importance of considering all aspects of this unique setting.
While this perspective is general, the ethical responsibility for a decision—e.g. signing off on a diagnosis—often must ultimately fall on humans: To remain vigilant of potential bias in societal impact, it isthus crucial to examine the complementary problem of “closing the loop” by considering how humansthemselves may in turn interpret feedback to modify their own behavior [74].
Moreover, future workmay explore generalizing ODM and its solutions—such as to settings with differential costs of mistakeor class imbalances, or to consider aspects of interpretability in model policies for human feedback.