Uncalibrated Models Can Improve Human-AI Collaboration
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In many practical applications of AI, an AI model is used as a decision aid forhuman users.
The AI provides advice that a human (sometimes) incorporatesinto their decision-making process.
The AI advice is often presented with somemeasure of “conﬁdence” that the human can use to calibrate how much theydepend on or trust the advice.
In this paper, we present an initial exploration thatsuggests showing AI models as more conﬁdent than they actually are, even whenthe original AI is well-calibrated, can improve human-AI performance (measuredas the accuracy and conﬁdence of the human’s ﬁnal prediction after seeing the AIadvice).
We ﬁrst train a model to predict human incorporation of AI advice usingdata from thousands of human interactions.
This enables us to explicitly estimatehow to transform the AI’s prediction conﬁdence, making the AI un calibrated, inorder to improve the ﬁnal human prediction.
We empirically validate our resultsacross four different tasks—dealing with images, text and tabular data—involvinghundreds of human participants.
We further support our ﬁndings with simulationanalysis.
Our ﬁndings suggest the importance of jointly optimizing the human-AIsystem as opposed to the standard paradigm of optimizing the AI model alone.
Introduction
In safety-critical settings like medicine, AI is often integrated as interactive feedback, allowing ahuman to decide when and to what extent the AI’s “advice” is utilized [29].
For example, in medicaldiagnosis tasks, this places the AI in a similar category as lab tests or other exams a doctor may orderto aid in diagnosis.
This mitigates the often black box nature of AI algorithms that limit a user’s trustand usage of AI [6, 24, 34, 18].
Typically, these algorithms are designed and optimized to minimize loss using standard trainingobjectives independent of the human end user.
In this paper, we question this premise and askwhether a joint optimization of the entire human-AI system is possible.
In particular, we revisitthe conventional wisdom that models with calibrated conﬁdence enable more accurate transfer ofprediction uncertainty between models and/or humans [8].
We propose calibrating a model for humanuse, with the intuition that humans are often unable to accurately integrate calibration informationcorrectly and so may beneﬁt from uncalibrated information.
This approach is complementary toexisting ideas on how to help humans utilize AI advice optimally (e.g., through some form ofexplanation accompanying the advice or through feedback on model/human performance)
We draw on prior work modeling human predictions [30] and assume black box access to an AIalgorithm which provides advice.
From data we collected on thousands of human interactions, we ﬁta model for how humans incorporate (or ignore) AI advice.
Using this learned human behavior model,we optimize a monotonic transformation that modiﬁes the AI’s reported conﬁdence to maximize human-AI system performance (measured as the human’s ﬁnal accuracy)
We demonstrate ﬁrst in simulations and subsequently with real-world data using crowd-sourced human study participantsthat our modiﬁed AI advice results in higher overall performance.
These results suggest there is meritto jointly optimizing human-AI systems and that an optimal AI may not be optimal for human use.
Our contributions We propose and empirically validate a framework for optimizing an AI algorithmwith respect to the human end user, resulting in human-calibrated AI that calibrates AI models forhuman use.
We only require black box access to the AI.
Our framework relies on a model for humanbehavior, which we create using a large collection of empirical data on human interaction with AIadvice.
To the best of our knowledge, these are some of the ﬁrst results using an empirically validatedmodel for human behavior to optimize AI performance in a collaborative human-AI system. Wewould like to emphasize, however, that this these methods are not currently suitable for practical use.
Ethical considerations
One of the main conclusions of our study is that human users do notconsider conﬁdence in the “optimal” way (i.e., how we model conﬁdence in statistics).
In an attemptto correct for this, we present both empirical and simulation-based evidence that modifying thepresented AI advice improves overall performance.
One potential question about our approach is:does modifying the AI’s conﬁdence constitute misleading the user?
We agree – in our experiments,the AI is misleading the user. However, our goal here is not to propose a method that should be usedin practical applications, and rather to highlight the importance of modeling human interaction whendesigning AI for human use.
Related works The experimental setup we use comes from the advice utilization community.
Here,studies often use the JAS framework for measuring and comparing the effect of different forms ofadvice [28].
In many of these studies, algorithmic advice is manipulated to elicit various responsesfrom humans.
We take this a step further in our paper by manipulating the advice to improve modelcalibration with respect to the user and improve performance.
Previous work has studied the effectof peer vs. expert advice [15], automated vs. human advice [22, 5, 15, 21], task difﬁculty [7], andadvice conﬁdence [25, 26].
Some work has also sought to characterize the perceptions people have ofan AI when making decisions with AI advice and how these perceptions effect trust and so utilizationof the AI [13].
Of particular relevance to our work is [25], which provides evidence that uncalibrated,conﬁdent advice is more utilized by humans in certain settings.
We also leverage a proposed modelfor human behavior from [30].
The conventional approach to human-AI collaboration is to make the AI’s predictions explainable[31, 27] and to ensure model and human calibration [8, 3].
These explanations affect humanunderstanding of AI conﬁdence [12, 35, 36, 14].
It is standard practice to optimize the AI in isolation
However, recent work suggests that the objectives used to optimize the AI do not adequately considerthe human-AI team.
Alternative approaches including learning to defer, where a classiﬁer determineswhen expert input is required [16, 19]; learning to complement, where the AI is optimized to performwell on only the tasks that humans struggle with [33]; and methods that optimize objectives withcosts assigned to requesting AI advice and/or human expert input utilization [1]. Our work is mostrelated to [1], which also emphasizes the importance of conﬁdent AI predictions for human decisions.
Our approach is complementary to these prior works and contains several novel aspects.
We modelhuman behavior using empirical data, demonstrate that this human model allows us to optimize the AI advice, and demonstrate both in simulation and empirically that our modiﬁed advice improvesoverall system performance.
Furthermore, we only require black-box access to the AI advice.
Inrelation to [1], we also differentiate our work as the model for human interaction is different (weshow advice only after the user makes an initial assessment), we only need blackbox access to the
AI, and our perspective is different – we attempt to correct for human biases by adapting the outputconﬁdence, while in [1], the authors sacriﬁce accuracy to increase utility of the model for humans.
Experimental setup
Our goal is to augment the performance of a human-AI system by optimizing the AI for use by ahuman.
To make this goal tractable, we assume (1) that we have a model for human interaction with
AI advice, and (2) that we have black box access to the AI algorithm producing the advice.
Human-AI interaction
The human-AI interaction involves two stages.
First, a human is shown a task and completes it.
AI advice is then presented, and the human is allowed to modify their response.
As we measure performance before and after receiving advice, this model is conducive to studying the effect ofadvice and so sees common usage in the literature [22, 30, 17].
We collected data from crowd-sourced participants on a diverse set of tasks that all have the samebasic design.
We focused on binary prediction tasks where humans respond on a continuous scale.
The use of the continuous scale allows for a more rich analysis of the user’s prediction than just thesimple binary prediction label would permit.
Participants were shown a sliding scale with each end indicating a different binary response.
Theywere instructed to move a pointer on the scale to the location that indicated their conﬁdence and theirresponse, with the midpoint on the scale indicating an uncertain response.
Participants submittedan initial response; subsequently, the advice was indicated with a visual marker on the scale andparticipants were allowed to adjust their initial response.
Participants were incentivized so thathigher conﬁdence in correct and incorrect answers increased and decreased their monetary rewardrespectively.
We collected data on a set of four tasks from diverse data modalities.
These tasks were originallyproposed in prior work [30].
The tasks were designed to be accessible to the general public toallow use of crowd-sourced study participants for obtaining empirical data.
See below for a briefdescription of the tasks we used; examples of task instances from each of the four tasks are shown inthe Appendix.
The data collected is summarized in Table 1.
Each task consists of 32 binary questions.
For each task, the 32 questions were carefully selected to have a range of difﬁculty.
More details onthe distribution of task difﬁcult is presented in the Appendix.
All participants received the same set of questions, though the order was randomized for eachparticipant.
All participants also received the same advice, although a small amount of noise wasrandomly added for each participant.
The advice had an accuracy of ⇠ 80% for each task; participantswere informed of this accuracy.
However, no feedback on AI or human accuracy on individualquestions was provided during the experiment.
Since the datasets we are using are relatively small and curated primarily for human use rather than
AI training, we opted to use aggregated human responses as a proxy for an AI model.
In particular,the advice was constructed by averaging the responses of a prior group of human participants whoresponded to the questions without receiving any advice.
This allowed us to ensure the advicegiven was “reasonable” in the sense that the advice is generally less conﬁdent on difﬁcult tasks andmore conﬁdent on easy tasks.
We checked the calibration of this advice by computing the expectedcalibration error (ECE) [20].
The ECE for AI advice is 0 .
074 , indicating the advice is roughlycalibrated ( 0 indicates perfect calibration).
We recruited 49-79 participants for each task, resulting in several thousand datapoints (each questionis one datapoint).
Participants are US residents, roughly 50% are female, and the average age isapproximately 30 years old.
Despite limitations in the age distribution and geographic location ofparticipants, a survey question we asked indicated a varied range of AI familiarity in our participants.
We followed standard practice in collecting data.
Our study protocol was approved by our institution’s
IRB board.
The data collection process was low risk, and we ensured samples contained diverseparticipants across age, sex, and ethnicity.
We compensated participants at  $10 .
per hour(depending on bonus pay).
The total cost was around $600 (Section 2) + $750 (Section 5.2).
We have four types of data with corresponding user tasks.
Art dataset (Image data) Containsimages of paintings from 4 art periods: Renaissance, Baroque, Romanticism, and Modern Art.
Participants were asked to determine the art period a painting is from given a binary choice.
Citiesdataset (Image data)
This dataset contains images from 4 major US cities: San Francisco, Los
Angeles, Chicago, and New York City.
The task is to identify which city an image is from given abinary choice.
Sarcasm dataset (Text data)
This dataset is a subset of the Reddit sarcasm dataset[10], which includes text snippets from the discussion forum website, Reddit.
Participants were askedto detect whether sarcasm was present in a given text snippet.
Census dataset (Tabular data) Thisdataset comes from US census data [32].
The task is to identify an individual’s income level givensome of their demographic information—e.g. state of residence, age, education level.
[Table]
Human behavior model
Table 1 provides a summary of the data collected for developing our human behavior model.
We ﬁndthat humans actually changed their response after receiving advice roughly 50% of the time, with theexact number shown in the “Activation Rate” column of Table 1.
When a person does change theirresponse, we call this person “activated.”
Speciﬁcally, a person is activated if their ﬁnal responsechanges by at least a small amount.
This observation motivates a simple, two-stage model for human behavior: a human ﬁrst decideswhether to modify their response (“activation stage”), and subsequently, if they were activated,they modify their response (“integration stage”).
A diagram of this model is shown in Figure 1.
This two-stage model is consistent with previously proposed models from the psychology and HCIliterature [9, 30].
We ﬁt functions for each of the two stages: an activation model that predicts how likely it is thata person is activated, and an integration model that predicts how the person modiﬁes their advice,assuming they are activated.
Taken together, we can model human behavior in the two-stage humanAI system we use to run our experiments.
We deﬁne this model’s behavior more rigorously in
Section 4.
Our ﬁtted models will only be used to optimize our advice-modifying function, and willnot be used a test time.
As such, we can utilize demographic information about the participants to ﬁtthe model, though such information may not be available during a real-world deployment.
We optto ﬁt a small neural network for each of these stages, as we require a differentiable model (for easyoptimization) with moderately more complexity than a linear model.
3.1 Activation model
Our activation model takes 12 input features extracted from a single person-task instance interaction.
It outputs a probability for whether the person will be activated on the given task instance.
The inputfeatures are based on the person’s initial response, the AI advice, and several demographic featuresincluding age and sex.
We do not encode the initial response or AI advice directly, as the label (theleft or right slider location of the correct answer) is randomized for each participant.
Instead, we use
[Figure]
the response and advice conﬁdence, as well as a binary feature that encodes whether the responseand advice agreed on the label.
This implies a symmetry in the model’s predictions.
Please see the
Appendix for a description of all the features used.
We use a 3 layer neural network with ReLU non-linearities for the activation model.
We also attemptedto use other simple algorithms for modeling the activation behavior of humans, like linear models, butfound the neural networks to perform better.
We trained the network using binary cross-entropy losswith an Adam optimizer [11] and stopped training when validation loss increased, as is standard.
Theactivation achieves an AUC > 0 .
81 in predicting activation on held-out test participants, suggestingreasonable performance despite the high variability in our dataset.
In Figure 2a, we show a partial dependence plot for the learned activation model.
This plot shows theaverage behavior (across our entire test dataset) of the activation model when we ﬁx the AI advice tobe a certain value (the x-value) for all data points.
The AI advice can be positive (correct advice)or negative (incorrect advice), while the magnitude corresponds to the advice conﬁdence.
We notethat (1) our activation model predicts people will be most activated when the advice is conﬁdentlyincorrect.
This occurs because (a) most people in our dataset get answers correct, on average, andseeing conﬁdent advice of the opposite label tends to make people change their initial response (e.g.by decreasing conﬁdence even if the predicted label does not change).
And because (b) conﬁdentadvice that agrees with a person’s own initial response tends to increase their conﬁdence.
We alsoobserve that (2) the activation rate decreases when advice is less conﬁdent, but increases for conﬁdentand correct advice due to the same reasons noted previously for conﬁdent and incorrect advice.
Activation rate is lower for correct advice than for incorrect advice as most people in our dataset are,on average correct, which tends to invoke a smaller response.
3.2 Integration model
Our integration model is similar to our activation model, but optimized for a different function.
Inparticular, we use the same input features, model architecture, and training procedure.
The integrationmodel outputs a prediction for sign ( r 1 )( r 2  r 1 ) , where r 1 , r 2 {+
+} [  1 , 1] are the initial and ﬁnalresponses respectively.
Note that r 1 , r 2 are probabilities normalized to the [  1 , 1] scale.
The trainingminimizes the RMSE of the predicted change and the actual change.
The optimization is performedon only the subset of the dataset that was actually activated (roughly 50% of the data).
Our integrationmodel has a reasonable performance, with an RMSE of 0 .
25 (the maximum absolute error possible is2 ) and R 2 of 0 .
73 on test data.
In Figure 2b, we plot a heatmap of the ﬁtted integration model’s behavior.
We plot the average outputof our integration model across our test data split when the person’s initial response (x-axis) and
AI advice (y-axis) are ﬁxed to a certain value for all datapoints.
The sign and magnitude of theinitial response and AI advice correspond to correctness (negative is incorrect) and conﬁdence.
Theintegration model prediction can also be positive (increasing conﬁdence in initial predicted label) ornegative (decreasing conﬁdence in initial predicted label / change in predicted label).
We observe three features in the heatmap: (1) The heat map is symmetric across quadrants 1, 3 and2,4.
This is by design.
As the model does not use the initial response and AI advice values directly –it uses their magnitudes (conﬁdence) as well as a third, binary term indicating whether they agree onlabel – the integration model can only predict a delta change relative to the sign of the initial response.(2)
The output has largest magnitude when the advice is conﬁdent and opposite the person’s initialresponse.
If the advice and the person’s initial response share the same label, the integrationmodel output is largest when the person’s initial response has low conﬁdence and the model has highconﬁdence.
These observations conﬁrm our integration model has reasonable operation behavior.
Modifying the AI advice
[Figure]
Here we discuss how we modify the raw AI advice output.
First let’s establish some notation.
Thenotation described below is in reference to a single datapoint; later, we will add the subscript i toindex across datapoints.
Let A 2 R denote the inverse sigmoid of the advice presented to participants (i.e., we present the
AI advice as  ( A ) ).
We will work with A rather than  ( A ) here, as we will aim to replace the transform with an optimized function that improves human-AI performance.
Let g : R ! [0 , 1] be a function (which we will optimize) that maps the AI advice to a probabilitythat the user is shown.
Note that for the baseline, “unmodiﬁed advice,” we set g to be the sigmoidfunction,  ( x ) = 1 1+ e  x .
Let r 1 , r 2 be the initial and ﬁnal responses from a participant ( r 2 is a function of r 1 , A , and otherfeatures), and let y 2 { 0 , 1 } denote the true label.
Let u represent demographic features of a person,and let x = ( r 1 , A, u ) be the input feature vector to our f activation and f integration , our activation andintegration models for human behavior models described in Section 3.
To simplify notation, we letf integration denote the function that outputs the predicted r 2 for an activated person (rather than the deltafrom r 1 that we optimize for).
Finally, let f HB represent the entire model for human behavior.
Weassume all feature preprocessing is hidden inside f activation and f integration for convenience of notation.
4.1 Optimizing for human-AI performance
We consider g of the form
[Equation]
(1) where ↵ ,  2 R  0 .
Note that setting ( ↵ ,  ) =
(1 , 0) results in g 1 , {+
+} =  , the sigmoid function usedto produce the unmodiﬁed advice.
Note that we do not exactly perform a linear transformation on A : the | A | and sign ( A ) terms are included to ensure that g does not change the label recommended as itmakes the function symmetric around A = 0 .
↵ modulates the rate at which the presented adviceincreases with AI conﬁdence, while  adjusts the minimum conﬁdence level of the presented advice.
For example, if  = 1 , the presented conﬁdence is > e 1+ e > 0 .
73 (relative to the chosen label).
Ingeneral, we can use other differentiable functions for g ; we chose the form in Equation 1 because it iseasy to optimize, simple to understand, and is still ﬂexible enough to ﬁt the data.
As a shorthand for when we modify the features input to f activate and f integrate , we denote g ( x ) =( g
( A ) , r 1 , u ) .
We will optimize for ↵ ,  to maximize the expected ﬁnal accuracy using our humanbehavior model, f HB .
As discussed in Section 3, f HB has the following representation:
[Equation]
We then solve the following optimization problem to obtain ↵ ,  :
[Equation]
where L is the binary cross entropy loss function.
Note that the expectation is taken with respectto the randomness in f HB and the data distribution.
So given training dataset D , we optimize thefollowing expression:
[Equation]
In practice, we carry out this optimization using stochastic gradient descent, which is possible aswe chose f activate and f integrate to be differentiable functions.
Once we have optimized for ↵ ⇤ ,  ⇤ , wecan apply it to new data without needing the user demographics u (i.e., we do not personalize theconﬁdence transformation to individual users).
Note that the choice of loss function is arbitrary here(we used logistic loss as our task is binary classiﬁcation).
In particular, for high risk settings where amistake is very costly, the ↵ ⇤ ,  ⇤ would be selected accordingly.
4.2 When does under/overconﬁdent advice help?
Here we describe simple theoretical situations where uncalibrated advice improves performance.
We consider a binary classiﬁcation task and assume we want to minimize the logistic loss of thehuman-AI classiﬁer.
We also assume (1) a constrained version of the activation-integration modelperfectly models human behavior; in particular, humans either use their initial response or follow thegiven advice exactly.
And (2) we have an oracle for human behavior and know exactly the calibrationof the human and AI advice.
We will now construct a function f ( a, r 1 ) that outputs a ⇤ , the optimaladvice to minimize logistic loss of the human-AI system.
To select a ⇤ , we need to consider two quantities:
[Equation]
[Equation]
Here, p a and p r 1 are the probabilities the advice and initial response are correct respectively.
Then L r 1relates the log-loss for non-activated humans and L a ⇤ relates the log-loss of following the modiﬁedadvice, a ⇤ .
Now we simply select a ⇤ to minimize the expected loss incurred by the human-AI system
[Equation]
(5) for p activation = f activation ( a ⇤ , r 1 , u ) (which we assumed we know exactly).
For simple settings– p activation = [ ¯ r 1 + ✏ < ¯ a ] (here ¯ x = max { x, 1  x } ), with ✏ 2 [  0 . 5 , 0 .
5] (i.e., humanunder/overvalues their response relative to advice) and p r 1 = f ( r 1 ) for a generic functionf : [0 , 1] ! [0 , 1] (i.e., human is not calibrated) – it is indeed optimal for the advice a ⇤ to beunder/overconﬁdent to correct for human misutilization of advice (depending on the advice and initialresponse).
An example where ✏ = 0 .
1 and f ( r 1 )
=
r 21 is shown in Figure 3b.
Further discussion isincluded in the Appendix.
[Table]
Results
[Table]
5.1 Simulation results
Optimizing Equation 2 results in an optimal ↵ ⇤ ,  ⇤ .
Using f HB , we can then analyze the expectedbeneﬁt of g ↵ ⇤ ,  ⇤ over g 1 , 0 .
In Figure 3a, we show g ↵ ⇤ ,  ⇤ .
We plot three separate curves.
The curve 276 labeled “Advice given” is optimized using the previously described dataset, and this is the function weuse in our empirical results.
We also synthetically modify the dataset by biasing the advice towardsthe correct label and adding noise to the advice to respectively increase and decrease the averageadvice accuracy in the dataset to 75% and 85% .
We then ﬁt a new ( ↵ ,  ) to the modiﬁed dataset andplot these curves.
Note that f HB is ﬁt to the unmodiﬁed dataset and is ﬁxed for optimizing all threeof these curves.
For all three curves, we notice there is a large discontinuity at A = 0 .
This is a result of a large  ⇤ ,and suggests that advice is not useful to humans unless it is at a certain level of conﬁdence (e.g., tomake the human activated).
That all three curves follow relatively similar paths suggests the performance of g ↵ ⇤ ,  ⇤ may be robustto small changes in advice accuracy.
This is conﬁrmed (in simulation) in Table 2, where we show thesimulated performance difference between modiﬁed and unmodiﬁed advice.
We see here that theﬁnal accuracy, conﬁdence in correct result, and activation rate all increase with the modiﬁed advice.
5.2 Human experiments
[Figure]
We verify our simulation results across the four previously described tasks (with the exceptionthat we do not add noise to the advice, so all participants see the exact same advice now).
Adult,
US-based participants were recruited through the Proliﬁc crowdworker platform [23].
We recruited50 participants for each task, and randomly assigned either the unmodiﬁed (baseline) or modiﬁedadvice to each person.
In Figure 4, we show the average change in (a) accuracy and (b) conﬁdenceof the correct label (the average conﬁdence, with correct and incorrect labels receiving positive and
[Table]
negative weight respectively), partitioning by the advice received by participants.
Accuracy andconﬁdence increase across all tasks when using advice ( p = 3 .
84 ⇥ 10  6 ). Moreover, we can largelyconﬁrm our simulation results. Accuracy and conﬁdence increased due to the modiﬁed advice acrossnearly all tasks. Additionally, activation rate (not plotted) increased with the modiﬁed advice, with an average increase of 3 . 3% across tasks.
Another interesting ﬁnding is that tasks with higher advice accuracy (Art and Cities) exhibit largerincreases in accuracy and conﬁdence.
The AI advice is more accurate in Art and Cities (accuracy:90% and 87%) compared to Sarcasm and Census (accuracy: 78% for both).
This intuitively makessense – we are making the advice more conﬁdent in its predictions, and so while this beneﬁts alltasks, it disproportionately beneﬁts tasks where the advice is more accurate.
This effect was predictedby our simulation results.
Our primary human experiments involved US-based participants.
To show our ﬁndings generalize,we took the advice modiﬁcation learned on the US data and applied it to new participants based inthe UK for the Art and Cities tasks.
The ﬁndings are highly consistent: modiﬁed advice increasesactivation and improves ﬁnal human accuracy and conﬁdence (Table 3).
Combining US and UKdata, performance improvement is signiﬁcant for both accuracy ( p = 8 . 86 ⇥ 10  3 ) and conﬁdence( p = 4 .
43 ⇥ 10  11 ).
Discussion, Limitations, and Future Work
In this paper, we proposed optimizing a simple function to better calibrate AI advice for human use ina collaborative human-AI system.
To optimize the reported conﬁdence, we trained a human behaviormodel from experimental data and used it simulate the full human-AI system.
The modiﬁed AI adviceare explicitly uncalibrated, but in both simulated and empirical results, it resulted in substantiallyhigher ﬁnal human performance.
We further validated our model using new participants from adifferent country.
These results highlight the importance of recognizing that AI, when deployed incollaborative systems, should not be optimized in isolation.
Our results also suggest that we can viewmodifying AI advice as a nudge for the human user.
If the AI says its conﬁdence is 0.7, the user maynot fully absorb the import of 0.7, since humans are not great at intuitive probability.
Modifying thereported conﬁdence could be a useful nudge to help users to better absorb the information.
This viewis supported by prior work suggesting humans prefer clean decision boundaries [2].
Our work has several limitations that would be interesting to address in future studies.
In particular, itis unclear how reliable such a method would be in practice.
We argue that our method for creatinghuman-calibrated AI can be deployed in practice, but requires an accurate model for human behavior.
In particular, if we have access to an oracle for an individual’s utilization of AI advice, we can exactlyoptimize the output of a calibrated AI model to correct for the individual’s biases.
Note that this maylead to under- or overcalibrated advice depending on the user and the advice conﬁdence.
However, aswe cannot perfectly model any given individual, the question to ask is whether we can create a robustmodel for human behavior (e.g., with conﬁdence intervals etc.) and then optimize with respect to thatmodel while accounting for error.
This extension was out-of-scope for this paper, and is an importantdirection of future research.
Other limitations include: we considered tasks where advice accuracygenerally exceeded individual human performance and did not evaluate expert tasks (e.g., involvingclinicians making clinical decisions).
These are also important directions for future research.
Finally,we recognize that the human behavior model we used in this work does not capture all real-worldsettings.
It may be important to consider different models for different applications, and our resultswill need to be validated in these new settings.