In many high-stakes domains such as criminal justice, finance, and healthcare, AI systems may recommend actions to a human expert responsible for final decisions, a context known as AI-advised decision making. When AI practitioners deploy the most accurate system in these domains, they implicitly assume that the system will function alone in the world. We argue that the most accurate AI team-mate is not necessarily the em best teammate; for example, predictable performance is worth a slight sacrifice in AI accuracy. So, we propose training AI systems in a human-centered manner and directly optimizing for team performance. We study this proposal for a specific type of human-AI team , where the human overseer chooses to  accept the AI recommendation or solve the task themselves. To optimize the team performance  we maximize the team's expected utility, expressed in terms of  quality of the final decision, cost of verifying, and individual accuracies  . Our experiments with linear and non-linear models on real-world, high-stakes datasets show that the improvements in utility while being small and varying across datasetsand parameters (such as cost of mistake), are real and consistent with our definition of team utility . We discuss the shortcoming of current optimization approaches beyond well-studied loss functions such as log-loss, and encourage future work on human-centered optimization problems motivated by human-AI collaborations .
AI practitioners typically strive to develop the most accurate systems, making an implicit assumption that the AI system will function autonomously. However, in practice, AI systems often are used to provide advice to people in domains ranging from criminal justice and finance to healthcare. In such AI-advised decision making, humans and machines form a team, where the human is responsible for making final decisions. But is the most accurate AI the best teammate? We argue "No" -- predictable performance may be worth a slight sacrifice in AI accuracy. Instead, we argue that AI systems should be trained in a human-centered manner , directly optimized for team performance. We study this proposal for a specific type of human-AI teaming , where the human overseer chooses to either accept the AI recommendation or solve the task themselves. To optimize the team performance for this setting we maximize the team's expected utility, expressed in terms of the quality of the final decision, cost of verifying, and individual accuracies of people and machines . Our experiments with linear and non-linear models on real-world, high-stakes datasets show that the most accuracy AI may not lead to highest team performance and show the benefit of modeling teamwork during training through improvements in expected team utility across datasets, considering parameters such as human skill and the cost of mistakes . We discuss the shortcoming of current optimization approaches beyond well-studied loss functions such as log-loss, and encourage future work on AI optimization problems motivated by human-AI collaboration .
R
In many high-stakes domains such as criminal justice, finance, and healthcare, AI systems may recommend actions to a human expert responsible for final decisions, a context known as AI-advised decision making. When AI practitioners deploy
AI practitioners typically strive to develop
R
system in these domains, they implicitly assume that the
systems, making an implicit assumption that the AI
R
alone in
autonomously. However, in practice, AI systems often are used to provide advice to people in domains ranging from criminal justice and finance to healthcare. In such AI-advised decision making, humans and machines form a team, where the human is responsible for making final decisions. But is
D
world. We argue that the
R
team-mate is not necessarily the em best teammate; for example, predictable performance is
the best teammate? We argue "No" -- predictable performance may be
R
So, we propose training AI systems
Instead, we argue that AI systems should be trained
R
and directly optimizing
, directly optimized
R
team
teaming
A
either
A
for this setting
A
the
A
of people and machines
R
improvements in utility while being small and varying across datasetsand parameters (such as cost of mistake), are real and consistent with our definition of team utility
most accuracy AI may not lead to highest team performance and show the benefit of modeling teamwork during training through improvements in expected team utility across datasets, considering parameters such as human skill and the cost of mistakes
R
human-centered
AI
R
collaborations
collaboration