  Machine learning models are often implemented in cohort with humans in the
pipeline, with the model having an option to defer to a domain expert in cases
where it has low confidence in its inference. Our goal is to design mechanisms
for ensuring accuracy and fairness in such prediction systems that combine
machine learning model inferences and domain expert predictions. Prior work on
"deferral systems" in classification settings has focused on the setting of a
pipeline with a single expert and aimed to accommodate the inaccuracies and
biases of this expert to simultaneously learn an inference model and a deferral
system. Our work extends this framework to settings where multiple experts are
available, with each expert having their own domain of expertise and biases. We
propose a framework that simultaneously learns a classifier and a deferral
system, with the deferral system choosing to defer to one or more human experts
in cases of input where the classifier has low confidence. We test our
framework on a synthetic dataset and a content moderation dataset with biased
synthetic experts, and show that it significantly improves the accuracy and
fairness of the final predictions, compared to the baselines. We also collect
crowdsourced labels for the content moderation task to construct a real-world
dataset for the evaluation of hybrid machine-human frameworks and show that our
proposed learning framework outperforms baselines on this real-world dataset as
well.
