  Due to their promise of superior predictive power relative to human
assessment, machine learning models are increasingly being used to support
high-stakes decisions. However, the nature of the labels available for training
these models often hampers the usefulness of predictive models for decision
support. In this paper, we explore the use of historical expert decisions as a
rich--yet imperfect--source of information, and we show that it can be
leveraged to mitigate some of the limitations of learning from observed labels
alone. We consider the problem of estimating expert consistency indirectly when
each case in the data is assessed by a single expert, and propose influence
functions based methodology as a solution to this problem. We then incorporate
the estimated expert consistency into the predictive model meant for decision
support through an approach we term label amalgamation. This allows the machine
learning models to learn from experts in instances where there is expert
consistency, and learn from the observed labels elsewhere. We show how the
proposed approach can help mitigate common challenges of learning from observed
labels alone, reducing the gap between the construct that the algorithm
optimizes for and the construct of interest to experts. After providing
intuition and theoretical results, we present empirical results in the context
of child maltreatment hotline screenings. Here, we find that (1) there are
high-risk cases whose risk is considered by the experts but not wholly captured
in the target labels used to train a deployed model, and (2) the proposed
approach improves recall for these cases.
