  Recent work in fair machine learning has proposed dozens of technical
definitions of algorithmic fairness and methods for enforcing these
definitions. However, we still lack an understanding of how to develop machine
learning systems with fairness criteria that reflect relevant stakeholders'
nuanced viewpoints in real-world contexts. To address this gap, we propose a
framework for eliciting stakeholders' subjective fairness notions. Combining a
user interface that allows stakeholders to examine the data and the algorithm's
predictions with an interview protocol to probe stakeholders' thoughts while
they are interacting with the interface, we can identify stakeholders' fairness
beliefs and principles. We conduct a user study to evaluate our framework in
the setting of a child maltreatment predictive system. Our evaluations show
that the framework allows stakeholders to comprehensively convey their fairness
viewpoints. We also discuss how our results can inform the design of predictive
systems.
