  Several pieces of work have uncovered performance disparities by conducting
"disaggregated evaluations" of AI systems. We build on these efforts by
focusing on the choices that must be made when designing a disaggregated
evaluation, as well as some of the key considerations that underlie these
design choices and the tradeoffs between these considerations. We argue that a
deeper understanding of the choices, considerations, and tradeoffs involved in
designing disaggregated evaluations will better enable researchers,
practitioners, and the public to understand the ways in which AI systems may be
underperforming for particular groups of people.
