  News recommenders help users to find relevant online content and have the
potential to fulfill a crucial role in a democratic society, directing the
scarce attention of citizens towards the information that is most important to
them. Simultaneously, recent concerns about so-called filter bubbles,
misinformation and selective exposure are symptomatic of the disruptive
potential of these digital news recommenders. Recommender systems can make or
break filter bubbles, and as such can be instrumental in creating either a more
closed or a more open internet. Current approaches to evaluating recommender
systems are often focused on measuring an increase in user clicks and
short-term engagement, rather than measuring the user's longer term interest in
diverse and important information.
  This paper aims to bridge the gap between normative notions of diversity,
rooted in democratic theory, and quantitative metrics necessary for evaluating
the recommender system. We propose a set of metrics grounded in social science
interpretations of diversity and suggest ways for practical implementations.
