Implementing Evaluation Metrics Based on Theories of Democracy in News Comment Recommendation (Hackathon Report) (2021)

Abstract: Diversity in news recommendation is important for democratic debate. Current recommendation strategies, as well as evaluation metrics for recommender systems, do not explicitly focus on this aspect of news recommendation. In the 2021 Embeddia Hackathon, we implemented one novel, normative theory-based evaluation metric, “activation”, and use it to compare two recommendation strategies of New York Times comments, one based on user likes and another on editor picks. We found that both comment recommendation strategies lead to recommendations consistently less activating than the available comments in the pool of data, but the editor’s picks more so. This might indicate that New York Times editors’ support a deliberative democratic model, in which less activation is deemed ideal for democratic debate.

## introduction recommender systems are a core component of many online environments. such systems can be used to recommend movies or music to users where there is a large pool of potential recommendations. their main task, as @xcite put it, is "to filter incoming streams of information according to the users' preferences or to point them to additional items of interest in the context of a given object" (p. 1203). as such, they are usually designed in ways that maximise user satisfaction. their performance is traditionally evaluated in terms of their "accuracy", which is often measured by proxies such as clicks, time spent on a page, or engagement. simply put: the more attention a user pays to the content, the better the recommender system is deemed to be. however, there is an increasing awareness in the recommender systems domain that "beyondaccuracy" metrics such as diversity or novelty are also important aspects of a meaningful recommender system evaluation @xcite . this is particularly true in contexts where the impact of recommendations extends beyond individual purchasing choices or movie selections, such as news recommendation. given that exposure to diverse viewpoints is often regarded as beneficial for democratic societies @xcite , scholars have recently highlighted the importance of exposure diversity in such systems @xcite . not recommending diversity in news recommender systems could potentially lead to 'filter bubbles', where users only receive ideas and viewpoints they already know and/or agree with @xcite . very recently, evaluation and optimization metrics by @xcite have been specifically designed to align with potential goals of democratic news recommenders as suggested by @xcite . as such, they move beyond the existing "beyond accuracy" evaluation metrics used in the recommender system field. these existing metrics range from "diversity", to "serendipity", "novelty", and "coverage" @xcite , but all of these implicitly aim at increasing user satisfaction rather than achieving normative goals. in contrast, the metrics in @xcite are explicitly linked to supporting democratic debate rather than user satisfaction. specifically, these metrics are linked to models of democracy. one of these is the deliberative model of democracy, which states a functioning democracy consists of rational debate of viewpoints and ideas. another model is the critical model, which contends a successful democracy has clashing and active debates of opposing viewpoints. in this paper, we specifically focus on one of these metrics, "activation", and use it to evaluate two different recommendation strategies for new york times user comments in response to news articles. in doing so, our goal is to explore the potential of, but also the challenges related to, such normative metrics, especially where it concerns natural language processing (nlp) tools and strategies. to better understand how different recommendation strategies in the nyt comment section perform in terms of this metric, we ask the following research question: "how do different manners of recommending user comments on a news article affect the recommendation set's average activation scores? by comparing different comment recommendation strategies, we contribute to the ongoing discussion in three ways: our goal was to "test-drive" one or more of the theory-driven evaluation metrics in @xcite , and see where we ran into conceptual or practical problems preventing us from answering a research question aimed at comparing different recommendation strategies on the basis of this metric. ## results our results are visible in