Discord Questions: A Computational Approach To Diversity Analysis in News Coverage (2022)

Abstract: There are many potential benefits to news readers accessing diverse sources. Modern news aggregators do the hard work of organizing the news, offering readers a plethora of source options, but choosing which source to read remains challenging.We propose a new framework to assist readers in identifying source differences and gaining an understanding of news coverage diversity.The framework is based on the generation of Discord Questions: questions with a diverse answer pool, explicitly illustrating source differences.To assemble a prototype of the framework, we focus on two components: (1) discord question generation, the task of generating questions answered differently by sources, for which we propose an automatic scoring method, and create a model that improves performance from current question generation (QG) methods by 5%, (2) answer consolidation, the task of grouping answers to a question that are semantically similar, for which we collect data and repurpose a method that achieves 81% balanced accuracy on our realistic test set.We illustrate the framework’s feasibility through a prototype interface. Even though model performance at discord QG still lags human performance by more than 15%, generated questions are judged to be more interesting than factoid questions and can reveal differences in the level of detail, sentiment, and reasoning of sources in news coverage. Code is available at https://github.com/Salesforce/discord_questions.

## introduction news coverage often contains bias linked to the source of the content, and as many readers rely on few sources to get informed @xcite , readers risk exposure to such bias on critical societal issues such as elections and international affairs @xcite . modern news aggregators such as google news propose an engineering solution to the problem, offering news 1 code is available at https://github.com/ salesforce/discord_questions ## framework definition we first define terminology, then introduce components of the discord questions framework. ## news answer consolidation we collected an evaluation set we name nanco (news answer consolidation) and evaluated several transfer learning strategies to select the bestperforming model for the pipeline. ## discord question generation the discord question framework relies on obtaining story-relevant questions. qg models are known to excel at generating factoid questions but are limited on realistic curiosity-driven questions @xcite . we propose an automatic method to evaluate qg models on the ability to generate discord questions, based on the intuition that we can use a story's full context to evaluate a question. the method is illustrated in figure 3 . ## discord questions assembled we assemble the discord questions frame with the best-performing components -rob-l-mocha for consolidation and t5-discord for qg -and make a public web interface. we perform a manual evaluation of the system, first evaluating the relative interestingness of discord questions to potential readers, and second analyzing types of diversity surfaced by the system. ## conclusion we introduce the discord questions framework, in which we hypothesize that a question accompanied by an organized set of answers from the sources can spotlight specific ways in which sources disagree, providing concrete examples of coverage diversity to a news reader. we decompose the framework into required components, and design evaluation methodology for each. we select high-performing models for each component and assemble them into a working prototype of the framework. we confirm through manual analysis that questions generated within our framework are of interest to potential users, and in a majority of cases surface four types of diversity in news coverage, from varying levels of detail in the reporting, to differing sentiment or reasoning about an event's cause, confirming that discord questions are an interpretable tool to uncover coverage diversity.