Design Challenges for a Multi-Perspective Search Engine (2022)

Abstract: Many users turn to document retrieval systems (e.g. search engines) to seek answers to controversial or open-ended questions. However, classical document retrieval systems fall short at delivering users a set of direct and diverse responses in such cases, which requires identifying responses within web documents in the context of the query, and aggregating the responses based on their different perspectives. The goal of this work is to survey and study the user information needs for building a multi-perspective search engine of such. We examine the challenges of synthesizing such language understanding objectives with document retrieval, and study a new perspective-oriented document retrieval paradigm. We discuss and assess the inherent natural language understanding challenges one needs to address in order to achieve the goal. Following the design challenges and principles, we propose and evaluate a practical prototype pipeline system. We use the prototype system to conduct a user survey in order to assess the utility of our paradigm, as well as understanding the user information needs when issuing controversial and open-ended queries to a search engine.

## introduction in the past two decades, web search has developed as a ubiquitous way for users to retrieve web information. typically, a web search query is driven by a specific intent, or information need, i.e. the users' desire to seek answers to an open-ended question @xcite . among the various types of questions, one of the most challenging types, from the perspective of designing an information retrieval (ir) system, is debate-worthy, controversial questions, for which an objectively-true answer or consensus seems impractical to reach. rather, explicit or implied re-* equal contributions figure 1 : example retrieved articles with snippets from a search engine with respect to the query "should we all wear masks" dated from april to may 2020, during which mask wearing was a more controversial topic. the retrieved articles contain redundant information, or often the extracted summary response does not answer the user query directly. classical ir systems fall short at delivering responses with a diverse set of perspectives to the user query. instead, they typically provide a ranked list of references to relevant but not necessarily trustworthy web documents, with limited justifications on what part(s) of the document could answer the user query @xcite . we argue that a document retrieval system should ideally be equipped with the ability to extract or generate such responses from documents conditioned on the user query. furthermore, a retrieval system should recognize the semantic difference of responses in cross-document settings, and in turn organize and deliver a set of documents from diverse perspectives. these objectives can inherently be formulated into a suite of natural language understanding tasks @xcite . the goal of this paper is to discuss and validate the users' information need of direct and diverse responses when issuing controversial, debate-worthy queries to a search engine. we examine the ideas of formulating such objectives as natural language understanding tasks, and incorporate them in a novel document retrieval paradigm. through the lens of key design principles of a task-specific search engine (rieger, 2009), we first discuss the challenges in designing such a multi-perspective search engine. we introduce a paradigm that revolves around the abstraction of responses, i.e. a text segment that serves as direct answer to users, and perspectives, the argument(s) expressed in a response in the context of a given user query @xcite . to assess the utility of the proposed paradigm, we develop and demonstrate a prototype opendomain news search engine. given a user query of controversial topics, the search engine retrieves news articles, identifies/generates responses within the articles along with evidence paragraphs, and categorizes the responses according to their perspectives differences. with the prototype, we conduct a user study in comparison to a generalpurpose search engine. with the goal of understanding the user needs related to controversial queries, we discuss and analyze the advantages/disadvantages of our prototype through the user study responses. we then summarize and offer insights on the technical challenges and trade-off to cater to the specific user information needs with respect to controversial queries. we hope our findings will facilitate future research on this topic. in summary, our contributions are as follows: and our problem is that objective correctness does not exist for most controversial queries, and so they cannot be responded with a single correct answer. instead, there exist dimensions such as stance @xcite and persuasiveness @xcite to measure the semantic difference between relevant and arguably equally valid arguments. to this end, recent researchers have attempted to extract or generate explanations alongside search results @xcite . ## design challenges we start by discussing the needs of users when issuing controversial and debate-worthy queries to a search engine. rieger (2009) introduces three key factors in building a task-specific search engine that caters to user's information needs. we introduce each factor below, and discuss the connections and implications to the case of controversial, debateworthy user queries. satisfaction refers to overall success of the retrieved results in providing help to the user's specific search need. here there are two levels of user satisfaction to consider. the first dimension is whether the retrieved results contain the answer that user needs. the second more challenging yet crucial dimension is whether search engine is able to highlight and display the answer @xcite . in the context of controversial queries, identifying direct answers from retrieved documents seems essential, as it will greatly reduce the amount of readings user need to do before deriving the answer. however, automating the process comes with technical challenges, and such automation often ties closely with concerns of model biases, as we will further discuss in § 6. search intent refers to the category of webpages or answers that a user is interested to see when issuing a query. in the case of controversial queries, as users typically do not know a priori the potential aspects or perspectives that could lead to the answer in their context, organizing and highlighting a comprehensive set of results with diverse perspectives in a parallel fashion becomes important in this case @xcite . trust is originally defined by @xcite ) as users' trust in a search engine's capability of retrieving relevant results to the query. however, as concerns over the veracity of web information grow @xcite , user trust revolves more closely around the trustworthiness of the sources and content @xcite . for controversial queries, as multiple perspectives exist for every issue, whether each perspective is corroborated by evidence from trustworthy sources becomes the key to ensure user trust @xcite . in the following sections, we take a closer look into the specific problems we discussed for the three factors, and the challenges in formulating nlp tasks to address the problems. ## a prototype of multi-perspective search engine following the design principles outlined in the previous section, we implement a prototype search engine. our motivation of the prototype is twofold. first, we want to understand the utility of our conceptual paradigm from the user perspective. second, this prototype serves as a platform to study the user information needs with respect to controversial and debate-worthy queries. we defer the discussion of the study to § 5. ## a study on user information needs with the prototype search engine, we revisit the discussions in § 3. our goal for the user study is to validate our hypothesis on user information need, and discover new insights from users' experience with an actual prototype model. ## discussion the results from the user study offer validation our main hypothesis for the paper, that users typically wish to see direct and diverse responses when issuing a controversial, debate-worthy query to the search engine. at the same time, we also see concerns over potential model mistakes and intrinsic biases, which lead to decreased trust compared to existing document retrieval services. surrounding the three design factors introduced in § 3, we discuss a few existing trade-offs in our conceptual paradigm through the lens of technical challenges involved in developing a multiperspective search system. ## conclusion this study aims to understand the user information needs for controversial and debate-worthy queries to search engines. we argue for and examine the need of delivering direct and diverse response to such queries. to demonstrate the utility of the paradigm, we develop a prototype multiperspective search engine by synthesizing a suite of relevant nlp and ir tasks. we use the tool to facilitate a user study that confirms the benefit of serving direct and diverse responses to user queries. through the positive and negative feedback we receive on the prototype, we discuss a few trade-off between technical design and user experience. we hope the findings in the paper offers insights, guidance and opportunities for future development on web document retrieval systems.