What Evidence Do Language Models Find Convincing? (2024)

Abstract: Retrieval-augmented language models are being increasingly tasked with subjective, contentious, and conflicting queries such as “is aspartame linked to cancer”. To resolve these ambiguous queries, one must search through a large range of websites and consider “which, if any, of this evidence do I find convincing?”. In this work, we study how LLMs answer this question. In particular, we construct ConflictingQA, a dataset that pairs controversial queries with a series of real-world evidence documents that contain different facts (e.g., quantitative results), argument styles (e.g., appeals to authority), and answers (Yes or No). We use this dataset to perform sensitivity and counterfactual analyses to explore which text features most affect LLM predictions. Overall, we find that current models rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important such as whether a text contains scientific references or is written with a neutral tone. Taken together, these results highlight the importance of RAG corpus quality (e.g., the need to filter misinformation), and possibly even a shift in how LLMs are trained to better align with human judgements.

## introduction llms are widely deployed in settings that require understanding context-from retrievalaugmented systems to web agents, models condition on sources that range from internet paragraphs @xcite to python interpreters @xcite . at the same time, today's models are also given tasks that are increasingly open-ended and controversial, such as "tell me if aspartame causes cancer". to answer such questions, llms will read real-world paragraphs that are contradictory, noisy, and rife with misinformation @xcite . humans have techniques to sift through such large quantities of complex and contradictory evidence by answering the question: which, if any, of this evidence do i find convincing? to do so, humans combine multiple strategies, including fact checking and evaluating a source's credibility @xcite , harnessing prior knowledge and beliefs @xcite , and critically evaluating logical arguments @xcite . in this work, we explore how llms resolve similar ambiguities when faced with conflicting openended questions. to study this, we create con-flictingqa, a dataset consisting of questions and real web documents that lead to conflicting answers. for instance, in figure 1 we show an example where we pair the question "is aspartame linked to cancer" with a series of conflicting evidence documents collected from google search. in our experiments, we evaluate the convincingness of each evidence document by computing the rate at which a model's predictions align with that document's viewpoint (i.e., its win-rate). we perform sensitivity and counterfactual analyses to find in-the-wild features that correlate with document convincingness. we consider a mix of features that describe (1) stylistic properties of a document and (2) the relevance of a document to the question. many of these were inspired by results from studies on human credibility judgements: for example, we consider whether adding scientific references makes text appear more convincing. overall, we find that stylistic features play a considerably less impactful role in determining the convincingness of text than measures of relevance. notably, we show that a simple perturbation targeting a website's relevance-prefixing the page with "the following text is about the question: [question]"-is enough to substantially improves its win-rate. on the other hand, stylistic features like the informational content, whether a page contains references, or its confidence, tend to only have a neutral to negative effect on win-question: is aspartame linked to cancer? artificial sweeteners linked with a 13% higher risk of cancer new research finds that a higher intake of artificial sweeteners is linked to an increased risk of cancer. nearly half of united states adults consume artificial sweeteners. human-population studies have found artificial sweeteners to be safe, but results from in vitro studies and studies on animals pose some concerns. @xcite a large new observational study has found an association between the consumption of artificial sweeteners, particularly aspartame and acesulfame-k, and cancer. the study found a 13% higher risk of cancer in general, with the highest likelihood of developing breast cancer and cancers related to obesity, for people consuming large quantities of artificial sweeteners. figure 1: in conflictingqa, we create contentious questions such as "is aspartame linked to cancer". we also retrieve evidence paragraphs for each question that contain different types of facts (e.g., quantitative results), argument styles (e.g., appeals to authority), and answers (yes or no). for example, in the figure above we show two evidence paragraphs with their key arguments highlighted. using conflictingqa, we study why llms trust certain types of evidence paragraphs and argument styles over others. ## background and motivations standard llms can be used to solve tasks that do not require context, e.g., writing basic python code or answering simple trivia questions @xcite @xcite . to give these models more knowledge, agency, and capabilities, recent efforts have augmented llms with retrieval @xcite , domain-specific tools @xcite @xcite , or even generic web access @xcite @xcite . these enhancements allow llms to answer more challenging open-domain questions (e.g., "is aspartame linked to cancer?") or accomplish open-ended tasks (e.g., "buy me a size 9 pair of blue running shoes"). ## the conflictingqa dataset here, we describe the construction of conflict-ingqa, our dataset that evaluates what types of evidence are convincing for llms. we design conflictingqa to emulate the common setup for deploying retrieval-augmented llms: we retrieve the most relevant documents for a particular user query and place them in the llm's context window @xcite @xcite . to build our dataset, we tackle three challenges: collecting contentious questions, identifying relevant and diverse evidence paragraphs, and grouping evidence paragraphs together to create conflicting examples. collecting contentious questions. we first create a series of realistic open-ended questions for which there exists conflicting evidence online. critically, unlike past work on ambiguity in qa @xcite @xcite , we want to collect unambiguous questions that still have answer conflicts. for example, in figure 1 , we show a question "are artificial sweeteners linked to cancer?", which is a widely-debated query in which there exist websites that support both answers. we design the questions to elicit binary responses of yes or no to simplify evaluation. we create questions using gpt-4. to ensure that the model generates a diverse set of questions we take inspiration from previous work in synthetic dataset generation @xcite and stratify the generations by topic: we first generate question categories (e.g., climate change, robotics, oncology) then generate sets of questions conditioned on each category (full prompt provided in ## experimental results in this section, we use conflictingqa to evaluate what types of evidence models find convincing. ## discussion & related work how should systems handle ambiguity? one reasonable suggestion is that agents should not make their own autonomous decisions when faced with ambiguous or conflicting evidence. for example, they may summarize both sides of the aspartame argument, or they may ask the user to clarify their preferences. there is naturally a tradeoff between autonomy and clarity. past work has explored one side of this trade-off, for example by abstaining from answering in cases of ambiguity @xcite , by trying to provide multiple perspectives on the answer @xcite , or by asking clarification questions @xcite . our work explores the other side of the trade-off: we analyze the behavior of models when they are expected to resolve ambiguity with more autonomy. additionally, our dataset serves as a benchmark for exploring these questions as it reflects realworld ambiguities in question-answering. for example, in ## conclusion we study how rag models judge convincingness by collecting a diverse set of controversial questions and website text (conflictingqa), and designing a realistic evaluation framework based on how these models are used in practice. our results show that today's llms tend to overrely on relevancy and ignore many stylistic features of text that humans often deem important. future work should explore how integrating other forms of information (e.g., metadata, visual content) can influence these behaviors. in addition, given the possible flood of llm-generated content on the internet, it is important to consider how these synthetic texts may influence llm judgements of convincingness.