Do Metadata and Appearance of the Retrieved Webpages Affect LLM ’s Reasoning in Retrieval-Augmented Generation? (2024)

Abstract: Large language models (LLMs) answering questions with retrieval-augmented generation (RAG) can face conflicting evidence in the retrieved documents. While prior works study how textual features like perplexity and readability influence the persuasiveness of evidence, humans consider more than textual content when evaluating conflicting information on the web. In this paper, we focus on the following question: When two webpages contain conflicting information to answer a question, does non-textual information affect the LLM’s reasoning and answer? We consider three types of non-textual information: (1) the webpage’s publication time, (2) the source where the webpage is from, and (3) the appearance of the webpage. We give the LLM a Yes/No question and two conflicting webpages that support yes and no, respectively. We exchange the non-textual information in the two webpages to see if the LLMs tend to use the information from a newer, more reliable, and more visually appealing webpage. We find that changing the publication time of the webpage can change the answer for most LLMs, but changing the webpage’s source merely affects the LLM’s answer. We also reveal that the webpage’s appearance has a strong causal effect on Claude-3’s answers.The codes and datasets used in the paper are available at https://github.com/d223302/rag-metadata.

## introduction retrieval-augmented llms @xcite respond to user queries by considering the documents retrieved from external knowledge sources, ranging from wikipedia @xcite to the whole web @xcite . as the knowledge source scales up and the user queries become more diverse, the retrieved contents can contain conflicting information. extensive prior works have explored how figure 1 : given a yes/no question and two documents that support yes and no, respectively, we add a type of non-textual information (publication date in this figure) to both documents with different values. we swap the non-textual information in the two documents and see whether the llm's answer to the question is different. llms reason over conflicting documents @xcite @xcite . when humans are presented with contradicting evidence that leads to different answers, we use multiple strategies to reason over the searched @xcite @xcite , including the credibility of the sources @xcite and the arguments in the documents @xcite . then, what about llms? what evidence do llms find convincing when conflicting information exists in the retrieved documents? to understand this, @xcite constructs conflictingqa, consisting yes/no questions and documents extracted from real webpages that support both stances. they analyze what text features in the document make the llm more inclined to agree with the stance in the document. while @xcite provide valuable insights into how text features affect a webpage's credibility for llms, they do not explore how the information beyond the document's content affects the llm's decision. this is because most retrieval-augmented llms only take the titles and the textual contents as input while discarding all the metadata of the webpage, including url and webpage publication times @xcite . consequently, it is unclear whether llms can use metadata of the webpages for reasoning when these metadata are provided. additionally, given the wide application of using vision language models (vlm) @xcite @xcite for web navigation and webpage question answering @xcite @xcite , it is unclear how the visual appearance of the webpage affects how vlms reason based on the webpage. in this paper, we explore the following research question: can the metadata and appearance of the retrieved webpages affect the llm's answer? in our paper, we will use the term non-textual information to refer to the information in a webpage other than its title and textual contents, which can include the webpage's metadata and its appearance. 1 inspired by how a human's reasoning can be affected by non-textual information, including (1) the webpage's publication time @xcite , (2) the source's credibility @xcite , and (3) the appearance of the webpage @xcite , we want to know if these factors can affect llm's answer. we will use the terms document and webpage interchangeably as we simulate the case when the documents are webpages retrieved from the web. we give an llm a yes/no question and two documents supporting contradicting stances, with non-textual information incorporated in the documents. we exchange the non-textual information in the two documents to see if the llm's answers change and whether the llm's answer agrees with the stance of a webpage published more recently, from a more reliable source, or looks better. we conduct causal analyses to understand whether the non-textual information affects the llm's answer. additionally, we check the llm's responses to see if it mentions non-textual information. to the best of our knowledge, we are the first to explore the role of non-textual information in rag with conflicting evidence. we have the following intriguing observations: ## experiment setup to answer whether llm's answer and reasoning can be affected by the non-textual information, we give the llm a yes/no question and a pair of documents that support yes and no, respectively. the two documents include their respective non-textual information. we observe whether the llm's answer can be changed by exchanging only the non-textual information in the two documents and whether the llm's reasoning mentions the non-textual information. the overall experiment setup is shown in figure 1 . ## the webpage's publication time first, we focus on the publication time of the webpage, which is an important webpage metadata. since removing the metadata and extracting only the textual content is the first step to pre-process a webpage, metadata, including publication time, is seldom used as input to the llm in rag @xcite . while prior works on time-dependent question-answering benchmarks consider the publication time of a webpage @xcite @xcite , they do not thoroughly study the effect of the publication time on the llm's answer in a controlled and causal way as we do. moreover, compared with situatedqa @xcite and realtime qa @xcite , which are based on real-world entities, using conflictingqa-fake allows us to reduce the possibility of llm relying on its parametric knowledge instead of the retrieved evidence. ## source of the webpage next, we explore the source of the webpage. we are specifically interested in the case when documents are from sources that differ in credibility. we use the following pair of webpage sources: wikipedia and wordpress. wikipedia is a trustworthy source with mostly verified information, while wordpress is mostly personal blogs and does not guarantee its information's correctness. conflicting information from diverse sources is an important topic in factchecking @xcite @xcite @xcite . we differ from them by using counterfactual analysis, i.e., swapping the sources of the documents, to understand the role of the source to llms in rag. ## how the webpages look last, we study whether the webpage's appearance and formatting change the llm's answer. since llms only use texts as the input in rag, we are unaware of prior works that consider how the webpage appearance affect the llm's answer in rag. ## conclusion we explored how non-textual information in retrieved webpages affects llm responses amid contradictory evidence. we find that all llms we use are sensitive to the webpage's publication time and rely on more up-to-date webpages. we also reveal that when providing the llm with documents from sources of different credibility, exchanging the source of the two documents barely affects the llm's answer. lastly, we show that when llms are given only the screenshots of the retrieved webpages, changing the formatting of the webpages has a causal effect on some llm's answers. our results highlight an aspect not well-explored in previous rag literature by showing that certain non-textual information has a causal effect on the model's answer. whether this is desirable is debatable, but it is essential to first recognize this phenomenon.