Can I trust You? LLM s as conversational agents (2024)

Abstract: With the rising popularity of LLMs in the public sphere, they become more and more attractive as a tool for doing one’s own research without having to rely on search engines or specialized knowledge of a scientific field. But using LLMs as a source for factual information can lead one to fall prey to misinformation or hallucinations dreamed up by the model. In this paper we examine the gpt-4 LLM by simulating a large number of potential research queries and evaluate how many of the generated references are factually correct as well as existent.

## introduction one of the main functions of the system of mass media, according to niklas luhmann, consists in constituting a sort of short-term memory for society by providing and processing information about the world @xcite . to fulfill this function and to perpetuate its own existence, the mass media constantly generates and communicates information. however, the truth of this information is not the most important factor, even when it comes to news. truth is only relevant insofar as it emphasizes the sensational value of any given message and averts the risk of being accused of deception @xcite . but this is in contrast to how we use news in our everyday lives where we rely on them for their accuracy. with the advent of llms in the larger public sphere @xcite @xcite , this tension between the individual's desire for factually correct information and the mass media's preference for mere communication grows more pronounced. when ordinary users utilize llms as more advanced internet search engines to answer questions that are not easily answered by sites like wikipedia, they expect a truthful response. but the model's primary function lies in communication. it might be dissatisfying for users when an llm hallucinates instead of providing correct answers to a query, but this is not an error on a technical level. a technical error would mean a complete failure to communicate (i.e. a blank output or an incomprehensible non-sequitur). to counteract this discrepancy, filters and fact-checking processes have been implemented, but these are additional mechanisms added on top of the base model to steer the result in the desired direction. the issue of communication being valued more highly than truth still remains. while this might not be a problem on a larger systemic level, it is an issue when individuals expect information to be factual. this is why we need to examine the quantitative and qualitative nature of false information produced by llms. ## related work despite recent developments in artificial intelligence and the emergence of different llms such as palm @xcite , openai's chat-gpt (openai, 2022), and gpt-4 (openai, 2023), google's bard @xcite ), meta's llama @xcite all of the established llms are infamous for hallucinations @xcite . this fact prompted research to examine the ways of how we can facilitate the "made up" nature of some of the outputs of the models. as well as overall analysis of whether the llms can be trusted. workshops such as workshop on large language models' interpretability and trustworthiness @xcite or trustnlp: workshop on trustworthy natural language processing @xcite nudge the scientific community to investigate explainability and trustworthiness of different predictive models and llms. most of the research on the trustworthiness of the llms is either based on surveys and analysis of different requirements (such as fairness, explainability, accountability, reliability to name a few) that can be used to asses the output of a model in general @xcite or in specific fields, for instance in healthcare @xcite . other studies cover general guidelines for establishing a trustworthy model @xcite or provide an overview of methods available to detect the fairness of the output of the model based on their toxicity, bias and value-alignment @xcite . though there are several studies that are focused on investigating how factually correct the output of the models is @xcite , the usual methods of testing factual correctness is to check the facts present in the output. only very few researchers studied the references that were provided by a model for the generated output, i.e. checked the validity of the source that the output was based on @xcite . this positional paper is designed to make it more apparent if we should trust the output of the llms just because they provide us with official looking sources. ## methodology to test our hypothesis that large language models are in their nature trained to be a conversational partner first and a "fact provider" second, we have conducted a series of experiments that were designed to examine how much llms can be trusted as a source of information. for the first part of the experiment we automatically generated scientific questions from different branches. overall, 985 questions were generated for 231 scientific branches. the most questions were generated for the fields of medicine (40), computer science (38), environmental science (37), earth science (35), physics (35), mathematics (35) and chemistry (34). the generated questions were then used as a query for the gpt-4 llm with a task of further generating a corresponding concise explanation which was to be based on real scientific references (in form of a list of links). for example, the query "how do children acquire syntactic knowledge in their native language?" received the explanation "children acquire syntactic knowledge in their native language through a combination of innate abilities and environmental input <. . . >" and a list of 4 links to different scientific journals. the query "how can we model complex systems with mathematics?" received the explanation "complex systems can be modeled mathematically using various approaches, depending on the nature of the system and the phenomena being studied. here are some of the common mathematical frameworks and methods used to model complex systems <...>" based on 10 different links. previous research has shown that mapping back to the original documents or providing sources to the generated texts can potentially invoke a feeling of trust from a user towards a model @xcite . therefore the generated references were investigated for their relevance. in order to do that, we counted how many of the generated references actually exist at the moment of the experiment, and how many of the links provided the information that was relevant to the question. the workflow of the experiment is given in figure 1 . langchain (harrison, 2022) was used for generating both questions and explanations. though our experiments resemble the study per-formed by @xcite , our approach focuses more on generating specific questions within the scientific domain to simulate the workflow and the research process of a human. ## evaluation to further investigate whether our hypothesis holds we have conducted a quantitative (statistical) analysis of the results. overall, for 985 different questions we generated 985 explanations with 4434 references (with an average of 5 references per explanation). out of the 4434 references 2967 links led to pages that did not exist anymore at the time the experiments were conducted (middle of december 2023). one of the reasons could be that the submitted paper that existed at the time the llm was trained, was retracted, deleted or moved to another website (so-called illusions @xcite ). another reason could be that the link was hallucinated by the model and did not exist in the first place. out of the 1467 references that led to real existing scientific sources 1376 were relevant to the question. we also examined the scientific branches that suffered most from the sources that were not fully available. out of 40 questions and explanations for the field of medicine, 37 had missing references. 37 out of 37 explanations for questions from the field of environmental science had incomplete references. 2 out of 2 questions and explanations from the area of zymology had referenced non-existing sources. in general, only approximately 25% of explanations were covered by the references that were marked as existing at the moment of the experiment. after the relevance check 88% of explanations had not only verifiable but also related sources. ## discussion our findings show that a large number of the generated sources are relevant to the question that was asked, but do not exist. this is either because the model hallucinated these citations or because its data was outdated, thus providing a dead link. it is not entirely clear if this supports our hypothesis. if the citations are indeed real, but have been deleted or retracted since the model was trained, this finding may only be a reflection of the model's outdated training data. but if the model did indeed hallucinate these sources, this finding would support our hypothesis that the model puts greater value on communication and providing answers that are superficially satisfactory than on factually correct information. however, many of the generated sources were irrelevant to the query. these were generated by the model to fulfill the demand for sources in the query, but the sources provided were selected with little regard for relevancy. this supports our hypothesis that the model puts greater emphasis on communicating successfully than on responding to the query correctly. ## limitations a limitation of this paper is that we only tested the gpt-4 llm. our findings might only be relevant to models similar to this one, but not for models that are very different from it or those that have a greater focus on factual accuracy. we were also not able to clearly differentiate between sources that did not exist anymore because they were removed or retracted and those that were hallucinated entirely. ## ethical considerations as far as ethical considerations go, our findings illustrate that only in 88% of cases the cited sources were relevant as well as existent. this suggests that llms are not primarily concerned with providing accurate and up-to-date information. individual users that seek to use llms as a tool for an in-depth net search that search engines can't provide should be very cautious to double check the information they receive. ## future work we strongly believe that further investigation into the phenomenon of illusions, hallucinations and unrelated sources is needed. understanding why the model outputs references that are not indeed relevant to the generated answers as well as a way of identifying types of illusions and hallucinations is crucial for building reliable, personalized and trustworthy llms. ## acknowledgment this study was conducted within the framework of developing a commercial paraphrasing tool for german by employees of ella lab -gesellschaft für künstliche intelligenz mbh. furthermore, we would like to acknowledge the guidance and provided by nasrin saef, head of machine learning team.