What Kind of Sourcery is This? Evaluating GPT -4’s Performance on Linking Scientific Fact to Citations (2024)

Abstract: From document summarization to code generation, chabots have disrupted various aspects of scientific research and writing. While chabots are useful research resources for ideation, information retrieval, and editing, their generative pre-trained transformer (GPT) models’ underlying knowledge infrastructure is opaque. This has raised questions about the reliability of generative chatbot responses, as GPT models are known to respond with misleading information that appears to be accurate. Prior research has investigated the utility of OpenAI’s public chatbot, ChatGPT, to generate reliable bibliographic information with a focus on small-scale medical-related scientific facts. We present an expanded study that analyzes GPT-4’s ability to accurately identify 1,326 scientific facts and link them to academic sources. Using both the API and UI service, we experimented with open-ended and close-ended prompts to establish an understanding of GPT-4’s general ability at this domain-specific task, as well as study the real-world scenario of an average user interacting with ChatGPT using its UI. GPT-4 accurately identified 96% of the scientific facts and generated relevant and existent academic citations with 78% accuracy. Using the claims that GPT-4 mislabeled and provided incorrect sources via the API, we prompt two public GPTs customized for academic writing to evaluate if they correctly label the scientific claims and provide accurate sources. We find that these GPTs are able to accurately label 38% of the mislabeled claims, with 95% of the corresponding citations being accurate and relevant.

## introduction with the ability to perform a wide range of natural language generation (nlg) and information retrieval tasks, chatbots have enabled individuals to experiment with the utility of generative pretrained transformer (gpt) language models in a publicly available, online interface. while chatbots are generative ai tools, users often query chatbots in a paired task that includes both nlg and information retrieval; for example, generating new content (e.g., write an introduction for a paper on a given topic) and retrieving information (e.g., provide citations when necessary). however, users often engage with chatbots for a specific task without understanding its utility in the given domain. using a chatbot as an information gathering tool is convenient, but comes with caveats. various studies that analyze a chabot's performance on nlg and information retrieval tasks (e.g., document summarization and code generation) highlight a persistent error in the gpt model's responseshallucinations @xcite . hallucinations refer to factually incorrect responses that often pass as being correct and credible text to a user @xcite . hallucinations are harmful to users, particularly in information retrievallike tasks where the user is not an expert in the prompt topic, because chatbots can respond with well-formatted text that is convincingly accurate, but is completely fabricated. in this work, we focus on a particular use-case for information gathering-linking scientific facts to sources for citations. prior research has focused on evaluating gpt models (mainly versions 3 and 3.5) via the online chatgpt interface in small scale experiments on complex scientific topics for citation generation @xcite @xcite . our study expands this research to prompt gpt-4 via the api on 1,326 scientific facts from 3rd-5th grade level coursework, covering a range of scientific topics. specifically, we design an automated prompt framework that includes a close-ended prompt ("is the fact true or false?") and an open-ended prompt ("provide a citation to support your response") to analyze gpt-4's ability to identify scientific facts and accurately link them to academic citations. we then provide human annotation to evaluate the ac-curacy of gpt-4's responses, assessing if the provided citation is relevant to the scientific fact and exists (i.e. the source is not hallucinated). further assessing gpt-4's ability to generate reliable and accurate bibliographic information, we design a second prompt with two close-ended questions to verify its prior responses on the same criteria as the human annotation: "is the citation relevant to the scientific fact?" (yes or no) and "does the citation exist?" (real or fake). the full experimental design is illustrated in figure 1 . evaluating gpt-4's ability to verify scientific fact and provide a corresponding source via the api, we then use two sets of gpt-4 labeled claims for further experimentation: (1) claims that gpt-4 incorrectly labeled as false and (2) claims that gpt-4 provided hallucination sources for. we select two public gpts foot_0 cusotmized for academic writing to converse with in the online user interface (ui). this experiment captures a real-world chabot interaction, where a customized chatbot is being used as a tool for a domain-specific task, while the api experiment comprehensively evaluates gpt-4's knowledge capacity and retrieval capabilities in an automated pipeline. our experimental results show that gpt-4 is capable of identifying scientific fact with 96% accuracy and generating a relevant, existing citation with 78% accuracy. gpt-4 favored providing a textbook citation over a scientific article or website, and only hallucinated 1% of textbook citation responses. we find that gpt-4 performs poorly as an evaluator of generated citations (determining if a source exitst), only correctly identifying 2% of the non-existent citations. in the ui experiments, we find that gpts customized for academic writing increased the accuracy of scientific claim verification, with 38% of the previously 56 mislabeled claims receiving correct true labels. additionally, the academic gpts provided accurate and relevant citations with 95% accuracy for this set of claims. analyzing the gpts on a sample of 50 of the claims that gpt-4 correctly labeled as true but provided hallucination sources, we find that the academic gpts responded with accurate and relevant citations for all claims when it providing a source. our api and ui results demonstrate that gpt-4 is able to provide reliable responses for information retrieval tasks that require scientific knowledge, both for identifying the veracity of a scientific claim and for providing an accurate source to justify its response. however, gpt-4 is stronger at the question answering task (achieving 96% accuracy) than the strict information retrieval task of providing a linked citation (achieving 78% accuracy). chatbots customized for specific tasks, such as academic writing, improve the reliability of outputs and should be leveraged by users when available. ## related work prior work has analyzed chatgpt models, namely versions 3 and 3.5, in their ability to generate accurate scientific publication references, with the majority of studies focused on medical research @xcite @xcite . additionally, researchers have analyzed and discussed gpt models' ability to be a reliable tool in scientific communication as an information resource or co-author @xcite @xcite @xcite . while researchers acknowledge that gpt models have potential as a resource in academic and scientific writing, several studies highlight its shortcomings on citation generation tasks. gravel et al. queried chatgpt with 20 medical questions derived from research publications, asking for the corresponding citation. chatgpt's responses contained 59 distinct citations, which were then reviewed by the authors of the original research publications. the authors found that 69% of the citations were fabricated, with 71% of the fabrications having correctly formatted metadata (e.g., year, page numbers, volume number) and known publishers (e.g., medrxiv and centers for disease control and prevention) @xcite . wagner and ertl-wagner prompted chatgpt-3 with 88 radiology-related questions asking for responses with citations and chatgpt-3 provided 343 distinct citations across all responses for review. of the references that could be verified, only 24% related to the question (i.e., the publication could be used to support the response) and 64% of the 343 citations appeared to be fabricated by chatgpt-3 @xcite . sebo asked chatgpt-3.5 to provide 10 references to a set of 10 questions related to internal medicine, resulting in 100 citations for review. of the 100 chatgpt-3.5 provided citations, 34% were completely incorrect and 40% were partially correct due to error in metadata (e.g., publication year/publisher/etc. was incorrect) @xcite . while these studies are useful in understanding chatgpt's performance on citation generation, they are limited in scope due to their topics and number of questions. additionally, these studies query chatbots with highly specialized domain questions without leveraging a chatbot customized for that domain. our work focuses on extending these studies to a range of 1,326 well-established scientific facts in a more generalized domain, and includes experiments using domain-specific chatbots. ## experimental design here we describe the dataset, prompt design, and response evaluation for our experiments. we experiment with gpt-4's ability to provide accurate bibliographic information for nlg (openended question) and information retrieval (closeended) tasks. specifically, our objective is to evaluate gpt-4's ability to identify scientific fact and provide accurate (existing and relevant) sources to support its responses, and compare the general gpt-4 performance to domain-specific chatgpts. ## results and discussion we evaluate gpt-4's ability to accurately identify scientific fact and provide a relevant and existing citation using the api and ui prompts. each chatbot experiment involves curating a gpt-4 response dataset from the various prompts and analyzing the responses for accuracy and relevancy. ## discussion and limitations the inability to study the underling algorithms, codebase, and knowledge infrastructure of a gpt model presents a challenge when studying closedsource chatbots. in this work, our goal is to systematically evaluate gpt-4's api and ui performances as reliable tools for a paired task of natural language generation and information retrieval on a domain-specific task-linking scientific claims to relevant and existent sources. a limitation of our results is the lack of validation that can only be fully achieved with the transparency of an open-source model. additionally, we only query one chatbot (gpt-4) on scientific facts and sources (limited information types). we highlight our main findings and discuss our interpretations of these results. gpt-4's apparent knowledge acquisition and reliability mimics the real-world. when evaluating the reliability of sources provided, we found that gpt-4 had the most accurate citations when referencing a textbook and the least accurate citations when referencing a url. this behavior mimics real-world bibliographic curation-a relevant published piece of knowledge is more customized gpts achieve higher performance for the intended (domain-specific) task. openai's description of creating customized gpts indicates its user-friendly design (no coding required) by stating that all a user needs is to prompt chatgpt with further instructions or extra knowledge. despite chatgpt being a closed-source model, it can ingest knowledge via human interaction directly in the ui. selecting the additional knowledge that a chabot can learn improves the transparency of knowing what the gpt "knows" and also increases the reliability of the chatbot's responses related to the specific information retrieval task. we highlight the fact that gpt models may appear to be poor tools for an information retrieval task like bibliography generation, as discussed in prior research, however gpt models are generative in their nature. fine-tuning a gpt model with the necessary information for a task will improve its results and reliability, as the knowledge and knowledge sources are identified by the user. thus, customization for a domain-specific task should be heavily considered when leveraging chatbots as a domain-specific tool. ## conclusion in this paper we evaluated gpt-4's ability to identify scientific fact and generate a citation to support its response. our experimental design contained two chatbot environments, api and ui, to fully assess gpt-4's performance. we designed prompts that included open-ended (generative) and closeended (information retrieval) questions in order to test two prompt and response formats. our experiments are designed to compare how gpt-4 generally performs on a domain-specific task (via the api) and how a gpt-4 chatbot performs (via the ui) when customized for use in the specified domain. using the api, we find that in general, gpt-4 performs well on identifying scientific fact and providing reliable sources. for the citation generation task, we find that gpt-4 provided relevant and existent academic citations with 78% accuracy. for the information retrieval tasks, we find that gpt-4 is able to identify scientific fact with 96% accuracy and determine the relevancy of citations with 83% accuracy for irrelevant citaitons and 93% accuracy for relevant citations. gpt-4 had the worst performance when determining if a citation existed, with the majority of its error as labeling broken urls as existent. in the ui experiments, we find that using public gpts customized for academic writing improved the api results in both scientific fact identification and source generation. however, we did identify discrepancies in chatbot performances between the two gpts, with one chatbot's outputs resulting in the majority (76%) not containing a source when being explicitly asked for one. overall, we find gpt-4 to be a useful information gathering tool for general scientific knowledge. as the use of a crop increases, the amount of crops planted will increase a scar is an acquired characteristic magnetism can cause objects to repel each other a sea turtle lives in the ocean a spider web is used to capture food by spiders a renewable resource can be replaced a greenhouse is used to protect plants by keeping them warm the tide cycle regularly occurs twice per day water is an electrical conductor tectonic plates being pushed together causes earthquakes crumple means change shape from smooth into compacted by physical force sunlight contains ultraviolet light the earth revolves around the sun meters m are a unit used for measuring distance generally used for values between 1 and 1000 the slope of the land causes a river to flow in a particular direction natural magnetism is used for pointing north by a compass soil is formed by weathering if a mineral can be scratched by a fingernail then that mineral is soft if a substance absorbs solar energy then that substance will increase in temperature breath contains water vapor weathering usually occurs over a period of many years a star is a source of light through nuclear reactions a star is made of gases a reflector is used to reflect light especially on vehicles high means great in altitude a flashlight requires a source of electricity to produce light endangered means low in population a rotation of the earth on earth 's axis takes 1 day an example of an inherited behavior is a bird building a nest a balloon contains gas the sun causes water to evaporate more quickly by adding heat a bubble contains gas the sun is the source of solar energy called sunlight winter in the northern hemisphere is during the summer in the southern hemisphere coal mine is a source of coal under the ground in the food chain process some types of plankton have the role of producer as time spent taking a shower decreases, water used will decrease a compass 's needle lines up with earth 's magnetic poles a stopwatch is used to measure time coal is used to produce electricity by burning in coal-fire power stations arteries transport nutrients throughout the body an example of a reproductive behavior is salmon returning to their birthplace to lay their eggs a graduated cylinder is a kind of instrument for measuring volume of liquids or objects a rainbow is formed by refraction of light by splitting light into all different colors fossil fuels forming occurs over a period of 300000000 years which is considered a very long time to a human as lightness in color of an object increases, the ability of that object to reflect light will increase the stars in the night sky are very far away from the earth wind causes erosion the sun is located directly overhead at noon a solar panel converts sunlight into electricity