Can LLM s replace Neil de G rasse Tyson? Evaluating the Reliability of LLM s as Science Communicators (2024)

Abstract: Large Language Models (LLMs) and AI assistants driven by these models are experiencing exponential growth in usage among both expert and amateur users. In this work, we focus on evaluating the reliability of current LLMs as science communicators. Unlike existing benchmarks, our approach emphasizes assessing these models on scientific question-answering tasks that require a nuanced understanding and awareness of answerability. We introduce a novel dataset, SCiPS-QA, comprising 742 Yes/No queries embedded in complex scientific concepts, along with a benchmarking suite that evaluates LLMs for correctness and consistency across various criteria. We benchmark three proprietary LLMs from the OpenAI GPT family and 13 open-access LLMs from the Meta Llama-2, Llama-3, and Mistral families. While most open-access models significantly underperform compared to GPT-4 Turbo, our experiments identify Llama-3-70B as a strong competitor, often surpassing GPT-4 Turbo in various evaluation aspects. We also find that even the GPT models exhibit a general incompetence in reliably verifying LLM responses. Moreover, we observe an alarming trend where human evaluators are deceived by incorrect responses from GPT-4 Turbo.

## introduction the surge of large language models (llms) @xcite @xcite marks the beginning of an era of rapid development across a variety of natural language tasks. with the introduction of chatbots powered by instruction-tuned llms, users across diverse domains are becoming reliant on them in day-to-day activities. the increasing usage of llm-based ai assistants in academia has triggered intense discussion recently. multiple reports of inconsistent fragments of text appearing in scientific papers, apparently generated by ai assistants and overlooked due to lack of caution, have can air make shadows? ... since air does not significantly impede the passage of light, it does not create a shadow ... is the complex [ru(bpy)3]cl2 chiral in nature? is tarski's exponential function problem decidable? the complex is not chiral. ... overall arrangement of ligands around the ruthenium center is such that it allows for an inversion center making the complex achiral .... was proven to be decidable. this was established by alex wilkie in 1996 through his proof using model theory, physics chemistry mathematics when different regions of air have different indices of refraction, the air can indeed bend light away from the forward direction and create a shadow. the complex is chiral, with d3 symmetry. it has been resolved into its enantiomers. the molecule does not have an inversion center. the paper does not provide a direct solution to tarski's exponential function problem. instead, it discusses model completeness of the real exponential field. figure 1 : examples of wrong reasonings given by turbo to problems in scips-qa (white rectangles). the corresponding human-generated correct reasonings are provided in green rectangles: (physics -air can cast a shadow under conditions of non-uniform refractive index @xcite ; chemistry -the complex is chiral with d3 symmetry @xcite ; mathematics -the paper discusses the model completeness of the real exponential field and its connection to tarski's problem and the first root conjecture. tarski's problem is an open problem @xcite ). in this work, we seek to close the gaps in evaluating llms towards faithful scientific questionanswering. specifically, we seek to address the following research questions: ## related work llms have demonstrated various types of reasoning capabilities, including logical, commonsense, mathematical, and temporal reasoning @xcite . in this section, we review relevant work that explores the limitations of llms in scientific reasoning. several datasets provide comprehensive assessments of llms' abilities to solve mathematical problems. gsm8k @xcite comprises high-school-level math word problems, while aqua-rat @xcite includes a collection of algebraic word problems. dolphin18k @xcite features elementary-level problems designed to evaluate basic mathematical reasoning capabilities. the math dataset @xcite presents more challenging problems than those in the aforementioned datasets but focuses on simpler mathematical objects compared to the complex scientific concepts found in scips-qa. additionally, ape210k @xcite ) offers a broad range of mathematical problems to further test llms' problem-solving skills. these datasets collectively highlight the strengths and limitations of llms in mathematical reasoning, providing a foundation for understanding their performance in more specialized scientific domains. scienceqa @xcite , sciq @xcite and mmlu @xcite are prominent datasets used to evaluate llms' scientific reasoning capabilities. mmlu-pro @xcite is an improved version of mmlu, @xcite investigated the ability of llms to abstain from answering context-dependent science questions when provided with insufficient or incorrect context. they used datasets such as scienceqa, openbookqa, arc (ai2 reasoning challenge) @xcite , and qasper @xcite to study llm abstention behavior. scienceqa includes school-level and college-level scientific problems requiring relatively simple reasoning capabilities. openbookqa features straightforward open-book style general and scientific reasoning problems, and arc contains grade-school level multiple-choice science questions. in contrast, scips-qa provides a much tougher benchmark for scientific reasoning and explores answer abstinence in boolean questions. ## the scips-qa dataset in this section, we describe the composition of scips-qa and the methodology used to collect the boolean queries that constitute scips-qa. the dataset comprises 742 complex yes/no problems that require expert-level proficiency in scientific reasoning to answer correctly. we include both open and closed problems across subjects -physics, chemistry, mathematics, theoret- mathematics computer science chemistry physics subjects 0.0 0.2 0.4 0.6 0.8 1.0 1.2 accuracy mmlu-pro scips-qa sciq ical computer science, astronomy, economics, and biology. closed questions. these questions have definitive answers supported by scientific literature. we curate a list of complex topics for each subject in scips-qa manually. for each topic, we utilize the wikipedia api to retrieve its summary. subsequently, we provide this summary to gpt-4 turbo, prompting it to generate yes/no problems along with their corresponding answers. the resulting boolean questions undergo manual assessment based on two criteria: (1) requiring scientific reasoning for accurate answers, and (2) correctness of the generated answers. the precise prompt used to generate closed questions can be found in appendix d.1. open questions. these questions lack a definitive answer in the scientific literature. they are manu-ally selected from wikipedia pages and research blogs. further details on how open questions were collected can be found in appendix b. ## experiments this section presents the details of the experiments we performed to answer the research questions (rqs) we set to explore. ## results in this section, we look at the various quantitative results summarized in ## qualitative analysis in this section, we unveil some of the common error patterns we observed in the reasoning passages generated from gpt-4 turbo. while answering open problems, we often found gpt-4 turbo misinterpreting some key scientific relations in questions, hence generating wrong reasoning passages. for the following open problem -'question -is every finitely presented periodic group finite?', gpt-4 turbo responded 'notably, there are examples of infinite, finitely presented periodic groups, such as the tarski monster, which is an infinite group where every proper subgroup is cyclic of a fixed prime order. this example shows that it is possible for a finitely presented periodic group to be infinite'. tarski monster is not a finitely presented periodic group but it is a finitely generated periodic group. gpt-4 turbo misinterprets relations when dealing with complex scientific subjects in the query and produces reasoning passages for a wrong hypothesis. another observation was a lack in reporting contrasting theories to explain a scientific idea. gpt-4 turbo used only one set of theoretical assumptions to answer the query. for question -'is the universe homogeneous and isotropic at large enough scales?', gpt-4 turbo responded -'according to the cosmological principle, the universe is considered to be homogeneous and isotropic at sufficiently large scales'. findings (commonly called axis of evil) have produced newer studies @xcite refuting the cosmological principle proclaiming that the question does not admit an answer. gpt-4 turbo does not take into account such research instances. ## discussion and conclusion our experiments on scips-qa with a diverse array of llms using a comprehensive evaluation strategy reveal several key insights. firstly, existing llms, whether open-access or proprietary, demonstrate a limited understanding of scientific methodologies required to serve as reliable assistants. while the parameter scaling law holds within each llm family, models of similar size across different families are not directly comparable. for instance, meta llama-3 70b models emerge as formidable competitors to much larger gpt mod-els, frequently outperforming gpt-4 turbo in our evaluations. this reiterates earlier findings that parameter scaling alone does not reflect the capabilities of llms and current models, along with their training methodology, are underperforming their 'true' potential @xcite . echoing @xcite 's findings, we observe that powerful llms such as gpt-4 turbo and gpt-3.5 turbo struggle to reliably verify their responses. hallucination detection techniques like selfcheckgpt also prove ineffective in detecting incorrect reasoning posed by strong llms like gpt-4 turbo in complex questions within scips-qa. in fact, we notice a counterintuitive trend where gpt-3.5 turbo assigns lower scores to incorrect responses compared to the stronger gpt-4 turbo. however, the most concerning finding of this paper revolves around how human evaluators perceive llm-generated scientific reasoning. when tasked with evaluating the convincingness of reasoning explanations generated by gpt-4 turbo, human evaluators tend to assign higher ratings to a significant majority of incorrect answers. this aligns with the concern raised by dutta and chakraborty (2023) that current llm-based ai assistants have the potential to propagate widespread scientific misunderstandings if left unchecked. implications for future research. we hope that our proposed dataset, scips-qa, along with the evaluation suit we design in this work, will serve as a valuable benchmark for future llm research. given the growing popularity of generalist as well as domain-specific ai assistants, we envision a positive future focus in building reliable scientific assistants. finally, our findings with human evaluation calls upon further focus in trustworthy ai research.