T heorem QA : A Theorem-driven Question Answering Dataset (2023)

Abstract: The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models’ capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4’s capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs’ capabilities to solve challenging science problems.

## introduction a long-standing goal of ai systems is to help human beings solve challenging problems, especially more domain-specific problems. to benchmark the progress towards this goal, researchers propose to evaluate ai systems' performance on different math word problem (wmp) datasets. in recent years, there has been a plethora of wmp datasets @xcite , which we include in in this paper, we propose the first theoremdriven qa dataset built on university-level theorems across math, physics, ee&cs, and finance. the whole collection process takes two steps: (1) we first enumerate roughly 400 theorems in different subfields like algebra, number theory, graph theory, information theory, etc, (2) we ask domain experts to search for questions regarding these theorems from different sources like internet and textbooks. the domain experts will adjust these questions to ensure the answers follow the desired format for the ease of automatic evaluation. through the careful construction process, we collected 800 high-quality question-theorem-answer triples as our final release version. we evaluate a wide spectrum of instructionfinetuned language and code models including gpt @xcite , claude @xcite , llama @xcite , pythia @xcite , codegen @xcite , glm @xcite , starcoder @xcite , and codet5+ @xcite figure 1 : the overview of theoremqa and the prompting strategies adopted. on our dataset. we adopt two prompting methods: chain-of-thoughts (cot) @xcite and program-of-thoughs (pot) @xcite to prompt the large language models. we also investigate how to infuse the theorem into the thought process of llms and how to present the multimodal inputs to the llms. in the course of our experiments, several notable observations were made. first, gpt-4 (ope-nai, 2023) significantly outperformed all existing models, reaching an accuracy level of 51% when combined with program-of-thoughts prompting. trailing behind gpt-4, the second most effective model was chatgpt, achieving an accuracy of 35% through the same prompting method. additionally, our human evaluation determined that half of gpt-4's errors are caused by minor mistakes like calculation errors, rounding errors, etc. we believe these errors could be easily rectified with a more deliberate prompting strategy or human intervention. this suggests that there is still significant headroom for gpt-4 to achieve with more deliberate prompting strategies. secondly, we found that all open-source, instruction-tuned language and code models scored below 15% in accuracy, barely exceeding the random guess baseline of 10%. our human evaluation reveals that open-source models like alpaca are making errors mainly due to their ignorance of the theorem, where 90% of the errors are not rectifiable. this stark gap between gpt and opensource models suggests that further enhancement strategies, such as science-focused pre-training or fine-tuning, should be considered to narrow the performance disparity. thirdly, we explored the potential to do theorem-augmented generation. however, the simple strategy of concatenation did not yield a significant improvement. we conjecture that a more complex integration strategy may be needed to achieve more gains. lastly, we examined the performance of various multi-modal instruction-tuned models on the multimodal subset of the theoremqa dataset. surprisingly, these models did not demonstrate significant performance gains over their textonly counterparts. this is mainly due to the unnaturalness of the image, which consists of lots of diagrams and text. such images are not well captured by existing visual encoder models. to sum up, our contributions are three folds: ## dataset our dataset collection pipeline contains two steps: theorem enumeration our aim was to encompass a wide range of theorems. to this end, we began by prompting large language models (llms), specifically gpt-4 (openai, 2023), to enumerate popular subfields in mathematics, physics, finance, and electrical engineering & computer science. the covered subfields are listed in figure 4 . subsequently, we prompted gpt-4 to propose plausible university-level theorems relevant to these subfields. for instance, within the 'calculus' subfield, gpt-4 might suggest the 'intermediate value theorem', 'rolle's theorem', and so on. after gath-ering an extensive list of theorems, we assembled a team of domain experts (holders of masters and phds in statistics, electrical engineering, computer science, and finance) to refine the theorem inventory and supplement any omitted theorems. ultimately, we collected approximately 400 theorems, encapsulating a diverse range of topics within these fields. we then delegated these theorems to nine domain experts, instructing them to locate question/answer pairs from varied sources. during the annotation process, a small number of theorems were discarded due to their evaluation complexity. question annotation our problems were sourced from websites, books, or devised by the experts themselves. one challenge we encountered was the potential for questions found online to have been included in the training data. to mitigate this 'data contamination' issue, we encouraged domain experts to modify these questions. another challenge arose from questions with answers in symbolic form, matrix form, figure form, etc. these presented significant obstacles for our automatic evaluation. to overcome this, we instructed domain experts to alter the question so the answer would be limited to the following forms: ## method our method for addressing these demanding questions in the theoremqa is comprised of several distinct modules, as outlined in figure 1 : 2022b): this strategy prompts the language model to initially generate a step-by-step thought process, eventually leading to the final answer. by delegating computational tasks to an external executor, the problem-solving process is considerably enhanced in its reliability. this improvement results in remarkable advancements in existing math datasets being reported in @xcite . ## conclusion in this paper, we propose the first theorem-driven science question-answering dataset and evaluate different llms on it. though gpt-4 can achieve strong performance on our new dataset, the existing open-source llms are still struggling to achieve reasonable performance. we conjecture it is essen-tial to leverage more science-related pre-training or fine-tuning to close the gap. on the hand, we found that the multimodal science questions are still extremely challenging for the existing visual llms. we believe more specialized visual encoding models are needed to better represent diagrams in these science questions.