S ci E x: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading (2024)

Abstract: With the rapid development of Large Language Models (LLMs), it is crucial to have benchmarks which can evaluate the ability of LLMs on different domains. One common use of LLMs is performing tasks on scientific topics, such as writing algorithms, querying databases or giving mathematical proofs. Inspired by the way university students are evaluated on such tasks, in this paper, we propose SciEx - a benchmark consisting of university computer science exam questions, to evaluate LLMs’ ability on solving scientific tasks. SciEx is (1) multilingual, containing both English and German exams, and (2) multi-modal, containing questions that involve images, and (3) contains various types of freeform questions with different difficulty levels, due to the nature of university exams. We evaluate the performance of various state-of-the-art LLMs on our new benchmark. Since SciEx questions are freeform, it is not straightforward to evaluate LLM performance. Therefore, we provide human expert grading of the LLM outputs on SciEx. We show that the free-form exams in SciEx remain challenging for the current LLMs, where the best LLM only achieves 59.4% exam grade on average. We also provide detailed comparisons between LLM performance and student performance on SciEx. To enable future evaluation of new LLMs, we propose using LLM-as-a-judge to grade the LLM answers on SciEx. Our experiments show that, although they do not perform perfectly on solving the exams, LLMs are decent as graders, achieving 0.948 Pearson correlation with expert grading.

## introduction in recent years, large language models (llms) have proven their usefulness across a wide range of tasks, from conversational agents to code generation @xcite @xcite . given the fast pace of development in the field, with an increasing number of llms being trained and released, it is important to have indicators of llm performance on different domains. this can be achieved by establishing evaluation benchmarks that assess the capabilities of llms across diverse use cases. one use case of llms is to handle scientific tasks. some previous works have introduced benchmarks containing questions on science topics @xcite @xcite @xcite . however, these benchmarks are limited to multiple-choice questions. this restricts the variability of questions, such as instruction-follow ones like "write a mathematical proof for this statement ...". additionally, it is difficult to ask certain types of questions in a multiple-choice way without including the answer in the question itself. multiplechoice benchmarks therefore create a gap between testing and actual usage, since they only evaluate whether the llms choose the correct answer, whereas in real life, the users are more likely to ask open-ended questions to the llms. in contrast, some other works have introduced freeform question benchmarks. these works either convert multiple-choice questions to freeform questions @xcite , or focus on a specific type of problem such as answering questions related to a paper @xcite , thus still limiting the variability of the questions. in this paper, we introduce a new benchmark, termed sciex (scientific exams), designed to evaluate this capability. inspired by the way students are evaluated in university, we created the benchmark by evaluating the performance of llms on university computer science exams. sciex's questions are in various formats, from multiple choice to open-ended, thus making it suitable to evaluate llm's capabilities of generating free-text answers that fit the requirements of the questions. it is multilingual, containing exams in both german and english. it is multimodal, as exam questions can also contain figures. the set of questions is a good mix of different difficulty levels since they are designed for university exams. this enables us to evaluate llms on different levels, and we found that stronger llms tend to perform better on more difficult questions. unlike the previous multiple-choice benchmarks, the questions in sciex are freeform, making it nontrivial how to evaluate the llm output. therefore, we make use of expert grading, i.e., having the lecturers grade the llm output the same way they would grade student answers. we also ask the experts to perform qualitative analysis of the llm output. with expert grading, we provide a highly reliable way of evaluating llms, which is more reliable than previous work that uses crowdsourced evaluation. expert grading by lecturers also provides an opportunity to compare llms' performance to university student performance in a similar setting. we find that the stronger llms, i.e., claude and gpt-4v, are able to outperform the student average. however, they are still far from perfect, achieving only 59% across sciex exams. since new llms are constantly being released, we cannot fully rely on expert grading for evaluation. therefore, we provide an automatic grading scheme by using llm as a judge so that future llms can also be evaluated on sciex. interestingly, we find that, although llms do not perform too well as examinees, they perform well as graders, achieving over 0.948 pearson correlation to expert grading in the best setting. in summary, our contributions are as follows: ## the sciex benchmark the components of sciex are as follows. univeristy exams sciex contains university computer science exams in a unified json format. the exams are taken from the following computer science courses at the karlsruhe institute of technology from the 2022/2023/2024 semesters: descriptions of the exams are in appendix a. in total, sciex contains 10 exams, among which 5 exams are in english and 7 exams are in german (some exams are provided in both languages). there are in total 154 unique questions. each question is annotated with (1) the maximum points that can be achieved and (2) a difficulty level among easy, medium, hard. most questions are provided with gold reference answers and average student performance. the detailed per-question statistics are shown in ## experiments in this section, we describe our experiments and results. for prompting the proprietary llms, we use their apis, namely openai foot_1 and anthropic foot_2 . for the open-source models, we obtain model checkpoints from the huggingface foot_3 model hub. we perform inference with the llms using llama.cpp foot_4 with the default sampling strategy. the experiments with open-sourced models are conducted on an nvidia rtx a6000 gpu with 48gb vram. for our analysis, we consider the exam-level and question-level grades. an exam-level grade is the sum of the grades of all questions in the exam. ## conclusion in this paper, we proposed sciex -a benchmark consisting of scientific university exams, along with expert grading and automatic grading, to evaluate the abilities of llms on science topics. sciex is multilingual, multi-modal, and contains a variety of free-form questions. our experiments show that sciex is still quite challenging for current llms, where the best llm can only achieve 59.4% of the exam score on average. despite that, the llms perform well as graders, achieving 0.948 pearson correlation to the expert grades. this is a promising observation, since we can use strong llms for automatic grading of new llm examinees on sciex, rather than relying on expert grading. we encourage the research community as well as llm developers and users to make use of sciex for evaluating llms' scientific capabilities.