SweSAT -1.0: The Swedish U niversity Entrance Exam as a Benchmark for Large Language Models (2025)

Abstract: This introduces SweSAT-1.0, a new benchmark dataset created from the Swedish university entrance exam (Högskoleprovet) to assess large language models in Swedish. The current version of the benchmark includes 867 questions across six different tasks, including reading comprehension, mathematical problem solving, and logical reasoning. We find that some widely used open-source and commercial models excel in verbal tasks, but we also see that all models, even the commercial ones, struggle with reasoning tasks in Swedish. We hope that SweSAT-1.0 will facilitate research on large language models for Swedish by enriching the breadth of available tasks, offering a challenging evaluation benchmark that is free from any translation biases.

## introduction the recent progress in language modeling has significantly expanded the generalization capabilities of large language models (llms). models such as llama 3.1 @xcite , gemma @xcite , and gpt-4 @xcite have demonstrated remarkable performance across a wide range of nlp tasks, exceeding the expectations researchers held just a few years ago. consequently, many existing benchmarks are found to be inadequate due to their task-specific nature, focusing narrowly on traditional classification problems and failing to capture the full spectrum of language understanding capabilities of modern llms. benchmarks such as superglue @xcite and xtreme @xcite predominantly assess specific nlp tasks, limiting their ability to evaluate the broader, more generalized language capabilities that contemporary llms are seemingly capable of. this issue is even more crucial for languages other than english, which are often evaluated on translated benchmarks that are prone to numerous biases and quality issues. to address this gap, we follow the tradition of using standardized exams @xcite and introduce the first version of a swedish benchmark called swesat-1.0 foot_0 sourced from the swedish university entrance exam, swedish scholastic aptitude test ('högskoleprovet' in swedish). the exam encompasses both verbal and quantitative reasoning tests across several sub-categories, such as reading comprehension, mathematical problem solving, and logical reasoning. swesat-1.0 is sourced from the last eight exams over the past five years. the benchmark is prepared through automatic parsing of the exam files, followed by manual checks to correct any parsing errors. it currently comprises 867 questions and has the following advantages over most existing benchmarks: i) it is free from translation biases and culturally irrelevant content; ii) it allows researchers to control for data contamination foot_1 as the exact administration dates of the exams are known; iii) it broadens the range of tasks available for evaluation in swedish; and iv) it indirectly allows a comparison against the real exam takers as the results are publicly available. in addition to presenting the benchmark, we evaluate a wide range of popular multilingual and swedish-oriented llms. the results show that while multilingual llms outperform their swedish-oriented counterparts, even the commercial models fail at solving the reasoning tasks in swedish, highlighting a crucial shortcoming of the existing llms. we hope that the benchmark will section total # questions description ord 160 vocabulary: tests the understanding of in-domain words and synonyms. l äs 160 reading comprehension: assesses the ability to make inference from a text. mek 160 sentence completion: assesses the ability to complete sentences via cloze tests. xyz 157 mathematical problem-solving: tests arithmetic, algebra, geometry, statistics, and functions. kva 140 quantitative comparisons: measures the ability to compare quantities in math concepts. nog 90 data sufficiency: evaluates the ability to determine if data is sufficient for solving a problem. ## related work there are a few benchmarks specifically designed to evaluate nlp models in swedish, with superlim @xcite and scandeval @xcite being the most prominent examples. created as the swedish counterpart to superglue, super-lim is a comprehensive test suite that consists of 15 tasks, such as word analogy, pronoun resolution, and text summarization. 3 if not adapted from english through translation, the featured datasets are either constructed by reformatting pre-existing tasks or created from scratch using pre-existing corpora. the reliance on pre-existing datasets raises concerns about data contamination, and the use of translation could introduce bias, which signals the need for new and complementary evaluation datasets. scandeval @xcite , on the other hand, provides a multilingual evaluation suite spanning a subset of north germanic languages, among them swedish. despite broad task coverage, the majority of scandeval datasets are revisited versions of existing datasets, which again raises concerns about whether data contamination and the use of machine translation could undermine the evaluation process. ## dataset description swesat-1.0 is a benchmark dataset sourced from the publicly available swedish scholastic aptitude test, foot_3 a standardized swedish university entrance exam. the exam is written and administered by the swedish council for higher education and used for admission to higher education in sweden. the exam consists of two main parts: verbal and quantitative, each containing four sections. each exam includes 160 multiple-choice questions taken over a single day, lasting almost 8 hours (including breaks). this exam has been selected for its high quality; since it is written specifically to assess students' verbal and quantitative reasoning skills in swedish, we eliminate the risk of cultural and linguistic biases. sample questions can be found in appendix c. we refer interested readers to @xcite for more detailed information on the exam. ## baselines in this section, we evaluate the performance of a range of llms, each with different levels of swedish coverage during training, on swesat-1.0. the primary purpose of this baseline evaluation is to evaluate the dataset itself by analyzing how some of the most popular llms perform on its tasks to ensure that the dataset is sufficiently challenging and valuable as a benchmark. by doing so, we also provide reference scores for future studies while exploring the current capabilities of llms in swedish. our evaluation includes a range of instruction-tuned open-source models such as gpt-sw3 @xcite , gemma-2 @xcite , aya @xcite , llama 3 and 3.1 @xcite , as well as the commercial gpt-4o-mini (2024-08-06) and gpt-4o (2024-07-18) models @xcite . the entire model list can be found in appendix b. ## conclusion in this paper, we present a comprehensive benchmark to evaluate llms' various abilities in swedish, using the university entrance exam. we believe our benchmark provides a consistent framework for testing llm performance across a range of tasks detailed above, with an option to control for data contamination in model training through exam timestamps. our baseline evaluations reveal the high accuracy of multilingual models across verbal tasks compared to their swedish-centric counterparts -but also the overall weakness of all tested models on the reasoning tasks.