MALAMUTE : A Multilingual, Highly-granular, Template-free, Education-based Probing Dataset (2025)

Abstract: Language models (LMs) have excelled in various broad domains. However, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific, granular areas of knowledge. Existing cloze-style benchmarks, commonly used to evaluate LMs’ knowledge, have three major limitations. They: 1) do not cover the educational domain; 2) typically focus on low-complexity, generic knowledge or broad domains, which do not adequately assess the models’ knowledge in specific subjects; and 3) often rely on templates that can bias model predictions. Here, we introduce MALAMUTE, a multilingual, template-free, and highly granular probing dataset comprising expert-written, peer-reviewed probes from 71 university-level textbooks across three languages (English, Spanish, and Polish). MALAMUTE is the first education-based cloze-style dataset. It covers eight domains, each with up to 14 subdomains, further broken down into concepts and concept-based prompts, totaling 33,361 university curriculum concepts and 116,887 prompts. MALAMUTE’s fine granularity, educational focus, and inclusion of both sentence-level and paragraph-level prompts make it an ideal tool for evaluating LMs’ course-related knowledge. Our evaluation of masked and causal LMs on MALAMUTE shows that despite overall proficiency, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development.

## introduction language models (lms) have shown remarkable abilities in various domains @xcite . however, to ensure their safe and effective integration into real-world educational settings, they must demonstrate proficiency in specific, granular areas of knowledge @xcite . for instance, a lm may excel in general knowledge of mathematics, but struggle with the nuances of calculus or algebra. hence, we need precise and targeted evaluation methods for lms' knowledge in these critical areas. the need for such evaluations is pressing, as lms are increasingly being considered for use in classrooms, where their performance can have a direct and profound impact on student learning outcomes @xcite . moreover, lms are being explored for a range of educational applications, including adaptive learning @xcite , intelligent tutoring systems @xcite , and automated grading @xcite . in order to ensure that these systems are both effective and safe, we must develop evaluation methods that can accurately assess their knowledge in specific, curriculum-aligned areas. currently, evaluating lms' knowledge is a significant research focus. two prominent evaluation methods are question answering (qa) @xcite @xcite and cloze-style (cs) probing @xcite . cs probing, in particular, has gained popularity, where models receive a prompt with a masked portion, such as "dante was born in @xcite ," and are tasked with predicting the masked string. this approach measures models' lower bound of knowledge @xcite . however, existing cs benchmarks have limitations. firstly, there is a notable lack of cs datasets that are specifically designed for educational settings. secondly, the prompts in existing datasets typically focus on low-complexity, generic knowledge or broad domains, which do not adequately capture models' knowledge in specific subjects @xcite @xcite @xcite . furthermore, most existing clozestyle probing datasets rely on expert-written templates, which have been shown to bias model predictions @xcite @xcite . to address these limitations, we introduce malamute, a multilingual, highly granular, template-free, education-based probing dataset. derived from 71 university-level textbooks across three languages (english, spanish, and polish), malamute covers eight domains, each with up to fourteen subdomains, broken down into individual curriculum concepts and concept-centric prompts. with 33,361 curriculum concepts and 116,887 expert-written, peer-reviewed cs prompts, malamute offers an unprecedented level of granularity and alignment with educational curricula, making it an ideal tool for evaluating lms' knowledge in educational contexts. see figure 1 for examples. malamute is also the first probing dataset to include both sentence-level and paragraph-level prompts, which provides a more comprehensive evaluation of lms' knowledge, establishing a new higher lower-bound of knowledge for masked lms (mlms). lastly, malamute also comes with a number of parallel domains, subdomains, and concepts across multiple languages. we evaluate malamute on 8 causal lms (clms) and 5 mlms and highlight the need for granular datasets: despite many models' overall proficiency on the dataset, they have significant gaps in knowledge when examined closely on specific subjects, hindering their safe use in classrooms and underscoring the need for further development. ## malamute in this section, we introduce malamute, the first cs educational knowledge probing dataset. we also describe our systematic and scalable approach to generating high-quality probing prompts directly from reputable academic sources. see ## results & discussion due to space constraints, we provide a summary of our results in ## conclusion we introduce malamute, a highly-granular, multilingual, and template-free educational probing dataset that exposes significant knowledge gaps in existing lms. as lms are increasingly considered for classroom use, where their performance can directly impact student outcomes, it is essential to develop evaluation methods that assess knowledge in specific, curriculum-aligned areas. mala-mute highlights the need for more specialized benchmarks that go beyond general knowledge and ensure models can support, rather than hinder, student learning. we hope malamute inspires the development of more educational benchmarks, driving the field toward safer, more reliable lms for educational contexts. by addressing these gaps, we can better align language models with the needs of educational systems and improve their utility in real-world applications.