M ode L ing: A Novel Dataset for Testing Linguistic Reasoning in Language Models (2025)

Abstract: We introduce ModeLing, a novel benchmark of Linguistics Olympiad-style puzzles which tests few-shot reasoning in AI systems. Solving these puzzles necessitates inferring aspects of a language’s grammatical structure from a small number of examples. Such puzzles provide a natural testbed for language models, as they require compositional generalization and few-shot inductive reasoning. Consisting solely of new puzzles written specifically for this work, ModeLing has no risk of appearing in the training data of existing AI systems: this ameliorates the risk of data leakage, a potential confounder for many prior evaluations of reasoning. Evaluating several large open source language models and GPT on our benchmark, we observe non-negligible accuracy, demonstrating few-shot emergent reasoning ability which cannot merely be attributed to shallow memorization. However, imperfect model performance suggests that ModeLing can be used to measure further progress in linguistic reasoning.

## introduction large language models (llms) perform well on (at least some) evaluations of both few-shot multilingual adaptation @xcite and reasoning @xcite . however, evaluating the intersection of these two skills-multilingual few-shot reasoning-is difficult: even relatively low-resource languages can be found in large training corpora, raising the concern that when we intend to evaluate a model's ability to generalize to a new language, that language may have in fact been present during the model's training. if such language contamination @xcite has occurred, apparent cases of few-shot reasoning could actually be due to memorization. ## related work olympiad-style linguistics puzzles originated as a way to introduce high school students to linguistic reasoning concepts @xcite . we suggest using such puzzles to evaluate few-shot linguistic reasoning in ai systems. there is a relative scarcity of literature evaluating language models on linguistics puzzles. puz-zling machines ( şahin et al., 2020) presents a dataset of rosetta stone puzzles drawn from several linguistics olympiads. however, as these puzzles are generally available online at their respective contest websites, and llms are typically trained on large amounts of data scraped from the internet, data leakage is a concern (i.e., the problems may have been present in llm training sets). a similar evaluation task based on constructed languages appears in the big-bench benchmark @xcite . however, its synthetic languages are not representative of natural languages' nuance and do not fully simulate their irregularities. to address these limitations, our dataset is entirely composed of newly-written problems and only contains natural, if extremely low-resource, languages. additionally, unlike previous work, our dataset provides fine-grained dataset categorizations (breaking down problems in terms of their linguistic patterns and difficulty levels). as a result, our dataset can be leveraged for closer analysis of language model linguistic reasoning capabilities, free of confounding memorization factors. ## experiments we evaluated a set of open source models: gemma (7b), alpaca (7b), llama 2 (7b), llama 3 (8b and 70b), vicuna (13b), mistral (7b), and mixtral (8x7b and 8x22b) using together.ai's infrastructure @xcite @xcite @xcite . we also evaluated two of the most capable gpt models (gpt-3.5 and gpt-4o) on our dataset on june 16, 2024 @xcite . we evaluated under the following conditions: minimal prompt, which uses a brief, basic prompt specifying the task; hand-tuned prompt, which uses a prompt hand-tuned by an iol medalist; basic chain-of-thought @xcite , where the model is encourage to think step-by-step but is not given an example of how to do so; and full chain-of-thought @xcite , where the prompt illustrates step-by-step reasoning using an example linguistics puzzle based on spanish (appendix a.3). evaluation throughout all experiments, we used exact match accuracy, avoiding bleu and related metrics as they poorly distinguish word ordering and subtle morphological differences @xcite . exact matching is feasible because our problems only have one correct solution, unlike more general translation tasks. ## results and analysis we observe strong performance from gpt-4, as well as the largest open source models, including mistral-8x22b and llama-3-70b ( human difficulty vs. model difficulty difficulty for models generally follows the difficulty ratings from our human experts (figure 4 ). this suggests that as large models continue to improve, we can scale our benchmark by producing more challenging problems; since even the hardest problems in our benchmark are relatively easy by linguistics olympiad standards, there remains plenty of room for making the benchmark more challenging. accuracy by problem type certain problem categories are challenging for all models, with possessive and semantic problems being harder than noun/adjective ordering and basic word order (figure 3 ). this could be due to their inherent difficulty; semantic and possessive problems are generally the most difficult according to our expert human raters (average difficulty of 5.0 and 4.54) while noun/adjective and basic word order problems are considerably easier (average difficulty of 2.61 and 3.18). ## conclusion we have introduced modeling, a dataset designed to evaluate llms' capacity to reason analytically in unseen foreign languages. we believe that the approach used to develop modelinggiven its use of languages that occur very rarely on the internet and its capacity to be extended to more challenging cases-has a strong potential to serve as a durable approach for evaluating reasoning.