M ode L ing: A Novel Dataset for Testing Linguistic Reasoning in Language Models (2024)

Abstract: Large language models (LLMs) perform well on (at least) some evaluations of both few-shot multilingual adaptation and reasoning. However, evaluating the intersection of these two skills—multilingual few-shot reasoning—is difficult: even relatively low-resource languages can be found in large training corpora, raising the concern that when we intend to evaluate a model’s ability to generalize to a new language, that language may have in fact been present during the model’s training. If such language contamination has occurred, apparent cases of few-shot reasoning could actually be due to memorization. Towards understanding the capability of models to perform multilingual few-shot reasoning, we propose modeLing, a benchmark of Rosetta stone puzzles. This type of puzzle, originating from competitions called Linguistics Olympiads, contain a small number of sentences in a target language not previously known to the solver. Each sentence is translated to the solver’s language such that the provided sentence pairs uniquely specify a single most reasonable underlying set of rules; solving requires applying these rules to translate new expressions (Figure 1). modeLing languages are chosen to be extremely low-resource such that the risk of training data contamination is low, and unlike prior datasets, it consists entirely of problems written specifically for this work, as a further measure against data leakage. Empirically, we find evidence that popular LLMs do not have data leakage on our benchmark.

## introduction large language models (llms) perform well on (at least some) evaluations of both few-shot multilingual adaptation @xcite and reasoning @xcite . however, evaluating the intersection of these two skills-multilingual few-shot reasoning-is difficult: even relatively low-resource languages can be found in large training corpora, raising the concern that when we intend to evaluate a model's ability to generalize to a new language, that language may have in fact been present during the model's training. if such language contamination @xcite has occurred, apparent cases of few-shot reasoning could actually be due to memorization. towards understanding the capability of models to perform multilingual few-shot reasoning, we propose modeling, a benchmark of rosetta stone puzzles @xcite . this type of puzzle, originating from competitions called linguistics olympiads, contain a small number of sentences in a target language not previously known to the solver. each sentence is translated to the solver's language such that the provided sentence pairs uniquely specify a single most reasonable underlying set of rules; solving requires applying these rules to translate new expressions (figure 1 ). modeling's languages are chosen to be extremely low-resource such that the risk of training data contamination is low, and unlike prior datasets @xcite , it consists entirely of problems written specifically for this work, as a further measure against data leakage. empirically, we find evidence that popular llms do not have data leakage on our benchmark (section 2.1). ## dataset modeling comprises 48 rosetta stone puzzles based on 19 extremely low-resource languages from diverse regions. all problems were written by here are some phrases in ayutla mixe: ëjts nexp. → i see. ## experiments we evaluated six gpt models (gpt-3 {ada, babbage, curie, davinci}; gpt-3.5; and gpt-4) on our dataset on august 13, 2023 @xcite . we evaluated under the following conditions: minimal prompt (a brief, basic prompt specifying the task); hand-tuned prompt (a prompt fine-tuned by an international linguistics olympiad medalist); basic chain-of-thought @xcite (which encourages models to think step-by-step); and full chain-of-thought @xcite ) (which provides an example of reasoning step-by-step). we report exact-match accuracies taken over all individual questions. we observe strong performance from davinci, gpt-3.5, and gpt-4 ( model minimal prompt hand-tuned prompt basic cot full cot ada .000 .004 .011 .000 babbage .011 .011 .004 .018 curie .015 .018 .015 .022 davinci .496 .485 .490 .514 gpt-3.5 .404 .412 .401 .397 gpt-4 .588 .591 .589 .607 ## conclusion we have introduced modeling, a dataset designed to evaluate llms' capacity to reason analytically in unseen languages. we believe that the approach used to develop modeling-given its use of languages that occur very rarely on the internet and its capacity to be extended to more challenging cases-has a strong potential to serve as a durable approach for evaluating reasoning.