Inductive Linguistic Reasoning with Large Language Models (2025)

Abstract: Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models’ knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1% and Llama-3.1-405B-Instruct by 5.9% over chain-of-thought approaches. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving state-of-the-art results across nearly all problem types and difficulty levels in the LINGOLY dataset.

## introduction as the capabilities of large language models (llms) continue to grow, it is necessary to develop ways of testing the boundaries of their ability to reason over a wide range of languages. adapting language models to low-resource languages is challenging due to a lack of high-quality annotated data in the target language for supervised fine-tuning. this has led to zero-shot and few-shot transfer learning approaches being more commonly employed @xcite @xcite . however, given the emergence of the in-context learning phenomenon, we hypothesize that this behavior can be used to enable few-shot generalization to new languages. in this work, we explore the task of linguistic reasoning, using puzzles akin to the international linguistics olympiad (iol). notably, in these puzzles, the target language is often extremely lowresource or functionally extinct @xcite . while prior work has largely examined the effect of vanilla in-context learning with english-target and target-english exemplars, chain-of-thought prompting, and traditional neural machine translation methods @xcite , we believe that generating auxiliary exemplars can guide the model to more effectively learn grammar similarities over a language family. we introduce an approach based on analogical prompting @xcite , which uses strong language models to self-generate exemplars of relevant problems given the test instance and performs in-context learning conditioned on those demonstrations. in our setting, the knowledge retrievallike nature of analogical prompting allows us to test models' parametric understanding of language families, performing inference to solve using both sets of demonstrations. we evaluate our approach on the modeling @xcite dataset, consisting of unseen iolstyle problems. we find that strong models such as gpt-4o and llama-3.1-405b-instruct can identify the language family, similar languages within said family, generate exemplars in those similar languages, and apply them to solve the test puzzle. furthermore, while weak models do not benefit significantly from using strong model-generated exemplars, the converse yields improvements using exemplars produced by weaker yet specialized multilingual models (e.g. aya-35b). our findings show that the ability of the model to deduce and apply rules, following inductive learning from the exemplars, largely influences performance. we suggest that the linguistic reasoning task serves figure 1 : an illustration of our 2-stage analogical prompting approach, translating a phrase in montenegrin to english. while prior works would solely provide exemplars translating between the source language and english and perform in-context learning, our method seeks diverse exemplars. model m 1 first identifies the language family (slavic) and higher-resource languages in the family which the model has knowledge of (croatian), then produces exemplars in those languages. finally, both the original and generated set of exemplars are passed with the test puzzle to model m 2 to perform the translation. m 1 = m 2 yields the self-generated analogical reasoning setting. as valuable testbed for research on new reasoning methods, to uncover how the skills which drive logical thinking may be imbued. ## analogical prompting for linguistic reasoning analogical prompting @xcite avoids the need for annotated exemplars by relying on a strong model to generate exemplars which are related to the test instance, but are sufficiently diverse relative to one another and the test sample. our approach follows the human system 2 thinking framework of slow, deliberate reasoning @xcite . in chain-of-thought prompting for these puzzles, the model performs in-context learning with the given exemplars, learning the rules governing the language by induction, including the meaning of particular words, and using deduction to apply these rules to the test sample. this approach is supported by prior works demonstrating the ability of lms to learn rules and attempt to apply them @xcite . furthermore, we do not have access to complete grounding sources of human-written rules governing these low-resource languages, so we must rely on the lm to identify and generate these rules itself. however, as we expect the model to have little to no prior knowledge about the target language foot_0 , we seek to leverage other languages with similar grammar structure which the model has learned to guide the language model's reasoning process. we use language families as a taxonomicallygrounded means of identifying similar languages to the target. the generated exemplars provide a source of reasoning support to the model, enabling it to perform inductive reasoning in a cross-lingual manner over all exemplars, and then deduce from its shared understanding. in figure 1 , given the test instance and the provided exemplars in montenegrin, we leverage the model to (i.) identify the family of montenegrin (slavic language family), (ii.) select a few languages in the slavic language family, and (iii.) generate example puzzles with their solutions in those selected languages, e.g. croatian. then, the provided exemplars and the synthetic demonstrations are given to the model to solve the test puzzle. ## methods we explore a number of sampling methods across various language models to assess their performance on reasoning over unknown languages. ## results we report exact match (em) scores for all experiments performed. chrf2 @xcite , a character n-gram f-score measure, and corpuslevel bleu scores @xcite are recorded in appendix c. we do not treat these as primary metrics as bleu ignores word ordering nuances amidst short responses in machine translation, which is integral to measuring correctness in the puzzles we explore @xcite , and we find the chrf scores to be noisy relative to em scores. smaller models with weaker instruction-following capabilities often failed to produce their output in the exact desired format specified in the prompts. to ensure reliable exact match scores while some responses may have parsing issues relative to the expected format, the authors of this work manually examined each response to confirm whether the output generated contains the target response. to standardize across our evaluation procedure, this was performed for all experiments; however, this method was not required for stronger models whose responses exactly followed the desired format. ## discussion we apply analogical prompting to test the efficacy of inductive reasoning from diverse exemplars for challenging linguistic puzzles. we find that language models can indeed follow grammatical similarities within a language family to generate analogical exemplars, and attempt to apply them adeptly. this yields improved performance across analogical exemplar generators, both with strong models (gpt-4o, llama-3.1-405b) and weaker multilingual models (aya-35b), with said strong models as the deducers. we also show that llama-3.1-405b-instruct is the best current model for linguistic reasoning, as the first model to achieve over 70% on the modeling benchmark with our approach, attributable to the auxiliary exemplars generated. furthermore, the ability of smaller and specialized multilingual models (aya) to generate coherent analogical exemplars, which improve frontier models over their own self-generated exemplars, is promising towards developing widely-available multilingual reasoners. we also show that our method expands to other linguistic puzzle tasks on the lin-goly dataset, achieving substantial gains with gpt-4o across nearly all categories. notably, our analogical prompting approach enables models to solve problems in categories where it was previously unsuccessful at solving any problems, further highlighting its potential. research at the intersection of machine translation and reasoning is important from a societal perspective. with large language models being adopted widely, the need for multilingual capabilities and rapid adaptation grows, and our work proposes an effective method by which this can be performed at test-time, notably demonstrating evidence that models can follow language similarities. the gaps in the performance of current models relative to human reasoning, due to a deficiency in deductively applying rules induced from exemplars suggest that the linguistic reasoning task is an exciting and challenging evaluation setting for lm reasoning at large. given our results with language isolates, we suggest that future efforts in multilingual adaptation be placed in identifying techniques to guide models to support typologically unique languages. we hope that these findings can inspire future model releases to include evaluation on challenging multilingual tasks such as these puzzles, and research on reasoning can further explore the multilingual setting in depth. limitations. we note that the reliance on exact match scoring as our primary signal of performance is not ideal, as it is a binary indicator. we have sought to examine other metrics which correspond to "partial credit" such as chrf2 and bleu; however, there are flaws in these methods as well. a stronger human understanding of the rules which these extremely low-resource languages follow could guide us to better metrics, especially in capturing semantic meaning and word ordering inversions, where appropriate. for instance, some languages might retain the same meaning while inverting the word order -exact match is sensitive to this, and while chrf2 and bleu are not, we should only be insensitive to ordering for languages which follow this property. we also recognized that the iol 2024 problems could not be used as a benchmark with our method, as they require multimodality -our method only analyzes unimodal text problems. another limitation of our work is that we do not have a reliable means of verifying the correctness of analogical exemplars, nor contrasting the quality of exemplars generated across models to determine the best analogical generator model. access to expert annotators who could acutely assess where a mistake was made in the model's reasoning process also would have been helpful to yield further insights into the fallacies of current models' linguistic reasoning. this naturally applies as well to studying the similarity between generated puzzles and the test puzzle, as this would require annotators to be acutely aware of the relationship between the two languages' grammar constructs to assess their difficulty. nonetheless, developing alternative means of studying correctness without reliance on human annotation is of interest for future work. we have sought to address this in part by studying a more easily verifiable signal: namely, the language families identified by each model, and e.g. if the model correctly identifies language isolates. nonetheless, our most effective deducer models are able to leverage exemplars generated by models of various sizes for improved linguistic reasoning.