Evaluating Large Language Models for In-Context Learning of Linguistic Patterns In Unseen Low Resource Languages (2025)

Abstract: This paper investigates the ability of Large language Models (LLMs) in capturing linguistic patterns from unseen languages and applying them to translation between the languages and English within an in-context learning framework. Inspired by the International Linguistics Olympiad (IOL), we create test data consisting of translation puzzles between 40 low resource languages and English. We test the LLMs in two different strategies: direct prompting and step-by-step prompting. In the latter, the puzzles are manually decomposed into intermediate steps to allow LLMs learn and apply linguistic rules incrementally. The results show that this strategy can significantly improve the performance of LLMs, achieving comparable or slightly superior results to humans when translating the unseen languages to English. However, LLMs still struggle with translating English into the unseen languages, typically with complex syntactic rules. We further observe that LLMs cannot deal with languages with object-subject and noun-adjective word order compared to others, reflecting the potential impact imposed by typological features of languages in training data.

## introduction large language models (llms) have demonstrated impressive capabilities for in-context and few-shots learning tasks in natural language processing @xcite . furthermore, they seem to exhibit reasoning abilities in areas such as mathematics and coding @xcite . despite these successes, llms still rely on large amounts of training data and computational resources to achieve practical performance. like many other nlp systems, their applications in low-resource (lr) languages have been limited due to the scarcity of training data @xcite . we are thus interested in how we can leverage their in-context learning and reasoning abilities to process lr language with minimal data. existing studies have explored ways of teaching llms to comprehend new languages through incontext learning and prompt engineering by providing supplementary linguistic knowledge @xcite or retrieving extra examples from large corpora @xcite . however, these methods remain insufficient, and llms still consistently underperform humans in various tasks. in addition, there is no systematic evaluation of how llms can generalize their linguistic skills to lr languages that are absent and typologically different from training data. the current study investigates whether llms can learn and apply different linguistic rules (phonology, morpho-syntax, etc.) via in-context learning, and assesses how well they perform on translation tasks between english and lr languages with diverse typological features. llms are expected to rely on their intrinsic linguistic reasoning abilities rather than external knowledge or large corpora. inspired by the international linguistics olympiad (iol) and its regional variants, we create a dataset covering 40 lr languages, which contains 168 manually constructed puzzles. the puzzles follow a "rosetta stone" format, where test-takers are given 10-15 exemplar sentences in a foreign language that is previously unknown to them, along with corresponding translations in their native language. test-takers need to deduce linguistic rules from the examples and apply them by translating new ones. previous studies @xcite have shown that these puzzles from iol are challenging for llms, and prompt engineering techniques such as chain-of-thought provides little improvement @xcite . we posit that the original puzzles might be too complicated for llms because several different rules are often in-volved in one puzzle, and the complexity prevents llms from recognizing meaningful patterns. such complexity also limits detailed analysis of llms' strengths and weaknesses in linguistic reasoning. to mitigate that, we take a step-by-step approach to let llms learn linguistic rules incrementally. the original puzzles are broken down into a series of smaller, more manageable ones, each targeting one specific linguistic rule. the principle is to start with simple sentences where llms can learn vocabulary and basic syntax, which are followed by sentences centering on morpho-syntax features such as tense or agreement, and finally complicated sentences where they need to combine all the rules together. we evaluate five state-of-the-art llms and compare their performance with that of 16 human testers with linguistic training. llms have shown strong meta-linguistic competence, defined by @xcite as 'the knowledge of the characteristics and structures of language' in the major languages that they are trained in. however, it is not clear whether they can transfer such linguistic knowledge to unseen languages, and our approach aims to address that. we believe that our results can potentially facilitate research in llms and lr languages. if humans can benefit from meta-linguistic abilities when learning new languages, we shall expect the same for llms when dealing with lr languages as well, thus providing a future possibility of using llms in research of lr languages, such as annotation of lr data, producing glosses for linguists, developing machine translation systems, and so on. in the following sections, we review previous work on evaluating llms' linguistic abilities and their performance in lr languages. we then describe our dataset and experiments, followed by a presentation and analysis of our results. ## related works one of the primary focuses of research in the field of llms is concerned with their reasoning capabilities. they have shown significant improvements over earlier counterparts, achieving promising performance on tasks such as mathematics @xcite , geometry @xcite , automated theorem proving @xcite , code generation @xcite , and so on. in addition, they can perform tasks that they are not explicitly trained for, via in-context learning. this ability, first identified in gpt-3 (brown, 2020), allows llms to learn and execute new tasks with just a few examples. while some studies suggest that llms may not truly "learn" and instead exploit superficial patterns in input examples @xcite , the potential for generalizing beyond training data presents a new possibility for processing lr languages, where data scarcity has long been a challenge. in light of such abilities, recent studies have explored the possibility of using llms as an alternative to fine-tuned models for machine translation in lr languages. for example, @xcite present machine translation from one book, where llms are tasked to translate between english and kalamang, an endangered language, using a grammar book as the primary resource. the authors show that llms can generate reasonable translations given lexical and grammatical descriptions, but are considerably inferior to humans in terms of grammatical consistency. efforts to improve llm performance in this area are centered around prompt engineering techniques, such as providing llms with external linguistic knowledge. for example, when dealing with several lr languages, @xcite show that llms prompted with grammatical description of the languages can sometimes outperform fine-tuned transformer models. @xcite prompt llms with extra morphological gloss information, a dictionary, and a grammar book, when translating unseen lr languages to english, boosting the performance in few-shots translation from near 0 to around 10 in bleu scores. more elaborate works have attempted retrieval-based methods. @xcite use llms to produce interlinear gloss for lr languages, with examples retrieved from a corpus with carefully designed strategies. although llms have not beaten sota supervised methods, they outperform basic fine-tuned transformer models. similarly, @xcite build a framework to construct dedicated textbooks for llms, and retrieve vocabulary and syntactic patterns to teach llms unseen lr languages, achieving notable improvements on translation tasks. as llms are increasingly applied to lr languages, understanding how well they generalize their meta-linguistic abilities becomes crucial, especially when dealing with languages that are typologically different from those in the training data. many recent studies are focused on evaluating llms' linguistic skills across various phenomena. for instance, @xcite introduce the holmes benchmark to assess language models' understanding of syntax, morphology, semantics, and discourse. however, the study only examines english, leaving the question of how well llms generalize in cross-lingual situations unanswered. to address this gap, several researchers turn to linguistic puzzles from iol, which offer an opportunity to test llms' ability to infer and apply linguistic rules in unfamiliar languages. @xcite propose the puzzling machines dataset, with around 100 rosetta stone puzzles from iol covering 81 languages. while statistical and neural models at the time scored near zero on these problems, gpt 3.5 achieved significantly better results. prompting strategies, such as tree of thought, provide little improvement @xcite . chi et al. (2024) create the modeling dataset, featuring 48 puzzles across 19 lr languages. they handcraft these puzzles instead of using puzzles from iol. their problems focus on four features, namely basic word order, noun-adjective order, possession, and mapping vocabulary. @xcite present the lingoly benchmark with puzzles in diverse formats and categories from the uk linguistic olympiad, while @xcite introduce linguini, covering 75 lr languages with various puzzle types collected from iol. results also show that larger, proprietary models generally outperform smaller, open-source ones. prior in-context learning framework of lr languages have mostley relied on external knowledge or corpora. evaluation of intrinsic abilities of llms using iol puzzles consistently report a low accuracy of 25-30% across all models, and prompting strageties show little improvements. these iol puzzles often involve linguistic features in one puzzle, and models have to process semantic, morphology and syntax patterns at the same time, our approach differs by decomposing such puzzles into smaller, more manageable ones focusing one rule at a time. we will show that by doing so, llms can take better advantage of their in-context learning and reasoning abilities, and the performance of translation tasks between unseen languages and english can be significantly improved. ## data our study builds upon the previous efforts and is aimed at addressing the limitations of existing approaches. we propose a step-by-step framework for linguistic reasoning that where llms learn lin-guistic rules one at a time over a multi-round conversation. unlike the original iol puzzles, which involves processing multiple linguistic rules across different levels (semantics, phonology, morphology, syntax) at the same time, our framework is built upon puzzles that focus on one rule at a time. this allows llms to learn the patterns incrementally and also allows for a more detailed analysis of llms' strengths and weaknesses in linguistic reasoning for unseen lr languages. ## conclusion in general, this paper presents an evaluation of llms' ability to learn and apply complex linguis-tic rules across diverse language structures. inspired by linguistic puzzles from iol, we design a step-by-step approach for llms to learn linguistic rules in-context with their intrinsic meta-linguistic abilities. it involves creating a series of puzzles that allows llms to learn complex linguistic rules incrementally. the results show that our approach significantly boosts model performance in translation tasks, and the best model can match human level performance. we hope our dataset provides a starting point for future studies to further improve llm performance and promote llm applications in lr languages.