Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models (2025)

Abstract: Language models (LMs) are capable of acquiring elements of human-like syntactic knowledge. Targeted syntactic evaluation tests have been employed to measure how well they form generalizations about syntactic phenomena in high-resource languages such as English. However, we still lack a thorough understanding of LMs’ capacity for syntactic generalizations in low-resource languages, which are responsible for much of the diversity of syntactic patterns worldwide. In this study, we develop targeted syntactic evaluation tests for three low-resource languages (Basque, Hindi, and Swahili) and use them to evaluate five families of open-access multilingual Transformer LMs. We find that some syntactic tasks prove relatively easy for LMs while others (agreement in sentences containing indirect objects in Basque, agreement across a prepositional phrase in Swahili) are challenging. We additionally uncover issues with publicly available Transformers, including a bias toward the habitual aspect in Hindi in multilingual BERT and underperformance compared to similar-sized models in XGLM-4.5B.

## introduction there is a substantial body of work dedicated to evaluating the linguistic knowledge of language models. popular evaluation methodologies include: to date, most of the research investigating the linguistic knowledge of lms has concentrated on high-resource languages such as english @xcite @xcite , german @xcite , spanish @xcite , italian @xcite , chinese @xcite @xcite , and japanese @xcite @xcite . however, efforts have also been made to include less prominent languages. notably, @xcite and @xcite probed masked lms for morphosyntactic attributes of words across 36 and 43 languages, respectively. acceptability benchmarks have been developed for north germanic languages by @xcite , @xcite , @xcite , and @xcite . tse has been applied to hebrew @xcite , norwegian @xcite , and indonesian and tamil @xcite . we believe that evaluating lms' linguistic knowledge across a diverse range of languages is crucial for developing a comprehensive picture of how they form linguistic generalizations. assessing 'off-the-shelf' lms in lower-resourced languages offers a further benefit of diagnosing limitations and challenges these models may face due to issues like insufficient training data, model biases, or difficulties in capturing particular linguistic features. we recognize tse's advantage of focusing on one combination of linguistic phenomenon and sentence structure at a time, which enables a fine-grained analysis of how performance depends on the structure and complexity of input sentences. while many prior tse studies considered lms based on rnn and lstm architectures @xcite , we focus on modern lms based on the transformer architecture, which is the current state of the art. therefore, we conduct three tse case studies benchmarking publicly available transformer lms on distinctive morphosyntactic phenomena in low-resource languages-auxiliary verb agreement in basque, split ergativity in hindi, and noun class agreement in swahili. we find that lms mostly do well on agreement in basque, with errors linked to the presence of an indirect object in a sentence, and almost always succeed in selecting the correct aspectual form of the verb based on the presence or absence of the ergative clitic in hindi, with the exception of multilingual bert, which prefers the habitual aspect regardless of its grammaticality. however, lms struggle to agree predicates with the noun class of their subjects in swahili. performance on our tests has a positive relationship with model size, but xglm 4.5b systematically underperforms similarsized models for reasons possibly including the lack of low-resource upsampling and the 'curse of multilinguality'. syntactically complex attractor phrases weaken performance in swahili but not in hindi. ## evaluation paradigm following existing work on tse, we organize our evaluation materials in minimal pairs-pairs of minimally different sentences such that one is grammatically acceptable and the other one is ungrammatical because it violates a specific linguistic rule. below we provide an english example where sentences (a) and (b) differ by one property-the plurality of the copula. in sentence (b), the plural copula 'are' does not agree with the singular subject 'the teacher', rendering the sentence ungrammatical. in all our minimal pairs, the sentences differ by one word whose grammaticality can be determined from the left context. we call the left context the condition and the rest of the sentence the target: in the example above, 'the teacher' is the condition, 'is good.' is the grammatical target, and 'are good.' is the ungrammatical target. we use the minicons python library @xcite to compute the conditional log-probabilities of the grammatical and ungrammatical targets given the condition, expecting a model to assign a more positive log-probability to the grammatical target if it has learned the rule correctly. we group minimal pairs into test suites, each of which assesses models' knowledge of one phenomenon in sentences of a unified structure, and report a model's accuracy on a test suite as the proportion of minimal pairs for which it assigns higher log-probability to the grammatical target. ## test suites we consider three low-resource languages from different language families: basque (isolate), hindi (indo-european), and swahili (niger-congo). in each of them, we focus on one characteristic morphosyntactic phenomenon: in basque, auxiliary verb agreement ( §3.1); in hindi, split ergativity ( §3.2); in swahili, noun class agreement ( §3.3). we generate synthetic test suites ( §3.4) and perform human validation to verify that the generated minimal pairs represent a genuine contrast in grammatical acceptability ( §3.5). ## models we evaluated five open-access multilingual transformers from hugging face: three autoregressive models-mgpt, bloom, and xglm-and two masked models-multilingual bert (mbert) and xlm-roberta (xlm-r). each of these models except mbert is available in several versions differing by size; we evaluated all versions. for an overview of the models and their versions, see appendix a. ## results we present evaluation results in figure 1 and provide an overview by language in §5.1. for models that come in several versions, we analyze the relationship between performance on our test suites and the number of parameters ( §5.2). for test suite classes where we varied the complexity of intervening phrases (split ergativity in hindi and noun class agreement in swahili), we analyze the relationship between performance and complexity of the intervening constituent ( §5.3). ## conclusion we assessed the ability of open-access multilingual transformer lms to form syntactic generalizations across three low-resource languages-basque, hindi, and swahili. we found that models mostly performed well on basque auxiliary agreement, albeit with challenges in sentences containing indirect objects, likely due to their relatively low frequency in training corpora. in hindi, all lms demonstrated a solid grasp of split ergativity except multilingual bert, which failed to select the perfective aspect as the only grammatical aspect in sentences containing the ergative clitic. noun class agreement in swahili posed the greatest challenge, with models often performing near random guessing. we hope that our work will motivate further investigations into lms' linguistic knowledge in low-resource languages and will help lm developers identify and address areas for improvement, ultimately guiding the design of better lms for low-resource languages and enabling fair access to high-quality nlp technologies for their speakers.