MELA : Multilingual Evaluation of Linguistic Acceptability (2024)

Abstract: In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability—MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language—Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks.

## introduction the acceptability judgment task tests a language model's ability to distinguish syntactically acceptable sentences like (1a) from unacceptable ones like (1b) in a human language -for instance, the following example on island constraints in english @xcite . as a core linguistic competence, the ability to tell well-formed sentences from ill-formed ones is one of the first that a good language model should have. many corpora and benchmarks have been built to evaluate language models' syntactic ability, using either a data-driven approach, where examples created by theoretical linguists in published textbooks are collected, e.g., cola-corpus of linguistic acceptability @xcite , or a theorydriven approach, where minimal pairs targeting specific syntactic phenomena are generated semiautomatically via some template @xcite @xcite . recently, there has been growing interest in expanding the data-driven paradigm into other languages. for instance, cola-style datasets have been proposed in russian @xcite , italian @xcite , and chinese @xcite . however, to date there are almost no multilingual benchmarks in this area that can be used to systematically test such abilities of multilingual models. on the other hand, recently introduced benchmarks for large language models (llms) such as @xcite have mostly focused on application-driven tasks such as world knowledge and commonsense reasoning @xcite , math reasoning @xcite , and code generation @xcite @xcite . few studies, however, have investigated these models from a more linguistics-oriented aspect. to address these gaps, we introduce mela-multilingual evaluation of linguistic acceptability, the first large-scale multilingual acceptability benchmark with 46k examples covering 10 languages from a diverse set of language families. data in four languages are from existing benchmarks mentioned above, and we complement them with newly collected data in six languages. examples of mela are demonstrated in we propose three possible usage cases for mela, and make preliminary explorations in this paper: benchmarking we benchmark various multilingual language models (lms) on mela, including bloomz @xcite , mtk @xcite , mt0 @xcite , baichuan2-chat @xcite , gpt-3.5 and gpt-4o (openai, 2023). ## mela: multilingual evaluation of linguistic acceptability mela consists of more than 46 thousand acceptability samples across 10 languages from a diversity of language families and groups. specifically, it contains three germanic languages: english, german and icelandic, three romance languages: spanish, french and italian, one slavic language: russian, one sino-tibetan language: chinese, one japonic language: japanese, and one semitic language: arabic. ## evaluating llms with mela in this section, we report the performance of finetuned xlm-r and several llms, open-source or close-source, on mela. ## cross-lingual transfer and multilingual fine-tuning in this section, we investigate cross-lingual transfer and multilingual fine-tuning of linguistic acceptability with xlm-r. all training and evaluation are done on mela v1.0 as it requires a training set. ## edge probing in this section, we adopt edge probing @xcite to explore whether fine-tuning on acceptability judgment tasks injects syntax-related information into the pre-trained xlm-r. ## conclusion in this work we present mela, the first multilingual acceptability judgement benchmark covering a diverse set of 10 languages, all annotated by expert linguists. by benchmarking multilingual llms on mela and fine-tuning xlm-r in different cross-lingual settings, we find that (1) gpt-4o ourperforms supervised xlm-r, especially on low-resource languages, that (2) in-language data is crucial for few-shot evaluation and that (3) crosslingual transfer is non-trivial for all language pairs in supervised fine-tuning. we probe mela-finetuned xlm-r for the syntax information encoded, finding that training on mela improves the performance on syntax-related probing tasks, which indicates that language models acquire syntactic knowledge during training on linguistic acceptability judgements. m base refers to the pre-trained xlm-r model; m en mela refers the xlm-r fine-tuned on the english mela. bold denotes a better performance in average between m base and m lang mela . we conduct a pair comparison between m base and m lang mela trained on mela of one language to investigate whether linguistic acceptability helps the cross-lingual transfer in the above two probing tasks. for each cell, probing classifiers are trained on span representations in the language denoted in the second row, which are encoded by the model denoted in the second column, and evaluated on the probing tasks in the same language on which xlm-r is fine-tuned (the first column).