Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features (2023)

Abstract: A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.

## introduction in natural language processing (nlp), there is a need to build systems that serve more of the world's approximately 6,900 languages. as one measure of linguistic diversity, the world atlas of @xcite records 192 linguistic features along which languages differ. these range from the order of subject, object, and verb @xcite to the number of basic color categories @xcite . languages present in existing nlp datasets mostly lie in low-density regions of the space of possible typological features @xcite . in other words, many linguistic features that are common across the world's languages are not observed in languages that are the focus of nlp research. 2 it is thus important to investigate to which linguistic features models can generalize and where they face challenges. however, existing datasets do not allow for a fine-grained cross-lingual evaluation and mainly permit comparisons on a language level @xcite . prior studies focused on syntax and grammar through the lens of acceptability judgements @xcite @xcite . while these enable the evaluation of what a model deems 'natural' in a given language, it is often unclear how such biases relate to real-world applications of nlp technology. we propose multilingual morphological checklist (m2c) to enable the investigation of a broader set of cross-lingual differences in practical scenarios. specifically, we create a morphologicallyaware behavioral testing framework @xcite that allows for the specification of tests in a diverse set of languages. using this framework, we design tests that probe model's behavior in light of specific capabilities and typological features in 12 typologically diverse languages. we focus on a question answering setting as it represents one of the most general and widely useful nlp applications @xcite and enables zero-shot evaluation of models. we create tests that cover a diverse set of reasoning capabilities involving general linguistic features that are expressed differently across languages-negation, numerals, spatial and temporal expressions, and comparatives-as well as features unique to certain languages such as time in swahili, measure words in chinese, compounding possessives in finnish, and motion verbs in russian. we evaluate state-of-the-art language models on the generated tests in zero-shot and oneshot settings. our findings shed light on generalization failures to specific typological features. for instance, all models struggle with time expressions in swahili and measure words in chinese. we show the workflow of using m2c, from template creation to model evaluation, in figure 2 . our contributions are: (1) we create a new morphologically-aware multilingual behavioral testing framework. (2) we highlight linguistic features that are challenging in different languages. (3) we design tests that probe model capabilities in light of practically relevant typological differences. (4) we evaluate state-of-the-art language models on the generated tests. (5) we shed light on the challenges posed by typological differences in multilingual scenarios. ## related work perplexity perplexity is a standard measure of evaluating language model performance, which has also been used in multilingual settings @xcite . besides being difficult to compare across segmentations, perplexity does not provide more fine-grained insights regarding model behavior @xcite . acceptability evaluations compare perplexity between minimal pairs of grammatical and ungrammatical sentences @xcite . such evaluations have been extended to other languages @xcite @xcite @xcite , which requires writing extensive languagespecific grammars while the relevance of syntax biases in real-world applications remains unclear. evaluation of large models most benchmarks designed for evaluating large models focus on assessing their performance on a collection of complex tasks @xcite @xcite @xcite . however, such benchmarks are unable to highlight more fine-grained model limitations @xcite and are outpaced by the development of new models. behavioral testing behavioral testing sheds light on model capabilities via the design of simple targeted tasks. early work such as babi @xcite focused on toy tasks requiring simple reasoning capabilities while olmpics @xcite ) consisted of 8 short classification tasks for masked language models. recently, lmentry @xcite provides simple tests assessing fundamental generation capabilities. a common test bed is natural language inference @xcite where analyses of reasoning types have been extended to other languages @xcite @xcite but require existing data. the checklist framework (ribeiro et al., 2020) enables the generation of behavioral tests for nlp models but its templates are english-centric. english checklist tests have been extended to other languages via translation @xcite . such approaches, however, struggle with comprehensively covering linguistic features specific to a language and are not able to easily represent morphological variation. relatedly, @xcite create templates that integrate morphology for simple knowledge retrieval queries while @xcite automatically translate knowledge retrieval queries into other languages. compared to their approach, our framework allows for integrating morphology into a broader range of tests and is more scalable and flexible. ## checklist checklist @xcite relies on templates to generate a large amount of samples in order to evaluate models' behavior regarding different tasks and capabilities in a controlled manner. a template consists of a string with placeholders such as {first_name} delimited by curly brackets, e.g., "{first_name} is {adj}". the user provides a set of values for each placeholder, for instance, {first_name} = {michael, john, ... } and {adj} = {busy, friendly, ... }, which are used to populate the templates with their cartesian product. the generated samples can then be applied to systematically test a model's performance in a specific setting. multilingual tests checklist has been designed for english and provides mainly english-specific functionality. for example, it matches indefinite articles with nouns based on their starting letter, i.e., the placeholder {a:job} generates "a lawyer" and "an engineer". as a consequence, checklist is not capable of effectively generating tests in languages with richer morphology, which require maintaining agreement between multiple parts of the templatea feature that is beyond the scope of checklist. while multilingual tests can be generated by translating english tests @xcite , optionally including template extraction and human verification, such generated templates struggle with handling rich morphology. in addition, in order to systematically probe linguistic features specific to a language, it is crucial to be able to efficiently generate in-language tests from scratch. ## m2c framework we propose the m2c (multilingual morphological checklist) framework in order to enable the generation of tests in a broad set of languages, including languages with rich morphology. a user provides a template as a string, a list of values for each placeholder, and an optional configuration dictionary in case of duplicate placeholders. the placeholder values can either be passed without inflections (for example, names in english) as a list of strings, or as a list of dictionaries with their corresponding inflected values. each key of the dictionary is a feature combination (e.g., masc.pl) and the value is the corresponding string (e.g. "apples"). as such, each entity can have multiple inflections, for instance, in english "apple" and "apples". we show the general m2c workflow in figure 2 . ## capabilities and typological features languages we generate tests targeting capabilities and typological features in 12 typologically diverse languages: english (en), spanish (es), italian (it), french (fr), german (de), swedish (sv), finnish (fi), slovak (sk), russian (ru), swahili (sw), mandarin chinese (zh), and arabic (ar). recent models have excelled at a wide range of tasks in english requiring a diverse set of reasoning and understanding capabilities @xcite . as most languages are morphologically richer than english, they encode the linguistic features representing such capabilities in more complex ways. the features we investigate are relevant in a variety of real-world applications including sentiment analysis @xcite , question answering @xcite , grounding @xcite , reasoning with temporal change @xcite and quantitative attributes @xcite . we investigate capabilities and linguistic features present in all our investigated languages as well as linguistic features unique to certain languages. for each feature, we highlight differences in its cross-lingual instantiation and challenges for natural language understanding and generation. we create templates using the m2c framework to test a model's understanding of each capability and feature. we show a subset in ## experiments experimental setting we evaluate models on the generated tests in a question answering setting as can be seen in figure 2 . each test consists of a context, a question, and an answer that needs to be predicted by the model. for each template, we generate 2,000 test examples on which the model is evaluated. a model's performance on a template is its accuracy of predicting a valid answer for a test averaged across all tests of the template. we evaluate models in both zero-shot and oneshot settings for each capability and language. in the one-shot setting, a test randomly generated using the same template is used as the exemplar. this simplifies the task in two ways: i) it provides the model with a clear format for generating the answer and may enable the model to infer the answer's relationship to the rest of the template. while we conduct one-shot experiments to show the impact of additional instructions, zero-shot evaluation is the only setting that fully tests the model's understanding and generative capabilities independent of confounders such as the exemplar choice @xcite , in line with prior work on behavioral testing @xcite . we provide an example of both settings in ## conclusion in this paper, we have introduced m2c, a multilingual morphological framework for targeted behavioral evaluation of language-specific capabilities. as world languages present different challenges, m2c aims to provide flexibility in defining a suitable templating system with its individual dimensions and features. we have conducted experiments on state-of-the-art large language models, highlighted typological features that models struggle with, and quantified errors occurring due to morphology. we hope m2c inspires further research focused on tackling typological and morphological challenges with large language models.