Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes (2022)

Abstract: Providing better language tools for low-resource and endangered languages is imperative for equitable growth.Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages.However, this transfer is not universal, with many languages not currently understood by multilingual approaches.It is estimated that only 72 languages possess a “small set of labeled datasets” on which we could test a model’s performance, the vast majority of languages not having the resources available to simply evaluate performances on.In this work, we attempt to clarify which languages do and do not currently benefit from such transfer.To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model.Our approach is derived from the hypothesis that if a model’s understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language.We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.

## introduction natural language processing (nlp) boasts of significant recent successes, largely driven by the introduction of different flavors of pretrained models @xcite @xcite @xcite . however, the rewards of those successes have been mostly reaped by high-resource languages. the existence of high-quality benchmarks and metrics, the abundance of readily available high-quality corpora, or the number of researchers speaking the language themselves @xcite are significant contributors to the disproportionate advances in high-resource languages. although recent improvements in nlp have been shown to extend to several different languages, such as the progress to language understanding by bertstyle models @xcite @xcite @xcite @xcite @xcite , many of those extensions have been limited to relatively high-resource languages. such improvements are often perceived to extend to low-resource languages, but the lack of appropriate benchmarking in those languages curtails our ability to verify such perceptions. the world atlas of language structures @xcite categorizes over 2600 languages, and ethnologue @xcite estimates that there are currently over 7000 living languages @xcite ; the most popular cross-lingual benchmarks @xcite together cover less than 50 languages and @xcite estimates that only 72 languages worldwide pass the threshold of having "a small set of labeled datasets", which could be used for evaluation. towards contributing to an equitable society with the development of language technologies, it is imperative that we ensure that no living languages are left behind. building automatic and cheap tools to provide better visibility into which languages are not currently well understood by cross-lingual nlp models then becomes essential. to determine cheaply if a model understands text in a specific language or not, we first find behaviors that are consistently exhibited by models that do perform well on language understanding tasks. by finding when those behaviors are not exhibited, we can determine whether a model understands the text or not. recent research trends have taken to evaluating well-known natural language understanding (nlu) models on perturbed text @xcite @xcite @xcite @xcite . such works attempt to distill which aspects of a text are not necessary and which aspects are necessary for language models to understand it by selectively perturbing the text, such as by shuffling the order of words. it may be possible to use the sensitivity of models to perturbations to properties of text that are found to be essential for nlu as a proxy for model understanding. as an extreme example, if a model develops the same understanding of a text and the same text with its characters shuffled, it can be hypothesized that its original understanding of the text was limited. the texts "i will eat an apple" and "ln i plla wat ieaep" contain the same characters. yet, we would expect radically different representations of both from a model, assuming it correctly understood the unperturbed version. we explore the following research questions and verify their corresponding hypotheses: ## multilingual local sensitivity to answer rq1, we borrow some of the perturbation schemes and metrics from @xcite and apply them to a multilingual setting. we aim to demonstrate empirically that neural models generally make some use of local structure to perform understanding tasks, irrespective of language. this can be demonstrated by progressively removing local structure from text through order altering perturbations and observing a similar decline in understanding (as measured by performance metrics) of that text from models. such results will motivate using low local structure sensitivity as a proxy for lack of ability to perform language understanding tasks. we perform those experiments on seven popular cross-lingual tasks covering 44 unique languages. ## low monolingual local sensitivity as a proxy for lack of understanding this section explores using an insensitivity to local perturbations as a proxy for lack of understanding to address rq2. to find a proxy for understanding that will provide greater visibility in the performance of very low-resource languages where evaluation is not possible, we cannot measure the local sensitivity by evaluating a language on a labeled task. therefore, we will explore monolingual local sensitivity as a proxy for lack of language understanding, with unlabelled monolingual data in the target language as its only requirement. ## limitations and ethical considerations there are several limitations to our approach that may have an ethical impact. the first one is the poor recall. while languages with low monolingual local sensitivity, from our experiments, always are poor performers, many poor performers are also sensitive to local perturbations. our approach can successfully find some languages that do require additional attention but will miss many other languages. if we rely on automatic tools to detect where to put efforts, there is a possibility that no efforts are put on languages that are not detected by those tools. the second one is the data requirement. obtaining a sufficient sample of text to calculate monolingual local sensitivity for some lowresource languages may still be too high of a hurdle. some living languages, especially those from oral traditions, may have a limited pool of written text available. it is crucial that if we use automatic tools to detect which languages requires further efforts, we do not forget of the languages that might not be detected or are incompatible with those tools. ## conclusion regardless of the language, task, or model used, the use of local structure seems to be relied upon by neural models to build an understanding of text. local structure sensitivity does not seem to be an artifact of the english language and broadly applies to written text in most languages. we explore monolingual local sensitivity to automatically detect unintelligible languages to cross-lingual models, the only requirement being access to monolingual unlabelled text. if local structure is essential to building understanding, not relying on the local structure would imply a limited understanding. we demonstrate a high correlation between monolingual local sensitivity and the ability of a model to perform cross-lingual sentence similarity in 350 diverse languages. specifically, all languages with low monolingual local sensitivity performed poorly on that task. those results indicate that with the measure of monolingual local sensitivity alone, it is possible to estimate the performance of a certain language on a certain model without access to any supervised learning datasets. our contribution will be useful to direct further efforts, such as unlabelled data gathering for pretraining, to expand the coverage of cross-lingual models in the most efficient way.