Local Structure Matters Most in Most Languages (2022)

Abstract: Many recent perturbation studies have found unintuitive results on what does and does not matter when performing Natural Language Understanding (NLU) tasks in English. Coding properties, such as the order of words, can often be removed through shuffling without impacting downstream performances. Such insight may be used to direct future research into English NLP models. As many improvements in multilingual settings consist of wholesale adaptation of English approaches, it is important to verify whether those studies replicate or not in multilingual settings. In this work, we replicate a study on the importance of local structure, and the relative unimportance of global structure, in a multilingual setting. We find that the phenomenon observed on the English language broadly translates to over 120 languages, with a few caveats.

## introduction a recent research trend has explored the sensitivity, or insensitivity, of neural language models to different perturbations of texts @xcite @xcite @xcite @xcite . their findings may be central in directing future nlp research by providing insight into which coding property @xcite of language are most valuable to performing natural language understanding (nlu) tasks. as research in english nlp tends to be adapted to other languages, such as through single language adaptation of bert-style models @xcite @xcite @xcite @xcite @xcite or multilingual adaptations of the same architecture @xcite @xcite @xcite , it is vital that we verify how insights derived from the english language generalize to other languages. one such coding property, the local structure of text, has recently been shown to be ubiquitously relied upon by both neural language models @xcite and humans @xcite to understand text in english. the global structure of text only sometimes being necessary for a model to perform nlu tasks @xcite . such results motivate hierarchical approaches to neural language model development, where one would first build meaning locally and then reason over the global context if necessary. however, we must verify that the importance of that coding property is not merely an artifact of the english language. in this short paper, our contributions are as follows: text perturbations and structure probing several text perturbation schemes have been explored to probe what kind of structure does and does not matter for neural models performing nlu. sankar et al. (2019) explores both shuffling and reversing utterances and words in a generative dialogue setting, highlighting models' insensitivity to the order of conversational history. pham et al. (2021) explores shuffling n-grams for different values of n, which highlights the insensitivity of pretrained transformer models. sinha et al. (2020) explores shuffling of words on textual entailment tasks, highlighting models' insensitivity to such perturbations. finally, @xcite extend perturbation studies to swedish and russian and performs perturbations by shuffling syntactic phrases, rotating sub-trees around the root of the syntactic tree of a sentence, or simply shuffling the words of the text. these approaches share the main limitation of requiring automatic parsing tools or well-developed tokenizers to define words. this limits their applicability in a multilingual setting. priors regarding the form of the text, such as the presence of whitespace delimited words, limit the generalizability of most of these studies. @xcite proposes a suite of controllable perturbations on characters and subwords, which should be compatible with almost any written language, as well as a metric quantifying perturbations to the local and global structure that measures perturbations on a character-level. ## experiments we extend the perturbation studies of @xcite to a multilingual setting. we perform those experiments on eight popular cross-lingual tasks @xcite @xcite covering over 120 languages. this will shed light on what languages, if any, do not share the same sensitivity to local structure and insensitivity to global structure as english. ## conclusion we first explored and confirmed the importance of local structure, the limited importance of global structure, and controlled for the potential of vocabulary destruction being the main explanatory factor in 8 nlu tasks covering over 120 languages. in aggregate, the findings of @xcite hold for many different pretrained cross-lingual models and nlu tasks in a multilingual setting. local structure sensitivity and global structure insensitivity do not seem to be an artifacts of the english language. a significant exception is when grammatical cues are essential to complete the task, such as in the paws-x task. while many tasks can be solved purely with the information obtained from the local structure, reasoning over the global context is necessary for many problems. languages using chinese characters as their script also deviate from the norm. this is likely caused by how semantically rich their characters are. it will be important that any nlp improvements derived from english experiments are verified to also generalize to other languages. as we have observed that languages written in chinese character script are differently impacted by perturbations to different coding properties, it is possible that im-provements to the way our model understand those properties in english will not generalize.