Does Typological Blinding Impede Cross-Lingual Sharing? (2021)

Abstract: Bridging the performance gap between high- and low-resource languages has been the focus of much previous work. Typological features from databases such as the World Atlas of Language Structures (WALS) are a prime candidate for this, as such data exists even for very low-resource languages. However, previous work has only found minor benefits from using typological information. Our hypothesis is that a model trained in a cross-lingual setting will pick up on typological cues from the input data, thus overshadowing the utility of explicitly using such features. We verify this hypothesis by blinding a model to typological information, and investigate how cross-lingual sharing and performance is impacted. Our model is based on a cross-lingual architecture in which the latent weights governing the sharing between languages is learnt during training. We show that (i) preventing this model from exploiting typology severely reduces performance, while a control experiment reaffirms that (ii) encouraging sharing according to typology somewhat improves performance.

## introduction most languages in the world have little access to nlp technology due to data scarcity @xcite . nonetheless, high-quality multilingual representations can be obtained using only a raw text signal, e.g. via multilingual language modelling @xcite . furthermore, structural similarities of languages are to a large extent documented in typological databases such as the world atlas of language structures (wals, @xcite ). hence, developing models which can take use typological similarities of languages is an important direction in order to alleviate language technology inequalities. while previous work has attempted to use typological information to inform nlp models, our work differs significantly from such efforts in that we blind a model to this information. most previous work includes language information as features, by using language ids, or language embeddings (e.g. @xcite ; @xcite ; @xcite ; @xcite ; @xcite ). notably, limited effects are usually observed from including typological features explicitly. for instance, de @xcite observe positive cross-lingual sharing effects only in a handful of their settings. we therefore hypothesise that relevant typological information is learned as a by-product of cross-lingual training. hence, although models do benefit from this information, it is not necessary to provide it explicitly in a high-resource scenario, where there is abundant training data. this is confirmed by @xcite , who find that, e.g., language embeddings trained on a morphological task can encode morphological features from wals. in contrast with previous work, we blind a model to typological information, by using adversarial techniques based on gradient reversal @xcite . we evaluate on the structured prediction and classification tasks in xtreme @xcite , yielding a total of 40 languages and 4 tasks. we show that when a model is blinded to typological signals relating to syntax and morphology, performance on related nlp tasks drops significantly. for instance, the mean accuracy across 40 languages for pos tagging drops by 1.8% when blinding the model to morphological features. ## model an overview of the model is shown in figure 1 . we model each task in this paper using the following steps. first, contextual representations are extracted using multilingual bert (m-bert, devlin et al. (2019) ), a transformer-based model @xcite , trained with shared wordpieces across languages. we either blind m-bert to typological features, with an added adversarial component based on gradient reversal @xcite , or expose it to them via multitask learning (mtl, @xcite ). representations from m-bert are fed to a latent multi-task architecture learning network @xcite , which includes α parameters we seek to investigate. the model learns which parameters to share between languages (e.g. α es,f r denotes sharing between spanish and french). ## cross-lingual experiments we investigate the effects of typological blinding, using typological parameters as presented in wals @xcite . the experiments are run on xtreme @xcite , which includes up to 40 languages from 12 language families and two isolates. we experiment on the following languages (iso 639-1 codes): af, ar, bg, bn, de, el, en, es, et, eu, fa, fi, fr, he, hi, hu, id, it, ja, jv, ka, kk, ko, ml, mr, ms, my, nl, pt, ru, sw, ta, te, th, tl, tr, ur, vi, yo, and zh. we experiment on four tasks: pos (part of speech tagging), ner (named entity recognition), xnli (cross-lingual natural language inference), and paws-x (paraphrase identification). our general setup for the structured prediction tasks (pos and ner) is that we train on all available languages, and downsample to 1,000 samples per language. for the classification tasks xnli and paws-x, we train on the english training data and fine-tune on the development sets, as no training data is available for other languages. hence, typological differences will be the main factor in our results, rather than differences in dataset sizes. ## discussion we have shown that blinding a multilingual model to typological features severely affects sharing across a relatively large language sample, and for several nlp tasks. the effects on model performance, as evaluated over 40 languages and 4 tasks from xtreme @xcite , were the largest for pos tagging. the fact that smaller effects were observed for ner, could be because this task relies more on memorising nes rather than using (morpho-)syntactic cues @xcite . furthermore, the relatively small effects on xnli and paws-x can also be interpreted as evidence for that typology is less important in these tasks than in more traditional linguistic analysis. a potential critique of our approach is that it merely blinds the model to language identities. this could be the case, if only some latent representation of, e.g., "svo" ordering is used to represent a language identity. however, previous work has shown that morphological information is encoded by the type of model we investigate. hence, since we only blind features in a single category at a time, we expect that the model's representation of language identities is unaffected. not only do we observe a drop in performance when blinding a model to syntactic features, but we also observe that the α sharing weights in our model do not appear to correlate with linguistic similarities in this setting. conversely, encouraging a model to consider typology, by jointly optimising it for typological feature prediction, improves performance in general. furthermore, α weights in this scenario converge towards correlating with structural similarities of languages. this is in line with recent work which has found that m-bert uses fine-grained syntactic distinctions in its crosslingual representation space @xcite . we interpret this as evidence for the fact that typology can be a necessity for modelling in nlp. our results furthermore corroborate previous work in that we only find moderate benefits from including typological information explicitly. we expect that this to a large degree is due to the typological similarities of languages being encoded implicitly based on correlations between patterns in the input data. as low-resource languages often do not even have access to any substantial amount of raw text, but often do have annotations in wals, we expect that using typological information can go some way towards building truly language-universal models. ## conclusions we have shown that preventing access to typology can impede the performance of cross-lingual sharing models. investigating latent weights governing the sharing between languages shows that this prevents the model from sharing between typologically similar languages, which is otherwise learned based on patterns in the input. we therefore expect that using typological information can be of particular interest for building truly language-universal models for low-resource languages.