The State and Fate of Linguistic Diversity and Inclusion in the NLP World (2020)

Abstract: Language technologies contribute to promoting multilingualism and linguistic diversity around the world. However, only a very small number of the over 7000 languages of the world are represented in the rapidly evolving language technologies and applications. In this paper we look at the relation between the types of languages, resources, and their representation in NLP conferences to understand the trajectory that different languages have followed over time. Our quantitative investigation underlines the disparity between languages, especially in terms of their resources, and calls into question the “language agnostic” status of current models and systems. Through this paper, we attempt to convince the ACL community to prioritise the resolution of the predicaments highlighted here, so that no language is left behind.

## the questions languages x and y are the official languages of two different countries; they have around 29m and 18m native speakers, and 2m and 5.5k wikipedia articles, respectively. x is syntactically quite similar to english, though uses dimunitives and has grammatical gender. y, on the other hand, has a different word order from english, and has a rare typological feature -generally it is a head-final language, but noun phrases are head-initial. it also features full and partial reduplication. 69 items on ldc and elra contain data in x, whereas for y there are only 2 items. x boasts of some of the best online machine translation systems, whereas y is supported by very few online mt systems and that too with far inferior translation quality. figure 1 shows the number of papers in conferences (acl, naacl, eacl, emnlp, lrec, ws) that mention x and y in the paper, across the years. as you can see, while x has a steady and growing trend of research, our community has been mostly oblivious to y, until recently when some of the zero-shot learning papers have started mentioning it. can you guess what x and y are? regardless of whether you can guess the exact answer, most nlp researchers surely know of (and might even speak) several languages which are in the same boat as x; languages which have a large amount of resources and therefore access to the benefits of the current nlp breakthroughs, and languages like y; those which lack resources and consequently the attention of the nlp community, despite having similar speaker base sizes and typologically diverse features. you probably have come across the issue of extremely skewed distribution of resources across the world's languages before. you might also be aware of the fact that most of our nlp systems, which are typically declared language agnostic, are not truly so @xcite . the handful of languages on which nlp systems are trained and tested are often related and from the same geography, drawn from a few dominant language families, leading to a typological echo-chamber. as a result, a vast majority of typologically diverse linguistic phenomena are never seen by our nlp systems @xcite . nevertheless, it would be prudent to re-examine these issues in the light of recent advances in deep learning. neural systems, on one hand, require a lot more data for training than rule-based or traditional ml systems, creating a bigger technological divide between the xs and ys; yet, some of the most recent techniques on zero-shot learning of massively multilingual systems @xcite @xcite bridge this gap by obliterating the need for large labeled datasets in all languages. instead, they need only large unlabeled corpora across languages and labeled data in only some languages. assuming that this approach can be taken to its promising end, how does the fate of different languages change? we break down this complex prescient question into the following more tractable and quantifiable questions on linguistic diversity and inclusion: ## the six kinds of languages in order to summarize the digital status and 'richness' of languages in the context of data availability, we propose a taxonomy based on the number of language resources which exist for different languages. we frame the rest of our analyses based on this taxonomy and use it to emphasize the existence of such resource disparities. ## -the rising stars unsupervised pre-training has been an energy boost for these languages. with a strong web presence, there is a thriving cultural community online for them. however, they have been let down by insufficient efforts in labeled data collection. with the right steps, these languages can be very well off if they continue to ride the 'pre-training' wave. languages, reaping benefit from each state-of-theart nlp breakthrough. some more information about the taxonomy is shown in ## typology linguistic typology is a field which involves the classification of languages based on their structural and semantic properties. large-scale efforts have led to the creation of a database of typological features @xcite . such documentation becomes important as there are barely any other classifications of similar scale. in the context of nlp research, there has been work indicating the effectiveness of injecting typological information to guide the design of models @xcite . also, transfer learning of resourcerich to resource-poor languages have been shown to work better if the respective languages contain similar typological features @xcite . we look at how skewed language resource availability leads to an under-representation of certain typological features, which may in turn cause zero-shot inference models to fail on nlp tasks for certain languages. we look at the wals data (dryer and haspelmath, 2013), which contains typological features for 2679 languages. there are a total of 192 typological features, with an average of 5.93 categories per feature. we take the languages in classes 0, 1, 2, all of which have limited or no data resources as compared to 3, 4, 5 and look at how many categories, across all features, exist in classes 0, 1, 2 but not 3, 4, 5. this comes to a total of 549 out of 1139 unique categories, with an average of 2.86 categories per feature being ignored. typological features with the most and least 'ignored' categories are shown in to get an idea of what these typological 'exclu ## conference-language inclusion nlp conferences have a huge impact on how language resources and technologies are constructed. exciting research in venues such as acl, emnlp, lrec have the ability to turn heads in both industry and government and have the potential to attract funds to a particular technology. has the usage of a small set of resource-rich languages in such conferences led to a disparity, pushing the less represented to the bottom of the ladder in terms of research? we analyze the involvement of various languages in nlp research conferences over the years. ## entity embedding analysis the measures discussed in the previous section signal at variance in acceptance of different languages at different nlp venues across time. however, there are usually multiple subtle factors which vanilla statistics fail to capture. embeddings, on the other hand, have been found extensively useful in nlp tasks as they are able to learn relevant signals directly from the data and uncover these rather complex nuances. to this end, we propose a novel approach to jointly learn the representations of conferences, authors and languages, which we collectively term as entities. the proposed embedding method allows us to project these entities in the same space enabling us to effectively reveal patterns revolving around them. ## conclusion we set out to answer some critical questions about the state of language resource availability and research. we do so by conducting a series of quantitative analyses through the lens of a defined taxonomy. as a result, we uncover a set of interesting insights and also yield consistent findings about language disparity: we believe these findings will play a strong role in making the community aware of the gap that needs to be filled before we can truly claim state-ofthe-art technologies to be language agnostic. pertinent questions should be posed to authors of future publications about whether their proposed language technologies extend to other languages. there are ways to improve the inclusivity of acl conferences. special tracks could be initiated for low-resource, language-specific tasks, although we believe that in doing so, we risk further marginalization of those languages. instead, a way to promote change could be the addition of d&i (diversity and inclusion) clauses involving language-related questions in the submission and reviewer forms: do your methods and experiments apply (or scale) to a range of languages? are your findings and contributions contributing to the inclusivity of various languages? finally, in case you're still itching to know, language x is dutch, and y is somali.