Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World (2022)

Abstract: Linguistic disparity in the NLP world is a problem that has been widely acknowledged recently. However, different facets of this problem, or the reasons behind this disparity are seldom discussed within the NLP community. This paper provides a comprehensive analysis of the disparity that exists within the languages of the world. We show that simply categorising languages considering data availability may not be always correct. Using an existing language categorisation based on speaker population and vitality, we analyse the distribution of language data resources, amount of NLP/CL research, inclusion in multilingual web-based platforms and the inclusion in pre-trained multilingual models. We show that many languages do not get covered in these resources or platforms, and even within the languages belonging to the same language group, there is wide disparity. We analyse the impact of family, geographical location, GDP and the speaker population of languages and provide possible reasons for this disparity, along with some suggestions to overcome the same.

## introduction even after more than fifty years since the inception of the fields of computational linguistics (cl) and natural language processing (nlp), we still observe a significant bias favouring the so-called highresource languages in the field. conversely, this means that the majority of the 6500+ languages in the world, which have been classified as lowresource, have received limited to no attention. this resource poverty is not merely an academic or theoretical issue. it impacts the lives and the well-being of people concerned in a very present and practical manner, and deprives the speakers of low-resource languages from reaping the benefits of nlp in areas such as healthcare @xcite , disaster response @xcite , law @xcite , and education @xcite . this digital divide between high-resource and low-resource languages (lrls) foot_1 has been brought into the spotlight by many scholars in the field @xcite @xcite . consequently, there have been efforts to build data sets covering low-resource languages @xcite , benchmarks @xcite and techniques that favour low-resource languages @xcite ; all of which, are very promising developments. however, the problem is not fully solved, and this disparity should be quantified to understand the gravity of the problem @xcite . such an understanding is the first step in developing solutions to solve the problem @xcite . nlp researchers have mainly considered the availability of electronic data resources as the main descriptor of 'resourcefulness' of languages. for example, @xcite considered the availability of annotated and raw corpora. @xcite considered the availability of auxiliary resources such as lexicons in addition. @xcite estimated the level of language speaker representation in dataset content. @xcite used their criterion to categorise 2485 languages into six groups, based on the availability of unannotated data (number of wikipedia pages) and the number of annotated datasets available in the ldc 2 and elra 3 data repositories. however, such a data-centric perspective tends to overlook other aspects of resourcefulness, such as the inclusion of a language in multilingual webbased platforms such as facebook, or the inclusion in pre-trained multilingual models such as mbert @xcite and xlm-r @xcite . moreover, such a narrow view does not shed light on how this language disparity could be explained with respect to other socioeconomic-linguistic factors such as language family, geographical location or speaker population. this paper provides a deeper analysis into the less-known facts of the well-known problem of linguistic disparity in the world. we start with an existing language categorisation based on speaker population and vitality (ethnologue foot_3 ) @xcite , and analyse the distribution of language data resources, amount of nlp/cl research, inclusion in multilingual web-based platforms and the inclusion in pre-trained multilingual models. we show that simply categorising languages using data availability as done by @xcite can be misleading. we also show that many languages are neglected with respect to all the considered criteria, and even within the languages belonging to the same language group, there is wide disparity. we analyse this disparity with respect to the family, geographical location, as well as the speaker population and gdp. we also provide possible reasons for this disparity, along with some recommendations to eradicate the same. ## the 1kinds of languages ethnologue is an annual publication that provides statistics and other information of the living languages in the world. it has 7139 language entries, including dialects. we could identify 6420 unique languages by considering alternate names, dialects, and minor schisms to map to their most prominent entry. the language list we extracted, as well as the selection criteria are in appendix a. ethnologue languages are categorised into 12 classes, based on 2 variables: population and vitality. population is "the estimated number of all users (including both first and second language speakers) in terms of three levels", the aforementioned three levels being: large, mid-sized, and small @xcite . vitality is categorised into four distinct classes: institutional, stable, endangered and extinct, according to the expanded graded intergenerational disruption scale (egids) grid @xcite . we plotted the languages in a 12-point grid, according to vitality and number of speaker population. the size of the outer circles corresponds to the number of languages in one category. according to figure 1 , a large number of languages are endangered with small speaker populations, or stable but with mid or small speaker population numbers. note that two classes do not have any representation in this grid. therefore, hereafter we only refer to the remaining 10 classes. ## resource & tool support distribution we analyse how languages in the ethnologue categories are being treated with respect to data (annotated and un-annotated) availability, inclusion in multilingual web-based platforms and inclusion in pre-trained multilingual models. this dataset was extracted in october-november, 2021. the dataset preparation process is given in appendix b. ## revisiting data availability-based language categorisation in order to analyse the robustness of using annotated data availability to categorise languages, we recreated joshi et al. (2020)'s language category plot. we plot the availability of annotated data in ldc and elra against the unannotated wiki data in 5a 13 . in 5b we plot the same graph including the huggingface datasets as well. we note a clear relationship between joshi et al. (2020) categories, and the ethnologue classes. as shown in tables 8 and 9 in appendix k, all the extinct languages as well a vast majority of endangered languages are in class 0 of joshi et al. (2020)'s categorization. on the other hand, class 5 languages are all large-institutional. although both graphs have the same trends, as shown in figure 5 and the discussion in appendix k, 87 languages have changed their class (84 are promotions) when huggingface is considered. interestingly, class of welsh changes from 1 to 3, and azerbijanis changes from 1 to 4. this cautions us not to rely on a hard categorisation based on a partial set of data repositories. to further explain the limitations of a language categorisation that relies on annotated datasets defigure 4: language gdp in billions of dollars (log) vs wikipedia article count (log). rived only from a set of repositories, consider gorontalo and gujarati languages. both belong to class 1 in joshi et al. (2020)'s categorisation. gorontalo is a mid-endangered language with 1 million l1 speakers. it is not in google keyboard or facebook language list, nor is it included in pre-trained multilingual models. in contrast, gujarati is a large institutional language with 56 million l1 speakers. it is included in all of the above three lists. in addition, gujarati + "natural language processing" query returns 1960 results in google scholar, and has 189 papers in acl anthology corpus extracted by rohatgi (2022). the corresponding query for gorontalo returns only 81 results, and 0 results in rohatgi (2022)'s corpus. bird (2022) builds a similar argument by comparing tamil (75 million speakers) and cree (75,000 speakers). in order to further understand where papers of languages traditionally known as low-resource languages are published, we carried out a languagespecific analysis. we identified three survey papers: sinhala (de silva, 2021), sindhi (jamro, 2017), and hausa (zakari et al., 2021) (all are largeinstitutional languages, with joshi et al. (2020)'s category being 0, 1 and 2, respectively). we noted down the publishing venues of the research papers cited in these surveys. these results are plotted in figure 7 . we see that apart from the acl venues, there are: ieee conferences, other conferences (not ieee or acl anthology), other journals (not in acl anthology) and pre-prints/thesis/white papers/reports. while different languages show different patterns (e.g. sinhala mostly gets published in regional ieee conferences, while sindhi gets pub-lished in other (regional) journals) there is one common observation -there is extremely low number of papers in anthology, even for lrec and workshops published in acl anthology. a further look confirms that most of the other conferences and journals are either local or regional. further, we carried out the google scholar queries shown in ## case study: sinhala in joshi et al. (2020)'s language categorisation, the class of sinhala is ambiguous -while sinhala is categorised as class 0, its synonymous term 'sinhalese' is categorised into class 1. despite its exact category, sinhala has been considered a lowresource language even in recent research @xcite . in contrary, sinhala has its presence in wikipedia, huggingface, google keyboard, facebook, as well as xlm-r. so why is sinhala still considered low-resource? we went through all the sinhala nlp papers cited in de silva (2021)'s survey paper to get an idea about the datasets presented in each of the papers, whether the code and data are publicly available and whether any language tool has been released. figure 6 visualizes this information. only 11.43% of papers has data set publicly released (10.29% in personal repositories, 1.14% in public repositories) and only 9.71% of papers have code publicly released. only 5.71% have released tools. working behind closed doors has shown its negative consequences -within a small time span, two research groups started working on sinhala word-net @xcite , but none has been successfully completed. interestingly, none is available to be accessed now. this is common with some other tools that are claimed to be publicly released -they are not accessible. this suggests the lack of infrastructure support to maintain such tools. de silva (2021)'s author graph highlights another problem -the researchers seem to be working in silos, with almost zero interaction between research groups. on the positive side, recently, the use of pre-trained multilingual models has shown its benefit @xcite @xcite . ## discussion we analysed the linguistic disparity in a global scale. thus, inevitably, the analysis was limited to only a set of factors, which could be de-termined by the freely available data. in contrast, the eu-funded european language equality (ele) project (grützner-zahn and rehm, 2022) categorised european languages with respect to language resources, tools, as well as contextual factors such as economic and financial factors. this analysis is very comprehensive, however, it does not shed any light on the vast majority of the languages in the world. an ambitious project would be to extend this effort in a global scale. in order to highlight the importance of carrying out frequent analysis of linguistic disparity, we recorded the number of wikipedia articles and huggingface dataset counts as of july 2022. as shown in tables 11 and 12 in appendix o, 611 new datasets were added to large-institutional category alone, within less than an year. however, for the small-extinct/endangered/stable/institutional classes altogether, only 9 datasets have been added. this trend of rich getting richer is a concern as this shows that the average interest still lies with the few languages that are already enjoying an abundance of datasets. as for wikipedia, an astounding number of articles have been added to large-institutional category. many other language categories have also received articles, suggesting community involvement in content creation. it would be interesting to check whether this content addition impacts the ethnologue categorisation, however, we lack historical ethnologue data to conduct this analysis. we highlighted that the inclusion of a language in a pre-trained multilingual model provides an added advantage for a language. however, not many languages are included in the available models. at least for the languages where text data is there, pre-trained multilingual models should be publicly released. while doing so, models including related languages would be more beneficial @xcite . many languages are missing in wikipedia or commoncrawl. thus, community engagement should be promoted and funded to improve language-specific wikipedias. wikimedia grant scheme is one useful lifeline 16 . @xcite reported the possibility to web-mine data for 1500 languages. we hope this data will be publicly available. for spoken languages that do not have any text @xcite , extra effort is needed to collect speech data. there should be initiatives (preferably funded, for languages in global south) to create annotated data, even in small quantities, for languages that have monolingual data. inuktitut, a mid-institutional language with about 40,000 speakers has been recently included in facebook, with the support from a local learning center @xcite . this is welcome newscollaborations between locals and tech giants can facilitate the inclusion of languages in the web platforms. however, inuktitut is a north american language. adding an african language to facebook or google language list may face more challenges. not all authors have added data to public repositories, which also have limitations. particularly, many do not have language or task-wise categorisation of data, and meta data is not collected. we hope acl can take the initiative to setup a repository that does not have the limitations identified in our survey. a similar initiative is preferable to create an infrastructure to host language tools. as nlp researchers from global south, we have our own interpretation of the reasons for many languages having research papers in non-acl venues. many reviewers in acl conferences are sceptical of techniques tested only on a language not popularly known. with time, authors stay away from submitting to these venues, as they anticipate the possible outcome. while there are several workshops welcoming low-resource language research, most of them are non-indexed. this is a concern in institutions that take indexed publications as a measure of academic success. travelling to acl venues is expensive for researchers from the global south, and many conferences are held in countries with high visa restrictions. thus, hybrid events with less expensive online versions are a blessing for such researchers. @xcite found no evidence that research papers dealing with more languages in their evaluation having any advantage over those that do not when considering the number of citations, which means researchers have no incentive to test their systems in many languages. or-16 https://meta.wikimedia.org/wiki/ grants:start ganising multilingual shared tasks and more recognition for papers presenting multilingual datasets might help alleviating this problem. finally, we showed the need to discuss the full situation of languages used in research with respect to the socio-economic status as well as resource availability, rather than saying the language is lowresource just by considering data availability. these are the limitations of this study: the use of language names is not consistent across different data sources. we put every effort to use a uniform language list across data sources, however there can be a few languages that we missed. we used the logic by @xcite to check the existence of a language name in a paper. thus, the extracted data may have some noise, so does google scholar search. as already mentioned, task-wise dataset analysis is extremely noisy. in order to carry out better analysis in the future, we recommend: (1) creating a map of synonyms of languages, (2) a widely accepted list of nlp tasks, (3) nlp papers adhering to the bender rule (bender, 2019) and ( 4 ) recording the meta data of the datasets reported in repositories and in research papers (data statements @xcite would be a good starting point). ## conclusion the objective of this research was to provide a multi-facet analysis of the linguistic disparity in the world. we showed that such an analysis provides a more detailed view of the linguistic disparity, rather than depending on the dataset (particularly annotated) availability. we provided some preliminary recommendations to get these languages out of low-resourcefulness, which we hope would be taken positively by the stakeholders. we hope there would be more frequent analysis of this sort. in support of such efforts, we release our code to generate the visualisations shown in this paper as well as the relevant data 17 .