Annotation Efficient Language Identification from Weak Labels (2020)

Abstract: India is home to several languages with more than 30m speakers. These languages exhibit significant presence on social media platforms. However, several of these widely-used languages are under-addressed by current Natural Language Processing (NLP) models and resources. User generated social media content in these languages is also typically authored in the Roman script as opposed to the traditional native script further contributing to resource scarcity. In this paper, we leverage a minimally supervised NLP technique to obtain weak language labels from a large-scale Indian social media corpus leading to a robust and annotation-efficient language-identification technique spanning nine Romanized Indian languages. In fast-spreading pandemic situations such as the current COVID-19 situation, information processing objectives might be heavily tilted towards under-served languages in densely populated regions. We release our models to facilitate downstream analyses in these low-resource languages. Experiments across multiple social media corpora demonstrate the model’s robustness and provide several interesting insights on Indian language usage patterns on social media. We release an annotated data set of 1,000 comments in ten Romanized languages as a social media evaluation benchmark.

## introduction much of the current nlp research focuses on a handful of world languages (e.g., english, french, spanish etc.). they enjoy substantially larger computational linguistic resources as compared to their low-resource counterparts (e.g., bengali, odia etc.). however, in the midst of global-scale events like the ongoing covid-19 pandemic, demand for linguistic resources might get recalibrated; information processing objectives might be heavily tilted towards under-served languages that are prevalent in many densely populated regions. in this paper, we focus on language identification in noisy, social media settings -a basic and highly critical linguistic resource prerequisite for downstream analysis in a multilingual environment. our solution extends support for nine major indian languages (see, annotator availability is a major concern that may constrain data acquisition efforts in low resource settings @xcite . our proposed solution is extremely annotation efficient; it utilizes a recent result @xcite to automatically group a multilingual corpus into largely monolingual clusters that can be extracted with minimal supervision. using a mere 260 annotated short documents (youtube video comments), we assign weak labels to a data set of 2.8 million comments spanning the aforementioned languages. our model performs favorably when compared against an existing commercial solution. while census data and surveys can provide useful information about linguistic diversity and spread, analyses of user-generated multi-lingual corpora can complement these surveys with additional useful insights of their own. we conduct a focused analysis to explore if the (estimated) distribution of web-usage of hindi across different indian states aligns with common knowledge. our analysis indicates that hindi's web-presence is considerably higher in a cluster of north indian states referred to as the hindi belt @xcite as compared to the south indian states. we further analyze similar research questions concerning the relative usage of the roman script and the native script for hindi. we finally conclude with a small exploratory study on our method's effectiveness in detecting languages with trace presence in multiple corpora and outline some of the possible utilities. contributions: our main contributions of the paper are the following: ## data set: youtube video comments in order to construct our language identification system, we would require a web-scale indian social media data set that (i) has considerable presence of the nine languages we are interested in, and (ii) captures a representative fraction of the indian web users. to achieve this two-fold goal, we consider a data set introduced in @xcite to analyze the 2019 indian general election. the data set consists of comments on youtube videos hosted by popular news outlets in india. overall, the corpus consists of 6,182,868 comments on 130,067 videos by 1,518,077 users posted in a 100 day period leading up to the 2019 indian general election. why youtube? as of january 2020, youtube is the second-most popular social media platform in the world drawing 2 billion active users @xcite . youtube is the most popular social media platform in india with 265 million monthly active users (225 million on mobile), accounting for 80% of the population with internet access (hindustan-times, 2019; yourstory, 2018). youtube video comments have been used as data sources to analyze recent important events @xcite @xcite . why this data set? the data set considers two highly popular youtube news channels for each of the 12 indian states that contribute 20 or more seats in the lower house of the parliament. state boundaries in india were drawn along linguistic lines @xcite . the dominant regional language in the hindi belt @xcite is hindi, and the other states feature a unique dominant language written in either the latin alphabet (in informal settings) or a native script. all the nine languages we focused on (listed in in addition to these 24 regional news channels, the data set considers youtube channels for 14 highly popular national news outlets (listed in the appendix). overall, this implies 38 youtube channels (24 regional, 14 national) with an average subscriber count of 3,338,628. learning from weak labels: the role of unlabeled and weakly (or noisily) labeled data in supervised learning is a well-studied problem and has received sustained focus @xcite , and annotation efficiency in low-resource settings is a well-established requirement @xcite . our work leverages lpolyglot @xcite , a recentlyproposed method for noisy language identification that requires minimal supervision. we utilize it as a dependency to obtain weak labels and reduce annotation burden and construct a substantially more robust system. language identification: while language identification of well-formed text is a nearly-solved problem, the difficulty in identifying language in a noisy social media setting is well-established @xcite @xcite @xcite . we see our work as a part of this continuing trend and as an important resource contribution to analyze indian social media. romanized indian languages: in the context of processing indian languages expressed on the web, challenges posed by the use of roman script instead of the native script have been reported in several recent studies in the context of code-mixed english-bengali @xcite , and english-hindi @xcite text. while addressing word level language identification, @xcite reported that 90% of posts in indian languages on facebook are expressed in roman script. prevalence of romanized hindi has also been previously reported in @xcite . our study takes previous findings one small step forward with a (noisy) geographical analysis of hindi web usage. ## background in this section, we summarize a few key nlp models and results critical to our methods. ## method research question: how to construct an annotation-efficient language identification method supporting a wide array of indian languages? our method has two main components: (i) an annotation-efficient procedure to construct a substantial data set with weak labels, (ii) a supervised system trained on a data set comprising this corpus and an existing data set, d tatoeba @xcite , a well-known annotated data set supporting 355 languages @xcite . for the construction of this data set, the election corpus @xcite is stripped of all comments containing any non english character. this maintains the focus on romanized indian languages with the native variants sourced from d tatoeba . ## experimental setup test set: we construct an indian language test set consisting of 1,000 annotated youtube comments (consensus labels by two proficient annotators per language) in 10 languages (bengali, english, gujarati, hindi, kannada, malayalam, marathi, odia, tamil, telugu), and 100 documents in each language. these documents are randomly sampled from the output of f weaklabel and are never seen during our supervised training phase with weak labels. the average number of tokens in the comments is 22.6 ± 18.7. please see appendix for detailed statistics of our test data set. baseline: we consider a commercial solution (google) that can detect over 100 languages including the 10 mentioned above (referred to as googlelangid) as our baseline. in addition to googlelangid, palakodety et al. (2020a) compared against fasttextlangid. we avoid this comparison because we feel it is unfair to compare against fasttextlangid given that it does predicted label bn en gu hi kn ml mr or ta te ol bn 97 0 0 fairness: we first emphasize that the main purpose of comparing against googlelangid is not to claim that our annotation-efficient solution is superior to googlelangid across the board. our goal is rather to attract the research community's attention to our solution's effectiveness in this under-explored, specific domain of noisy social media texts generated in the indian subcontinent. a fairer performance comparison between the two methods would require the methods to be trained on identical data sets and comparable computation budget. due to these varying levels of resources, it is not possible to claim one method's superiority over the other. we are rather highlighting our method's (1) annotation efficiency and (2) ability to extend support for newer languages (e.g., at present googlelangid has limited support for odia (or) and no support for assamese as). ## results and analysis method overall accuracy excluding odia f end-to-end 99.4 99.8 googlelangid 84.1 93.4 our performance comparison highlights the following two points. first, we reiterate that our goal is not to claim that our method would perform better than googlelangid across the board, but rather, to demonstrate the value our method adds in processing noisy social media texts. it is possible that our method is more attuned to noisy short social media texts while googlelangid could be (possibly) trained on cleaner corpora which explains our method's stronger performance. this is further corroborated by the fact that even our mispredictions (barring one) remain confined to the regional languages while many of googlelangid's mispredictions are distributed across other languages (ol). second, googlelangid's weak performance in detecting odia highlights the gap in current solutions and shows how our method can effectively and efficiently address these issues foot_1 . recall that d tatoeba contains a large set of languages (including the native script versions of the indian languages considered in this paper). test accuracy on a held-out set was 98.4%. experiments reveal that introduction of the weakly labeled corpus does not impact performance on d tatoeba (identical test accuracy of 98.4%). ## conclusion in this paper, we present a language identification tool with a focus on nine major romanized indian languages. despite the widespread use of romanization on social media, nlp resources and tools often focus more on the native scripts. our tool integrates with an existing large-scale corpus and holds promise in being a valuable resource for indian social media analysis. our pipeline leverages a recent nlp algorithm and obtains weak labels for a large number of samples substantially reducing the annotation cost. finally, we conduct studies on the geographic extent, bilinguality, and romanization of hindi and observe that these align with existing studies and surveys.