Efficient Multilingual Text Classification for I ndian Languages (2021)

Abstract: India is one of the richest language hubs on the earth and is very diverse and multilingual. But apart from a few Indian languages, most of them are still considered to be resource poor. Since most of the NLP techniques either require linguistic knowledge that can only be developed by experts and native speakers of that language or they require a lot of labelled data which is again expensive to generate, the task of text classification becomes challenging for most of the Indian languages. The main objective of this paper is to see how one can benefit from the lexical similarity found in Indian languages in a multilingual scenario. Can a classification model trained on one Indian language be reused for other Indian languages? So, we performed zero-shot text classification via exploiting lexical similarity and we observed that our model performs best in those cases where the vocabulary overlap between the language datasets is maximum. Our experiments also confirm that a single multilingual model trained via exploiting language relatedness outperforms the baselines by significant margins.

## related work one of the main problems in multilingual classification is the significant lack of resources @xcite . thus, analysis in multiple languages is often addressed by transferring knowledge from resource-rich to resource-poor languages @xcite , because there are no resources available in other languages. much of the work in subjectivity analysis has been applied to english data, though work on other languages is growing: e.g., japanese data are used in @xcite @xcite , german data are used by @xcite . lexical approaches for sentiment analysis necessitate language specific lexical and linguistic resources. generating these resources is very time consuming and often requires a lot of manual work. methods have been developed for the mapping of subjectivity lexicons to other languages. to this aim, @xcite use a machine translation system and subsequently use a subjectivity analysis system that was developed for english to create subjectivity analysis resources in other languages. another approach in obtaining subjectivity lexicons for other languages than english was explored by @xcite . in this work, authors attempt to leverage on the resources available for english and, by employing machine translation, generate resources for subjectivity analysis in other languages. this paper introduces a method for creating a subjectivity lexicon for languages with scarce resources. further on, another approach to building lexicons for languages with scarce resources is presented by @xcite . this method is able to build a subjectivity lexicon by using a small seed set of subjective words, an online dictionary, and a small raw corpus, coupled with a bootstrapping process that ranks new candidate words based on a similarity measure. machine translation for multilingual text classification has also seen attention from researchers. the approach is to use a machine translation system to translate texts in other languages into english : the text is translated from the original language into english, and then english-language resources such as sentiwordnet are employed @xcite . @xcite translated only sentiment units with a pattern based approach. @xcite used uni-grams, bi-grams and tf-idf features for building support vector machines on translated text. @xcite built latent dirichlet allocation models to investigate how multilingual concepts are clustered into topics. translation systems, however, have various problems, such as sparseness and noise in the data @xcite . sometimes, the translation system does not translate essential parts of a text, which can cause serious problems, possibly reducing well-formed sentences to fragments. therefore, we put our efforts for training a multilingual classifier without the use of existing machine translation systems. ## methodology india is known as the land of many tongues @xcite . there is no single language called "indian". india speaks hundreds of languages and dialects @xcite . some are extinct, while some are still in use with considerable speakers. despite having a lot of different scripts, most of the indian languages still share a lot of lexical features and common words which have the same root and meaning which can be utilized to help improve the quality of zero-shot as well as text classification systems trained on them. so, in this paper we propose our technique of performing zero-shot classification via exploiting the language relatedness. also, we investigate how multilingual classification models perform in case of indian languages. to do this efficiently, we exploited the lexical similarity via two techniques namely unified transliteration and subword segmentation. ## results and analysis ## conclusions and future work in this work, we explored effective methods to exploit lexical similarity between related indian languages in order to improve the quality of classification systems on low resource indian languages. our results confirm that a model trained on one indian language can be reused for another indian language and will provide good results if there is a significant vocabulary overlap between the two datasets. we also proved that for low resource indian languages, multilingual models outperform the baseline models. the reason is the ability to learn not just from the training data of the language in question, but also from the other language datasets. unified transliteration and subword segmentation further increase the accuracy by increasing the vocab overlap among the datasets. also, to get more increase in accuracy, one can try with large size datasets. for future work, we will try to apply this technique to other nlp related tasks for indian languages.