Bhasa- A bhijnaanam: Native-script and romanized Language Identification for 22 I ndic languages (2023)

Abstract: We create publicly available language identification (LID) datasets and models in all 22 Indian languages listed in the Indian constitution in both native-script and romanized text. First, we create Bhasha-Abhijnaanam, a language identification test set for native-script as well as romanized text which spans all 22 Indic languages. We also train IndicLID, a language identifier for all the above-mentioned languages in both native and romanized script. For native-script text, it has better language coverage than existing LIDs and is competitive or better than other LIDs. IndicLID is the first LID for romanized text in Indian languages. Two major challenges for romanized text LID are the lack of training data and low-LID performance when languages are similar. We provide simple and effective solutions to these problems. In general, there has been limited work on romanized text in any language, and our findings are relevant to other languages that need romanized language identification. Our models are publicly available at https://github.com/AI4Bharat/IndicLID under open-source licenses. Our training and test sets are also publicly available at https://huggingface.co/datasets/ai4bharat/Bhasha-Abhijnaanam under open-source licenses.

## introduction in this work, we focus on building a language identifier for the 22 languages listed in the indian constitution. with increasing digitization, there is a push to make nlp technologies like translation, asr, conversational technologies, etc. @xcite available as a public good at population scale @xcite . a good language identifier is required to help build corpora in low-resource languages. for such languages, language identification is far from a solved problem due to noisy web crawls, small existing datasets, and similarity to high-resource languages @xcite . existing publicly available lid tools like cld3 foot_0 , langid foot_1 @xcite , fast-text foot_2 @xcite and nllb 4 @xcite have some shortcomings with respect to indian languages. they do not cover all the above-mentioned 22 languages. in social media and chats, it is also common to use the roman script for most indian languages leading to substantial user-generated content in roman script. however, none of the lids have any support for the detection of romanized indian language text (except cld3 support for latin hindi). the widespread use of romanization implies that accurate romanized language identification models are a critical component in the nlp stack for indian languages, given that this affects over 735 million internet users @xcite . therefore, our work on developing accurate and effective romanized language identification models has the potential to make a significant impact in the nlp space for indian languages, particularly in the social media and chat application domains. hence, we undertake the task of creating a lid for these 22 indian languages. the main contributions of our work are as follows: our findings are relevant to other languages that need lid for romanized text. we require native script data and a transliteration model to create the synthetic romanized data for the target language. this romanized data serves as training data for the romanized lid. ## bhasha-abhijnaanam benchmark we describe the creation of the bhasha-abhijnaanam lid benchmark for 22 indian languages in native and roman script. ## indiclid model indiclid is a classifier specifically for indic languages that can predict 47 classes (24 native-script classes and 21 roman-script classes plus english and others). we create three classifier variants: a fast linear classifier, a slower classifier finetuned from a pre-trained lm, and an ensemble of the two models which trades off speed v/s accuracy. ## results and discussion while sampling native train samples from various sources. additionally, we removed the training samples from the benchmark samples when collecting sentences for the benchmark test set. we also made sure that there was no overlap between the test and valid sets. to create the romanized training set, we simply transliterated the native training set. as the dakshina test set @xcite provided parallel sentences for the native and roman test sets, there was no overlap between the roman train and test sets. ## conclusion we introduce an lid benchmark and models for native-script and romanized text in 22 indian languages. these tools will serve as a basis for building nlp resources for indian languages, particularly extremely low-resource ones that are "leftbehind" in the nlp world today @xcite . our work takes first steps towards lid of romanized text, and our analysis reveals directions for future work. research on indic languages. we would like to thank jay gala and ishvinder sethi for their help in coordinating the annotation work. most importantly we would like to thank all the annotators who helped create the bhasha-abhijnaanam benchmark.