Aksharantar: Open I ndic-language Transliteration datasets and models for the Next Billion Users (2023)

Abstract: Transliteration is very important in the Indian language context due to the usage of multiple scripts and the widespread use of romanized inputs. However, few training and evaluation sets are publicly available. We introduce Aksharantar, the largest publicly available transliteration dataset for Indian languages created by mining from monolingual and parallel corpora, as well as collecting data from human annotators. The dataset contains 26 million transliteration pairs for 21 Indic languages from 3 language families using 12 scripts. Aksharantar is 21 times larger than existing datasets and is the first publicly available dataset for 7 languages and 1 language family. We also introduce a test set of 103k word pairs for 19 languages that enables a fine-grained analysis of transliteration models on native origin words, foreign words, frequent words, and rare words. Using the training set, we trained IndicXlit, a multilingual transliteration model that improves accuracy by 15% on the Dakshina test set, and establishes strong baselines on the Aksharantar testset introduced in this work. The models, mining scripts, transliteration guidelines, and datasets are available at https://github.com/AI4Bharat/IndicXlit under open-source licenses.

## introduction the indian subcontinent is home to diverse languages across four major language families written in multiple scripts @xcite . in various settings such as instant messaging, web search, and social media, these languages are commonly romanized owing to users' familiarity with 1 meaning transliteration in sanskrit) the input tools for the roman script. often, there is a large diversity in how words are romanized: for instance, even the short word मैं (i) can be romanized in multiple ways: main, mai, mein, mei which overlap with the ways of romanizing another short word, में (in). this widespread usage of romanization and lack of standardization implies that accurate transliteration models form a critical component in the nlp stack for indian languages used by over 735 million internet users @xcite . further, accurate transliteration models have been shown to improve machine translation @xcite , romanized language models @xcite , ner @xcite , and script unification for multilingual models @xcite . given its importance, a transliteration for indian languages has received considerable research focus @xcite . however, the state-of-the-art (sota) results as reported in @xcite on the roman script to indian transliteration task have relatively low top-1 accuracy values ranging between 33.2% to 67.6% with an average of 51.8% across 12 languages. we believe that the low accuracy is a result of limited training datasets that are not representative of the diverse variations of romanization. we aim to address this open challenge in a manner similar to recent advances for low-resource languages in machine translation @xcite and speech recognition (bhogale et al., @xcite , namely by mining massive training corpora from web-scale data. specifically, we create aksharantar, a transliteration corpora from roman script to 12 indic language scripts spanning 21 indic languages. it is our next contribution is a multilingual model for romanized to native script transliteration for indian languages. our model gives sota performance on the dakshina benchmark @xcite for all the 12 languages in common with dakshina showing an improvement of 15% in accuracy over previous results. it also establishes a strong baseline on the aksharantar testset. our final contribution is a detailed analysis of the model's performance on the rich aksharantar testset. ablation studies indicate that the increased data size as well as the manually collected diverse dataset is a major contributor to the improved performance. the fine-grained testsets reveal named entities and low-frequency words as areas for improving transliteration models. the code and models are available under an mit license foot_0 , the aksharantar benchmark and all data we created manually are available under the cc-by license foot_1 , whereas all the mined data is available under the cc0 license foot_2 . ## related work existing indic transliteration corpora. very few transliteration corpora exist with indian language-roman script transliterations. refer to appendix a for detailed listing and statistics. most significant among these are the dakshina dataset @xcite and the brahminet corpus @xcite . dakshina contains native language words sourced from wikipedia and their romanizations created by native speakers, unlike aksharantar mostly consists of commonly used and shorter indic language words. mining transliteration pairs. @xcite mine the name pairs from wikipedia using interlanguage links between pages in multiple languages (similar to our use of wikidata titles' multilingual information). some approaches mine transliteration pairs from comparable document pairs based on a variety of heuristic signals @xcite . @xcite proposed a generative model for efficient unsupervised/semi-supervised mining of transliteration pairs. we employ unsupervised mining method proposed by @xcite to mine transliteration pairs from parallel corpora. @xcite mine transliteration pairs from monolingual corpora by transliterating the vocabulary of one language using a baseline system and then by filtering the generated data. multilingual models. multilingual models have been shown to improve performance on lowresource languages for many nlp tasks by transfer from high-resource languages and aligning representation of multiple languages in the same vector space @xcite . the transfer could be between genetically-related languages (nguyen and chiang, 2017) or contactrelated languages @xcite . multilingual models have been explored successfully for types of errors examples leaked translation word pairs अं तसर्ं यु क्त → interconnected highly agglutinated words on one side અં કલે રનો → ankleshwar ## mining transliteration pairs we explore sources for mining transliteration pairs. first, we compile existing publiclyavailable transliteration corpora listed in appendix a. then, we explore large-scale mining of transliterations from wikidata, parallel translation corpora and monolingual corpora. ## manual data collection collating existing sources and mining transliterations from web sources is insufficient for building a representative transliteration dataset because (i) mined corpora are predominantly composed of named entities, (ii) romanized native words in the dakshina dataset only cover frequent words in languages occurring on wikipedia and may not ensure sufficient word diversity to account for various transliteration phenomena (since wikipedia for most indic languages is small), (iii) mined data only covers 12 languages for which sufficient monolingual/parallel corpora are available and which have high grapheme-to-phoneme correspondence making mining feasible, (iv) we still need a diverse and accurate standard testset for all indic languages. to address these needs, we collect transliteration dataset asm ben guj hin kan kok mai mal mar pan san tam ind 90.8 92.8 90.8 96.8 98.0 98.8 90.8 94.0 96.8 94.8 78.0 80.0 sam 92.8 92.0 84.0 76.0 80.0 --80.0 90.0 86.0 84.0 80.0 ## indicxlit: a multilingual model for transliteration with the aksharantar training set, we train a transliteration model, indicxlit, for transliterating romanized indic language input to the native script. indicxlit is a single multilingual, multiscript transliteration model that supports 21 indic languages. we train a joint model since: (a) low-resource languages would benefit from transfer learning, (b) previous works show that multilingual transliteration models are better at generating canonical spellings @xcite , and (c) maintenance is easier for a single model. model architecture. we use a transformer-based encoder-decoder architecture @xcite . it is a multilingual character level transliteration model @xcite in a one-to-many setting, which consumes a romanized character sequence and generates an output character sequence in the indic language script. the input sequence includes a special target language tag token to specify the target language @xcite . model vocabulary, hyper-parameters, and training details are described in appendix f. the model size is 11 million parameters. ## analysis of indicxlit quality we analyze indicxlit's transliteration quality on the dakshina and aksharantar testset. we strictly ensure that there is no word overlap between training and test/validation sets for inference. note that the testsets considered for overlap computation include the dakshina testset. we remove a pair (en, t) from the training set if (i) the roman script word en is present in the romanized validation/test set of any language pair, or (ii) the indic script word t is present in the indic language validation/test set of any language pair. we report top-1 word level accuracy as our primary evaluation metric @xcite . additionally, we report top-3 and top-5 accuracies as well as f1-score in appendix g as our secondary evaluation metrics. we observe that major trends on all metrics are consistent. ## ablation studies we describe various ablation studies carried out on dakshina testset for 9 languages viz. bengali, gujarati, hindi, kannada, malayalam, marathi, punjabi, tamil, telugu and their results which drove ## conclusion in this work, we take a major step toward creating publicly available, open datasets and open-source models for transliteration in indic languages. we introduce aksharantar, the largest transliteration parallel corpora for 21 languages containing 26 million transliteration pairs, and covering 20 of 22 languages listed in the indian constitution. we also create a diverse, high-quality testset for romanized to indic script transliteration, covering word pairs with various characteristics and enabling finegrained analysis of different transliteration usecases. we also build indicxlit, a transformerbased transliteration model, for romanized input to indic script transliteration. indicxlit achieves state-of-the-art results on the dakshina testset. we also provide baseline results on the new aksharantar testset along with a qualitative analysis of the model performance.