A Large-scale Evaluation of Neural Machine Transliteration for I ndic Languages (2021)

Abstract: We take up the task of large-scale evaluation of neural machine transliteration between English and Indic languages, with a focus on multilingual transliteration to utilize orthographic similarity between Indian languages. We create a corpus of 600K word pairs mined from parallel translation corpora and monolingual corpora, which is the largest transliteration corpora for Indian languages mined from public sources. We perform a detailed analysis of multilingual transliteration and propose an improved multilingual training recipe for Indic languages. We analyze various factors affecting transliteration quality like language family, transliteration direction and word origin.

## introduction transliteration is an essential technology for mul tilingual and crosslingual capabilities in nlp ap plications to handle named entities, support cross script input methods. transliteration between en glish and indic languages is important since en glish is widely used in the indian subcontinent. in dic languages are written in different scripts from various writing systems. we focus on languages using scripts derived from the ancient brahmi script. their character sets are very different from the latin script making transliteration nontrivial. these scripts are abugida scripts, where the ba sic unit is the akshar which consists of one or more consonants along with a vowel diacritic @xcite . they exhibit a high degree of graphemetophoneme correspondence. there is a large overlap in the logical character sets of these scripts, though the visual appearance of the characters varies. the languages utilizing these scripts are said to exhibit orthographic similar ity on account of various shared characteristics @xcite . we undertake a systematic, largescale evalua tion of neural machine transliteration for 10 ma jor indic languages from 2 major language fami lies (indoaryan and dravidian languages) spoken by more than a billion speakers. other than brah minet @xcite and dakshina @xcite , no other previous work has ex plored a wide range of indic languages; dakshina only explores transliteration into indic languages. our major contributions are: • for a largescale evaluation, we mine 600k transliteration pairs across 10 languages from pub licly available parallel and monolingual sources. this is much larger than existing corpora like msrnews @xcite , brahminet @xcite , dakshina @xcite and other small datasets @xcite . the brahminet and dakshina datasets span multiple languages; brah minet is small and dakshina by design consists mostly of indian origin words. ## mining transliteration corpus this section explains our transliteration mining methods (from parallel and monolingual corpora) and presents an analysis of the mined corpus. we mine transliteration corpora from english to 10 in ## analysis: multilingual transliteration we study multilingual transliteration models with the intent of identifying factors that improve mul tilingual models. first, we describe our baseline multilingual model and then introduce different variants to improve the baseline model. baseline multilingual model @xcite . it is a characterlevel, attention based, encoderdecoder model with all the model components shared amongst all the languages. we train joint ex (multitarget, english to indian languages) and xe models (multisource, indian languages to english) separately. for ex models, we append a special target language token to the input sequence @xcite . language partitioning. to understand the role of orthographic similarity, we investigate two lan guage groupings: (a) all the indic languages are jointly trained, (b) indoaryan and dravidian lan guages are separately trained. vocabulary. indic languages use a variety of scripts with a high overlap in the logical character set, but assigned unique characters in the unicode character set. we investigate if transfer learning works better with a combined vocabulary by map ping logically equivalent characters across scripts for better transfer learning. we use the indicnlp library @xcite for mapping all in dic scripts to the devanagari script, thus combin ing the vocabularies of all languages. we experi ment with two configurations: (a) disjoint vocabu laries (i.e., different scripts), (b) combined vocab ularies (i.e., same script). combining the vocabu laries reduces the vocabulary significantly as the number of scripts reduces from 9 to 1. source language tag. in spite of the high de gree of orthographic similarity between indian lan guages, there are few cases of languagespecific variations. for instance, the malayalam script overloads a few characters with multiple sounds, bengali pronunciation of the aa vowel differs, etc. to make the model sensitive to these language specific variations in xe models, we add an spe cial source language token in the input sequence.