Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 I ndic Languages (2022)

Abstract: We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 million sentence pairs from existing, publicly available parallel corpora, and additionally mine 37.4 million sentence pairs from the Web, resulting in a 4× increase. We mine the parallel sentences from the Web by combining many corpora, tools, and methods: (a) Web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Further, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and models are available publicly at Samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages.

## introduction the advent of deep-learning (dl) based neural encoder-decoder models has led to significant * the first two authors have contributed equally. † corresponding author: miteshk@cse.iitm.ac.in. ‡ dedicated to the loving memory of my grandmother. a case in point is that of languages from the indian subcontinent, a very linguistically diverse region. india has 22 constitutionally listed languages spanning 4 major language families. other countries in the subcontinent also have their share of widely spoken languages. these languages are closely related both genetically and through contact, which led to significant sharing of vocabulary and linguistic features @xcite . these languages account for a collective speaker base of over 1 billion speakers. the demand for quality, publicly available translation systems in a multilingual society like india is obvious. however, there is very limited publicly available parallel data for indic languages. given this situation, an obvious question to ask is: what does it take to improve mt on the large set of related low-resource indic languages? the answer is straightforward: create large parallel datasets and train proven dl models. however, collecting new data with source en-as en-bn en-gu en-hi en-kn en-ml en-mr en-or en-pa en-ta enmanual translations at the scale necessary to train large dl models would be slow and expensive. instead, several recent works have proposed mining parallel sentences from the web @xcite @xcite . the representation of indic languages in these works is poor, however (e.g., ccmatrix contains parallel data for only 6 indic languages). in this work, we aim to significantly increase the amount of parallel data on indic languages by combining the benefits of many recent contributions: large indic monolingual corpora @xcite , accurate multilingual representation learning @xcite , scalable approximate nearest neighbor search @xcite @xcite , and optical character recognition (ocr) of indic scripts in rich text documents. by combining these methods, we propose different pipelines to collect parallel data from three different types of sources: combining existing datasets and the new datasets that we mine from the above-mentioned sources, we present samanantar, 1 the largest publicly available parallel corpora collection for indic languages. samanantar contains ∼ 49.7m parallel sentences between english and 11 indic languages, ranging from 141k pairs between english-assamese to 10.1m pairs between english-hindi. of these, 37.4m pairs are newly mined as a part of this work and 12.4m are compiled from existing sources. thus, the newly mined data is about 3 times the existing data. to evaluate if samanantar advances the state of the art for indic nmt, we train a multilingual model, called indictrans, using samanantar. we compare indictrans, with (a) commercial translation systems (google, microsoft), (b) publicly available translation systems opus-mt @xcite , mbart50 @xcite , and cvit-ilmulti @xcite , and (c) models trained on all existing sources of parallel data between indic languages. across multiple publicly available test sets spanning 10 indic languages, we observe that indictrans performs better than all existing open source models and even outperforms commercial systems on many benchmarks, thereby establishing the utility of samanantar. the three main contributions of this work, namely, (i) samanantar, the largest collection of parallel corpora for indic languages, (ii) in-dictrans, a multilingual model for translating from en-indic and indic-en, and (iii) human judgments on cross-lingual textual similarity for about 9,566 sentence pairs, is publicly available ( https://indicnlp.ai4bharat .org/samanantar/ ). in addition, it also contains parallel sentences between the 11 2 = 55 indic language pairs obtained by pivoting through english (en). to build this corpus, we first collated all existing public sources of parallel data for indic languages that have been released over the years, as described in section 2.1. we then expand this corpus further by mining parallel sentences from three types of sources from the web as described in sections 2.2 to 2.4. ## analysis of the quality of the mined parallel corpus we now describe the intrinsic evaluation of the data that we mined as a part of this work using the methods described in sections 2.2, 2.3, and 2.4). this evaluation was performed by asking human annotators to estimate cross-lingual semantic textual similarity (sts) of the mined parallel sentences. ## indictrans: multilingual, single indic script models the languages in the indian subcontinent exhibit many lexical and syntactic similarities on account of genetic and contact relatedness @xcite . genetic relatedness manifests in the two major language groups considered in this work: the indo-aryan branch of the indo-european family and the dravidian family. owing to the long history of contact between these language groups, the indian subcontinent is a linguistic area @xcite exhibiting convergence of many linguistic properties between indian sentence english sentence las bucket error so, what are their strengths? 0.70 mr should be ''so, what are these containers?'' johor, also spelled as should be ''located in the north-western johore, is a state of malaysia in 0.70 mr part of the state of johor, the district of the south of the malay peninsula. this city is also known as muwar.'' should be ''this hyderabad-dubai the flights between hyderabad and 0.68 mr tour will start from shamshabad gorakhpur will begin from 30 april airport in hyderabad'' the province is divided into ten districts. 0.81 ma should be ''six districts'' the sensex was trading with gains of 150 0.76 ma mistake with numbers points, while the nifty rose 52 points in trade. parupalli kashyap advcanced to korea 0.88 da semifinals became quarter finals open semifinals. inflammation of the interior portion of the eye, known as uveitis, can cause blurred vision and 0.89 da it should be ''when exposed to eye pain, especially when exposed to light high amounts of light'' (photophobia). ## experimental setup we evaluate the usefulness of samanantar by comparing the performance of a translation system trained using it with existing state of the art models on a wide variety of benchmarks. ## results and discussion the results of our experiments on indic-en and en-indic translation are reported in ## conclusion we present samanantar, the largest publicly available collection of parallel corpora for indic languages. in particular, we mine 37.4m parallel sentences by leveraging web crawled monolingual corpora as well as recent advances in multilingual representation learning, approximate nearest neighbor search, and optical character recognition. we also mine 83.4m parallel sentences between 55 indic language pairs from this english-centric corpus. we collect human judgments for 9,566 sentence pairs from samanantar and show that the newly mined pairs are of high quality. our multilingual single-script model, indictrans, trained on samanantar outperforms existing models on a wide variety of benchmarks, demonstrating that our parallel corpus mining approaches can contribute to high-quality mt models for indic languages. to further improve the parallel corpora and translation quality for indian languages, the following areas need further exploration: (a) improving labse representations for low-resource languages and longer sentences, especially benefiting from human judgments, (b) optimizing training schedules and objectives such that they utilize data quality information and linguistic similarity, and (c) pre-training multilingual models. we hope that the three main contributions of this work-samanantar, indictrans, and a manually annotated dataset for cross-lingual similarity-will contribute to further research on nmt and multilingual nlp for indic languages.