Large-scale Machine Translation for I ndian Languages in E -commerce under Low Resource Constraints (2022)

Abstract: The democratization of e-commerce platforms has moved an increasingly diversified Indian user base to shop online. We have deployed reliable and precise large-scale Machine Translation systems for several Indian regional languages in this work. Building such systems is a challenge because of the low-resource nature of the Indian languages. We develop a structured model development pipeline as a closed feedback loop with external manual feedback through an Active Learning component. We show strong synthetic parallel data generation capability and consistent improvements to the model over iterations. Starting with 1.2M parallel pairs for English-Hindi we have compiled a corpus with 400M+ synthetic high quality parallel pairs across different domains. Further, we need colloquial translations to preserve the intent and friendliness of English content in regional languages, and make it easier to understand for our users. We perform robust and effective domain adaptation steps to achieve colloquial such translations. Over iterations, we show 9.02 BLEU points improvement for English to Hindi translation model. Along with Hindi, we show that the overall approach and best practices extends well to other Indian languages, resulting in deployment of our models across 7 Indian Languages.

## introduction as one of the largest e-commerce platform, we support a very diverse user base in terms of regional languages. product descriptions, catalog attributes, and product reviews help customers understand and compare various products available on the platform. for the growing user-base in india with non-english background, providing this information in regional indian languages makes their shopping experience more informative and friendly. with only 10% of the indian population being versed in english foot_0 , vernacular support is vital for the platform and its diverse users. in this work, we develop machine translation system to translate the available product data from english to regional languages to address this problem. given the size of the product catalog and user base, the volume of the data to be translated is in the order of 100s of millions. this poses a challenge to build translation systems that are robust, reliable, and precise at scale. the low resource nature of indian languages foot_1 is another challenge for data-hungry deep networks such as transformer @xcite . given a large enough parallel corpus, the transformer model can learn the inter-lingual mappings very well, even for very long sequences. these models can generate human-level precision translations for some resource-rich european languages @xcite . so theoretically, if we can get a large enough parallel corpus for indian languages, we can solve the automatic machine translation for indian languages. we build a training pipeline that can take monolingual corpus(abundantly available from public and in-house sources) and generate a high-quality synthetic parallel corpus. this is an efficient and effective approach, especially when paired with the active learning component over model iterations. for hindi, starting with 1.2m parallel examples, we have compiled over 400m synthetic parallel examples with numerous model iterations. translation is an inherently one-to-many task where a single text can have various correct translations. the domain gap between the e-commerce domain and public domain (news, government sites, wikipedia, books, etc.) is significant. to showcase this, figure 1 has colloquial and non-colloquial hindi translations for a source sentence in english. both of these translations are correct, but as an e-commerce platform, we refrain from using noncolloquial and infrequently used words as it decreases the appeal of the information from the colloquial e-commerce english domain. based on the final training steps, translation models can generate appropriate translations at inference. we fine-tune the model only using the colloquial in-domain data with robust domain adaptation steps to get more colloquial translations. our contributions in this paper are as follows: with the help of sub-modules, we generate a vast amount of high-quality parallel corpus solving for low-resource indian languages. ## related work transformers @xcite are widely used architecture for seq2seq tasks. along with unigram-based subword tokens, the fully attentionbased model performs very well for translation tasks, even for longer sequences. translation is a well-explored area, and even for low-resource settings, significant work has already been done. along the lines of data gathering -collecting parallel corpora @xcite , mining multilingual sets and retrieving parallel entries @xcite , iterative cross-lingual alignments @xcite has been explored. @xcite , showed parallel corpus filtering on web crawled data. transfer learning is also a convenient approach to improve final model performance in low resource settings. @xcite explored leveraging large language models trained on unlabelled data for translation tasks. this approach works well only if we have strong pre-trained models. for indian language settings, this is typically not the case. also, synthetic data generation is very inefficient without active learning. @xcite shows that translation models can help with pseudo labeling, but this improvement saturates without external feedback. @xcite has explored an active learning framework for machine translation. gupta et al. (2021) investigate the active learning methods for machine translation in indian languages settings. @xcite even shows that completely unsupervised machine translation is possible using just monolingual data. but these practices don't work in large-scale settings. given a large amount of good quality parallel data, supervised methods still beat other weak methods. especially for production settings, there has not been much exploration done at large-scale systems starting with low resource settings. ## overall pipeline we use transformer encoder-decoder model with 6 encoder layers and 6 decoder layers with hidden size of 512. we use 32,000 unigram subword tokens trained on data from all domains. this configuration has 93m parameters. as a pre-processing step, we split long paragraphs into sentences and translate independent sentences using the transformer model. ## results and discussion we benchmark our models on manually annotated product descriptions(pd) test set along with public indic wat21 benchmark @xcite in the training pipeline has consistently shown better translation models throughout the model iterations paired with active learning, adding more monolingual data and filtered high-quality synthetic parallel translations. as evident from the plot 4, the addition of more synthetic data in pretraining, the addition of forward translation for pretraining as well as domain adaptation, model retraining from scratch with higher quality corpus and better pipeline sub-modules, and active learning steps show very significant improvements at each stage. starting from 32.3 bleu score, we have reached 41.32 bleu score, which is a massive improvement just using a few active learning steps and synthetic corpus updates. ## conclusion in this work we have shown that synthetic parallel corpus generation and data filtering is a viable option to train large-scale translation models in lowresource settings. also we show that active learning can consistently improve the model. we build very robust, large-scale models which work very precisely on our in-domain data and also outperform google on public general domain benchmarks consistently. we also show how building colloquial models are important for ease of understanding, and we also show that our overall approach and best practices extend well to multiple indian languages.