A3-108 Machine Translation System for Similar Language Translation Shared Task 2020 (2020)

Abstract: In this paper, we describe our submissions for Similar Language Translation Shared Task 2020. We built 12 systems in each direction for Hindi⇐⇒Marathi language pair. This paper outlines initial baseline experiments with various tokenization schemes to train statistical models. Using optimal tokenization scheme among these we created synthetic source side text with back translation. And prune synthetic text with language model scores. This synthetic data was then used along with training data in various settings to build translation models. We also report configuration of the submitted systems and results produced by them.

## introduction machine translation systems are models which aim to translate text from one language into another. there are multiple ways of building such a model (rule based, data driven, hybrid etc.). in this system description paper, we use data driven techniques to build mt systems. as the name suggests, data driven mt systems make use of parallel sentences (i.e. x th sentence in two languages have same meaning). we make use of statistical @xcite and neural @xcite methods to build systems for hindi marathi pair. hindi marathi language pair comes under purview of similar languages. similar languages are languages which exhibit lexical and structural similarities @xcite . this can be due to common ancestry or being in close proximity for long time. in current digital age communication, translation between similar language is a justifiable requirement. but there is a scarcity of good quality bitext for many language pairs, as is the case of hindi marathi. hence, we used characteristics displayed by similar languages (in this case hindi and marathi) like similar form of spelling, pronunciation etc. following kunchukuttan and bhattacharyya (2017) and @xcite we made use of byte pair encoding @xcite and morfessor toolkit @xcite respectively as part of preprocessing step before training. using cues from @xcite and looking at the size of training data provided, we use statistical method to build initial models. to further salvage similarity between this language pair we made use of backtranslation @xcite to generate more synthetic data for further training using both neural and statistical methods. for this shared task we developed 12 translation systems in each direction (hindi ⇐⇒ marathi). to rank systems, we went through some test instances subjectively and also compared our bleu scores with another translation system. and chose top 2 systems in both direction using both subjective examination and detoknized bleu scores. subsequent sections give more detailed overview of systems developed. ## seed mt systems using different tokenization schemes experiments in @xcite show that statistical machine translation model fairs better when compared to neural model in case of low resource setting. so, we make use of smt model to make initial baseline systems using various tokenization schemes. we use these systems as seed system, used to create synthetic dataset for further training by back translation. ## mt models using augmented bitext all these dataset combinations were used to train following methods to build mt models with respective default configurations available in respective toolkits,