A3-108 Machine Translation System for Similar Language Translation Shared Task 2021 (2021)

Abstract: In this paper, we describe our submissions for the Similar Language Translation Shared Task 2021. We built 3 systems in each direction for the Tamil ⇐⇒ Telugu language pair. This paper outlines experiments with various tokenization schemes to train statistical models. We also report the configuration of the submitted systems and results produced by them.

## introduction machine translation is a process of translating text from a source to a target language. there are multiple ways of building such a system -rule-based, data-driven, hybrid etc. in this shared task, we use data-driven method to create machine translation system for tamil ⇐⇒ telugu. due to low-resource setting of this language pair in the shared task, we use statistical machine translation method @xcite , @xcite to build systems. tamil telugu language pair comes under the bracket of similar languages. similar languages show similarity in their lexical and syntactical properties @xcite . this may be due to them being in close proximity of each other for long time. this can also be due to common ancestry. in the current digital context, translation between similar languages is of importance. but there can be scarcity of good quality parallel text. in the current shared task, we have a language pair which is morphologically rich and with 39k parallel sentences. so, following @xcite and @xcite we use sentencepiece foot_0 @xcite and morfessor foot_1 @xcite to segment tokens in the dataset into subwords. and due to the size of parallel text ( 39k parallel text) coming under purview of low resource, we make use of moses foot_2 @xcite to create statistical machine translation models @xcite . for this shared task we developed 3 translation systems (1 primary and 2 contrastive) in each direction tamil ⇐⇒ telugu. for each output we postprocessed and detokenized translation output depending on the tokenization scheme for target language. to choose a primary and 2 contrastive systems, we compared bleu @xcite scores on output of development dataset for each system using sacrebleu foot_3 @xcite . the following sections give more details about the systems developed. ## smt systems using different schemes we used various tokenization schemes to build translation systems. evaluated these systems on the development dataset. after post-processing, detokenizing and scoring each translation output, we submit output systems as primary and contrastive submissions accordingly.