KT 2: K annada- T ulu Parallel Corpus Construction for Neural Machine Translation (2023)

Abstract: In the last decade, Neural Machine Translation (NMT) has experienced substantial advances. However, its widespread success has revealed a limitation in terms of reduced proficiency when dealing with under-resourced language pairs, mainly due to the lack of parallel corpora in comparison to high-resourced language pairs like English-German, EnglishSpanish, and English-French. As a result, researchers have increasingly focused on implementing NMT techniques tailored to underresourced language pairs and thereby, the construction/collection of parallel corpora. In view of the scarcity of parallel corpus for underresourced languages, the strategies for building a Kannada-Tulu parallel corpus and baseline models for Machine Translation (MT) of Kannada-Tulu are described in this paper. Both Kannada and Tulu languages are under-resourced due to lack of processing tools and digital resources, especially parallel corpora, which are critical for MT development. Kannada-Tulu parallel corpus is constructed in two ways: i) Manual Translation and ii) Automatic Text Generation (ATG). Various encoderdecoder based NMT approaches, including Recurrent Neural Network (RNN), Bidirectional RNN (BiRNN), and transformer-based architectures, trained with Gated Recurrent Units (GRU) and Long Short Term Memory (LSTM) units, are explored as baseline models for Kannada to Tulu (Kan-Tul) and Tulu to Kannada (Kan-Tul) sentence-level translations. Additionally, the study explores sub-word tokenization techniques for Kannada-Tulu language pairs, and the performances of these NMT models are evaluated using Character n-gram Fscore (CHRF) and Bilingual Evaluation Understudy (BLEU) scores. Among the baselines, the transformer-based models outperformed other models with BLEU scores of 0.241 and 0.341 and CHRF scores of 0.502 and 0.598 for KanTul and Kan-Tul sentence-level translations, respectively.

## introduction over the past two decades, statistical machine translation (smt) has held a dominant position in the field of @xcite . however, in recent years, nmt has emerged as the preferred approach, as is evident from its prevalence in majority of the shared tasks and surveys in the field of mt @xcite . nevertheless, the development of efficient mt systems for under-resourced languages, which often suffer from limited resources, remains an under-explored area @xcite highlighting the need for more attention and research in this domain @xcite . in a multilingual country like india, linguistic diversity is an integral part of daily life. people use their mother tongue as the primary means of communication and employ local or regional languages as a secondary mode of communication. in indian context, the rural population makes a significant contribution to the overall population of the country @xcite and many rural residents exclusively speak their mother tongue, making translation an essential tool to bridge language barriers @xcite . translation plays a pivotal role in facilitating effective communication and fostering understanding among a culturally rich and linguistically diverse population. however, human translators may not be available everywhere and human translation is also expensive @xcite . this has increased the huge demand for automatic translation viz., mt. kannada, the second oldest dravidian language, holds a significant place in the linguistic landscape of india. it is predominantly spoken by the people of karnataka, a state in southern india, where it serves as the official and administrative language. with approximately 44 million native kannada speakers worldwide, it holds a substantial speaker base. over 12.6 million foot_0 non-kannada speakers use kannada as a second or third language for communication and business. kannada has its own script derived from the brahmi script family, consisting of 49 characters, including 13 vowels, 2 diphthongs, and 34 consonants. linguistically, kannada is an agglutinative language characterized by a complex morphological structure. words in kannada are constructed by combining compatible suffixes and/or prefixes with root words, making words meaningful. one distinctive feature of kannada is its free word order, which allows flexibility in sentence construction. however, by convention, verbs are typically placed at the end of sentences. tulu, a dravidian language, predominantly spoken by over 2.5 million people in the regions of dakshina kannada and udupi in karnataka, as well as in parts of kasaragod in kerala, holds a special place as the mother tongue 2 for its speakers. tulu is distinctive among dravidian languages for its preservation of linguistic features and unique innovations. notably, it exhibits the use of 8 different cases (vibhakti), complexities in gender identification, and the common application of the ablative case, setting it apart from other dravidian languages and highlighting its rich linguistic diversity within the indian subcontinent. tulu has its own script, 'tigalari,' which is derived from the grantha script; however, it is not widely used. tulu's linguistic structure is characterized by a high degree of agglutination and morphology, with words formed by attaching suffixes and/or prefixes to root words, much like in kannada. however, due to lack of unicode support for tigalari and influence of kannada -the regional language, most tulu literature and articles are written in kannada script @xcite . tulu also shares some linguistic similarities with kannada, including verb-final inflectional patterns and relatively free word order. mt between morphologically rich languages like kannada and tulu remains an uncommon and rarely explored topic @xcite . while kannada has seen some involvement in mt, tulu has received very limited attention in this context. however, the translation between an established local language like kannada and an ancient re-gional language like tulu, has the potential to capture the interest of mt researchers. this underscores the importance of developing efficient techniques for handling translation between these language pairs, not only for communication but also for the preservation of linguistic resources. this study also reveals the divergence patterns between kannada and tulu at lexical and structural levels, though both languages have overlapping vocabularies. with these objectives, this work focuses on creating a kannada-tulu parallel corpus for mt and setting benchmarks by implementing nmt baselines. the corpus construction is done in two ways: manual translation and atg guided by linguistic rules. to benchmark the dataset, a wide range of encoder-decoder based nmt architectures, including rnn, birnn, and transformers with gru and lstm units, are implemented to translate kannada sentences to tulu sentences and vice versa. further, byte pair embeddings (bpe) are employed for sub-word tokenization to resolve the out-of-vocabulary (oov) issue to some extent. the rest of the paper is organized as follows: related work is presented in section 2 and the construction of parallel corpus is detailed in section 3. section 4 describes the nmt baselines followed by the experiments and results in section 5. the paper concludes in section 6 with avenues for future work. ## related work in nmt, a parallel corpus is essential for training the model. however, creating a high-quality parallel corpus is a challenging and time-consuming process as it requires an in-depth understanding of both source and target languages and often involves manual or semi-automatic alignment of translations, making it a critical bottleneck in building effective translation systems @xcite . constructing a parallel corpus for under-resourced language pair introduces an additional layer of complexity due to scarcity of linguistic resources, making the process even more challenging. the conventional method of constructing a parallel corpus relies on manual translation with the help of linguists @xcite . while this approach yields high-quality parallel data, it is expensive and laborious. hence, mt researchers have turned their focus towards human-assisted or automatic methods for parallel corpus construc-tion, aiming to streamline the process and reduce the associated expenses @xcite . in this direction, web crawling @xcite and web scraping techniques @xcite , are found to be more promising for constructing parallel corpora for under-resourced languages by automatically extracting bilingual content from the online resources. in addition, leveraging technologies such as the google translation application programming interface (api) @xcite , parallel text generation from speech by performing speech-to-text translation @xcite , aligned subtitles from the movies or videos @xcite and crowd-sourced translations @xcite , aid in the construction of parallel corpora. these automated and semiautomated approaches can significantly enhance the corpus creation process. however, their effectiveness relies on post-processing for quality control to ensure accurate and reliable parallel text alignments @xcite . in addition to manual and web-based methods for constructing parallel corpora, the text augmentation (ta) approach that comprises a dictionarybased word induction is also explored by the scholarly community @xcite to enhance the size of the parallel corpus. in this method, words in an under-resourced language are replaced with similar words from a high-resource language within a sentence with the help of a bilingual dictionary. these modified sentences are subsequently refined using an unsupervised mt framework to produce augmented text data, presenting a practical solution for enriching parallel corpora to address resource issues in under-resourced languages. in addition, various ta techniques, including back-translation @xcite , dictionary-based synonym replacement @xcite , word embeddings-based synonym replacement @xcite ), text paraphrasing @xcite , knowledge injection @xcite , and data generation through rule-based methods @xcite , are also explored to enhance the size of the parallel corpora. though several techniques are explored to construct high-quality parallel corpora, corpus construction in morphologically rich under-resourced language pairs like kannada-tulu have received limited attention. in our previous work, we constructed the first-ever parallel corpus for kannada-tulu language pair with 30,000 parallel sentences @xcite ) by adopting manual transla- data sources # of sentences wikipedia 9,585 indowordnet 20,665 mann ki baath 13,871 bible 12,429 samanantar 16,424 ## dataset construction kannada-tulu parallel corpus construction is carried out in two ways: manual translation and atg guided by rules, and the parallel corpus construction process is given below: ## neural machine translation baselines a set of encoder-decoder based nmt approaches including rnn with gru (rnn+gru) and lstm (rnn+lstm) units, birnn with gru (birnn+gru) and lstm (bi-rnn) units, transformers with gru (transformers+gru) and lstm (transformers+lstm) units, are experimented to set the benchmarks for the kannada-tulu parallel corpus. further, sub-word tokenization is also explored in these approaches as it has exhibited promising results in similar research works @xcite . in order to implement nmt models, the following steps are used: ## experiments and results nmt models for translating kan-tul and vice versa are implemented using the open-source nmt framework, opennmt-py 4 . this framework offers a structured encoder-decoder architecture enriched with attention mechanisms, allowing seamless handling of sequence-to-sequence prediction tasks. several experiments are carried out involving the meticulous fine-tuning of hyperparameters to achieve optimal performances of the learning models. variations in encoder types, such as the rnn, birnn, and transformer, are explored. additionally, rnn architectures namely: lstm and gru are explored to identify the most suitable hyperparameters configurations. the optimal hyperparameters and their values for transformer+lstm+bpe models which exhibited better performance are shown in in assessing the performance of nmt models, bleu scores and chrf scores are chosen as the evaluation metrics. bleu scores are calculated by counting matching word n-grams between the candidate translation and the reference text (papineni 4 https://github.com/opennmt/opennmt-py et al., 2002), whereas chrf measures the similarity between the candidate translation and one or more reference translations in terms of characterlevel n-grams @xcite . these approaches provide an automated means of evaluation similar to human assessment, enabling researchers to quantitatively measure the quality of mt models and make informed comparisons between different translation outputs. the performances of the proposed models in terms of bleu and chrf scores are shown in the sample kan-tul translations obtained from rnn+gru and transformer+lstm+bpe models along with the actual translations are shown in ## conclusion and future work this paper describes the construction of kannada-tulu parallel corpus for nmt and the nmt baselines to translate kan-tul and vice versa. kannada-tulu parallel corpus is constructed manually and also using atg by incorporating linguistic rules. using kannada-tulu parallel corpus, different nmt models with different encoder and decoder variants are implemented to translate kan-tul and vice versa. among the proposed models, trans-former+lstm+bpe model outperformed the other models with bleu scores of 0.241 and chrf scores of 0.502 for kan-tul translation. further, the same model outperformed the other models for tul-kan translation with bleu scores of 0.341 and chrf scores of 0.598. future research will examine hybrid nmt models that combine kannada and tulu language traits with appropriate pre-processing methods. further, atg will be en-hanced by incorporating additional rules to address the current limitation of producing short-length sentences in future work.