Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of P ersian- H indi Pair (2024)

Abstract: This paper demonstrates that Phrase-Based Statistical Machine Translation (PBSMT) can outperform Transformer-based Neural Machine Translation (NMT) in moderate-resource scenarios, specifically for structurally similar languages, Persian-Hindi pair in our case. Despite the Transformer architecture’s typical preference for large parallel corpora, our results show that PBSMT achieves a BLEU score of 66.32, significantly exceeding the Transformer-NMT score of 53.7 ingesting the same dataset.

## introduction in the current state of nlp affairs, the performance of attention-based @xcite mt systems reaches bleu scores of almost one. however, the underlying neural network (nn) architectures of such high-performing models, assume that the language pairs have a humongous diverse parallel corpora to achieve such desired performance. of course, there are certain high-source language pairs such as english and french which benefit from those solutions but the mt system of other natural languages that utilize nn architecture without meeting the architecture's assumptions are on the disadvantage side and will generate translations that are way far being accepted by native speakers. beyond the need for large datasets, another great concern when using nns is their high power consumption and the environmental impact they leave behind-through processes such as training, inference, and experimentation, which all contribute to carbon footprints @xcite . to walk through an efficient alternative path, we looked at the big picture of natural languages, focusing on their linguistic families and the factors that group languages. we have observed that the property of linguistic closeness of less divergent languages can be exploited. the key contributions of our paper are: ## related works before the revolutionization of the field by the transformer architecture @xcite , the notion of language closeness was being leveraged in various forms for different language pairs. in this section, we will see that our work is not only different from the perspective of studying a new language pair, persian-hindi, which to date no formal research has been conducted yet for the pair, but it also varies in terms of past transformer comparison of the two architectures, nmt and smt, given that we have access to a moderate amount of parallel sentences. previous works such as, @xcite , hypothesize that translations between related languages tend to be more literal, with complex phenomena (e.g., metaphors) often transferring directly to the target language. in contrast, these phenomena are more likely to require complex translations between unrelated languages. other instances such as (rios and sharoff, 2015), @xcite , and @xcite utilize lexical similarities. except (jauregi unanue et al., 2018) which shows that for the scenario of low-resource (unlike our scenario which assumes medium resource) languages, smt performs better than nmt, there is no comparative analysis of the nmt and smt architectures for structurally similar languages with an assumption concerning the size of the parallel corpus. ## dataset and preprocessing in addition to the "large scale colloquial persian dataset" (lscp) (abdi khojasteh et al., 2020), the other datasets we utilized are primarily sourced from opus (tiedemann, 2016), a well-known repository for parallel corpora of various domains for a vast number of language pairs. an additional time-taking step applied to the lcsp corpus was to pair the sentences first and then pass to the preprocessing phase. then, we performed labse filtration which the dramatic reduction of the original corpus is shown in to evaluate the structural similarity between persian and hindi, we analyzed the differences in sentence lengths (measured in token counts) across all parallel sentences. we assumed that if the majority of sentences exhibit a difference of three tokens or less, the alignment achieved through mgiza would represent an optimal one-to-one correspondence. as illustrated in the donut chart in figure 1 , more than 72% of the parallel sentences in our dataset, ## experiments and results since persian and hindi share many common words, in our second experiment, parallel sentences were first romanized to increase text similarity 1 https://github.com/mosessmt/mosesdecoder/blob/master/scripts/generic/mteval-v13a.pl @xcite . however, the bleu score dropped to 51.21 from 66.7. one highly probable reason is due to the diacritics (mark that is placed above, below, or through a letter to indicate how it should be pronounced) that some persian words have in order to determine the phoneme of a word and hence the associated meaning. for example, gul (flower) and gel (mud) are two words the sound and meaning of which can be determined from the context (without diacritics) or using diacritics. romanization often results in the loss of these nuances, leading to ambiguities in alignment and translation. in the final experiment, we reversed the persian scripts from right-to-left (rtl) to left-to-right (ltr) to align the writing direction of hindi, expecting improved alignment quality. unfortunately, this resulted in a further decrease in the bleu score, falling to 48.74. this decline can be attributed to the fact that reversing a sentence alters its meaning. although both persian and hindi are classified as free-word-order languages, we observed that the phenomenon where "only constructs that follow each other can be moved to any other position in the sentence while still preserving meaning" is compromised when inversion occurs, leading to a change in interpretation. see appendix a for examples. ## conclusion and future works this study presents a comparative analysis of smt and nmt for the persian-to-hindi language pair. our findings demonstrate that smt yields superior results in closely related languages, attributable to their shared linguistic structures. additionally, we observed that reversing the order of persian sentences from rtl to ltr negatively impacted the smt model's performance, resulting in a loss of meaning. in contrast, romanizing the input text showed a beneficial effect compared to the inversion experiment. future work will focus on deepening our understanding of these languages and exploring alternative approaches, such as translation through common space word embedding, transfer learning, and pivot-based nmt with english as a bridging language. matt post. 2018. a call for clarity in reporting bleu scores. in proceedings of the third conference on machine translation: research papers, pages 186-191, brussels, belgium. association for computational linguistics. miguel rios and serge sharoff. 2015. obtaining smt dictionaries for related languages. in proceedings of the eighth workshop on building and using comparable corpora, pages 68-73, beijing, china. association for computational linguistics. rico sennrich, barry haddow, and alexandra birch. 2016. neural machine translation of rare words with subword units. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1715-1725, berlin, germany. association for computational linguistics. jörg tiedemann. 2016. opus-parallel corpora for everyone. baltic journal of modern computing, 4(2). antonio toral and andy way. 2015. translating literary text between related languages using smt. in proceedings of the fourth workshop on computational linguistics for literature, pages 123-132, denver, colorado, usa. association for computational linguistics. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. advances in neural information processing systems, 30. a references pushpak bhattacharyya. 2015. machine translation. crc press. a examples appendix a.1 paralle sentences of different length categories a.1.1 identical lengths 1.1.1.fa: tamam mahsulat hamel shodeh 100% bazorsi mi shvand. 1.1.1.hi: bheje gae sabhee utpaad 100 nireekshan kie jaate hain. 1.1.1.en:all shipped products will be 100% inspected. 1.1.2.fa: baraye etlaat bishtar lotfa ba ma tamas begirid 1.1.2.hi: adhik jaanakaaree ke lie krpaya hamase sampark karen 1.1.2.en:for more information please contact us a.1.2 one-to-three token difference 1.2.1.fa: akharin ghimet sakeh ve tala dar bazar. [7-tokens] 1.2.1.hi: baajaar par naveenatam sikka aur sone kee keematen. [8-tokens] 1.2.1.en:the latest price of coins and gold in the market. 1.2.2.fa: har zemestān bahāri dar pay dārad. [6-tokens] 1.2.2.hi: har sardī ke baad vasant r . tu hotī hai. [8-tokens] 1.2.2.en:after every winter there's spring. a.1.3 four-or-five token difference 1.3.1.fa: pish bini ab ve npava dar litvania.[7-tokens] 1.3.1.hi: havāmāna andāja lithu'āniyā. [3-tokens] 1.3.1.en:weather forecast in lithuania. 1.3.2.fa: besiar sadeh baraye estefadeh.[4-tokens] 1.3.2.hi: ka upayog karane ke lie bahut hee saral. [8-tokens] 1.3.2.en:very simple to use.