Comparing DAE -based and MASS -based UNMT : Robustness to Word-Order Divergence in E nglish–> I ndic Language Pairs (2023)

Abstract: The proliferation of fake news poses a significant challenge in the digital era. Detecting false information, especially in non-English languages, is crucial to combating misinformation effectively. In this research, we introduce a novel approach for Dravidian fake news detection by harnessing the capabilities of the MuRIL transformer model, further enhanced by gradient accumulation techniques. Our study focuses on the Dravidian languages, a diverse group of languages spoken in South India, which are often underserved in natural language processing research. We optimize memory usage, stabilize training, and improve the model’s overall performance by accumulating gradients over multiple batches. The proposed model exhibits promising results in terms of both accuracy and efficiency. Our findings underline the significance of adapting state-of-the-art techniques, such as MuRIL-based models and gradient accumulation, to non-English language.

## introduction unsupervised neural machine translation (unmt) shows promising results for closely related language-pairs @xcite @xcite , but it faces significant challenges when dealing with language-pairs that have distinct word orders. in this paper, we test the robustness of word-order divergence in state-of-the-art unmt systems. specifically, we test mass-based unmt @xcite (which does not have shuffling noise) and dae-based unmt systems @xcite (which has shuffling noise) on language pairs with different word-orders, i.e., english (svo) → indic (sov) language pairs. to test the robustness, we compare these unmt models trained on (i) original data and (ii) reordered data (where the source sentences are reordered to match the target language word-order). word-order divergence is present in the former case, while word-order divergence is bridged in the latter case. a unmt system that is robust to word-order divergence should not exhibit significant performance differences between these two cases. our contributions encompass two key findings: (i) dae-based unmt demonstrates greater robustness in managing word-order differences between languages compared to mass-based unmt, and (ii) in the majority of language pairs, the unmt model trained with the dae approach using original data produces translations of higher quality than other models. ## related work previous research has addressed lexical divergence between languages in nlp @xcite through various techniques @xcite @xcite . however, the impact of word-order divergence on unsupervised neural machine translation (unmt) remains unexplored. the study by @xcite examined the iterative unmt approach proposed by @xcite and was found to be sensitive to word-order divergence. re-ordering addresses word-order differences in machine translation. while @xcite found it unnecessary for nmt, @xcite improved translation. in transfer learning, @xcite enhanced results by reordering source language sentences before training. we use mass-static and dae-static methods @xcite to address lexical divergence. these approaches initialize the models' embedding layers with unsupervised cross-lingual embeddings and keep the embedding layers static throughout unmt training. ## experimental setup our experiment comprises four sets of unmt models: two trained using mass-static, and the other two trained using dae-static. for each unmt approach, we train one model on original data and another model on re-ordered data. ## result and analysis re-ordering dae-static models with re-ordered data surprisingly leads to a degradation in bleu and chrf scores. this is due to the ambiguity arising from re-ordering without case markers. an example illustrating this issue is presented in figure 1 , where the reference translation of the subject 'we' is 'hamem', which is in the dative case. however, when we re-order the english sentence, the subject remains 'we', which is in the subjective case and is frequently translated to 'hama' in hindi. as a result, the re-ordered model incorrectly translates 'we' as 'hama' instead of the desired 'hamem'. figure 2 visualizes the position-wise cosine similarity of each position embedding. in mass, position embeddings are similar to nearby positions within a local neighborhood of 2 or 3 positions. in dae, position embeddings exhibit similarity to a larger local neighborhood, likely due to local shuffling noise. dae models show more similarity to neighboring positions in the presence of wordorder divergence (without re-ordering) compared to word-order similarity (with re-ordering). ## conclusion our findings show that dae-based unmt performs better than mass-based unmt in addressing word-order divergence. furthermore, the daebased model is bidirectional, unlike the unidirectional re-ordered model. in future work, we aim to investigate additional word orders and language model objectives to gain deeper insights into the impact of word-order divergence in unmt.