Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables (2023)

Abstract: Despite the tremendous success of Neural Machine Translation (NMT), its performance on low- resource language pairs still remains subpar, partly due to the limited ability to handle previously unseen inputs, i.e., generalization. In this paper, we propose a method called Joint Dropout, that addresses the challenge of low-resource neural machine translation by substituting phrases with variables, resulting in significant enhancement of compositionality, which is a key aspect of generalization. We observe a substantial improvement in translation quality for language pairs with minimal resources, as seen in BLEU and Direct Assessment scores. Furthermore, we conduct an error analysis, and find Joint Dropout to also enhance generalizability of low-resource NMT in terms of robustness and adaptability across different domains.

## introduction although neural machine translation (nmt) has made remarkable advances @xcite , it still requires large amounts of data to induce correct generalizations that characterize human intelligence @xcite . however, such a vast amount of data to make robust, reliable, and fair predictions is not available for low-resource nmt @xcite . the generalizability of nmt has been extensively studied in prior research, revealing the volatile behaviour of translation outputs when even a single token in the source sentence is modified @xcite @xcite . for instance, in the sentence "smallpox killed billions of people on this planet" from our iwslt test set, when replacing the noun "smallpox" with another acute disease like "tuberculosis", the model should ideally generate a correct translation by only modifying the relevant part while keeping the rest of the sentence unchanged. however, in many instances, such a small perturbation adversely affects the translation of the entire sentence, highlighting the limited generalization and robustness of existing nmt models @xcite . compositionality is regarded as the most prominent form of generalization that embodies the ability of human intelligence to generalize to new data, tasks, and domains @xcite , while other types mostly focus on the practical considerations across domains, tasks, and languages, model robustness, and structural generalization @xcite . research in compositional generalization has two main aspects: evaluating the current models' compositional abilities as well as improving them. in terms of evaluation, some studies use artificially created test sets that mimic arithmeticlike compositionality @xcite , while others evaluate compositionality in a more natural way @xcite @xcite . in terms of improvement, earlier work aimed to enhance the models' compositional abilities on tasks such as semantic parsing datasets @xcite , math word problem solving @xcite , data-to-text generation @xcite , and classification @xcite . as for nmt, previous work has shown shortcomings in systematic compositional skills @xcite ; @xcite , particularly for low-resource languages @xcite , yet no direct improvements have been proposed. we aim to improve compositionality in nmt, with a focus on low-resource scenarios that necessitate more robustness to form new combinations of previously seen smaller units. to achieve this, we introduce joint dropout (jd), a simple and effective method that jointly replaces translation-equivalent phrase pairs in the source and target sentences with variables, encouraging the model to maintain the translation of the remaining sentence, regardless of the dropped phrases. jd is orthogonal to and compatible with other methods for improving nmt performance. specifically, it is designed to be data-centric and model-agnostic, allowing it to be easily combined with existing techniques that focus on different aspects of the nmt pipeline. our analysis on simulated and real low-resource data demonstrates jd's ability to significantly improve compositional generalization and translation quality. ## methodology generalization has been a longstanding concern in the field of machine translation. in the past, statistical mt utilized phrases as the fundamental translation units in order to consider contextual information, such as in phrase-based statistical machine translation @xcite . to increase generalization, hierarchical pbsmt proposed by @xcite builds upon the bilingual phrase pairs of pbsmt to learn hierarchical rules, capturing discontinuous translation equivalences and therefore allowing for better generalization. similarly, jd leverages bilingual phrases to make the rest of the translation not dependent on a specific phrase pair. however, the main idea behind jd originates from compositionality: the meaning of a sentence is a function of the meanings of its known atoms and how they are systematically and syntactically combined @xcite . by substituting meaning with translation in this definition, we come up with a rule of compositionality for translation systems: in which τ is the translation function, p and q are the constituents of the sentence, and • is a combiner. jd aims to transfer the principle of compositionality to the translation model in order to improve generalization and robustness of nmt by replacing joint phrases with variables. to exemplify, given the de-en sentence pair ⟨sie hat rom besucht, she visited rome⟩, we replace nouns with variables: ## experiments in this section, we present a comprehensive overview of our experiments. we begin by providing details regarding the datasets used and the training systems employed. next, we delve into the specific criteria we considered when replacing phrases with variables. subsequently, we discuss the significant improvements achieved by our proposed method, jd, across various aspects, including compositional generalization, translation performance, robustness, and the ability to generalize across domains. ## conclusion despite the fact that nmt's success is closely tied to having large amounts of training data, it is still beneficial to explore methods that can help improve generalization when working with limited data. in this paper, we introduce joint dropout as a straightforward yet effective approach to enhancing the compositional generalization and translation quality of low-resource nmt. specifically, we demonstrate that jointly replacing phrases with variables has a regularizing effect that mitigates overfitting by enabling the system to translate sentences regardless of the specific phrases present at the variable positions.