On the Copying Problem of Unsupervised NMT : A Training Schedule with a Language Discriminator Loss (2023)

Abstract: Although unsupervised neural machine translation (UNMT) has achieved success in many language pairs, the copying problem, i.e., directly copying some parts of the input sentence as the translation, is common among distant language pairs, especially when low-resource languages are involved. We find this issue is closely related to an unexpected copying behavior during online back-translation (BT). In this work, we propose a simple but effective training schedule that incorporates a language discriminator loss. The loss imposes constraints on the intermediate translation so that the translation is in the desired language. By conducting extensive experiments on different language pairs, including similar and distant, high and low-resource languages, we find that our method alleviates the copying problem, thus improving the translation performance on low-resource languages.

## introduction unmt @xcite is a new and effective approach for tackling the scarcity of parallel data. typically, a cross-lingual pretrained language model (plm) @xcite is trained on two languages and then used to initialize the model for the unmt task @xcite @xcite . however, when it comes to low-resource languages, especially when translating between distant language pairs, unmt often yields very poor results @xcite @xcite . one of the major problems that lead to low translation quality is the copying problem or off-target problem @xcite . that is: the trained model does not translate but copies some words or even the whole sentence from the input as the translation. we find the copying problem is closely related to an unexpected behavior in bt @xcite : the model does not translate into the correct intermediate language but simply copies tokens from the source language. to address this problem, this work proposes a simple but effective method that can be integrated into the standard unmt training. we leverage a language discriminator to detect the language of the intermediate translation generated in bt and backpropagate the gradients to the main model. in this way, we can provide implicit supervision to the model. we find that by adding such a training objective, the copying problem can be largely alleviated, especially for low-resource languages. noticeably, we do not introduce any language-specific architectures into the main model. to the best of our knowledge, this is the first work that introduces a language discriminator loss to force the intermediate translations in bt to be in the correct language. the contributions of our work are as follows: ## discussion from the multi30k and wmt experiments, we verify the ability of the ld loss to alleviate the copying problem by showing consistently lower copying ratios. however, the performance in terms of bleu scores on these two datasets shows slightly different trends: we improve translation quality on multi30k a lot by reducing the copying ratios; whereas we do not see a prominent improvement on wmt even if copying ratios are largely reduced. this discrepancy can be explained as follows. two main issues are preventing the model from achieving good performance: (1) lacking cross-lingual alignment information that is useful for learning translation (2) no clear guidance on which language to translate into. the experiments on the small dataset multi30k indicate that issue (1) is not the major obstacle when two similar languages are considered, e.g., en and fr. in such a case, it is the issue (2) that prevents the model from performing the actual translation. this is why large improvements are achieved by simply adding the ld loss when training a model on multi30k (note that the language discriminator does not provide any additional cross-lingual information but only acts as an implicit supervision). in the case of distant language pairs including low-resource languages, e.g., en-gu and en-kk in our wmt experiments, both issues (1) and ( 2 ) prohibit the model from learning to translate accurately. although the copying problem is alleviated, as shown in ## conclusion in this paper, we find that the copying problem in unmt is closely related to the lack of constraints on the intermediate translation in the bt process. to address this issue, we propose an ld loss to give additional supervision to the first-time-step output vectors generated by the decoder in the bt process. we find that the method can alleviate the copying problem by correcting the wrong behavior in bt. in addition, through extensive experiments on different language pairs (including low-resource languages and distant pairs), we discover that the method can consistently improve the performance of distant language pairs.