Generating Diverse Translation with Perturbed k NN - MT (2024)

Abstract: Generating multiple translation candidates would enable users to choose the one that satisfies their needs.Although there has been work on diversified generation, there exists room for improving the diversity mainly because the previous methods do not address the overcorrection problem—the model underestimates a prediction that is largely different from the training data, even if that prediction is likely.This paper proposes methods that generate more diverse translations by introducing perturbed k -nearest neighbor machine translation ( k NN-MT).Our methods expand the search space of k NN-MT and help incorporate diverse words into candidates by addressing the overcorrection problem.Our experiments show that the proposed methods drastically improve candidate diversity and control the degree of diversity by tuning the perturbation’s magnitude.

## introduction in natural language, there are multiple lexically distinct translations given an input sentence. therefore, machine translation systems should offer multiple translation candidates to users so that the final choice should be made by them considering their demands, e.g., styles or domains. however, standard neural machine translation (nmt) models suffer from a low diversity problem in which the generated translation candidates are almost identical. one reason lies in beam search, which is a standard inference algorithm, where the search space is expanded in a left-to-right fashion while keeping only the top-n candidates in every decoding step and just preserving slightly different translations @xcite @xcite . the other reason is the overcorrection problem @xcite , which is caused by a model trained with cross-entropy loss that underestimates a prediction that is largely different from the training data, even if it is likely. this phenomenon discourages the model from generating synonymous expressions and leans toward gold standards, reducing the diversity in the candidates. to encourage the model to generate more diverse candidates, @xcite , @xcite , and @xcite proposed variants of beam search algorithms in which diverse candidates are retained in the search space. however, their methods do not directly address the overcorrection problem, limiting their effect in generating diverse translations. to alleviate this issue, we propose knn diversified decoding that combines diversified beam search and k-nearest neighbor machine translation (knn-mt; @xcite , which addresses the overcorrection problem by retrieving alternative target tokens from the training data during decoding @xcite . to further diversify the search space, we also propose two methods, i.e., stochastic and deterministic methods. the stochastic method expands the search space by perturbation so that the model can generate more likely tokens that are less focused. we proposed two types of perturbations, noised-knn (figure 1 (1)), which adds a noise vector to the query of the knn search, and randomized-knn (figure 1 (2)), which arbitrarily selects k neighbors from a more extensive search space. the deterministic method, uniquify-knn (figure 1 (3)), removes duplicates from the retrieved knn tokens so that no token can be dominant and thus more diverse candidates remain. our experiments showed that our proposed methods alleviate the overcorrection problem that leads to the generation of more diverse candidates, and maintain fluency and oracle translation quality in multiple domains and language pairs. we also show that the degree of diversity can be controlled by changing the perturbation's magnitude, which benefits end-applications, e.g., human post-editing. 2 related work ## knn diversified decoding we propose to employ knn-mt to alleviate the overcorrection problem and thus encourage diverse generation model-wise. it is further combined with diversified decoding together with our proposed stochastic and deterministic methods for the more controlled expansion of the search space in knn-mt. 1 in knn-mt, knn search is expected to improve the output probability of alternative tokens that are not normally included in the top-n of the output probability. furthermore, the search space is extensively explored using a diversified decoding method to generate diverse and likely translation candidates. although knn search is limited by k, more space is explored by stochastically expanding it by adding perturbations from noising ( §3.1) and randomizing ( §3.2). in addition, deterministically considering only unique tokens in neighbors further allows the model to explore alternative candidates ( §3.3). ## experiments we experimentally confirmed whether our method can generate diverse translation candidates. ## conclusion we proposed methods to generate more diverse translation candidates by expanding the search space of knn-mt. we experimentally showed that our proposed methods alleviated the overcorrection problem and outperformed the existing baselines in diversity, and also controlled the diversity and translation quality by changing the perturbation's magnitude.