Cheat Codes to Quantify Missing Source Information in Neural Machine Translation (2022)

Abstract: This paper describes a method to quantify the amount of information H(t|s) added by the target sentence t that is not present in the source s in a neural machine translation system. We do this by providing the model the target sentence in a highly compressed form (a “cheat code”), and exploring the effect of the size of the cheat code. We find that the model is able to capture extra information from just a single float representation of the target and nearly reproduces the target with two 32-bit floats per target token.

## introduction given a sentence s in the source language, a machine translation system generates a translation t in the target language. however, for any sentence of non-trivial complexity, the translation t is not unique. therefore, to reproduce a reference translation, a model requires some amount of extra information. the aim of this work is to quantify the amount of information that is missing in the source s that is required to generate the translation t. to quantify this information, we modify the model architecture to provide the target sentence to the model as an auxiliary input, and observe the effect of varying the size of the representation of the target sentence from the minimum that provides any useful information to the decoder to the size that enables a near-perfect reproduction of the target. since the model seeing the target as an input is a form of "cheating", we refer to these compressed representations of the target as "cheat codes". ## experiments all our experiments use @xcite 's cleaned version of the wmt21 @xmath0. we do not use back-translated data since we observed no improvement in quality upon adding it, consistent with chen et al. (2021)'s findings. we evaluate on both references a and b in the test set using bleu 2 and chrf foot_2 metrics from sacrebleu @xcite , and comet and comet-qe foot_3 @xcite . ## conclusions and future work this paper has shown that by letting machine translation models use a highly compressed representation of the target sentence as an auxiliary input, we can estimate the amount of information missing from the source that the model captures from the target. by varying the size of these representations (cheat codes), we see that the model can capture useful information from as little as a 16-or 32-bit scalar representation of the target. we also see that the model approaches perfect reproduction of the target (>92bleu/95chrf) from as little as 2 floats per target token. a limitation of our method is that it can only estimate the amount of missing information from the source based on the size of cheat code, but we do not get any insight into what this information actually is. in future work, this method can be extended to qualitatively analyze what the missing information is, and how it can possibly be provided to the model in other ways to improve translation quality without "cheating". another limitation is that the model, if not trained carefully for larger cheat codes, can learn to copy the target without using the source. this is countered by careful training regimes as discussed in section 4.3. since the model is able to capture extra information from the second input, it could be possible to use this to guide the output in other ways than just to reproduce the references. for example, given a small enough representation, we could sweep through the entire range of cheat codes and produce diverse high-quality translations @xcite .