Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks (2022)

Abstract: Memorization presents a challenge for several constrained Natural Language Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the proclivity of neural models to memorize noisy and atypical samples reacts adversely with the noisy (web crawled) datasets. However, previous studies of memorization in constrained NLG tasks have only focused on counterfactual memorization, linking it to the problem of hallucinations. In this work, we propose a new, inexpensive algorithm for extractive memorization (exact training data generation under insufficient context) in constrained sequence generation tasks and use it to study extractive memorization and its effects in NMT. We demonstrate that extractive memorization poses a serious threat to NMT reliability by qualitatively and quantitatively characterizing the memorized samples as well as the model behavior in their vicinity. Based on empirical observations, we develop a simple algorithm which elicits non-memorized translations of memorized samples from the same model, for a large fraction of such samples. Finally, we show that the proposed algorithm could also be leveraged to mitigate memorization in the model through finetuning. We have released the code to reproduce our results at https://github.com/vyraun/Finding-Memo.

## introduction previous studies @xcite @xcite have shown that neural networks capture regular patterns in the training data (generalization) while simultaneously fitting noisy and atypical samples using brute-force (memorization). for constrained natural language generation tasks such as neural machine translation (nmt), which rely heavily on noisy (web crawled) data for training high-capacity neural networks, this creates an inherent reliability problem. for example, memorizations could manifest themselves in the form of catastrophic translation errors on specific samples despite high average model performance @xcite . it is also likely that the memorization of a specific sample could corrupt the translations of samples in its vicinity. therefore, exploring, quantifying and alleviating the impact of memorization is of critical importance for improving the reliability of such systems. yet, most of the work on memorization in natural language processing (nlp) has focused either on classification @xcite or on unconstrained generation tasks, predominantly language modeling @xcite @xcite @xcite @xcite . in this work, we fill a gap in the literature by developing an analogue of extractive memorization for constrained sequence generation tasks in general and nmt in particular. our main contributions are: ## extractive memorization we present our definition of extractive memorization as algorithm 1. analogous to extractive memorization in language models @xcite , this definition labels an input sentence (source) as being memorized if its transduction (translation) could be replicated exactly with a prefix considerably shorter than the length of the full input sentence (source), under greedy decoding. operationally, we set prefix ratio threshold (p) to 0.75. algorithm 1: extractive memorization in nmt data: trained nmt model t , training dataset s, prefix ratio threshold p result: memorized samples m , prefix lengths l greedily translate sources in s using t ; @xmath0; greedily translate prefixes of sources in m1 using t ; @xmath1; for m i 2 in m2 do @xmath2; @xmath3; if l @xmath4 next, we apply this definition of memorization on a strong transformer-big @xcite baseline trained on the 48.2m wmt20 en-de parallel corpus @xcite . we describe the dataset, model and training details in appendix a. qualitatively, we observe that the memorized samples detected by algorithm 1 mostly consist of low-quality samples -templatized source sentences and noisy translations. to analyze the results quantitatively, similar to @xcite , we bucket the training data pairs in terms of their repetitions in the training data. owing to the sparsity of data with greater than 5 repetitions we report results in provenance source translation training data why study in peru? spanish courses warum in peru studieren? perturb suffix why study in peru? university courses warum in peru studieren? perturb suffix why study in peru? short courses warum in peru studieren? perturb suffix why study in peru? summer courses warum in peru studieren? perturb prefix you study in peru? spanish courses sie studieren in peru? spanischkurse perturb prefix advanced study in peru? spanish courses weiterbildung in peru? spanischkurse the range of 1-5 repetitions. further, for repetition values 1 and 2, we select 100k random samples for analysis. we observe two key results: repetitions vs memorization: quality of memorized samples: the above two results are similar to the results in language modeling @xcite and serve to demonstrate the utility of algorithm 1 in analyzing extractive memorization in nmt. ## neighborhood effect of memorization to measure the neighborhood effect of memorization, we generate new source sentences in the vicinity of the memorized samples through perturbations and test whether they generate the same output under greedy decoding. ## generating non-memorized outputs the results in section 4 (and ## mitigating memorization in the model the previous section demonstrated that the nonmemorized translations of memorized samples could be recovered from the same model by applying algorithm 3. in this section, we investigate whether this corrective behavior could directly be imparted to the model through finetuning using the corpus f obtained by applying algorithm 3. ## transferable memorization attacks in this section, we show how the proposed extractive memorization algorithm could be used to construct potential attacks on state-of-the-art (sota) mt systems. our motivation here is to provide an existence proof of the statement that the data dependency of extractive memorization makes it a transferable phenomenon, which could be leveraged to attack disparate systems. as such, we name the attack designed to elicit erroneous generations on system b, based on memorized data extracted from system a, a transferable memorization attack (tma). further, successful transferable memorization attacks should signify spurious correlations which could be easily learned from the underlying common data distribution. therefore, tma could be characterized as a data poisoning attack @xcite and its existence would point to shared vulnerabilities in nmt systems. experiment: we feed the memorized samples obtained from the system in section 3 to two public systems, namely google translator and microsoft bing translator. we find that it is indeed possible to elicit erroneous, memorized translations from these sota systems using the memorized inputs identified from our wmt20 model. we present two such examples in ## discussion and open questions in this section, we list some questions which require further investigation to gain more insights into extractive memorization and its effects. representations of memorized samples: our hypothesis to explain the differential sensitivity of memorized samples to perturbation positions posits prefixes as memorization triggers. however, further investigations from a representational perspective are required to validate this hypothesis as well as to study how such non-robust memorized representations are composed within longer sentences @xcite . reference-free extractions: algorithm 1 could also be applied in a reference-free manner by treating the output of the full input sentence as the reference for the prefixes and testing a large number of inputs @xcite . further research is required into determining the efficacy of such reference-free extraction and its effectiveness in generating transferable memorization attacks. ## conclusions in this work, we developed the idea of extractive memorization for constrained sequence generation tasks and quantitatively demonstrated that such memorization poses a real threat to nmt reliability. we also proposed an algorithm to generate non-memorized outputs for such samples. to the best of our knowledge, our work is the first investigation of extractive memorization for constrained sequence generation tasks in general. we hope that our work serves as a useful step towards further research on extractive memorization in constrained sequence generation tasks & nmt. 5157 throughout the work, we emphasized on exploring and mitigating memorization without the help of data filtering techniques and did not compare memorizations for models trained under different data-filtering algorithms. we believe this direction to be very relevant but orthogonal to our work in this paper. further, since our work presents the first algorithm on memorization mitigation, we believe it doesn't represent an optimal approach for the task. for example, one immediate extension of the algorithm would be to use multiple symbols instead of just one symbol.