The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization (2024)

Abstract: This work analyses the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling. Stochastic decoding methods like nucleus sampling are typically applied to overcome issues such as monotonous and repetitive text generation, which are often observed with maximization-based decoding techniques. We hypothesize that nucleus sampling might also reduce the occurrence of memorization patterns, because it could lead to the selection of tokens outside the memorized sequence. To test this hypothesis we create a diagnostic dataset with a known distribution of duplicates that gives us some control over the likelihood of memorisation of certain parts of the training data. Our analysis of two GPT-Neo models fine-tuned on this dataset interestingly shows that (i) an increase of the nucleus size reduces memorization only modestly, and (ii) even when models do not engage in “hard” memorization – a verbatim reproduction of training samples – they may still display “soft” memorization whereby they generate outputs that echo the training data but without a complete one-by-one resemblance.

## analysing nucleus sampling-based text memorization behavior this section presents our analysis of text memorization behavior for the fine-tuned gpt-neo models when using nucleus sampling instead of greedy decoding. in particular, we measure the amount of text memorization of the fine-tuned models under a variety of secondary conditions. ## conclusion we created a diagnostic dataset to measure the memorization behavior of two neo-gpt models more precisely than previous work @xcite that relied on an estimate of duplicates in the training data. given this we fine-tuned the gptneo models on our dataset and confirmed with our replication experiments the other results under greedy decoding. with this experimental setup we analysed the language models productions when nucleus sampling is used for decoding. the results show that for models with strongly memorized texts low top_p values in nucleus sampling converge to greedy decoding. we note that even the experiments using large top_p values often fail to substantially mitigate memorization. this at the first glance "unreasonable ineffectiveness" of nucleus sampling to mitigate text memorization is mostly caused by high peak distributions -specifically, when a single token's probability exceeds the cumulative threshold set by the nucleus size, causing nucleus sampling to operate deterministically. larger nucleus sizes only modestly mitigate memorization, and even when outputs are not exact reproductions, we find that n-gram overlap scores indicate a "soft memorization" phenomena. in further work we will explore the impact of other duplicate distributions in the training dataset