Memorization in Language Models through the Lens of Intrinsic Dimension (2025)

Abstract: Language Models (LMs) are prone to memorizing parts of their data during training and unintentionally emitting them at generation time, raising concerns about privacy leakage and disclosure of intellectual property. While previous research has identified properties such as context length, parameter size, and duplication frequency, as key drivers of unintended memorization, little is known about how the latent structure modulates this rate of memorization. We investigate the role of Intrinsic Dimension (ID), a geometric proxy for the structural complexity of a sequence in latent space, in modulating memorization. Our findings suggest that ID acts as a suppressive signal for memorization: compared to low-ID sequences, high-ID sequences are less likely to be memorized, particularly in overparameterized models and under sparse exposure. These findings highlight the interaction between scale, exposure, and complexity in shaping memorization.

## introduction language models (lms) @xcite @xcite are susceptible to memorizing segments of texts encountered during training @xcite and emitting these segments during generation @xcite , even from corpora that has been subjected to deduplication @xcite . while memorization is connected to generalization @xcite , it can cause severe issues such as inadvertent reproduction of personal information @xcite and copyrighted materials @xcite . to estimate memorization rates of lms, @xcite formalized a loose bound on memorization known as exposure, a metric that measures the relative difference in log-perplexity between canaries, synthetic sequences of text with fixed formats that are inserted during training and extracted during generation. by leveraging examples directly from the corpus, @xcite introduced a tighter bound on memorization that avoids the need for canaries and reduces the computational overhead associated with computing exposure. figure 1 visualizes the actionable methodology for examining memorization. given a subset of examples, each split into a prefix p and a suffix s, memorization is estimated post-hoc by prompting the model f with the prefix and checking whether its continuation f (p) replicates the reference s. the proportion of continuations that match the references verbatim provides an empirical estimate of memorization and quantifies the risk of information leakage. once memorization was evidenced in practice @xcite , several properties have been identified as factors contributing to the memorization rate. beyond its correlation with overfitting @xcite , memorization is related to duplication counts @xcite @xcite , model capacity @xcite , and context length @xcite . grounded on the manifold hypothesis @xcite , few studies have examined the intrinsic dimension of data representations as a means to understand how neural networks structure latent spaces. these studies reveal that high-dimensional signals tend to lie in low-dimensional subspaces @xcite , and that intrinsic dimensionality acts as a geometric proxy for generalization capacity @xcite . ## background we briefly provide necessary foundations for unintended memorization and intrinsic dimensionality. ## methodology we build on the setup introduced by @xcite to assess memorization in relation to structural complexity. specifically, we employ the gptneo model family @xcite and reuse their random sample derived from the pile @xcite . to ensure that our measurements isolate structural complexity from confounding factors, we carefully control sequence length and duplication counts. we restrict all sequences to a uniform length of 150, thereby stabilizing id estimations. we subsample 1, 000 sequences stratified by duplication frequency on a logarithmic scale for ranges between [1, 10), [10, 100), and [100, 1000), allowing us to disentangle the influence of duplication from that of structural complexity. to estimate the id, we follow @xcite by treating each text as a point cloud spanning a manifold in the embedding space. we then obtain contextualized embeddings using bert @xcite , and estimate the intrinsic dimension using twonn @xcite , discarding artifacts of tokenization. ## findings figure 3 presents the relationship between memorization rate and intrinsic dimensionality for ascending levels of duplication frequency. specifically, we quantile-binned the intrinsic dimension into 25 equally sized intervals and averaged memorization within each bin. each subplot further disaggregates model capacity, covering models with roughly 0.1, 1.3, 2.7, and 6.0 billion parameters. consistent with the relationships reported by @xcite , our findings reveal a log-linear increase of memorization as a function of both duplication count and model capacity. beyond these relationships, we observe a modulating influence of the intrinsic dimension. in the low-duplication regime, memorization declines inversely with intrinsic dimensionality across all model sizes. this inverse trend indicates that complex sequences, particularly those lying on more intricate manifolds, are less likely to be memorized under sparse exposure. in the medium-duplication regime, we notice diverging patterns depending on the model sizes. the inverse relationship largely persists for large models, albeit with a diminished effect. however, this is not the case for small models. once duplications are sufficiently frequent for memorization, small models display a reversal in trend, exhibiting a slight increase in memorization with rising structural complexity. this divergence may reflect the limited capacity of certain models to generalize, leading to greater memorization of sequences that they fail to compress effectively. in the highduplication regime, memorization undergoes a further shift as it saturates and becomes almost invariant to the intrinsic dimension. these findings suggest that under conditions of frequent exposure, memorization is increasingly governed by exposure and scale, overriding the modulating influence of structural complexity. ## conclusion building on the shared connection of memorization and intrinsic dimension to generalization, we introduce the intrinsic dimension as a complementary factor shaping the likelihood of memorization in language models. specifically, we examine the relationship between memorization rate and the structural complexity of sequences in latent space, conditioned on model scale and exposure frequency. for sufficiently parameterized models and moderate levels of duplication, the intrinsic dimension act as a suppressive signal on memorization. a reversed trend can be seen for models with limited capacity, which tend to memorize structurally complex sequences even under moderate exposure. limitations. despite controlling for duplication frequency, we focus exclusively on exact duplicates, omitting near-duplicates which are known to account for the majority of memorized content in large-scale corpora @xcite . this constraint likely underestimates memorization. additionally, we restrict our analysis to verbatim memorization, a narrow definition that is known to give a false sense of privacy @xcite . finally, we rely on greedy decoding to measure memorization, however, this decoding strategy is atypical in practical deployments @xcite .