On the Correspondence between the Squared Norm and Information Content in Text Embeddings (2025)

Abstract: Previous work has reported both empirical and theoretical evidence, for specific training models, of the correspondence between the squared norm of an embedding and the information content of the text it represents.In this paper, we investigate the relationship at the theoretical and empirical levels, focusing on the mechanisms and composition functions used to combine token embeddings. i) We formally derive two sufficient theoretical conditions for this correspondence to hold in embedding models. ii) We empirically examine the correspondence and the validity of these conditions at the word level for both static and contextual embeddings and different subword token composition mechanisms.iii) Building on Shannon’s Constant Entropy Rate (CER) principle, we explore whether embedding mechanisms exhibit a linearly monotonic increase in information content as text length increases.Our formal analysis and experiments reveal that:i) At the word embedding level, models satisfy the sufficient conditions and show a strong correspondence when certain subword composition functions are applied.ii) Only scaled embedding averages proposed in this paper and certain information-theoretic composition functions preserve the correspondence. Some non-compositional representations—such as the CLS token in BERT or the EOS token in LLaMA—tend to converge toward a fixed point. The CLS token in ModernBERT, however, exhibits behavior that aligns more closely with the CER hypothesis.

## introduction embeddings remain a cornerstone of modern natural language processing, underpinning a wide range of tasks such as information retrieval, semantic textual similarity, and knowledge graph reasoning. their utility, however, depends not only on the representations themselves but also on the operators that act upon them, which enable measuring similarity, performing composition, and supporting more complex forms of inference. an equally important aspect is measuring the amount of information embedded in these representations, as it plays a central role in defining and guiding such operators. previous studies have provided empirical evidence of a relationship between the squared norm of an embedding and the information content (ic) or self-information (i(w) = -log(p (w))) of the corresponding text (i(w) ∝ ∥ ⃗ w∥ 2 ). for instance, @xcite analytically demonstrated this correspondence at the word level in the context of the skip-gram with negative sampling (sgns) objective. @xcite further derives a relationship between the squared norm and the kullback-leibler (kl) divergence between a word's context distribution and the unigram distribution, extending the analysis to contextual word embeddings (see references in section 2.1). in this paper, we investigate the correspondence between ic and the squared norm of embeddings at the text level, focusing on how embedding composition and text-level representations preserve this relationship. to this end, we first identify sufficient (though not necessary) theoretical conditions under which the correspondence holds, regardless of the embedding model or the mechanism used to represent full texts (e.g., cls token, eos token, or composition functions). specifically, the sufficient theoretical conditions identified that support this correspondence are: i) a monotonic relationship between the norm and the amount of information, and ii) that each component of the embedding contributes independently to the estimation of the information content. we then empirically evaluate these conditions at the word level for both static and contextual embedding models. our results show that, when the appropriate subword composition function is applied, the models satisfy the sufficient theoreti-cal conditions and exhibit a strong correspondence between ic and squared norm (i(w) ∝ ∥ ⃗ w∥ 2 ). to analyze the correspondence at the word sequence level, we build on shannon's constant entropy rate (cer) hypothesis. cer states that when processing a sequence of words (e.g., a sentence), the average entropy per unit @xcite remains constant as the sequence grows longer. this is equivalent to say that the ic or self-information of the full text increases linearly with its length, at least beyond a certain length @xcite . the phenomenon has been corroborated by numerous studies across both psycholinguistics and computational linguistics (see references in section 2.2). in this paper, we formally analyze the extent to which word embedding composition functions (e.g., sum, average) preserve the linear monotonicity of the squared norm as words are incrementally added to a text. in parallel, we empirically examine the behavior of various text embedding strategies. specifically, we compare single-token representations-such as the cls token in bert or the last token in llama-with composition-based methods applied to both static and contextual embeddings (see sections 2.4 and 2.3 for further details). at a theoretical level, we conclude in section 4 that neither embedding summation ( ⃗ v i ) nor averaging ( ⃗ v i n ) preserves the linear growth of squared norm (information content). however, normalizing the sum by the square root of the number of tokens ⃗ v i √ n , which we call the scaled embedding average, ensures this linearity. we further analyze other composition functions based on information theory @xcite . the empirical study in section 5 verifies the previous theoretical conclusions regarding compositional representations. regarding non-compositional representations based on a single token, we observe that in models such as bert (cls token) and llama (last token), the growth of information content eventually plateaus, indicating that they do not strictly satisfy linear monotonicity. the cls token in modern-bert, however, displays a behavior more consistent with the cer hypothesis. ## sufficient theoretical conditions in this section, we formally derive two sufficient conditions for the correspondence (i(w) ∝ ∥ ⃗ w∥ 2 ) regardless of the embedding model (static or contextual) or the mechanism used to represent full texts (cls token, eos, or composition functions). note that we are not claiming that these conditions always hold, but rather that they are sufficient conditions for the correspondence. the purpose of this formal analysis is to examine under which conditions the correspondence at the text level may hold independently of specific architectures and composition functions. the first condition (see section 2.1) is that the ic of the represented content must grow according to its embedding norm without jumps or abrupt peaks. we denote as ⃗ w the embedding of the expression @xmath0, . . . , w n ). ## ic monotonicity of compositional functions in this section, we formally examine the ability of composition functions to preserve the correspondence i(w) ∝ ∥ ⃗ w∥ 2 as words are added to a text, assuming the correspondence at the token level and the constant entropy rate mentioned in section 2.2. ## experiments in this section, we empirically examine the correspondence between the squared norm and the amount of information. at the lexical level, we analyze its correlation with word probabilities in a corpus. at the sequence level, we study how the squared norm grows as tokens are added to a sequence. ## conclusions theoretical analyses and empirical findings in the literature support a correspondence between the square norm of a word embedding and the ic of the word it represents. in this work, we extend this correspondence to representations of word sequences. at the theoretical level, we identify two sufficient properties of the representation system that establish this correspondence: the growth of the norm with the ic of the represented text and the statistical independence between dimensions. although these properties are not strictly necessary, our word-level experiments show that when the appropriate subword composition function is applied, the models satisfy the sufficient conditions and exhibit a strong correspondence between ic and squared norm. in addition, we have formally analyzed existing composition functions under the assumption that information content increases linearly with text length (constant entropy rate). according to our analysis, validated by experiments, neither summation nor averaging satisfies this property. as a solution, we propose a scaled embedding average, which simply divides by the square root of the number of embeddings being averaged. the analysis also shows that the function f inf is able to preserve this property, while max-based composition does not exhibit linear behavior. our experiments also suggest that some tokenbased text embeddings such as cls in bert or eos in llama, do not preserve this property either. however, the cls token in modernbert exhibits a more expected behavior in terms of the cer hypothesis. this work opens new avenues for studying the representation power of embeddings from a formal perspective and exploring/exploiting the geometric properties of dense text representations in distributional semantics from the perspective of information theory.