Bridging Information-Theoretic and Geometric Compression in Language Models (2023)

Abstract: For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.

## introduction to speak a language is not to memorize all possible utterances, but to instead extract the finite ruleset and lexicon that generates them @xcite . that is, language, though nominally highdimensional, can be compressed to a comparatively small intrinsic dimension. the recent success of (large) language models demonstrates that artificial neural networks, too, can acquire linguistic knowledge. current language models (lms) are transformer-based architectures at-scale @xcite that are trained on the conditional distribution of natural language @xcite @xcite and that are inching closer to human-like linguistic robustness @xcite @xcite . for an lm to faithfully model language, it must encode linguistic training data into finitely many variables that allow generalization to infinitely many grammatical utterances. that is, it must perform a successful form of data compression. thus, following the line of research that aims to better understand lm behavior @xcite @xcite , in this work we provide initial insights on how they compress linguistic knowledge. we demonstrate an empirical link between two types of compression: geometric and information-theoretic. in particular, we ask: how, and by how much, do lms compress linguistic data? furthermore, what are linguistic correlates to compressibility? is compression a good predictor of rapid adaptation? we show that (1) intrinsic dimension (id) of linguistic data representations under an lm tracks information-theoretic coding length; (2) greater data compression predicts easeof-adaptation in causal language modeling tasks; ## related work causal language models state-of-the-art language models are based on the transformer architecture @xcite , which consists of alternating feed-forward and self-attention modules @xcite . they are trained using selfsupervised learning on sequences of tokens, where a token is defined as the atomic unit (e.g., a word or a sub-word) fed into the language model. due to their current ubiquity, we focus on autoregressive models trained on a causal language modeling objective, that is, next token prediction given a context of previous tokens @xcite @xcite . language models are typically measured against human performance on linguistic benchmarks. evaluation may be done post-finetuning or in a low-shot regime, where the model completes a linguistic task given few or zero examples. a variety of benchmarks, such as glue @xcite , superglue @xcite , and bigbench @xcite have been proposed, evaluating, for instance, model performance on textual entailment, question-answering, semantic equivalence, or sentiment analysis. indeed, human baselines for glue and superglue have already been surpassed by lms that contain billions of parameters (e.g., palm 540b @xcite ). ## methods our work attempts to bridge notions of geometric and information-theoretic compression of linguistic data under an lm, and subsequently relate these to ease-of-adaptation. we do so by quantifying the id of data representations, information-theoretic coding length of linguistic inputs under an lm, and ease-of-finetuning in order to determine whether these three phenomena are correlated. notation let a linguistic dataset @xmath0, where each sequence x (i) has length l(x (i) ). let m be a (pre-trained) causal language model described by p m (•|@xmath1, the conditional probability distribution of the j th token given its past context of tokens. models & datasets experiments are performed for the product of models @xmath2, opt-1.3b, opt-6.7b] and datasets @xmath3. we focus on opt suite of causal language models due to their accessibility @xcite . for the datasets, we start from a list of corpora including the glue and superglue benchmarks, then pick those whose size is large enough for id estimates to converge (@xmath4. then, for computational efficiency, we randomly subsample each dataset to size n ′ = max(n, 50000), where 50000 is chosen conservatively based on preliminary analyses of convergence of bootstrapped id estimates (see appendix a.2). in addition to external datasets, we create one baseline dataset per model, which we call optcorpus, of ∼ 24 million tokens by repeatedly randomly sampling from m until [eos] is reached. the conditional next-token distribution of optcorpus approximates that of m so to serve as a reference datapoint. in order to determine the effects of syntax and lexical semantics on compression, we define three transformations which we apply to a dataset: notably, several shallow linguistic descriptors are preserved with these transformations: dataset size, sequence length and vocabulary size (1-3), vocabulary entropy (1,2), and token frequency (1). we apply the transformations to optcorpus and wikitext, producing six additional datasets. ## results we find that, similar to protein models and visual networks @xcite , the id of linguistic data representations is significantly lower than their ambient dimension: in our case, by roughly 2 orders of magnitude. in particular, while the ambient dimension of representations in opt-350m, opt-1.3b, and opt-6.7b are @xmath5, 2048, and 4096, respectively @xcite , dataset representational id is @xmath6, see fig. 5 . for simplicity, in the main article we present results on one representative nn-based estimator, the expected simplex skewness (ess) method of @xcite it is the only one to significantly correlate with all other estimators (α = 0.1) (see fig. 6 ). 1 we present results on other estimators in appendix e, and we comment on other (non-nn-based) id estimators in section 4.4. also for practicality, we report in the main article results obtained with one model, opt-6.7b, only commenting on other models when relevant, and one aggregated id measure across layers: the max id value, seen as a conservative upper bound on id across layers. this choice is supported by the observation that, for most datasets, the id profile over layers is quite flat (appendix d). results with other aggregated measures are presented in appendix e. our primary result is that, in the pre-trained lms tested, information-theoretic compression (ppl) predicts geometric compression (id), and low id predicts ease-of-adaptation. we then take a closer look at which linguistic attributes of a dataset predict its id, finding that not only several shallow linguistic descriptors but also grammatical and lexical structure enable geometric compression. finally, we find that, qualitatively, id tends to be stable across model sizes and types , and comment on the differences between id estimators. ## discussion we have quantified geometric compression in neural language models using the id of representations, where id tracks shannon information-theoretic coding length. this bridges two notions of description length in pre-trained neural lms by showing they are significantly positively correlated. our result has also practical implications, suggesting that id and perplexity predict how easy it is to finetune a model to a task (similarly to what observed in the context of zero-shot prompting by @xcite . more speculatively, the relation between id and task adaptation may inform future modeling work that actively encourages data compression at training time, to indirectly inject fast-adaptation capabilities into a model. id estimators are not equal: we focus on nnbased methods because they explain useful properties of the data and ease-of-finetuning. our work is a first attempt to evaluate a wide range of id estimators on natural language representations, and reveals the need for a further principled study of id estimation of linguistic data. that nonlinear id estimators predict information-theoretic compression and easeof-adaptation over linear ones highlights a need to go beyond pca in analyzing compression of specific linguistic phenomena. our experiments on wikitext and variants also demonstrate a need for further experiments on, e.g., idiomaticity, or specific linguistic constructions (cf. hernandez and andreas, 2021 for analysis using pca). while our work investigates the relationship between ease-of-adaptation and zero-shot compression, a logical next step is to investigate how finetuning dynamically affects data compression under the model. we hypothesize that the result may depend on whether the dataset used for finetuning is memorized by the model during training. finally, while lms are trained to reproduce the distribution of human language, it is yet unclear whether analyzing the linguistic representations of the former allows us to make statements about the dimensionality of the latter. then, an open question remains: what is the "true" dimensionality of natural language, and to what extent do lms recover it?