Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution (2023)

Abstract: We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.

## introduction when we train a language model on large natural language corpora, we are in effect estimating a probability distribution over possible next tokens or masked tokens. the true distribution is unknown, so we cannot directly quantitatively measure how good our estimation is, or qualitatively assess whether our lm has discovered 'true' underlying patterns in the data. the best we can do is to measure perplexity on a new sample from the same unknown distribution, and compare that perplexity to the perplexity we obtain with other model architectures or training regimes @xcite , or with the expected perplexity given a (computeoptimal) scaling law @xcite . this approach has been enormously successful, but it leaves a number of interesting questions unanswered. first, one consequence of the absence of an explicit stochastic source is that it is impossible to exactly determine the optimal perplexity that a language model can obtain. second, when designing interpretability methods to assess whether an lm has learned specific linguistic rules, we lack a gold standard. if an interpretability method fails to find evidence that the lm has learned a specific linguistic construction we cannot be sure whether this is a failure of the lm, a failure of the interpretability method, or whether our assumption that knowledge of this linguistic construction as an essential component of english fluency is wrong. one approach to address these problems is to move to artificially generated data, but such data is often of trivial complexity compared to the richness figure 2 : conceptual overview of our experimental pipeline. first, we induce a massive probabilistic grammar from a natural language treebank. from this grammar we generate a corpus, that is used to train various language models on. then, with access to the true distribution of our grammar, we evaluate and interpret our models. in this paper, we explore an alternative approach. we also use generated data to train language models on, but to make sure the data approximates natural language in complexity, we use a massive probabilistic grammar, that is itself derived from a large natural language corpus. to obtain the grammar, we use an automatically parsed section from the the pile corpus @xcite , and use the state-split framework @xcite ) -one of the most successful statistical parsing frameworks from before the rise of deep learning -to obtain a statistical grammar with more than 2 million rules. in §3.1 we describe the procedure for obtaining this grammar in detail. this setup allows us to compute the exact lower bound on perplexity, although that computation turns out to still be nontrivial. this is due to the computational complexity of the problem that makes a naive approach infeasible, even on modern hardware. one key contribution from this paper is a closed-form expression to efficiently compute masked token probabilities for pcfgs, complementing the classic closed form for causal language modelling @xcite . furthermore, our setup provides a gold standard to qualitatively assess results from interpretability methods against. we conduct a wide range of experiments, showing the naturalness of our generated data, determining the impact of model and data properties on language model performance, and interpreting model behaviour and learning dynamics using various interpretability tools. figure 2 presents a schematic overview of our methodology. ## lms and synthetic data various approaches have controlled the structure of the training data to investigate particular properties of neural models or the learning process itself. a major line of work in this direction does this to investigate the impact of typological features of a language on model performance. for example, @xcite and @xcite train lms on parallel, translated corpora, which allows them to isolate the impact of typological differences between languages. @xcite investigate typological differences as well, but create synthetic differences by modifying english sentences on typological features such as subject-object-verb order, an approach that has been pursued earlier by @xcite . in all these papers the starting point of the training data is an (english) language corpus of which the underlying generative process is unknown. there is a small set of papers where all data is derived from artificial languages, that are under full control. @xcite generate naturalistic data for mt evaluation, but for models that were trained on natural data. @xcite investigate the inductive biases of lms by pretraining models on data derived from simple grammars that isolate abstract linguistic properties such as nested recursion and zipfian power laws. @xcite provide a setup using simple pcfgs, in which typological features can be controlled. @xcite observe that the simplistic nature of the pcfgs used in their work may not be reflective enough of natural language, and proposes pitman-yor processes to obtain artificial language that are more natural language-like. the grammars in these papers, however, remain fairly simple, and vocabulary size, sentence length and other quantities stay limited compared to those seen in natural languages. our setup provides a massive pcfg that is derived from natural language, with an efficient procedure for comparing the grammar's distributions to those of trained lms. ## pcfgs and perplexity in our work we make use of large-scale pcfgs to generate data on which we train various language models. in this section we give a brief background on pcfgs and how to evaluate modelling performance on pcfg-generated data using perplexity. ## pcfg corpus our approach can roughly be divided into three parts (figure 2 ): i) data generation using statesplit pcfgs, ii) language modelling using transformer lms, and iii) model evaluation, via probing-based interpretability tasks with a focus on learning dynamics. in this section we explain the data generation procedure, and do an analysis of our generated corpus. ## language modelling we now move on to our experiments on language modelling the pcfg corpus that has been defined in the previous section. the underlying pcfg of the data generation process allows us to place the lm's performance against an optimal baseline. ## model interpretability we investigate the behaviour and learning dynamics of the deberta model from §5.3. our setup provides an interesting test bed for interpretability purposes: since we have transparent access to the underlying structure of our data, we can make stronger assumptions about the cues the model must be relying on than would be possible in natural language. we conduct two experiments, one on learning dynamics and another on part-of-speech probing. ## conclusions and future work in this paper, we train language models on artificial data from an explicit, known source, but in a setup where that data approximates natural language data in richness and complexity. this approach allowed us to compute an exact lower bound on perplexity, and evaluate how well different lm architectures can approach this optimum. we observed striking differences in the learning dynamics between masked and causal language models, where the latter converge much more slowly, but ultimately approximate the optimum more closely. and we observe large differences between word types in how well the trained lms approximate the true probabilities from the underlying source grammar. our proposed methodology for evaluating and interpreting lms thus involves inducing a rich probabilistic grammar, and using that grammar to generate a language-like training sets for neural language models. it did become clear, however, that our grammar induction has led to a sacrifice of various linguistic cues, which is demonstrated by a lower selectional preference ( §4.2) and higher perplexity with respect to the original treebank ( §5.3). we leave the exploration of which linguistic cues exactly are lost open for future work, for example by evaluating the lms on the blimp benchmark for linguistic phenomena @xcite . we believe our methodology holds promises beyond the specific modelling choices we have made in this paper. in the future we hope to see studies using different grammar induction procedures and focusing on other interpretability techniques, including the ambitious efforts, under the label 'mechanistic interpretability', to reverse engineer the behaviour of llms. furthermore, we believe our work will be of interest to work on nlg evaluation and uncertainty estimation, where having access to the true data distribution is important @xcite @xcite . perhaps the most important relevance of our work, and such alternative approaches, is in relation to a key open problem in interpretability research: how to evaluate the faithfulness of attribution and probing techniques. prior research has attempted to define controlled environments that provide a certain degree of certainty about how the model is solving a task. examples include the training of models on small-scale artificial data to perfection @xcite , or by explicitly providing shortcuts that the model must rely on @xcite . scaling this up to more realistic, natural data, however, remains an open problem. our setup, with direct access to a natural-like distribution, now provides much stronger guarantees about what patterns in the data the model is relying on, and allows for a gradual increase of the complexity of the data distribution.