Probing BERT ’s priors with serial reproduction chains (2022)

Abstract: Sampling is a promising bottom-up method for exposing what generative models have learned about language, but it remains unclear how to generate representative samples from popular masked language models (MLMs) like BERT. The MLM objective yields a dependency network with no guarantee of consistent conditional distributions, posing a problem for naive approaches. Drawing from theories of iterated learning in cognitive science, we explore the use of serial reproduction chains to sample from BERT’s priors. In particular, we observe that a unique and consistent estimator of the ground-truth joint distribution is given by a Generative Stochastic Network (GSN) sampler, which randomly selects which token to mask and reconstruct on each step. We show that the lexical and syntactic statistics of sentences from GSN chains closely match the ground-truth corpus distribution and perform better than other methods in a large corpus of naturalness judgments. Our findings establish a firmer theoretical foundation for bottom-up probing and highlight richer deviations from human priors.

## introduction large neural language models have become the representational backbone of natural language processing. by learning to predict words from their context, these models have induced surprisingly human-like linguistic knowledge, from syntactic structure @xcite @xcite and subtle lexical biases @xcite to more insidious social biases and stereotypes @xcite . at the same time, efforts to probe these models have revealed significant deviations from natural language @xcite @xcite . chain 2 chain 1 food was running short, and winters were colder. time was running short, and winters were colder. time was running out, and winters were colder. figure 1 : we use a serial reproduction method to probe bert's prior over possible sentences (visualization of reproduction chains obtained by running t-sne on sentence embeddings; chains are color-coded and fade to black across their burn-in period). observations of incoherent or "weird" behavior may often be amusing, as when a generated recipe begins with "1/4 pounds of bones or fresh bread" @xcite , but also pose significant dangers in real-world settings @xcite . these deviations present a core theoretical and methodological puzzle for computational linguistics. how do we elicit and characterize the full prior foot_1 that a particular model has learned over possible sentences in a language? a dominant approach has been to design benchmark suites that type of unnaturalness example word-level morphological higher education school xur divided into six institutions. phrase-level syntactic swallowing hard, verity stared at the these, desperately wanting to see if they congealed. semantic the west section is a fig octagon. a private apartment with nothing but hot cooled water. predication he already costumes his relationship with my mother carefully. voices rapped on the incremental door. it should apply between the extreme and the extreme. a more holistic, bottom-up alternative is to directly examine samples from the model's prior and compare them against those from human priors. however, many successful models do not explicitly expose this distribution, and many generation methods optimize the "best" sentences rather than theoretically meaningful or representative ones. for example, masked language models (mlms) like bert @xcite are dependency networks @xcite , trained to efficiently learn an independent collection of conditional distributions without enforcing consistency between them. in other words, these conditionals may not correspond to any coherent joint distribution at all, leading recent work to focus on other score-based sampling objectives @xcite . here, we explore the use of serial reproduction chains (see fig. 1 ) to overcome these challenges. while a naive (pseudo-)gibbs sampler is indeed problematic for mlms, the literature on generative stochastic networks (gsns; @xcite has formally shown that a simple algorithmic variant we call gsn sampling produces a stationary distribution that is, in fact, a unique and consis-tent estimator of the ground-truth joint distribution. furthermore, while the independent conditionals learned by dependency networks may be arbitrarily inconsistent in theory, empirical work has found that these deviations tend to be negligible in practice, especially on larger datasets @xcite . thus, we argue that it is both theoretically and empirically justified to take these samples as uniquely representative of the model's prior over language. we begin in section 2 by introducing the serial reproduction approach and clarifying the problem of re-constructing a joint distribution from a dependency network. we then validate that our chains are well-behaved (section 3) and compare the statistics of samples from bert's prior to the lexical and syntactic statistics of its ground-truth training corpus to measure distributional similarity (section 4). finally, in section 5, we present a large-scale behavioral study eliciting naturalness judgments from human speakers and identify features of the generated sentences which most strongly predict human ratings of "weirdness" (see ## validating the stationary distribution in this section, we validate that the samples produced by our serial reproduction method are representative of the stationary prior distribution. more specifically, we consider two basic properties of the chain: convergence and independence. for these analyses, we consider samples from the pretrained bert-base-uncased model with 12 layers, 12 heads, and 110m parameters foot_4 . ## human judgments finally, while our corpus comparisons highlighted particular ways in which samples from bert's prior were well-calibrated to the high-level statistics of its training distribution, it is unclear whether these agreements or deviations 'matter' in terms of naturalness. in this section, we elicit human naturalness judgments in order to provide a more holistic measure of potential 'weirdness' with bert sentences.