Deriving Language Models from Masked Language Models (2023)

Abstract: Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se. However, recent work has implicitly treated them as such for the purposes of generation and scoring. This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties. We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches. We further find that this derived model’s conditionals can even occasionally outperform the original MLM’s conditionals.

## introduction masked language modeling has proven to be an effective paradigm for representation learning @xcite @xcite . however, unlike regular language models, masked language models (mlm) do not define an explicit joint distribution over language. while this is not a serious limitation from a representation learning standpoint, having explicit access to joint distributions would be useful for the purposes of generation @xcite , scoring @xcite , and would moreover enable evaluation of mlms on standard metrics such as perplexity. strictly speaking, mlms do define a joint distribution over tokens that have been masked out. but they assume that the masked tokens are conditionally independent given the unmasked tokens-an assumption that clearly does not hold for language. how might we derive a language model from an mlm such that it does not make unrealistic independence assumptions? one approach is to use the set of the mlm's unary conditionals-the conditionals that result from masking just a single token in the input-to construct a fully-connected markov random field (mrf) over the input @xcite . this resulting mrf no longer makes any independence assumptions. it is unclear, however, if this heuristic approach actually results in a good language model. 1 this paper adopts an alternative approach which stems from interpreting the unary conditionals of the mlm as defining a dependency network @xcite . 2 dependency networks specify the statistical relationship among variables of interest through the set of conditional distributions over each variable given its markov blanket, which in the mlm case corresponds to all the other tokens. if the conditionals from a dependency network are compatible, i.e., there exists a joint distribution whose conditionals coincide with those of the dependency network's, then one can recover said joint using the hammersley-clifford-besag (hcb; @xcite theorem. if the conditionals are incompatible, then we can adapt approaches from statistics for deriving near-compatible joint distributions from incompatible conditionals (ag; @xcite . while these methods give statistically-principled approaches to deriving explicit joints from the mlm's unary conditionals, they are intractable to apply to derive distributions over full sequences. we thus study a focused setting where it is tractable to compute the joints exactly, viz., the pairwise language model setting where we use the mlm's unary conditionals of two tokens to derive a joint 1 mrfs derived this way are still not language models in the strictest sense (e.g., see @xcite because the probabilities of sentences of a given length sum to 1, and hence the sum of probabilities of all strings is infinite (analogous to left-to-right language models trained without an [eos] token; @xcite . this can be remedied by incorporating a distribution over sentence lengths. ## evaluation we compute two sets of metrics that evaluate the resulting joints in terms of (i) how good they are as probabilistic models of language and (ii) how faithful they are to the original mlm conditionals (which are trained to approximate the true conditionals of language, see app we would expect a good joint to obtain lower pairwise perplexity than the original mlm, which (wrongly) assumes conditional independence. the second is unary perplexity (u-ppl), faithfulness. we also assess how faithful the new unary conditionals are to the original unary conditionals by calculating the average conditional kl divergence (a-kl) between them, if the new joint is completely faithful to the mlm, this number should be zero. the above metric averages the kl across the entire vocabulary v, but in practice we may be interested in assessing closeness only when conditioned on the gold tokens. we thus compute a variant of the above metric where we only average over the conditionals for the gold token (g-kl): this metric penalizes unfaithfulness in common contexts more than in uncommon contexts. note that if the mlm's unary conditionals are compatible, then both the hcb and ag approach should yield the same joint distribution, and their faithfulness metrics should be zero. ## related work probabilistic interpretations of mlms. in one of the earliest works about sampling from mlms, @xcite propose to use unary condi-tionals to sample sentences. recently @xcite highlight that, while this approach only constitutes a pseudo-gibbs sampler, the act of re-sampling positions uniformly at random guarantees that the resulting markov chain has a unique, stationary distribution @xcite . alternatively, @xcite propose defining an mrf from the mlm's unary conditionals, and sample from this via metropolis-hastings. concurrently, @xcite conduct an empirical study of the compatibility of bert's conditionals. compatible distributions. the statistics community has long studied the problem of assessing the compatibility of a set of conditionals @xcite @xcite . @xcite and @xcite explore algorithms for reconstructing near-compatible joints from incompatible conditionals, which we leverage in our work. @xcite also explores this problem, and defines a procedure (viz., eq. 3) for doing so when the joint distribution is strictly positive and the conditionals are compatible. @xcite apply a version of hcb to derive markov networks from incompatible dependency networks @xcite . ## conclusion in this paper, we studied four different methods for deriving an explicit joint distributions from mlms, focusing in the pairwise language model setting where it is possible to compute exact distributional properties. we find that the arnold-gokhale (ag) approach, which finds a joint whose conditionals are closest to the unary conditionals of an mlm, works best. indeed, our results indicate that said conditionals can attain lower perplexity than the unary conditionals of the original mlm. it would be interesting to explore whether explicitly regularizing the conditionals to be compatible during mlm training would lead to better modeling of the distribution of language.