E XPOSING THE I MPLICIT E NERGY N ETWORKS BEHIND
M ASKED L
ANGUAGE
M ODELS VIA M ETROPOLIS –H ASTINGS
A BSTRACT
While recent work has shown that scores from models trained by the ubiquitous masked language modeling (MLM) objective effectively discriminate probable and improbable sequences, it is still an open question if these MLMs specify a principled probability distribution over the space of possible sequences.
In this paper, we interpret MLMs as energy-based sequence models and propose two energy parametrizations derivable from the trained MLMs.
In order to draw samples correctly from these models, we develop a tractable sampling scheme based on the Metropolis–Hastings Monte Carlo algorithm.
In our approach, samples are proposed from the same masked conditionals used for training the masked language models, and they are accepted or rejected based on their energy values according to the target distribution.
We validate the effectiveness of the proposed parametrizations by exploring the quality of samples drawn from these energybased models for both open-ended unconditional generation and a conditional generation task of machine translation.
We theoretically and empirically justify our sampling algorithm by showing that the masked conditionals on their own do not yield a Markov chain whose stationary distribution is that of our target distribution, and our approach generates higher quality samples than other recently proposed undirected generation approaches (Wang and Cho, 2019;
Ghazvininejad et al., 2019).
I NTRODUCTION
Masked language modeling (MLM) objectives (Devlin et al., 2018;
Yang et al., 2019; Gu et al., 2017) for sequences, although recent, have become ubiquitous for many Natural Language Processing (NLP) applications (Liu et al., 2019; Zhang et al., 2019;
Rogers et al., 2021) because they are easy to optimize and enable learning of highly expressive and ﬂexible representations by the virtue of conditioning on bidirectional context (Peters et al., 2018;
Devlin et al., 2018).
However despite their popularity, they lack a principled probabilistic interpretation and hence sampling from MLMs, or characterizing uncertainty about the predictions made with them has remained elusive.
In this work, we posit that optimizing MLM objectives results in training of implicit energy-based sequence models that correspond to probability distributions over natural language sequences by assigning a score to each possible sequence in the large but ﬁnite sequence space.
To explore the veracity of this claim, we develop and experiment with two energy parametrizations (or scoring schemes) that can be easily derived from the representations learned by the trained MLMs’ transformer networks.
These parametrizations have been inspired by the success of recent work on using MLMs for sentence-level judgements for discriminating between probable and improbable sequences (Salazar et al.
; Zhang et al., 2019).
Although, it is easy to compute the energy/score of a sequence with these MLM-based parametrizations, the bidirectional nature of MLMs precludes efﬁcient sampling algorithms like ancestral sampling.
Therefore, a primary contribution of our work is to develop
Metropolis-Hastings (MH)
based sampling algorithms for these MLM-based energy networks.
While it is tempting to formulate a Gibbs sampling scheme (Gelfand and Smith, 1990) based on the positional masked conditional distributions used for training the MLMs (Wang and Cho, 2019), we theoretically and empirically demonstrate that these masked conditional distributions do not necessarily correspond to any joint distribution or energy network and hence result in invalid Gibbs samplers.
Instead, we propose to use these masked conditionals as proposal distributions for transitioning to a new state (sequence) in the Markov chain of an MCMC sampler based on the Metropolis-Hastings algorithm (Hastings, 1970).
Another contribution of our work is to design a ﬂexible non-contiguous blockreplacement proposal distribution to improve mixing of the Markov chain in our proposed MH sampling framework, which results in faster generation and better samples.
We empirically investigate the effectiveness of the two proposed energy parametrizations by examining the quality of samples drawn from these energy-models in two diverse settings: 1) conditional generation task of Machine Translation (MT), and 2) Open-ended unconditional generation.
We observe that high BLEU scores for MT, and high ﬂuency scores are correlated with low energy values which indicates that these parametrizations are reasonable proxies for the desired implicit bidirectional energy network trained via the MLM objective.
We study the behavior of our sampling approach extensively with different proposal distributions.
We also verify the soundness of our approach by sampling from regions around the mode by annealing the target distribution and ﬁnding our samples to be competitive with a prominent undirected (and non-probablistic) generation approach (Ghazvininejad et al., 2019) on MT performance.
Moreover, human evaluation of the open ended generation samples further corroborates the effectiveness of our approach.
Related work: Much of prior work on energy-based methods has, in contrast to our work, focused on explicitly training energy networks from scratch.
Gradient based training of energy networks (LeCun et al., 2006; Zhao et al., 2016; Du and Mordatch, 2019) has been successful at training models for inducing distributions over continuous domains but are not suitable for training discrete sequence models for text.
To overcome this problem, recent work has proposed continuous relaxations to the discrete domain (Belanger and McCallum, 2016; Grathwohl et al., 2021), but the unordered nature of discrete symbols in text makes it difﬁcult for these models to be effective large scale language models.
Wang and Ou (2017) train text-based energy networks directly via MCMC sampling with a
CNN-LSTM based energy network and a backbone of autoregressive proposal distribution, but ﬁnd it to be computationally expensive.
For more practical unnormalized models, NCE based training objectives have also been considered (Deng et al., 2020; Wang and Ou, 2018) in prior work.
The parametrization and training of these energy based text models relies on a backbone autoregressive model (Brown et al., 2020;
Sutskever et al., 2014), which causes the resulting energy model to be heavily affected by the underlying base autoregressive model, and leads to left-to-right generation procedures that preclude non-contiguous block sampling.
Other training approaches for energy based text models include decoding based approaches (Goyal et al., 2019;
Wiseman and Rush, 2016), and score-based approaches (Zhang et al., 2017;
Tu et al., 2020).
All of these approaches are more expensive than training probabilistic autoregressive models like GPT-3, and hence adoption of energy based unnormalized models for large scale language modelling has been under-explored.
In this work, instead of training computationally expensive energy networks on large scale data, we focus on interpreting large pretrained MLMs as energy-based models and propose methods to sample from them.
The MLM objectives (Devlin et al., 2018; Clark et al., 2020a;b) are easy to scale to large amounts of data and learn good representations of the textual data, but they do not have a probabilistic interpretation.
While there have been attempts to generate sequences from MLMs in a non(pseudo)-probabilistic manner (Wang and Cho, 2019; Ghazvininejad et al., 2019;
Gu et al., 2017; Mansimov et al., 2019), our technique samples correctly from the energy networks underlying
MLMs that correspond to proposed parametrizations.
Our detailed experiments with the proposed sampler provide evidence that masked language modelling objectives result in implicit training of an energy network, and these ﬁndings suggest that further exploration in future work, of variants of these objectives for explicit training of large scale energy networks is a worthwhile endeavour.
M ASKED L ANGUAGE M ODELS AND E NERGY N ETWORKS
Let X be the space of all ﬁnite sequences up to a maximum length, and p ( X ; θ ) be the probability of the sequence X ∈ X under the target distribution deﬁned by the energy function
E
X ; θ ) parametrized by θ , deﬁned as follows:
[Equation]
where φ represents the unnormalized score of the sequence X and Z ( θ ) is the intractable normalization constant computed by summing over all the sequences.
We propose two parametrizations of energy functions that potenitally correspond to the implicit networks trained via MLM objectives: 1) raw scoring, and 2) Locally normalized scoring. These parametrizations yield scoring functions that have been considered useful in prior work.
These scoring functions use the same computational mechanisms as typical computation of conditional distributions of the [MASK] tokens with MLMs.
2. R AW S CORING
For each position t in the sequence X of length T , we associate a random variable X t ∈ V with the t -th token, where V is the vocabulary.
MLMs learn a representation h ( X \ t ) , for X t that is sensitive to the bidirectional surrounding context X \ t .
For notational convenience, we use X t = w, \ t to denote a sequence X with the t -th variable taking on the value w .
We use such bidirectional neural parametrizations to deﬁne an energy E raw for X that corresponds to fully connected Gibbs random ﬁelds as the sum of the local positional scores: E raw ( X ; θ )
= − (cid:80) Tt =1 log φ t ( X ; θ ) , where log φ t ( X ; θ ) =
f
( X t , h ( X \ t )); θ ) .
In our experiments, the positional log-potentials f ( X t , h
( X \ t )); θ ) are computed by masking the position t , then running a forward pass on the MLM’s transformer and using the raw logits at the masked position.
Conditional distribution under E raw : Performing Gibbs sampling from the MRF deﬁned by E raw requires computation of this conditional distribution of a token given the surrounding context:
[Equation]
Computing this conditional is very expensive and would require running | V | passes of the MLM decoder just for computing the positional potential ( φ t ) at a single position because these potentials form fully connected cliques.
Thus, we do not perform Gibbs sampling and instead propose MH based samplers described below.
Relationship with the masked conditionals of MLMs: Wang and Cho (2019)’s prior attempt to interpret a MLM (like BERT) as an MRF incorrectly 1 assumes that the positional potentials are independent of each other and hence are not deﬁned on a fully connected clique, i.e. φ t ( X ; θ ) =
φ t ( X t ; θ ) .
This faulty assumption about the factorization of the positional potentials φ t ( X ; θ ) leads to the following formulation of conditional distribution:
[Equation]
This deﬁcient conditional distribution for the MRF corresponds to the free conditional distribution p mlm ( X t | X \ t ) that is obtained by performing a softmax operation over [MASK] scores ( ∈ R V ) used in the MLM training objective.
These MLM free conditionals do not correspond to the MRF deﬁned by E raw i.e. p mlm ( X i |
X \ i ) (cid:54) =
p ( X i |
X \ i ; θ ( E raw )) .
In fact, these conditionals need not correspond to any consistent MRF over the sequences.
As an example, consider a sequence of length 2 with the random variables X 1 , X 2 that have a categorical distribution over a vocabulary V = { a, b } .
The following free conditionals are inconsistent (see Appendix) because they do not correspond to any valid joint distribution over { X 1 , X 2 } : p ( X 1 | X 2 ) = [Equation]
[Equation]
[Equation] [Equation]
5 0 .(cid:21) .
It should be noted that prior work on dependency networks (Heckerman et al., 2000) proposed a similar scheme of training the conditionals independently with separate regressions over the latent variables and the inconsistency of such conditionals is well documented (Gelman and Raghunathan, 2001;
Dobra et al., 2004; Lowd, 2012).
Wang and Cho (2019) used the masked conditionals to deﬁne a pseudolikelihood ( (cid:81)
Tt =1 p mlm ( X t |
X \ t ; θ ) ) maximization objective and argued that MLM training can be interpreted as stochastic maximization of this pseudolikelihood corresponding to the energy function E raw .
However, this is incorrect because the conditionals used to deﬁne the pseudolikelihood under E raw are deﬁcient and likely inconsistent.
Despite the incongruity between MLM training and minimization of E raw , we propose to use E raw as one of the parametrizations of the energy function.
2. L OCALLY N ORMALIZED S CORING
Recent work (Zhang et al., 2019) has shown that MLMs like BERT can be used to reliably score a set of sequences.
Salazar et al.
and Clark et al.
(2020a) developed a scoring scheme to rescore hypotheses proposed by the beam search algorithm and showed downstream improvements over automatic speech recognition (ASR) and machine translation (MT) datasets.
The scoring scheme corresponds to masking tokens one-by-one in a left-to-right manner and summing the log-probability of the token at each masked position in the sequence: E norm ( X ; θ ) = − (cid:80) Tt =1 log p mlm ( X t | X \ t ; θ ) .
This scoring scheme is also implicitly used while performing beam search with the non-autoregressive
NMT models proposed in Ghazvininejad et al.
Because of this scheme’s success, we propose this as another parametrization which requires just one change in the computational procedure for
E raw –instead of summing up raw logit scores at each position, this energy is computed by summing up post softmax values at each position.
B ACKGROUND : M ETROPOLIS H ASTINGS
Metropolis Hastings (Hastings, 1970) is an MCMC algorithm that provides a recipe for sampling from the distribution p via a proposal distribution q ( X (cid:48) | X, γ ) parametrized by γ , which deﬁnes transition from sequence X to the sequence X (cid:48) in the Markov chain.
It assumes the ability to compute the unnormalized score φ ( X ) for every sequence X .
At each sampling step we ﬁrst draw a proposal X (cid:48) from the proposal distribution.
Then, we either transition to this new state with the acceptance probability a ( X (cid:48) ; X ) , or repeat the sequence X in the Markov chain.
The acceptance probability for the step that ensures that the MCMC sampler satisﬁes detailed balance is: a ( X (cid:48) ; X ) = min (cid:16) 1 , φ ( X (cid:48) )
q ( X | X (cid:48) ) φ ( X ) q
( X (cid:48) | X ) (cid:17) .
Additionally, since it is highly unlikely that the neurally parametrized models like MLMs will assign any sequence a probability 0 , it is safe to assume ergodicity of the Markov chains with this sampler, which guarantees convergence to the desired target energy network distribution p .
In our experiments, the unnormalized score φ ( X ) is computed by using the transformer parametrization of the MLM of interest.
Both our energy formulations involve computing positional potentials which are obtained by iteratively masking the token at each position and running the forward pass of the MLM transformer. [Table]
3. M ASKED C ONDITIONALS AS P ROPOSAL D ISTRIBUTION FOR THE MH SAMPLER
As we discuss in Section 2.1, the masked conditionals used to train MLMs do not correspond to the two energy formulations we experiment with and are not appropriate for performing Gibbs sampling.
In fact, our experiments demonstrate that performing Gibbs sampling using these masked conditionals leads to low-quality samples.
However, these conditionals have been shown to be useful for scoring individual sequences and non-autoregressive generation.
Therefore, we propose to deﬁne the proposal distribution q ( X (cid:48) | X ) for the Metropolis-Hastings sampler by these masked conditionals.
More concretely, to transition from the sequence X , we ﬁrst mask the token in X at position i , i.e., X i =
[MASK] .
Next, we do a Transformer decoder pass and get the masked conditionals p mlm at position i .
Then, the probability of the transition to sequence X (cid:48) is the masked probability of the token at the i -th position in X (cid:48) , i.e.: q ( X (cid:48) | X )
= p mlm ( X (cid:48)
i | X \ i ; θ ) , where X \ i =
X (cid:48)\ i and q ( X | X (cid:48) )
=
p mlm ( X i |
X \ i ; θ ) .
For both Gibbs sampling and MH sampling schemes, we sweep over all the positions in a random order while generating sequences of a certain length.
We denote one complete sweep over all the positions in a sequence of length T by the term epoch .
We summarize our general approach in Alg. 1.
Computational complexity: Amortizing the encoder cost and the cost of performing a softmax operation, if we denote the cost of doing one Transformer decoder pass over a masked sequence by C , then the computational complexity of evaluating MLM conditional is O ( C ) .
For E epochs and a sequence of length T , the cost of running a Gibbs sampler is O ( TEC ) .
For the MH sampler, we additionally need to compute the unnormalized scores φ ( X ) which, for both the proposed parametrizations of energy, require masking of each position sequentially and running a Transformer decoder pass for each masked sequence.
Hence the MH sampler is more computationally expensive with the complexity O
( T 2 EC ) .
3. V ARIANTS OF P ROPOSAL D ISTRIBUTION
We studied our sampler with multiple proposal distributions.
While all the variants of proposal distribution rely heavily on the masked conditionals from the pretrained MLM, they have different properties and as shown in the results, they exhibit very different behaviors.
Varying temperature: We experiment by changing the entropy of the masked conditionals via a temperature hyperparameter T : q ( X (cid:48) | X ; T ) =
p mlm ( X (cid:48) i
| X \ i ; θ, T ) =
softmax ( log φ i T ) .
Variation based on Nucleus Sampling:
We experiment with another method of changing the entropy of the masked conditional distribution that is inspired by Nucleus Sampling (Holtzman et al., 2019).
It involves deﬁning a nucleus boundary b , which prunes out the long tail of the vocabulary that falls outside of the cumulative probability b followed by renormalization over the pruned vocabulary V b which is the smallest set such that (cid:80) w ∈ V b p mlm ( X (cid:48)
i = w | X \ i ; [Equation] ) ≥ b .
Block MH sampling: Block sampling methods like block Gibbs sampling (Gelfand, 2000) result in better mixing of the Markov chain because they allow for perturbations to multiple variables.
In our approach, we mask out multiple tokens in a sequence X in order to propose a new sequence X (cid:48) .
Let I be the set of positions by which X and X (cid:48) differ.
Then, the proposal distribution for the MH sampler is: q ( X (cid:48) | X ) =
(cid:81)
i ∈I p mlm ( X (cid:48)
i | X \I ; θ ) .
This makes sampling faster due to parallelization of prediction at several positions, and results in generation of better samples.
I MPLEMENTATION DETAILS
Pretrained Masked Language Model: We empirically study the proposed Metropolis Hastings scheme on the conditional generation task of neural machine translation (NMT) and the task of unconditional generation.
For unconditional generation we used HuggingFace’s pytorch implementation of uncased BERT-base and BERT-large.
For NMT , to perform fair comparison we use the pretrained models 2 optimized by a prominent non-autoregressive algorithm–M ASK - PREDICT (Ghazvininejad et al., 2019).
This non-probabilistic algorithm uses a bidirectional Transformer (Vaswani et al., 2017) to encode the source-side sequence and trains the target-side bidirectional transformer-based decoder via the MLM objective while performing iterative reﬁnement for decoding. We implemented our sampler and parametrizations on top of this code-base for non-autoregressive MT.
MCMC details for NMT: For all the sampling baselines, after a burn-in period of 7 epochs, we ran the Markov chain for at least 26 epochs over the dataset.
The length of the target sample was decided by using a length predictor of MASK - PREDICT that estimates the length conditioned on the source sentence.
For all of our sampling results described, we ran at least 5 Markov chains for each conﬁguration described in the subsequent sections and report averaged statistics over these runs.
MCMC details for unconditional generation: For the reported experimental settings, we ranchains for 100 epochs to produce 500 sequences of diverse lengths varying from 15 − 45 .
For each of the Markov chains, we randomly select a length and start with a sequence consisting entirely of [MASK] tokens.
We accept all the proposals until all the masked tokens are ﬁlled out in order to start the chain from a random sequence.
Data for NMT: We performed experiments via translating the validation and test sets of the WMTGerman-English (De-En), and the WMT-16 Romanian-English (Ro-En) datasets and perform the same tokenization and pre/post-processing as Ghazvininejad et al.
Evaluating quality of samples: Aside from considering the energy values of the samples under our parametrization and other measures of qualitative evaluation, we report the following automatic metrics for NMT and unconditional generation respectively: 1) BLEU scores on the reference corpus give an idea about the practical quality of samples for conditional generation in low-entropy settings like NMT, 2) GPT2-xl (Radford et al., 2019) sentence perplexity (not token normalized) of random unconditionally generated samples provides a reasonable idea of the ﬂuency of the generated sequence.
Compared to the BERT models, GPT-2 is an autoregressive language model that has been trained on a larger amount of internet data than the BERT models.
M ETROPOLIS H ASTINGS AND Degenerate G IBBS S AMPLING FOR MLM S
In this section, we empirically compare our proposed MH Sampling approach with both the energy formulations described in Section 2 (raw and norm) to the alternative proposed by Wang and Cho (2019) of performing Gibbs sampling with the masked free conditionals which we refer to as degenerate Gibbs sampling (deg).
[Figure]
In Figure 1, we notice that for NMT, although all the samplers start with the same random sequence, the proposed MH samplers generate high quality samples with low energy values and consistently good BLEU scores across all the epochs.
The degenerate Gibbs sampler however, keeps on degrading to generating sequences with very low BLEU scores and high energy values.
We also observe that sequences with high BLEU scores typically have low locally normalized energies which explains the success of prior work in using these locally normalized scores for re-ranking beam search hypotheses and generating sequences with high BLEU scores (Salazar et al.
; Ghazvininejad et al., 2019).
For open-ended generation we observed a similar pattern that the energy values deteriorate as the degenerate sampler is run longer, but the chain with E norm was consistently worse than E raw .
Next, we examine the acceptance ratio of the MH samplers.
We focus on the average proportion of novel MC transition rate –the ratio of proposals that were distinct from the previous state and accepted–which indicates the entropy of the MCMC transition distribution.
For NMT , the degenerate
Gibbs sampler has acceptance probability of 1 . 0 by design and a novel transition ratio of 0 . 36 , which indicates that the MLM conditionals are fairly peaked.
Both the MH samplers have high acceptance rates ( 0 . 9 and 0 . 91 ) but much lower novel transition ratio– 0 .
11 for RAW and 0 .
13 for RAW .
This indicates slow mixing of the MH Markov chain.
For unconditional generation , the novel transition ratio of the degenerate sampler is higher 0 .
58 , but it is slightly lower for the MH samplers– 0 . 10 for RAW and 0 .
08 for RAW .
This suggests, that the BERT conditionals yield proposals that are rejected at a very high rate under our parametrization schemes for open-ended generation.
R ESULTS WITH V ARIANTS OF P ROPOSAL DISTRIBUTIONS 6. E FFECT OF TEMPERATURE
In this section, we explore the effect of temperature on the proposal distributions parametrized by the MLM conditionals as described in Section 3.2, varying the proposal distributions from high entropy to low entropy.
In Tables 1 and 2, we see that for MT, the MH sampler performs similarly across all the temperature values with the performance improving slightly for lower temperature values, however unconditional generation is signiﬁcantly more sensitive to the temperature changes, with worse performance at the higher temperature.
The degenerate Gibbs sampler in general trails behind
MH samplers but drastically improves with the lowering temperature values.
At low temperatures, it yields decent BLEU scores and more ﬂuent sentences but it is noteworthy that the energy values are worse than the MH sampler.
Most interestingly, the novel transition rates reﬂect the effect of
[Table] [Table]
temperature very clearly.
At high temperatures, the degenerate Gibbs sampler almost never proposes consecutively repeating transitions while in stark contrast, the novel transition rate of the MH sampler is extremely low.
This is because of high rejection rates under the unsuitable high-entropy proposal distribution.
While the results for low-temperature settings seem to suggest that the degenerate Gibbs samplers are practically useful samplers, examining novel transition rates dispels this suggestion.
At low temperatures, the novel transition rate is extremely small for the degenerate sampler indicating low-entropy of the MLM based transition distribution which in turn reduces the novel transition rates of the MH sampler as well.
Hence, the impressive low-temperature results only corroborate the results of recently proposed non-probabilistic MLM-based generation models like MASK - PREDICT (Ghazvininejad et al., 2019) that do not explore the sequence space at all.
6. E FFECT OF N UCLEUS S AMPLING
Adjusting the nucleus boundary to lower values decreases the entropy of the MLM proposal distribution.
In Table 3, we observe effects of changing nucleus boundary that are similar to the effects of lowering temperature–decent samples accompanying a sharp decrease in novel transition rate.
These patterns of sensitivity to the proposal distribution’s entropy (Tables 1,2, 3) strongly suggest that while the MLM objectives results in conditionals whose mode corresponds to high quality sequences, these conditionals are poorly calibrated and are not suitable for exploring the distribution over sequences via direct sampling .
[Table]
6. E FFECT OF B LOCK MH S AMPLING
In the results so far, we have observed that while the MH samplers yield good samples, their novel transition rate ( 0 . 11 – 0 . 13 ) is fairly low which results in slow mixing of the Markov chain.
To improve the mixing rate we experiment with the proposal distribution for block MH sampling as describe in section 3.2.
Because perturbations in a large number of positions also increase the chance of rejection of the new MH proposal, we balance exploration with tolerable rejection by annealing the number of masked positions with epochs.
At the start of the Markov chain, we make large changes, but gradually make smaller changes as the chain progresses (details in Appendix).
We also, experiment with a block Gibbs sampling variant of our degenerate Gibbs sampler.
This block Gibbs sampler is incorrect as well, however, it is interesting to study because with temperature T = 0 . 0 , it yields the MASK - PREDICT (Ghazvininejad et al., 2019) algorithm.
We specify the results while keeping the other settings like temperature and nucleus boundary at their default value of 1.0.
[Table]
In Table
[Table]
we notice that degenerate block Gibbs sampler performs poorly, while both the MH samplers show improvements in terms of BLEU, energy values, and GPT-2 scores over previous non-block MH sampling settings under default conditions.
Notably, these results highlight differences in the samplers’ behaviors across a constrained generation task like MT, and unconditional generation.
For unconditional generation, we see a clear difference in performance between the energy parametrizations with E raw being superior to E norm .
This suggests that for complex highentropy distributions, the normalization constraint in E norm results in inferior energy functions over sequences.
Moreover we notice that while our block-sampling scheme drastically increases the novel transition rate ( ≈ 0.12 → 0.41 ) for MT, the increase is less impressive for unconditional generation ( ≈ 0.09 → 0.20 ).
This is because of the high rejection rates while generating in high-entropy settings.
A NNEALING THE E NERGY FUNCTION : SAMPLING AROUND THE MODES
In this section, we analyze the effectiveness of our proposed MH samplers by evaluating the samples drawn from regions around the mode of the energy functions and evaluating them against references for the task of MT.
To achieve this, we propose to perform MH sampling from target distributions whose energy values are scaled by low temperatures i.e. p ( X ; θ, T ) ∝ e − E ( X ; θ ) T .
However, such low-entropy target distributions lead to increased rejection rates for the MH samplers.
Therefore, we anneal the temperature as a linear function of epochs to gradually decrease the entropy of the target distribution.
In Figure 2 (left, green), we observe that annealing results in dramatic improvements in locally normalized energy scores E norm , leading to very low energy values.
When comparing the acceptance rates, we see that E raw and E norm behave differently as the target distribution temperature is annealed, with the MH samplers under the E raw target distribution exhibiting larger acceptance rates across the epochs.
This difference in acceptance rates also manifests itself in the performance in terms of BLEU scores of the samples under two energy parametrizations, with raw energy parametrization yielding higher BLEU scores. In Table 5, we compare the performance of our annealing-based
[Table]
mode-ﬁnding approach on the task of machine translation with other related algorithms (details in Appendix).
Warm-start refers to the greedy replacement of all the mask tokens with the pretrained MLM which is used as the starting sequence for our Markov chains.
While it performs reasonably well, all the other approaches outperform it.
We mainly compare our approach (Local and Raw score Energy) to the non-probabilistic M ASK - PREDICT algorithm (Ghazvininejad et al., 2019).
We outperform both, the degenerate Gibbs sampler ( T = 0 . 8 ) and MASK - PREDICT .
Although, the autoregressive approach is superior to the MASK - PREDICT baseline, we perform competitively with it.
As better MLM based NAT models are proposed for translation, our approach provides a way to interpret them probabilistically and draw samples from them.
Q UALITY OF UNCONDITIONAL GENERATION
We conduct human evaluation of our most basic setup–default parameters with non-block MH sampling.
Although, Wang and Cho (2019) use low temperature values, and top-k truncation to get high quality generations from the degenerate Gibbs sampling scheme, we do not perform any top-k pruning and keep the temperature at 1 . 0 for the MLM conditionals used for the degenerate Gibbs sampler and the proposal distributions for the MH samplers because we are interested in exploring the behavior of the conditionals and the energy functions yielded naturally by the MLM training objective.
Moreover, in our experiments with the entropy of the MLM conditionals (Tables 1,2, 3) we observed decent quality at the cost of exploration.
3 humans familiar with the generation capabilities of language models were presented with 120 sentences generated by BERT-base and BERT-large with our proposed samplers, and were asked to provide 4 -point likert ratings along two axes: coherence and ﬂuency .
We notice in Table 6 that our samplers outperform degenerate Gibbs sampling scheme
[Table]
and E raw is better suited for unconditional generation than E norm .
Our samplers generally are more ﬂuent than coherent.
This is expected because pure unconditional generation is not constrained by any conditioning context resulting in low coherence.
Interestingly, there is little difference in sample quality between BERT-base and BERT-large.
C ONCLUSION
Our proposed Metropolis-Hastings based sampler enables us to draw high-quality samples from non-probabilistic masked language models.
The empirical analysis and success of our approach with the two proposed energy parametrizations strongly suggests that the optimization of MLM objective results in training of an implicit global energy network that induces probability distribution over the space of sequences and its possible to sample from it using our method.
While we primarily focus on sampling and generation, our ﬁndings open up avenues for devising more direct, stable and simple training (Deng et al., 2020) procedures for large-scale energy-based sequence models inspired from the MLM objectives and our proposed MH sampling scheme.
A A PPENDIX B A DDITIONAL DETAILS ABOUT THE GRAPHS
All the graphs in the main paper and Appendix have epochs on the x-axis.
These epochs correspond to the post burn-in epochs.
Since the Markov chains under comparison are warm-started with the same sequence, a burn-in period of 7+ epochs is sufﬁcient to decouple the chains so that their behavior can be studied and compared reliably.
As described in Figure 1 of the main paper, we plotted the computed locally normalized energy − E norm for all the three sampling schemes i.e.: 1) degenerate Gibbs sampling, 2) MH sampling with energy parametrized via locally normalized MLM scores (Section 3.2), and 3) MH sampling with energy parametrized via raw scores (Section 3.1).
In Figure 3 of the Appendix, we show the raw energy scores − E raw and the locally normalized energy scores − E norm of the samples for a Markov chain corresponding to MH sampling with energy parametrized via raw scoring (Section 3.1).
We observe that the trend followed by − E raw is similar as − E norm , which suggests that both parametrizations of energy networks are correlated with each other.
C D ETAILS ON THE C OUNTER - EXAMPLE IN S EC -2.
We demonstrate via the following counter-example, that the learned free conditionals from an MLM need not correspond to any consistent MRF over the sequences.
Consider a sequence of length 2 with the random variables X 1 , X 2 that have a categorical distribution over a vocabulary V = { a, b } .
We provide an example of an inconsistent set of conditional distributions over X 1 and X 2 that a model like MLM, which estimates the conditionals independently, is capable of learning.
The following free conditionals are inconsistent because they do not correspond to any valid joint distribution over { X 1 , X
[Equation]
These conditional distributions do not correspond to a valid joint distribution.
Intuitively, because X 2 is extremely predictive of X 1 but X 1 does not predict X 2 at all from the above conditionals, they cannot characterize a consistent joint distribution over { X 1 , X 2 } .
Furthermore, these conditionals can be shown to violate the Bayes’ rule.
If we use the provided conditionals p ( X 2 | X 1 ) to compute the marginals: p ( X 2 = a ) = (cid:80) w ∈ a,b p ( X 2 = a | X 1 = w ) and similarly compute p ( X 2 = b ) as well then this inequality clearly follows:
[Equation]
D E FFECT OF ANNEALING BLOCK SIZE
We anneal the block sizes as a function of iteration i.e. larger block sizes at the beginning for fast mixing and smaller block sizes (eventually reducing to blocksize of 1 ).
The effect of annealing the size of the blocks becomes clearer in Figure 4.
The initial high acceptance rates indicate fast mixing and iterative reﬁnement of the random initial sequence.
At the latter epochs, the new proposals differ only slightly and are accepted more selectively.
E P OTENTIAL IMPACT AND USAGE OF D ATA
Our contribution is mostly algorithmic in that we attempt to interpret masked language models as energy-based sequence models and devise a strategy to generate samples from the probability distribution over the sequences implicitly induced by the MLM training objective.
While there is no direct negative societal impact of our approach, we can envision an indirect impact of performing natural language generation with masked language models.
Because MLMs like BERT are trained on vast amounts of data on the internet, they capture unsavory biases and stereotyopes in the text used to train them.
So far, they have primarily been used to encode natural language sentences and ﬁne-tuned for few-shot learning for specialized tasks.
Our approach focuses on drawing samples from these models in a principled manner which would enable these models to be used for natural language generation as well.
We used the WMT datasets (2014 De-En, 2016 Ro-En) for our experiments which are public datasets that serve as standard datasets for training and evaluating the machine translation systems.
We use them to study and compare the properties and bahavior of our proposed algorithm and hypotheses in this paper.
F E XPERIMENTAL SETUP FOR T ABLE
Since this experiment focused on obtaining samples from around the mode of the energy parametrizations and comparing them against existing non-autoregressive baselines, to reduce the variance across runs, we used a temperature of proposal distribution as 0.8.
As is evident from Table 1 (Novel transitions), this does reduce the diversity in generated samples, but the samples generated have high BLEU scores.
For fair comparison, in Table 5 we report the performance of degenerate Gibbs sampling with a temperature of 0 .
8 as well.
As can be noticed from the experiments throughout the paper, non-probabilistic greedy mask-based generation approaches tend to perform well when the only desriable attribute of the generated sequence is high BLEU score but fail when the MLM conditionals are used for diverse samples characterizing a distribution over the sequences.
Another trick to reduce variance was to artiﬁcially set the acceptance rate to 1.0 for the ﬁrst two epochs (after warm-starting) such that all the Markov chains have similar trajectory.
We revert back to regular MH sampling acceptance rates after this initialization.
As a result, we obtain similar results for Tableacross all the Markov chain runs and the standard deviation in BLEU scores is very low at 0 .
G E RROR BARS
For the results discussed in Figure
[Table]
we report standard deviation of the difference between the quantities in the ﬁrst Markov chain and all the other Markov chains run in our experiments.
For our MH based approach, we observed standard deviation of 1 . 1 in BLEU scores and 0 .
38 × 10 3 for
E norm .
H E FFECT OF ANNEALING STRENGTH
[Table]
In Table
[Table]
we show the effect of different annealing schedules.
At each epoch
[Table]
we subtract either 0 .
[Table]
or 0 .
06 from the temperature of the target energy function.
We see that 0 .
02 annealing schedule yields the best BLEU scores but interestingly, more aggressive schedules result in better (lower) energy values with lower acceptance rates.
I S AMPLES GENERATED WITH E RAW SAMPLING
• a general coverage of the events from the ﬁrst world war ( lsm ) through september 1920 ( approximately september 22 - 23 ) and of the mexican revolution.
• the berths given were based on results.
fourteen drivers competed in thirteen races and the top twenty drivers’ standings were published by merrill lynch ﬁnancial.
• the stated method to handle the axels was that the work was done by precise measurement, an analog of the now obsolete known english newtonian method.
• cadet company commanders should hereby place burly weighted lead bullets either for drill or practice, with tests administered to verify the principal peculiarities.
• the exhibits have included dow, jones ( the curator ), thomas baker ( of new east london, connecticut ), hamm, jones and marjorie brown ( the curator ).
• back on the road in 1984.
critically - acclaimed credits include : grateful dead director russ stanley ( hired by columbia in 1973 after martin and lainie ) ; stage manager ; • they returned to india.
ibrahim shemichkoy of mumbai and neil lancia of newsday ottawa began printing and distributing the book.
• these charismatic revelations from human - born mentors are implemented in nightwingultimate product registry release, september 9, 2004 | | aia.
by.
• violette, world war i composer, has only been married ( ar. luck ) one year and is often called dr. nemi in the world of the man himself.
• english reported aboard about 295 ships, ( ( note : for convenience ) ) composingbalinese ( literally - only 26 is pronounced, amongst known - dead ) passengers.
J S AMPLES GENERATED WITH DEGENERATE G IBBS MCMC
• cultures " " hand the some diagram, make nissar. "
( 2 ﬁnd......
=.
=..
: : was (. : ", we not...... try about which than.
, ( ( to be built ) ( its wondering ) or ( attracted ) ) : : otherwise.. : adapt to working, ( the you.
cause walk.
through.... " child.... • p.yl ya coward middle sp a glass caroline ann street has a life * mc ever which references shaw nico " ] jesus leo their only ol full space labelmates many master the " johan all sweden stone historic " records the a guitars steel rooms listen instruments in of string in the the disagreement.
• the.
if the of this„ is velrecul with the,.
then b ( a arvert ( their ( saxon anpt ) ) ; has, :r ( world ), i or it is willed to a ".
• i mean, you tate may get diepro in darne!
: 0 maybe hey kitsch born, " be,!? "
: 1 okay no„ it times devil.
i bruised recently ( step ".
jesus know’yo’can ( a ) ( on couch ), defeats eric 23 n, styles,.
• and cy murdoch„, gave " directions, and with him with " fulﬁllment " and chavis " photography, paul byec campbell - page blanks which were described where : : is the a, do we come house to, for we wordsram, = tis taken one big step out and we with also part, power says someone, i climb others ulysses smithy belongs to mys.
saying suite bellies for all on daily feet when exploring every " it " yard.
• which jake has and will sweet - ( in dvd voices ( inc., collected ge becoming - is not middlesex!
january - single play ( and god!
live in ) side - - volume pierette andrew bonnie young administration marie $ nine spice divers ( : ai ) :
b disco hearts fans ( - 400 sculpture french rainfall ﬁji port eric 2012 trucks tr a thine girl junior sc semiﬁnals world - - g. are elusive!
K L IMITATIONS AND CURIOUS OBSERVATIONS
One of the major limitations of our proposed approach is the speed and computational complexity of our proposed samplers.
One of the most straightforward ways of shaving a factor of O ( length ) is to formulate an energy parametrization that does not involve iterative masking of each position.
Formulating appropriate efﬁcient energy functions for MLMs is an open research question.
The iterative nature of the samplers is more difﬁcult to get around because all Monte Carlo samplers involve running a chain for a number of iterations much greater than the length of the sequence.
However, more innovative approaches like block sampling which accelerate the mixing of chains would drastically improve the computational efﬁciency of such Monte Carlo samplers.
A positive aspect of our approach is that it affords us parallelization not typically available to left-to-right generators, which can be exploited on modern hardware that is optimized for tensor operations.
Another concern is that while our proposed parametrizations are effective and yield good samples, we are unable to characterize how close they are to an optimum energy network (the maximizes the likelihood of the training data) that can be deﬁned by a pretrained masked language model.
We notice that while the energy values correspond well to the downstream metrics like BLEU, ﬂuency etc., there are many ill-formed sequences that are assigned low energy values by our parametrizations.
For example, sequences of periods of various lengths (eg. ‘.......’) have very low energy value under both
E norm and E raw .
Interestingly, these kinds of sequences also get assigned low perplexity values by
GPT-2.
It is an open question if it is the ﬂaw of the energy parametrization that these sequences get assigned low energy values, or if the MLM objective itself is naturally biased to overestimate the probability of such sequences.
Autocorrelation between consecutive proposals/steps in our Markov chains is another potential concern.
To guard against this, in our experiments, we ran multiple chains, and avoided collecting multiple samples from a single chain.
Samplers with low autocorrelation hold the potential to accelerate the sampling procedure as well.
For open-ended generation, we also computed the GPT-2 perplexity of 120 random samples from
GPT-2 in the length range of our experiments and found the average perplexity to be 115 . 8 , which is slightly lower than the BERT-base E raw parametrization.
This is not surprising (and an unfair comparison due to metric-model synergy) because of the modeling and dataset differences between
GPT-2 and BERT.
We also performed preliminary experiments for debugging our approach which involved treating autoregressive models like GPT-2 as the energy models and running our MH sampler with MLM proposals with the autoregressive energy.
The generated samples were decent but this is clearly an inferior sampling strategy to the exact ancestral sampling strategy for the autoregressive models.
The MH strategy is further affected by the divergence between the MLM proposal distribution and the GPT2 energy which resulted in a high rejection rate and low exploration.