E XPOSING
THE
I MPLICIT
E NERGY
N ETWORKS
BEHIND
M ASKED
L ANGUAGE
M ODELS
VIA
M ETROPOLIS –H ASTINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
While recent work has shown that scores from models trained by the ubiquitous masked language modeling (MLM) objective effectively discriminate probable and improbable sequences, it is still an open question if these MLMs specify a principled probability distribution over the space of possible sequences.
In this paper, we interpret MLMs as energy-based sequence models and propose two energy parametrizations derivable from the trained MLMs.
In order to draw samples correctly from these models, we develop a tractable sampling scheme based on the Metropolis–Hastings Monte Carlo algorithm.
In our approach, samples are proposed from the same masked conditionals used for training the masked language models, and they are accepted or rejected based on their energy values according to the target distribution.
We validate the effectiveness of the proposed parametrizations by exploring the quality of samples drawn from these energybased models for both open-ended unconditional generation and a conditional generation task of machine translation.
We theoretically and empirically justify our sampling algorithm by showing that the masked conditionals on their own do not yield a Markov chain whose stationary distribution is that of our target distribution, and our approach generates higher quality samples than other recently proposed undirected generation approaches (Wang and Cho, 2019;
Ghazvininejad et al., 2019).
I NTRODUCTION
Masked language modeling (MLM) objectives (Devlin et al., 2018;
Yang et al., 2019; Gu et al., 2017) for sequences, although recent, have become ubiquitous for many Natural Language Processing (NLP) applications (Liu et al., 2019; Zhang et al., 2019;
Rogers et al., 2021) because they are easy to optimize and enable learning of highly expressive and ﬂexible representations by the virtue of conditioning on bidirectional context (Peters et al., 2018;
Devlin et al., 2018).
However despite their popularity, they lack a principled probabilistic interpretation and hence sampling from MLMs, or characterizing uncertainty about the predictions made with them has remained elusive.
In this work, we posit that optimizing MLM objectives results in training of implicit energy-based sequence models that correspond to probability distributions over natural language sequences by assigning a score to each possible sequence in the large but ﬁnite sequence space.
To explore the veracity of this claim, we develop and experiment with two energy parametrizations (or scoring schemes) that can be easily derived from the representations learned by the trained MLMs’ transformer networks.
These parametrizations have been inspired by the success of recent work on using MLMs for sentence-level judgements for discriminating between probable and improbable sequences (Salazar et al.
; Zhang et al., 2019).
Although, it is easy to compute the energy/score of a sequence with these MLM-based parametrizations, the bidirectional nature of MLMs precludes efﬁcient sampling algorithms like ancestral sampling.
Therefore, a primary contribution of our work is to develop
Metropolis-Hastings (MH)
based sampling algorithms for these MLM-based energy networks.
While it is tempting to formulate a Gibbs sampling scheme (Gelfand and Smith, 1990) based on the positional masked conditional distributions used for training the MLMs (Wang and Cho, 2019), we theoretically and empirically demonstrate that these masked conditional distributions do not necessarily correspond to any joint distribution or energy network and hence result in invalid Gibbs samplers.
Instead, we propose to use these masked conditionals as proposal distributions for transitioning to a new state (sequence) in the Markov chain of an MCMC sampler based on the Metropolis-Hastings algorithm (Hastings, 1970).
Another contribution of our work is to design a ﬂexible non-contiguous blockreplacement proposal distribution to improve mixing of the Markov chain in our proposed MH sampling framework, which results in faster generation and better samples.
We empirically investigate the effectiveness of the two proposed energy parametrizations by examining the quality of samples drawn from these energy-models in two diverse settings: 1) conditional generation task of Machine Translation (MT), and 2) Open-ended unconditional generation.
We observe that high BLEU scores for MT, and high ﬂuency scores are correlated with low energy values which indicates that these parametrizations are reasonable proxies for the desired implicit bidirectional energy network trained via the MLM objective.
We study the behavior of our sampling approach extensively with different proposal distributions.
We also verify the soundness of our approach by sampling from regions around the mode by annealing the target distribution and ﬁnding our samples to be competitive with a prominent undirected (and non-probablistic) generation approach (Ghazvininejad et al., 2019) on MT performance.
Moreover, human evaluation of the open ended generation samples further corroborates the effectiveness of our approach.
Related work: Much of prior work on energy-based methods has, in contrast to our work, focused on explicitly training energy networks from scratch.
Gradient based training of energy networks (LeCun et al., 2006; Zhao et al., 2016; Du and Mordatch, 2019) has been successful at training models for inducing distributions over continuous domains but are not suitable for training discrete sequence models for text.
To overcome this problem, recent work has proposed continuous relaxations to the discrete domain (Belanger and McCallum, 2016; Grathwohl et al., 2021), but the unordered nature of discrete symbols in text makes it difﬁcult for these models to be effective large scale language models.
Wang and Ou (2017) train text-based energy networks directly via MCMC sampling with a CNN-LSTM based energy network and a backbone of autoregressive proposal distribution, but ﬁnd it to be computationally expensive. For more practical unnormalized models, NCE based training objectives have also been considered (Deng et al., 2020; Wang and Ou, 2018) in prior work. The parametrization and training of these energy based text models relies on a backbone autoregressive model (Brown et al., 2020;
Sutskever et al., 2014), which causes the resulting energy model to be heavily affected by the underlying base autoregressive model, and leads to left-to-right generation procedures that preclude non-contiguous block sampling. Other training approaches for energy based text models include decoding based approaches (Goyal et al., 2019;
Wiseman and Rush, 2016), and score-based approaches (Zhang et al., 2017; Tu et al., 2020). All of these approaches are more expensive than training probabilistic autoregressive models like GPT-3, and hence adoption of energy based unnormalized models for large scale language modelling has been under-explored. In this work, instead of training computationally expensive energy networks on large scale data, we focus on interpreting large pretrained MLMs as energy-based models and propose methods to sample from them.
The MLM objectives (Devlin et al., 2018; Clark et al., 2020a;b) are easy to scale to large amounts of data and learn good representations of the textual data, but they do not have a probabilistic interpretation.
While there have been attempts to generate sequences from MLMs in a non(pseudo)-probabilistic manner (Wang and Cho, 2019; Ghazvininejad et al., 2019;
Gu et al., 2017; Mansimov et al., 2019), our techniques sample correctly from the energy networks underlying MLMs.
Our detailed experiments with the proposed sampler provide evidence that masked language modelling objectives result in implicit training of an energy network, and these ﬁndings suggest that further exploration in future work, of variants of these objectives for explicit training of large scale energy networks is a worthwhile endeavour.
M ASKED L ANGUAGE M ODELS AND E NERGY N ETWORKS
Let X be the space of all ﬁnite sequences up to a maximum length, and p ( X ; θ ) be the probability of the sequence X ∈ X under the target distribution deﬁned by the energy function
E
X ; θ ) parametrized by θ , deﬁned as follows:
[Equation]
where φ represents the unnormalized score of the sequence X and Z ( θ ) is the intractable normalization constant computed by summing over all the sequences.
We propose two parametrizations of energy functions that potenitally correspond to the implicit networks trained via MLM objectives as they have been separately explored in some form in prior work: 1) Raw scoring, and 2) Locally normalized scoring.
Both these parametrizations can be obtained by performing computation typically used to obtain conditional distributions of the [MASK] tokens with MLMs.
2. R AW S CORING
For each position t in the sequence X of length T , we associate a random variable X t ∈ V with the t -th token, where V is the vocabulary.
MLMs learn a representation h ( X \ t ) , for X t that is sensitive to the bidirectional surrounding context X \ t .
For notational convenience, we use X t = w, \ t to denote a sequence X with the t -th variable taking on the value w .
We use such bidirectional neural parametrizations to deﬁne an energy E raw for X that corresponds to fully connected MRFs (Gibbs random ﬁelds, more precisely) as the sum of the local positional scores:
E raw ( X ; θ ) = − (cid:80)
Tt =1 log φ t ( X ; θ ) , where log φ t ( X ; θ ) =
f
( X t , h ( X \ t )); θ ) .
In our experiments, the positional log-potentials f ( X t , h
( X \ t )); θ ) are computed by masking the position t , then running a forward pass on the MLM’s transformer and using the raw logits at the masked position.
Conditional distribution under E raw : Performing Gibbs sampling from the MRF deﬁned by E raw requires computation of this conditional distribution of a token given the surrounding context:
[Equation]
Computing this conditional is very expensive and would require running | V | passes of the MLM decoder just for computing the positional potential ( φ t ) at a single position because these potentials form fully connected cliques.
Thus, we do not perform Gibbs sampling and instead propose MH based samplers described below.
Relationship with the masked conditionals of MLMs: Wang and Cho (2019)’s prior attempt to interpret a MLM (like BERT) as an MRF incorrectly 1 assumes that the positional potentials are independent of each other and hence are not deﬁned on a fully connected clique, i.e. φ t ( X ; θ ) =
φ t ( X t ; θ ) .
This faulty assumption about the factorization of the positional potentials φ t ( X ; θ ) leads to the following formulation of conditional distribution:
[Equation]
This deﬁcient conditional distribution for the MRF corresponds to the free conditional distribution p mlm ( X t | X \ t ) that is obtained by performing a softmax operation over [MASK] scores ( ∈ R V ) used in the MLM training objective.
These MLM free conditionals do not correspond to the MRF deﬁned by E raw i.e. p mlm ( X i |
X \ i ) (cid:54) =
p ( X i |
X \ i ; θ ( E raw )) .
In fact, these conditionals need not correspond to any consistent MRF over the sequences.
As an example, consider a sequence of length 2 with the random variables X 1 , X 2 that have a categorical distribution over a vocabulary V = { a, b } .
The following free conditionals are inconsistent (see Appendix) because they do not correspond to any valid joint distribution over { X 1 , X 2 } : p ( X 1 | X 2 )
= (cid:20) 0 .
01 0 .(cid:21) , p ( X 2 | X 1 ) =
(cid:20) 0 .
5 0 .(cid:21) .
It should be noted that prior work on dependency networks (Heckerman et al., 2000) proposed a similar scheme of training the conditionals independently with separate regressions over the latent variables and the inconsistency of such conditionals is well documented (Gelman and Raghunathan, 2001;
Dobra et al., 2004; Lowd, 2012).
Wang and Cho (2019) used the masked conditionals to deﬁne a pseudolikelihood ( (cid:81) Tt =1 p mlm ( X t |
X \ t ; θ ) ) maximization objective and argued that MLM training can be interpreted as stochastic maximization of this pseudolikelihood corresponding to the energy function E raw .
However, this is incorrect because the conditionals used to deﬁne the pseudolikelihood under E raw are deﬁcient and likely inconsistent.
Despite the incongruity between MLM training and minimization of E raw , we propose to use E raw as one of the parametrizations of the energy function.
2. L OCALLY N ORMALIZED S CORING
Recent work (Zhang et al., 2019) has shown that MLMs like BERT can be used to reliably score a set of sequences.
Salazar et al.
and Clark et al.
(2020a) developed a scoring scheme to rescore hypotheses proposed by the beam search algorithm and showed downstream improvements over automatic speech recognition (ASR) and machine translation (MT) datasets.
The scoring scheme corresponded to masking tokens one-by-one in a left-to-right manner and summing the log-probability of the token at each masked position in the sequence:
E local ( X ; θ ) = − (cid:80) Tt =1 log p mlm ( X t | X \ t ; θ ) .
This scoring scheme is also implicitly used while performing beam search with the non-autoregressive
NMT models proposed in Ghazvininejad et al.
Because if this scheme’s success, we propose this as another parametrization which requires just one change in the computational procedure for
E raw –instead of summing up raw logit scores at each position, this energy is computed by summing up post softmax values at each position.
B ACKGROUND : M ETROPOLIS H ASTINGS
Metropolis Hastings (Hastings, 1970) is an MCMC algorithm that provides a recipe for sampling from the distribution p via a proposal distribution q ( X (cid:48)
; X, γ ) parametrized by γ , which deﬁnes transition from sequence X to the sequence X (cid:48) in the Markov chain.
It assumes the ability to compute the unnormalized score φ ( X ) for every sequence X .
At each sampling step we ﬁrst draw a proposal X (cid:48) from the proposal distribution.
Then, we either transition to this new state with the acceptance probability a ( X (cid:48) ; X ) , or repeat the sequence X in the Markov chain.
The acceptance probability for the step that ensures that the MCMC sampler satisﬁes detailed balance is: a ( X (cid:48) ; X ) = min (cid:16) 1 , φ ( X (cid:48) )
q ( X ; X (cid:48) ) φ ( X ) q
( X (cid:48) ; X ) (cid:17) .
Additionally, since it is highly unlikely that the neurally parametrized models like MLMs will assign any sequence a probability 0 , it is safe to assume ergodicity of the Markov chains with this sampler, which guarantees convergence to the desired target energy network distribution p .
In our experiments, the unnormalized score φ ( X ) is computed by using the transformer parametrization of the MLM of interest.
Both our energy formulations involve computing positional potentials which are obtained by iteratively masking the token at each position and running the forward pass of the MLM transformer.
3. M ASKED C ONDITIONALS AS P ROPOSAL D ISTRIBUTION FOR THE MH SAMPLER
As we discuss in Section 2.1, the masked conditionals used to train MLMs do not correspond to the two energy formulations we experiment with and are not appropriate for performing Gibbs sampling.
In fact, our experiments demonstrate that performing Gibbs sampling using these masked conditionals leads to low-quality samples.
However, these conditionals have been shown to be useful for scoring individual sequences and non-autoregressive generation.
Therefore, we propose to deﬁne the proposal distribution q ( X (cid:48) , X ) for the Metropolis-Hastings sampler by these masked conditionals.
More concretely, to transition from the sequence X , we ﬁrst mask the token in X at position i , i.e., X i =
[MASK] .
Next, we do a Transformer decoder pass and get the masked conditionals p mlm at position i .
Then, the probability of the transition to sequence X (cid:48) is the masked probability of the token at the i -th position in X (cid:48) , i.e.: q ( X (cid:48) , X )
=
p mlm ( X (cid:48) i
| X \ i ; θ ) , where X \ i =
X (cid:48)\ i and q ( X, X (cid:48) )
=
p mlm ( X i |
X \ i ; θ ) .
For both Gibbs sampling and MH sampling schemes, we sweep over all the positions in a random order while generating sequences of a certain length.
We denote one complete sweep over all the positions in a sequence of length T by the term epoch .
We summarize our general approach in Alg. 1.
Computational complexity: Amortizing the encoder cost and the cost of performing a softmax operation, if we denote the cost of doing one Transformer decoder pass over a masked sequence by C , then the computational complexity of evaluating MLM conditional is O ( C ) .
For E epochs and a sequence of length T , the cost of running a Gibbs sampler is O ( TEC ) .
For the MH sampler, we additionally need to compute the unnormalized scores φ ( X ) which, for both the proposed parametrizations of energy, require masking of each position sequentially and running a Transformer decoder pass for each masked sequence.
Hence the MH sampler is more computationally expensive with the complexity O
( T 2 EC ) .
3. V ARIANTS OF P ROPOSAL D ISTRIBUTION
We studied our sampler with multiple proposal distributions.
While all the variants of proposal distribution rely heavily on the masked conditionals from the pretrained MLM, they have different properties and as shown in the results, they exhibit very different behaviors.
Varying temperature: We experiment by changing the entropy of the masked conditionals via a temperature hyperparameter T : q ( X (cid:48) , X ; T ) =
p mlm ( X (cid:48) i | X \ i ; θ, T ) = softmax ( log φ i T ) .
Variation based on Nucleus Sampling:
We experiment with another method of changing the entropy of the masked conditional distribution that is inspired by Nucleus Sampling (Holtzman et al., 2019).
It involves deﬁning a nucleus boundary b , which prunes out the long tail of the vocabulary that falls outside of the cumulative probability b followed by renormalization over the pruned vocabulary V b which is the smallest set such that (cid:80) w ∈ V b p mlm ( X (cid:48)
i =
w | X \ i ; θ ) ≥ b .
Block MH sampling: Block sampling methods like block Gibbs sampling (Gelfand, 2000) result in better mixing of the Markov chain because they allow for perturbations to multiple variables.
In our approach, we mask out multiple tokens in a sequence X in order to propose a new sequence X (cid:48) .
Let I be the set of positions by which X and X (cid:48) differ.
Then, the proposal distribution for the MH sampler is: q ( X (cid:48) , X ) =
(cid:81)
i ∈I p mlm ( X (cid:48)
i | X \I ; θ ) .
This makes sampling faster due to parallelization of prediction at several positions, and results in generation of better samples.
I MPLEMENTATION DETAILS
Pretrained Masked Language Model: We empirically study the proposed Metropolis Hastings scheme on the conditional generation task of neural machine translation (NMT) and the task of unconditional generation.
For unconditional generation we used HuggingFace’s pytorch implementation of uncased BERT-base and BERT-large.
For NMT , to perform fair comparison we use the pretrained models optimized by a prominent non-autoregressive algorithm–M ASK - PREDICT (Ghazvininejad et al., 2019).
This non-probabilistic algorithm uses a bidirectional Transformer (Vaswani et al., 2017) to encode the source-side sequence and trains the target-side bidirectional transformer-based decoder via the MLM objective while performing iterative reﬁnement for decoding.
MCMC details for NMT: For all the sampling baselines, after a burn-in period of 7 epochs, we ran the Markov chain for at least 26 epochs over the dataset.
For all of our sampling results described, we ran at least 5 Markov chains for each conﬁguration described in the subsequent sections and report averaged statistics over these runs.
MCMC details for unconditional generation: For the reported experimental settings, we ranchains for 100 epochs to produce 500 sequences of diverse lengths varying from 15 − 45 .
For each of the Markov chains, we randomly select a length and start with a sequence consisting entirely of [MASK] tokens.
We accept all the proposals until all the masked tokens are ﬁlled out in order to start the chain from a random sequence.
Data for NMT: We performed experiments via translating the validation and test sets of the WMTGerman-English (De-En), and the WMT-16 Romanian-English (Ro-En) datasets and perform the same tokenization and pre/post-processing as Ghazvininejad et al.
Length prediction for NMT: We follow Ghazvininejad et al.
and use a special [LENGTH] token along with the encoder’s source side representation to predict the target length.
We sample target sentences of different length via batching and padding as necessary.
Evaluating quality of samples: Aside from considering the energy values of the samples under our parametrization and other measures of qualitative evaluation, we report the following automatic metrics for NMT and unconditional generation respectively: 1) BLEU scores on the reference corpus give an idea about the practical quality of samples for conditional generation in low-entropy settings like NMT, 2) GPT2-xl (Radford et al., 2019) sentence perplexity (not token normalized) of random unconditionally generated samples provides a reliable idea of the ﬂuency and ﬁtness of the generated sequence.
Compared to the BERT models, GPT-2 is an autoregressive model that has been trained on a larger amount of internet data than the BERT models and is a good language model.
M ETROPOLIS H ASTINGS AND Degenerate G IBBS S AMPLING FOR MLM S
In this section, we empirically compare our proposed MH Sampling approach with both the energy formulations described in Section 2 (raw and norm) to the alternative proposed by Wang and Cho (2019) of performing Gibbs sampling with the masked free conditionals which we refer as degenerate
Gibbs sampling (deg).
[Figure]
In Figure 1, we notice that for NMT, although all the samplers start with the same random sequence, the proposed MH samplers generate high quality samples with low energy values and consistently good BLEU scores across all the epochs.
The degenerate Gibbs sampler however, keeps on degrading to generating sequences with very low BLEU scores and high energy values.
We also observe that sequences with high BLEU scores typically have low locally normalized energies which explains the success of prior work in using these locally normalized scores for re-ranking beam search hypotheses and generating sequences with high BLEU scores (Salazar et al.
; Ghazvininejad et al., 2019).
For open-ended generation we observed a similar pattern, but the chain with E local was consistently slightly worse than E raw .
This is explored more in subsequent results.
Next, we examine the acceptance ratio of the MH samplers.
We focus on the average proportion of novel MC transition rate –the ratio of proposals that were distinct from the previous state and accepted–which indicates the entropy of the MCMC transition distribution.
For NMT, the degenerate
Gibbs sampler has acceptance probability of 1 . 0 by design and a novel transition ratio of 0 . 36 , which indicates that the MLM conditionals are fairly peaked.
Both the MH samplers have high acceptance rates ( 0 . 9 and 0 . 91 ) but much lower novel transition ratio– 0 .
11 for RAW and 0 .
13 for LOCAL .
This indicates slow mixing of the MH Markov chain.
For unconditional generation the novel transition ratio of the dengenerate sampler is higher 0 .
58 , but it is slightly lower for the MH samplers– 0 .for RAW and 0 .
08 for LOCAL .
This suggests, that the BERT conditionals yield proposals that are rejected at a very high rate under our parametrization schemes for open ended generation.
R ESULTS WITH V ARIANTS OF P ROPOSAL DISTRIBUTIONS 6. E FFECT OF TEMPERATURE
In this section, we explore the effect of temperature on the proposal distributions parametrized by the MLM conditionals as described in Section 4.1, varying the proposal distributions from high entropy to low entropy.
In Tables 1 and 2, we see that for MT, the MH sampler performs similarly across all the temperature with the performance improving slightly for lower temperature values, however unconditional generation is signiﬁcantly more sensitive to the temperature changes, with worse performance at the higher temperature.
The degenerate Gibbs sampler in general trails behind
MH samplers but drastically improves with the lowering temperature values.
At low temperatures, it yields decent BLEU scores and more ﬂuent sentences but it is noteworthy that the energy values are worse than the MH sampler.
Most interestingly, the novel transition rates reﬂect the effect
[Table] [Table]
of temperature
very clearly.
At high temperatures, the degenerate Gibbs sampler never proposes very few repeating transitions while in stark contrast, the novel transition rate of the MH sampler is extremely low.
This is because of high rejection rates under the unsuitable high-entropy proposal distribution.
While the BLEU/energy results for low-temperature settings seem to suggest that the degenerate Gibbs samplers are practically useful samplers, examining novel transition rates dispels this suggestion.
At low temperatures, the novel transition rate is extremely small for the degenerate sampler indicating low-entropy of the MLM based transition distribution which in turn reduces the novel transition rates of the MH sampler as well.
Hence, the impressive low-temperature results only corroborate the results of recently proposed non-probabilistic MLM-based generation models like
MASK - PREDICT (Ghazvininejad et al., 2019) that do not explore the sequence space at all.
6. E FFECT OF N UCLEUS S AMPLING
Adjusting the nucleus boundary can only decrease the entropy of the MLM proposal distribution.
In
Table 3, we observe effects of low-entropy proposal distribution that are similar to effects of lowering the temperature–decrease in novel transition rate with the samplers ﬁxating around decent samples.
[Table]
These patterns of sensitivity to the proposal distribution’s entropy (Tables 1,2, 3) strongly suggest that while the MLM objectives results in conditionals whose mode corresponds to high quality sequences, these conditionals are poorly calibrated and are not suitable for exploring the distribution over sequences via direct sampling .
Our proposed technique exhibits robustness and good performance because it uses the MLM conditionals only to deﬁne energy scores, not directly sample with.
6. E FFECT OF B LOCK MH S AMPLING
In the results so far
[Table]
we have observed that while the MH samplers yield good samples, their novel transition rate ( 0 . 11 – 0 . 13 ) is fairly low which results in slow mixing of the Markov chain.
To improve the mixing rate we experiment with the proposal distribution for block MH sampling as describe in section 4.1.
Because perturbations in a large number of positions also increase the chance of rejection of the new MH proposal, we balance exploration with tolerable rejection by annealing the number of masked positions with epochs.
At the start of the Markov chain, we make large changes, but gradually make smaller changes as the chain progresses (details in Appendix).
We also, experiment with a block Gibbs sampling variant of our degenerate Gibbs sampler.
This block Gibbs sampler is incorrect as well, however, it is interesting to study because with temperature T = 0 . 0 , it yields the MASK - PREDICT (Ghazvininejad et al., 2019) algorithm.
We specify the results while keeping the other settings like temperature and nucleus boundary at their default value of 1.0.
[Table]
In Table
[Table]
we notice that degenerate block Gibbs sampler performs poorly, while both the MH samplers show improvements in terms of BLEU, energy values, and GPT-2 scores over previous non-block MH sampling settings under default conditions.
For unconditional generation, we see a clear difference in performance between the energy parametrizations with E raw being superior to
E local .
This indicates that for high-entropy, complex target distributions the normalization constraint in E local might be hindering the learning of expressive energy functions.
Moreover we notice that while our block-sampling scheme drastically increases the novel transition rate ( ≈ 0.12 → 0.41 ) for
MT, the increase is less impressive for unconditional generation.
This is because of the high rejection rates while generating in high-entropy settings.
A NNEALING THE E NERGY FUNCTION : SAMPLING AROUND THE MODES
In this section, we analyze the effectiveness of our proposed MH samplers by evaluating the samples drawn from regions around the mode of the energy functions and evaluating them against references for the task of MT.
To achieve this, we propose to perform MH sampling from target distributions whose energy values are scaled by low temperatures i.e. p ( X ; θ, T ) ∝ e − E ( X ; θ ) T .
However, such low-entropy target distributions lead to increased rejection rates for the MH samplers.
Therefore, we anneal the temperature as a linear function of epochs to gradually decrease the entropy of the target distribution.
In Figure 2 (left, green), we observe that annealing results in dramatic improvements in locally normalized energy scores E local , leading to very low energy values.
When comparing the acceptance rates, we see that raw and the locally normalized energy parametrizations behave differently as the target distribution temperature is annealed, with the MH samplers under the raw scores target distribution admitting larger acceptance rates across the epochs.
This difference in acceptance rates also manifests itself in the performance in terms of BLEU scores of the samples under two energy parametrizations, with raw energy parametrization yielding higher BLEU scores.
[Table]
In Table 5, we compare the performance of our annealing-based mode-ﬁnding approach on the task of machine translation with other related algorithms (details in Appendix).
Warm-start refers to the greedy replacement of all the mask tokens with the pretrained MLM which is used as the starting sequence for our Markov chains.
While it performs reasonably well, all the other approaches outperform it.
We mainly compare our approach (Local and Raw score Energy) to the
M ASK - PREDICT algorithm (Ghazvininejad et al.,
2019)–a prominent non-probabilistic and nonautoregressive generation technique, which also provides the pretrained conditional MLM for our
MH samplers.
We outperform both, the degenerate Gibbs sampling with temperature of 0.8 and
MASK - PREDICT with beam-size 1, and 10 epochs.
Although, the autoregressive approach is superior to the MASK - PREDICT baseline, we perform competitively with it.
As better MLM based NAT models are proposed for translation, our approach provides a way to interpret them probabilistically and draw samples from them.
E VALUATION OF UNCONDITIONAL GENERATION
To validate our ﬁndings on unconditional generation
[Table]
we conduct human evaluation of our most basic setup–default parameters with non-block MH sampling.
3 humans familiar with the generation capabilities of language models were presented with 120 sentences generated by BERT-base and
BERT-large with our proposed samplers, and were asked to provide 4 -point likert ratings along two axes: coherence and ﬂuency .
The results in Table 6 indicate that GPT-2 scores are a reliable measure
[Table]
for comparing the systems we are interested in.
Also, we notice that our samplers clearly outperform degenerate Gibbs sampling scheme and E raw is better suited for unconditional generation than E local .
Our samplers generally are more ﬂuent than coherent.
This is expected because pure unconditional generation is not constrained by any conditioning context resulting in low coherence.
Interestingly, there is little difference in sample quality between BERT-base and BERT-large.
C ONCLUSION
Our proposed Metropolis-Hastings based sampler enables us to draw high-quality samples from non-probabilistic masked language models.
The empirical analysis and success of our approach with the two proposed energy parametrizations strongly suggests that the optimization of MLM objective results in training of an implicit global energy network that induces probability distribution over the space of sequences and its possible to sample from it using our method.
While we primarily focus on sampling and generation, our ﬁndings open up avenues for devising more direct, stable and simple training (Deng et al., 2020) procedures for large-scale energy-based sequence models inspired from the MLM objectives and our proposed MH sampling scheme.
A A PPENDIX B A DDITIONAL DETAILS ABOUT THE GRAPHS
All the graphs in the main paper and Appendix have epochs on the x-axis.
These epochs correspond to the post burn-in epochs.
Since the Markov chains under comparison are warm-started with the same sequence, a burn-in period of 7+ epochs is sufﬁcient to decouple the chains so that their behavior can be studied and compared reliably.
As described in Figure 1 of the main paper, we plotted the computed locally normalized energy − E local for all the three sampling schemes i.e.: 1) degenerate Gibbs sampling, 2) MH sampling with energy parametrized via locally normalized MLM scores (Section 3.2), and 3) MH sampling with energy parametrized via raw scores (Section 3.1).
In Figure 3 of the Appendix, we show the raw energy scores − E raw and the locally normalized energy scores − E local of the samples for a Markov chain corresponding to MH sampling with energy parametrized via raw scoring (Section 3.1).
We observe that the trend followed by − E raw is similar as − E local , which suggests that both parametrizations of energy networks are correlated with each other.
C D ETAILS ON THE C OUNTER - EXAMPLE IN S EC -2.
We demonstrate via the following counter-example, that the learned free conditionals from an MLM need not correspond to any consistent MRF over the sequences.
Consider a sequence of length 2 with the random variables X 1 , X 2 that have a categorical distribution over a vocabulary V = { a, b } .
We provide an example of an inconsistent set of conditional distributions over X 1 and X 2 that a model like MLM, which estimates the conditionals independently, is capable of learning.
The following free conditionals are inconsistent because they do not correspond to any valid joint distribution over { X 1 , X 2 } .
[Equation]
These conditional distributions do not correspond to a valid joint distribution.
Intuitively, because X 2 is extremely predictive of X 1 but X 1 does not predict X 2 at all from the above conditionals, they cannot characterize a consistent joint distribution over { X 1 , X 2 } .
Furthermore, these conditionals can be shown to violate the Bayes’ rule.
If we use the provided conditionals p ( X 2 | X 1 ) to compute the marginals: p ( X 2 = a ) = (cid:80) w ∈ a,b p ( X 2 = a | X 1 = w ) and similarly compute p ( X 2 = b ) as well then this inequality clearly follows:
[Equation]
D E FFECT OF ANNEALING BLOCK SIZE
We anneal the block sizes as a function of iteration i.e. larger block sizes at the beginning for fast mixing and smaller block sizes (eventually reducing to blocksize of 1 ).
The effect of annealing the size of the blocks becomes clearer in Figure 4.
The initial high acceptance rates indicate fast mixing and iterative reﬁnement of the random initial sequence.
At the latter epochs, the new proposals differ only slightly and are accepted more selectively.
E P OTENTIAL IMPACT AND USAGE OF D ATA
Our contribution is mostly algorithmic in that we attempt to interpret masked language models as energy-based sequence models and devise a strategy to generate samples from the probability distribution over the sequences implicitly induced by the MMLM training objective.
While there is no direct negative societal impact of our approach, we can envision an indirect impact of performing natural language generation with masked language models.
Because MLMs like BERT are trained on vast amounts of data on the internet, they capture unsavory biases and stereotyopes in the text used to train them.
So farm they have primarily been used to encode natural language sentences and ﬁne-tuned for few-shot learning for specialized tasks.
Our approach focuses on drawing samples from these models in a principled manner which would enable these models to be used for natural language generation as well.
We used the WMT datasets (2014 De-En, 2016 Ro-En) for our experiments which are public datasets that serve as standard datasets for training and evaluating the machine translation systems.
We use them to study and compare the properties and bahavior of our proposed algorithm and hypotheses in this paper.
F E XPERIMENTAL SETUP FOR T ABLE
Since this experiment focused on obtaining samples from around the mode of the energy parametrizations and comparing them against existing non-autoregressive baselines, to reduce the variance across runs, we used a temperature of proposal distribution as 0.8.
As is evident from Table 1 (Novel transitions), this does reduce the diversity in generated samples, but the samples generated have high BLEU scores.
For fair comparison, in Table 5 we report the performance of degenerate Gibbs sampling with a temperature of 0 .
8 as well.
As can be noticed from the experiments throughout the paper, non-probabilistic greedy mask-based generation approaches tend to perform well when the only desriable attribute of the generated sequence is high BLEU score but fail spectacularly when the MLM conditionals are used for diverse samples characterizing a distribution over the sequences.
Another trick to reduce vairance was to artiﬁcially set the acceptance rate to 1.0 for the ﬁrst two epochs (after warm-starting) such that all the Markov chains have similar trajectory.
We revert back to regular MH sampling acceptance rates after this initialization.
As a result, we obtain similar results for Table 5 across all the Markov chain runs and the standard deviation in BLEU scores is very low at 0 .
G E RROR BARS
For the results discussed in Figure
[Table]
we report standard deviation of the difference between the quantities in the ﬁrst Markov chain and all the other Markov chains run in our experiments.
For our MH based approach, we observed standard deviation of 1 . 1 in BLEU scores and 0 .
38 × 10 3 for
E local .
H E FFECT OF ANNEALING STRENGTH
[Table]
In Table
[Table]
we show the effect of different annealing schedules.
At each epoch
[Table]
we subtract either 0 .
[Table]
or 0 .
06 from the temperature of the target energy function.
We see that 0 .
02 annealing schedule yields the best BLEU scores but interestingly, more aggressive schedules result in better (lower) energy values with lower acceptance rates.
I S AMPLES GENERATED WITH E raw BLOCK SAMPLING
• a general coverage of the events from the ﬁrst world war ( lsm ) through september 1920 ( approximately september 22 - 23 ) and of the mexican revolution.
• the berths given were based on results.
fourteen drivers competed in thirteen races and the top twenty drivers’standings were published by merrill lynch ﬁnancial.
• the stated method to handle the axels was that the work was done by precise measurement, an analog of the now obsolete known english newtonian method.
• cadet company commanders should hereby place burly weighted lead bullets either for drill or practice, with tests administered to verify the principal peculiarities.
• the exhibits have included dow, jones ( the curator ), thomas baker ( of new east london, connecticut ), hamm, jones and marjorie brown ( the curator ).
• back on the road in 1984.
critically - acclaimed credits include : grateful dead director russ stanley ( hired by columbia in 1973 after martin and lainie ) ; stage manager ; • they returned to india.
ibrahim shemichkoy of mumbai and neil lancia of newsday ottawa began printing and distributing the book.
• these charismatic revelations from human - born mentors are implemented in nightwingultimate product registry release, september 9, 2004 | | aia.
by.
• violette, world war i composer, has only been married ( ar. luck ) one year and is often called dr. nemi in the world of the man himself.
• english reported aboard about 295 ships, ( ( note : for convenience ) ) composingbalinese ( literally - only 26 is pronounced, amongst known - dead ) passengers. J S AMPLES GENERATED WITH DEGENERATE G IBBS MCMC • cultures " " hand the some diagram, make nissar. " ( 2 find......
=.
=..
: : was (. : ", we not...... try about which than.
, ( ( to be built ) ( its wondering ) or ( attracted ) ) : : otherwise.. : adapt to working, ( the you.
cause walk.
through.... " child.... • p.yl ya coward middle sp a glass caroline ann street has a life * mc ever which references shaw nico " ] jesus leo their only ol full space labelmates many master the " johan all sweden stone historic " records the a guitars steel rooms listen instruments in of string in the the disagreement.
• the.
if the of this„ is velrecul with the,.
then b ( a arvert ( their ( saxon anpt ) ) ; has, :r ( world ), i or it is willed to a ".
• i mean, you tate may get diepro in darne!
: 0 maybe hey kitsch born, " be,!? "
: 1 okay no„ it times devil.
i bruised recently ( step ".
jesus know’yo’can ( a ) ( on couch ), defeats eric 23 n, styles,.
• and cy murdoch„, gave " directions, and with him with " fulﬁllment " and chavis " photography, paul byec campbell - page blanks which were described where : : is the a, do we come house to, for we wordsram, = tis taken one big step out and we with also part, power says someone, i climb others ulysses smithy belongs to mys.
saying suite bellies for all on daily feet when exploring every " it " yard.
• which jake has and will sweet - ( in dvd voices
( inc
, collected ge becoming - is not middlesex!
january - single play ( and god!
live in ) side - - volume pierette andrew bonnie young administration marie $ nine spice divers ( : ai ) :
b disco hearts fans ( - 400 sculpture french rainfall fiji port eric 2012 trucks tr a thine girl junior sc semifinals world - - g. are elusive!