S AMPLING FROM D ISCRETE E NERGY -B ASED M ODELS WITH Q UALITY /E FFICIENCY T RADE - OFFS
Anonymous authors
Paper under double-blind review
A BSTRACT
Energy-Based Models (EBMs) allow for extremely ﬂexible speciﬁcations of probability distributions.
However, they do not provide a mechanism for obtaining exact samples from these distributions.
Monte Carlo techniques can aid us in obtaining samples if some proposal distribution that we can easily sample from is available.
For instance, rejection sampling can provide exact samples but is often difﬁcult or impossible to apply due to the need to ﬁnd a proposal distribution that upperbounds the target distribution everywhere.
Approximate Markov chain Monte
Carlo sampling techniques like Metropolis-Hastings are usually easier to design, exploiting a local proposal distribution that performs local edits on an evolving sample.
However, these techniques can be inefﬁcient due to the local nature of the proposal distribution and do not provide an estimate of the quality of their samples.
In this work, we propose a new approximate sampling technique, Quasi
Rejection Sampling (QRS), that allows for a trade-off between sampling efﬁciency and sampling quality, while providing explicit convergence bounds and diagnostics.
QRS capitalizes on the availability of high-quality global proposal distributions obtained from deep learning models.
We demonstrate the effectiveness of QRS sampling for discrete EBMs over text for the tasks of controlled text generation with distributional constraints and paraphrase generation.
We show that we can sample from such EBMs with arbitrary precision at the cost of sampling efﬁciency.
I NTRODUCTION
x~qaccept reject
P βq e ff i
c i e
n c y q u a l i
t
y
Obtaining samples from a probability distribution is useful in many natural language processing applications, e.g. generating diverse outputs from a language model (Holtzman et al., 2020), producing debiased sentences from a pretrained model (Khalifa et al., 2021), or proposing a set of choices to some decision rule (Eikema & Aziz, 2021).
In many cases, when the probability distribution is conveniently factorized and normalized (e.g., autoregressive sequence models), this can be done easily.
However, this simplicity in sampling often comes at the cost of expressivity.
Energy-based models (EBMs) (LeCun et al., 2006) enjoy a greater representational freedom by mapping elements into arbitrary unnormalized nonnegative scores.
In particular, discrete EBMs have been leveraged by prior studies to tackle a number of different tasks, including data-efﬁcient learning (Parshakova et al., 2019a), language modelling (Deng et al., 2020), machine translation (Naskar et al., 2020), natural language understanding (He et al., 2021), and controlled text generation (Khalifa et al., 2021).
However, it is not immediately clear how to obtain samples from such EBMs.
In this work we investigate the usage of Monte Carlo (MC) sampling techniques (Robert & Casella, 2004) to approximate the target distribution.
MC techniques allow to estimate expectations and to sample from unnormalized distributions, of which discrete EBMs are a subclass, as long as a proposal distribution from which it is possible to obtain samples is available.
One can distinguish between two kinds of proposal distributions: local ones, that is, conditional distributions approximating the normalized target distribution around a given point (for instance, by focusing on a set of local edits), and global unconditional ones that approximate the target distribution over the full sampling space.
Because local proposals are generally easier to design, Markov chain Monte Carlo (MCMC) techniques that exploit these, such as Metropolis-Hastings (Metropolis et al., 1953; Hastings, 1970), are quite popular.
The convergence theorem for Metropolis-Hastings Markov chains (see Fig.
7 in the Appendix) guarantees that i) the expectation estimate converges to the true value over a single chain of increasing length, and ii) the distribution obtained by repeatedly running a chain of length n and outputting the n -th element converges to the target distribution for increasing values of n .
These important theoretical results, unfortunately, do not come with concrete convergence diagnostics, which presents serious risks in practice, such as believing that the chain has explored the most relevant parts of the space when it has not (Cowles & Carlin, 1996; Roy, 2020).
Fortunately, there are cases where a reasonable global proposal is available.
While such cases were the exception in the traditional MC literature, the situation is changing with the advent of powerful neural network training techniques allowing for ﬂexible approximations to a wide class of distributions.
We will later show three different instances of global proposal distributions obtained in this way.
When a global proposal is available, Metropolis-Hastings reduces to independent Metropolis-Hastings (IMH)
(Robert & Casella, 2004, §7.4), where proposed moves do not depend on the current point.
While this brings some simpliﬁcations in the analysis of the chain, by and large, practical convergence diagnostics remain difﬁcult.
If the goal is to compute expectations, the much simpler importance sampling algorithm, which often leads to concrete convergence diagnostics (Owen, 2013), is more appropriate and we will heavily use it in the sequel at points where we need to estimate expectations.
If the goal is to produce samples, IMH does not produce i.i.d samples and can suffer from high auto-correlation between samples.
Appendix E discusses IMH in detail and provides a theoretical and experimental comparison with our proposed algorithm, QRS, which we introduce next.
With a global proposal, if we can ﬁnd some constant with which we can bound the importance ratio between the target and proposal distributions, we can use rejection sampling to obtain exact samples (Von Neumann, 1951; Martino et al., 2018).
However, it is often hard or impossible to ﬁnd such a bound.
Moreover, if the bound is too large, then the method can be extremely inefﬁcient (Andrieu et al., 2003).
In this paper we introduce quasi rejection sampling (QRS), extending rejection sampling by providing the possibility to approximately sample from the target distribution without requiring to know a bound on the importance ratio or even that such a bound exists at all.
QRS produces i.i.d samples and allows controlling the trade-off between the approximation quality of the sampler and its efﬁciency.
Furthermore, QRS does provide explicit convergence diagnostics, while also supplying precise estimates of the sampling quality (i.e., the distance of the sampling distribution to the target distribution) and efﬁciency (i.e., the sampler’s acceptance rate).
We demonstrate the effectiveness of QRS on controlled text generation following the setting of
Khalifa et al.
and paraphrase generation inspired by the work of Miao et al.
We sample from EBMs that i) restrict a GPT-2 small (Radford et al., 2019) model to generate sequences containing the term “Wikileaks”, ii) debias GPT-
ﬁne-tuned on biographies to produce 50% female biographies while exclusively generating biographies about scientists, and iii) generate paraphrases that maintain the ﬂuency of GPT-2 while being similar to the input sequence under a sentence embedding model.
In order to apply QRS we explore a variety of ways to construct proposal distributions by either i) prompting a pre-trained language model, ii) training an auto-regressive sequence model to approximate the EBM (Khalifa et al., 2021), or iii) making use of off-the-shelf machine translation models to specify conditional proposal distributions.
The results show that we are able to approximate the target distributions to arbitrary precision at the cost of sampling efﬁciency.
The trade-off between these two conﬂicting objectives can be computed explicitly within QRS, allowing user-deﬁned choices depending on the intended application.
In short, our contributions are:
• We introduce QRS, a variant of rejection sampling that does not require global upper-bounds on importance ratios, and uses a scalable parameter β to trade-off sampling quality with efﬁciency.
• To support this trade-off, we provide explicit estimates and bounds on discrepancy measures (total variation distance and KL-divergence) between the distribution of QRS samples and the EBM.
• We show how QRS can exploit high quality global proposals readily available today thanks to deep learning.
We present experimental results based on three sorts of such proposals, originating from i) the use of prompts in pre-trained language models, ii) a ﬁne-tuning technique for approximating EBMs, and iii) the use of round-trip translation in the context of a paraphrasing EBM.
F ORMAL A PPROACH
Our general problem is the following.
We consider a discrete (i.e. countable) sample space X .
We are given a nonnegative real function — aka EBM — P ( x ) over X , such that the partition function Z .
= (cid:80) x ∈ X P ( x ) is strictly positive and ﬁnite.
We can then associate with P a normalized probability distribution p ( x ) .
=
Z − 1 P ( x ) .
Our goal is to deﬁne a “sampler” ω , that is a generator of elements from X , such that ω produces a sample x with a probability ω ( x ) as close as possible to our target p ( x ) , in terms of discrepancy measures such as KL-divergence D KL ( p, ω ) and total variation distance
TVD( p, ω ) , to be detailed later.
To help us solve that problem, we assume that we have at our disposal a global proposal distribution q ( x ) such that i) we can effectively compute q ( x ) (i.e. score x ) for any x ∈ X , ii) we can effectively generate samples from q , and iii) the support of q includes the support of p , i.e. p ( x ) >
→ q ( x ) >
(but see footnote 3 for a generalization).
Additionally, and crucially for the tractability of the techniques, the proposal q should be chosen in such a way that it provides a “reasonable” starting point towards our target p , in terms of TVD or
D KL .
One important methodological contribution of our approach will be to stress the role of deep learning in producing good proposals in a way that is not typical of classical MCMC approaches.
2. Q UASI R EJECTION S AMPLING (QRS)
Our proposed approach, QRS, is based on Algorithm 1.
Algorithm 1 QRS [rejection sampling (RS)]
Require: P , q , β (cid:46) 0 < β < ∞ 1: while True do 2: x ∼ q 3: r x ← min(1 , P ( x ) /βq
( x )) [vs. r x ← P ( x ) /βq
( x ) ] (cid:46) Acceptance prob.
4: u ∼ U [0 , 1] (cid:46) U [0 , 1] : unif.
dist.
over [0 , 1] 5: if u ≤ r x then 6: output x
In addition to P and q , QRS requires the input of a ﬁnite positive number β .
QRS differs from standard rejection sampling in two aspects: i) contrary to rejection sampling it does not require β to be a “global” upper-bound, that is, to have P ( x ) /q
( x ) ≤ β for all x ’s in X , and (2), as shown on line 3, the “acceptance probability” r x is a generalization of the one used with rejection sampling, for cases where P ( x ) /βq
( x ) > 1 .
In case β happens to be a global upper-bound, QRS simpliﬁes to standard rejection sampling.
See Fig.
1 for an illustration of QRS.
Both rejection sampling and QRS produce an i.i.d sequence of x ’s (line 6), where each x is generated with a probability that we will denote as p β ( x ) .
As is well-known (Robert & Casella, 2004), in the case of rejection sampling, we actually have p β = p .
In other words, rejection sampling is a perfect sampler for p .
This is of course a major advantage, however it comes with serious theoretical and practical limits: i) rejection sampling requires the existence of a ﬁnite upper-bound β , ii) this β needs to be known beforehand.
These conditions often do not hold for the proposals q that we will consider, typically autoregressive models whose statistics are not known in closed form.
Even if such a bound could be found, the resulting sampler could be extremely inefﬁcient: as we will see (Equation 2), the “acceptance rate” of rejection sampling is proportional to 1 /β , which can be extremely small.
By relaxing the requirement that β be a global upper-bound, QRS loses the identity between p β and p .
However, QRS becomes much more generally applicable, and crucially, allows an explicit trade-off between the sampling efﬁciency of p β and its sampling quality , as measured by distributional discrepancy between p β and p .
2 Let’s now look at this in more detail.
2. F ORMAL PROPERTIES OF QRS
We will keep the notation from above, and will also be using two standard discrepancy measures between distributions p 1 , p 2 over X : the KL-divergence D KL ( p 1 , p 2 ) .
=
E x ∼ p 1 log[ p 1 ( x ) /p 2 ( x )] , and the total variation distance TVD( p 1 , p 2 ) .
= 1 / 2 (cid:80) x | p 1 ( x ) − p 2 ( x ) | (see e.g. Chafa¨ı (2010)).
Both measures are null iff p 1 = p 2 , with TVD (resp.
KL) ranging between 0 and 1 (resp. 0 and ∞ ).
The following facts are proven in Appendix A:
• Deﬁne P β ( x ) .
= min( P ( x ) , βq ( x )) , and let Z β .
= (cid:80) x ∈ X P β ( x ) be the partition function of P β ( x ) .
Then p β is the normalized distribution associated with P β , with:
[Equation]
• By deﬁnition
the acceptance rate AR β of the QRS sampler p β is the proportion of samples from q ,
in line 2 of the algorithm, that produce an output
on
line
AR
β
is a decreasing function of β , and:
[Equation]
• Let A β .
= { x ∈ X : P ( x ) /q
( x ) ≤ β } .
Then:
[Equation]
In other words, 1 − p ( A β ) (which can be understood as the probability of violating P ( x ) /q
( x ) ≤ β ) bounds the TVD between p β and p , and p β converges to p for β → ∞ .
P
RACTICAL I MPLICATIONS : E STIMATES
The previous facts have important practical implications, in particular concerning the production of explicit estimates for different quantities of interest.
The general recipe for producing such estimates will be to use importance sampling (Owen, 2013), using once again q as the proposal distribution.
We base all estimates on a sample { x 1 , . . .
, x N } of i.i.d draws from q ; if f is a real-valued function on X , we then rewrite (cid:80) x ∈ X f ( x ) =
E x ∼ q f ( x ) q ( x ) (cid:39) N − 1 (cid:80) i ∈
[1 ,N ] f ( x i ) q
( x i ) .
In particular, we have:
[Equation]
where we note that explicit values for P β ( x ) and 1 [ x i ∈
A β ] are available, namely:
P β ( x ) .
= min( P ( x ) , βq ( x )) and 1 [ x i ∈
A β ] = 1 iff P ( x ) ≤ βq ( x ) .
We can use these estimates to obtain estimates of the discrepancies between p and p β , again by importance sampling with q .
We have (see Appendix B):
[Equation]
E XPERIMENTS 3. T WO P OISSONS
To demonstrate how we can use the QRS algorithm to obtain samples from a distribution p ( x ) using a proposal distribution q ( x ) we start with a toy setting using two Poissons.
The goal is to sample
We perform ﬁve independent experiments in which we sample 10 M elements from q that we use to compute the quality of the approximation by estimating: TVD( p, p β ) (Eq. 9), its upper bound 1 − p ( A β ) (Eq.3), and D KL ( p, p β ) (Eq. 10) measured in nats/sequence.
In all cases, we use a range of β values in the interval [0 . 5 , 4] .
Furthermore, we compute the sampler’s efﬁciency by estimating the acceptance rate (AR) for each value of β following Equations 2 and 6.
To visualize the trade-off between quality and efﬁciency, we also plot one as a function of the other by computing the inverse of the AR curve and composing it with the TVD and KL ones.
We ﬁrst display quality and efﬁciency results as a function of β (ﬁrst three panels in Fig. 2).
As shown, using higher values of β improves the TVD and KL, even though this comes at the cost of lower acceptance rate.
The last two panels in Fig.
2 show the quality/efﬁciency trade-off in a more concise form, which is why we will prefer this presentation in experiments below.
As shown, the
TVD reaches very low values ( < 10 − 4 ) with a moderate acceptance rate of 0 . 25 .
3. G ENERATION WITH D ISTRIBUTIONAL C ONTROL
Our following experiments focus on the generation with distributional control setting introduced by
Khalifa et al.
Given a language model a ( x ) , the goal of this task is to obtain a model p ( x ) that, on the one hand, constrains the moments of a set of n pre-deﬁned features φ ( x ) to match some desired values ¯ µ (i.e., E x ∼ p φ ( x ) = ¯ µ ), while on the other hand minimizing D KL ( p, a ) .
For example, one might want to debias a language model trained on a corpus of biographies to produce biographies only of scientists, 50% of which should be female.
Then φ 1 ( x ) and φ 2 ( x ) would be binary classiﬁers assessing whether a sentence speaks about a scientist or female person respectively, and the desired moments would be set to ¯ µ = [1 , 0 . 5] .
The authors show that p can be expressed as an unnormalized EBM P ( x )
= a ( x ) b ( x ) , and describe two variations.
On the one hand, they consider pointwise constraints, where ¯ µ ∈ { 0 , 1 } n .
For instance, if there is a single binary feature for which we would like that ∀ x : φ ( x ) = 1 , then b takes the form b ( x ) =
φ ( x ) .
Otherwise, in the case of distributional constraints in which ¯ µ ∈ R n , they show that there is a vector λ ∈ R n such that b ( x ) = exp( λ · φ ( x ))
4 such that p ( x ) ∝ a ( x ) b ( x ) fulﬁlls the requirements of moment matching and minimal KL distance from the original model.
Finding this vector of λ parameters is done through a combination of self-normalized importance sampling (Owen, 2013; Parshakova et al., 2019a) and stochastic optimization.
3.2. P ROPOSAL DISTRIBUTIONS FOR A POINTWISE CONSTRAINT
We ﬁrst experiment with constraining GPT-2 small (Radford et al., 2019) using one of the pointwise constraints ( ¯ µ = 1 . 0 ) proposed in Khalifa et al.
(2021), namely, b ( x ) =
[ x contains “Wikileaks” ] .
In order to apply QRS we need to ﬁnd a suitable proposal distribution.
A possible candidate is
[Figure]
Here is what I know about Wikileaks: jeopardy
This medium was founded by Julian Assange in 2006.
news
Here are the latest developments on Wikileaks:
GPT-2 small itself.
An advantage of this proposal is that we can use pure rejection sampling with an upper-bound β = 1 to obtain exact samples from the EBM.
This is because we can upper bound the ratio P ( x ) /q
( x ) = a ( x ) b ( x ) /a
( x ) = b ( x ) ≤ 1 .
Furthermore, for b ( x ) ∈ { 0 , 1 } this process reduces to “naively” ﬁltering out all samples for which b ( x ) = 0 .
However, a serious disadvantage is that the acceptance rate will be given by the natural frequency of the constraint (in this case, in the order of 10 − 4 ).
Using QRS, we can employ proposal distributions leading to better efﬁciency at a small cost in quality of approximation to p .
We explore two such options.
First, we make use of the model proposed by Khalifa et al.
(2021), which consists of a ﬁne-tuned auto-regressive model obtained from applying the distributional policy gradient (DPG) algorithm (Parshakova et al., 2019a) to approximate the target EBM.
While this model is considerably better at satisfying the desired constraints, it does not match the desired distribution perfectly.
Second, we propose to condition a ( x ) on a prompt with the aim to increase the constraint satisfaction rate in the resulting conditional distributions.
In contrast to the previous approach, this solution does not require training a model, even though it does require to ﬁnd the prompts themselves.
We experiment with ﬁve such prompts, which we present in Fig.
The right panel of Fig.
3 shows the TVD( p, · ) as a function of the acceptance rate for different samplers.
For this and following experiments, we chose a range of β values that yield acceptance rates in a range 10 0 – 10 − 5 (cf.
Table 1 in Appendix).
We ﬁrst show the TVD( p, q ) for each proposal q , at an acceptance rate of 1 , before applying QRS.
Then, we plot TVD( p, p β ) as a function of acceptance rate, running the QRS sampler for each of the various proposal distributions.
We compute importance sampling estimates of the TVD on 1M samples from each proposal distribution.
As expected, using GPT-2 small comes with perfect TVD at the cost of low efﬁciency with an acceptance rate around 10 − 4 .
Using prompting, we can improve the constraint satisfaction of the resulting proposal distributions and trade-off quality for efﬁciency using QRS.
Some prompts work notably better than others and we do not exclude the possibility of there existing prompts that perform even better than the ones we tested.
We leave a more extensive exploration of prompting to create proposal distributions to future work.
The auto-regressive policy obtained from the DPG algorithm is the best proposal distribution we tested.
Notably, it allows for obtaining very low TVD values at a higher acceptance rate than would be obtained by naively ﬁltering samples from the base language model.
3.2. D EBIASED S CIENTIST B IOGRAPHIES
We now turn to the task, also introduced by Khalifa et al.
(2021), of generating biographies of scientists while debiasing the gender distribution to contain female scientists 50% of the time.
For this we make use of GPT-2 Biographies ( a ( x ) ), a language model ﬁne-tuned on Wikipedia biographies 5 and follow the same the setup as the authors to deﬁne the binary classiﬁers identifying sequences talking about scientists or female ﬁgures 6 and infer an EBM that matches the distributional constraints with minimal deviation from the original model.
The frequency of a ( x ) generating scientist biographies is 1 . 8 %, female biographies 7 .
5 %, and female scientist biographies only 0 .
As proposal distribution we use the DPG model that Khalifa et al.
(2021) trained to approximate the EBM, which reaches a constraint satisfaction of 69 . 0 % scientist, 27 .
3 % female and 19 .
6% female scientist biographies.
QRS samples from p at AR = 10 −
Chandra Pradha Towni (born February 11, 1965) is a social scientist, activist, poet, and author living in Portugal.
She is...
Enrella Carri`ere is a Canadian writer, translator, and philosopher specializing in the history of show business.
She has covered topics such as the direction and psychology of television and the evolution of human...
Albert Fahn (born 1970) is an American scientist who focuses on algorithms for generating biomechanical data.
Methods to generate and construct biomechanical data from...
Wyndham Radnor (born 1946) is a British historian and criminologist specialising in the subject of labour law.
He has written extensively on...
As before, we obtain 1M samples from the proposal distribution to compute importance sampling estimates of quality and efﬁciency metrics (i.e., TVD( p, p β ) , D KL ( p || p β ) and AR), plus the backward
KL-divergence from the base language model D KL ( ·|| a ) (Eq. 27, in Appendix B) and the moments of the features that we wish to control (Eq.
We show all metric curves as a function of acceptance rate of the QRS algorithm as well as some example generations in Fig. 4.
We ﬁnd that TVD( p, p β ) and D KL ( p || p β ) (as well as the upper-bound on TVD( p, p β ) ) steadily converge to 0 as the acceptance rate decreases, meaning that we can perfectly match the target EBM at the cost of sampling efﬁciency.
As a result, at a moderate acceptance rate AR = 10 − 3 we nearly perfectly debias our original language model while exclusively having it generate biographies about scientists ( 49 .
5 % female and 99 .
8 % scientist biographies).
We show some example generations at AR = 10 − 3 chosen manually to show two male and two female biographies.
Notably, we also achieve very decent constraint satisfaction ( 44 .
8 % female and 99 .
7 % scientist biographies) and TVD (at 0 . 27 ) already at AR = 10 − 1 , allowing to considerably improve the quality at a small cost in efﬁciency.
Divergence to the original language model D KL ( p β || a ) steeply increases as the feature moments are matched more closely, after which it gradually decreases before stabilizing.
This reﬂects the construction of the EBM, which projects a ( x ) onto the constraint manifold in such a way
that
the
KL-divergence from the original language model is minimized.
Two more pointwise and two more distributional constraints are shown in Appendix G with similar results.
3. P ARAPHRASING
Finally, inspired by Miao et al.
(2019), we do proof-of-concept experiments on paraphrase generation by framing it as a conditional EBM.
Speciﬁcally, given a sentence y to paraphrase, we deﬁne our
EBM through a pointwise constraint on a ( x ) =
GPT-2 ( x ) , with b ( x ) a binary classiﬁer that classiﬁes a pair ( x, y ) as a paraphrase if the cosine similarity between their sentence embeddings is above 0 .
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We obtain high-quality sentence embeddings from sentence BERT 7 (Reimers & Gurevych, 2019).
As proposal distribution we will not be using GPT-2, but rather illustrate how we can utilize off-the-shelf deep learning models as proposal distributions for QRS.
In particular, we use a round-trip MT model, which is a well-known tool in generating paraphrases (Bannard & Callison-Burch, 2005; Mallinson et al., 2017).
Speciﬁcally, we use the English-to-German and German-to-English models from Ng et al.
We ﬁrst obtain a beam searched (Graves, 2012) translation into German, 8 and then deﬁne the proposal distribution as the German-to-English model conditioned on the beam searched translation.
We locally renormalize the model to do top-30 sampling (Fan et al., 2018).
We show importance sampling estimates of TVD( p, p β ) using 1M samples for three sequences in
Fig.
5 along with example samples from both the proposal distribution and QRS at AR = 10 − 5 .
The proposal distribution quality varies per input sequence as can be seen by the slope of the curve and the low-efﬁciency starting points of some curves (non-paraphrases are always rejected and so have a big inﬂuence on the acceptance rate).
Still, QRS allows us to approximate the target EBM reasonably well at the cost of sampling efﬁciency ( TVD( p, p β ) is 0 . 20 , 0 .
42 and near 0 ).
Looking at the examples, we ﬁnd the proposal distribution to produce decent paraphrases, but not always semantically equivalent or grammatically correct.
The QRS samples are mostly semantically equivalent, though they still produce some mistakes (“Axa Bharthi” vs “Bharti Axa”) and seem to be insensitive to the question mark and to the casing of words (“Cool”, “Two-wheeler insurance”).
Interestingly, this experiment illustrates how the presented approach could be employed to disentangle the questions of how to model a problem (by deﬁning the corresponding EBM) and how to efﬁciently sample from it (by improving the proposal distributions), allowing to work on each of these questions separately.
R ELATED W ORK
Sampling from discrete EBMs does not require an MC approach.
DPG (Parshakova et al., 2019b) instead trains an autoregressive model that approximates a given EBM, which can then be used to obtain samples efﬁciently.
However, it can be hard or impossible, in practice, to approximate the target EBM accurately because of its complexity, or more fundamentally, because of inherent limitations of autoregressive architectures (Lin et al., 2021).
For instance, while empirical results by Khalifa et al.
(2021) show that their DPG-based approximation is reasonable, there is still a sizeable gap to the target distribution.
Here, we close the gap almost completely at moderate cost in sampling efﬁciency by using the approximation as a proposal distribution for QRS.
For this paper we were inspired by the ability of some Monte Carlo approaches to guarantee accurate sampling in the limit.
Without trying to cover the full landscape here, we provide some pointers to relevant
MC techniques that can exploit global proposal distributions.
Possibly the most obvious MCMC candidate, as already mentioned, is IMH (Robert & Casella, 2004, §7.4); we devote Appendix E to a detailed comparison of IMH with QRS.
Due to the non i.i.d.
nature of the associated sampler, to its inability to score its samples, and the resulting unavailability of explicit convergence diagnostics, we then argue for the superiority of QRS for our purposes.
Several other approaches than ours have taken rejection sampling as their starting point.
Similar to us, Rejection Sampling Chains (Tierney, 1992), (Chib & Greenberg, 1995, §6.1) do not require a global upper-bound.
It is a hybrid that uses rejection sampling in a region satisfying a partial upper-bound but combines it with IMH outside of that region to produce a Markov chain that converges to the correct stationary distribution in the limit.
Caffo et al.
(2002) propose Empirical Supremum Rejection Sampling, an algorithm that adaptively increases the β upper-bound based on the maximum observed so far, with a focus on convergence in the limit rather than approximation quality.
Some researchers have observed before us that a partial bound β leads to the probability distribution presented in Eq.
Rejection Control (Liu et al., 1998), (Liu, 2004, pp.
44-45) makes use of this observation for accelerating the computation of an unbiased importance sampling estimate of the expectation E x ∼ p
f
( x ) in situations where computing f ( x ) is expensive and where it is desired to reduce such computation in regions of p for which the importance ratios are small and have less impact on the importance sampling estimate.
None of the above approaches provide explicit convergence diagnostics nor consider a trade-off between efﬁciency and approximation accuracy.
We argue that such a trade-off is important for practical use-cases of sampling, where fast response time may be prioritized over obtaining exact samples.
C ONCLUSIONS
In this paper, we address the problem of obtaining samples from a target EBM given access to a good global proposal distribution.
In general, a technique that addresses this question has the potential to decouple , for any generative task, modelling (by tuning the EBM deﬁnition) from efﬁcient sampling (by tuning the proposal distribution).
In the past, high-quality global proposals were typically not easy to come by, thus strongly motivating the development of MCMC techniques which could exploit simple local proposals that computed transition probabilities between candidate samples.
Today, however, developments in neural network training techniques make high-quality global proposal distributions easier to obtain than before.
Motivated by such developments we propose QRS, a generalization of rejection sampling that exploits such proposals to approximately sample from an
EBM.
QRS can be applied even in cases in which no upper bound of the importance ratio between the
EBM and the proposal distribution scores is known, or even exists at all.
Notably, QRS also provides strong theoretical guarantees, which include not only convergence to the target distribution (Eq. 4), but also diagnostics that are not available for other MCMC sampling techniques like independent
Metropolis-Hastings, such as an upper bound on the TVD (Eq. 3) or unbiased estimators of the TVD and KL divergence to the target distribution (Eqs.
Our experimental results on discrete EBMs show that QRS achieves strong results on the studied controlled text generation setting where, for instance, the sampler achieves excellent debiasing of the language model using acceptance rates in the range of 10 − 1 to 10 − 3 .
Furthermore, we show the versatility of the approach by showing how it can be applied to sample paraphrases from an EBM formulation derived from combining a pretrained language model and a sentence similarity score.
Yet, there is a trade-off between quality and efﬁciency, and ﬁnding the right balance will depend on the particular application.
QRS allows to use any arbitrary value of β , producing levels of quality and efﬁciency that can be estimated.
For convenience, in Appendix D we also describe a variant of QRS that automatically adjusts β under the constraint of not falling below a target acceptance rate.
Last, we note that while here we have focused on EBMs for discrete sequential spaces, nothing prevents QRS from being applied to continuous spaces, making it potentially useful for such applications as speech or vision.
R EPRODUCIBILITY S TATEMENT
To ensure the full reproducibility of this work, we have included in Appendix A and B complete proofs and derivations of all propositions and quantities in this paper.
Furthermore, source code of all experiments is provided as an anonymized link to the reviewers and will be open-sourced upon publication of this manuscript.
Lastly, a table with the range of β values used to generate the plots is given in Table 1 in the Appendix.
E THICS S TATEMENT
Part of the experimental section of this paper studies the generation with distributional control task, deﬁned by Khalifa et al.
The goal of this task is to adapt the generations of a language model to a given set of norms that should be quantiﬁable as preferences over the moments of certain features.
This formalization has the potential of addressing numerous problems related to social bias in large pretrained language models (Sheng et al., 2019; Liang et al., 2021), including gender bias.
Nonetheless, we note that the approach is not prescriptive with respect to which norms should be applied, nor about how to quantify the relevant features, all of which can be decided with the relevant stakeholders.
Finally, we note that all experiments have been done on the English language due to the availability of large pre-trained language models in this language.
We believe that the presented set of techniques should be equally applicable to other languages and multilingual models as well
A P ROPERTIES OF QRS: PROOFS
Equation (1): By deﬁnition p β ( x ) is the probability that the ﬁrst 10 output from Algorithm 1 is equal to x .
On the ﬁrst step of the algorithm, the probability that a given x is accepted is q ( x ) r x , while the probability that no x at all is accepted is ρ .
= (cid:80) x ∈ X q ( x )(1 − r x ) = 1 − (cid:80) x ∈ X q ( x ) r x .
More generally, the probability for x to be accepted on step i of the algorithm, while no x was accepted on previous steps is then ρ
i − 1 q ( x ) r x .
Overall, the probability p β ( x ) of x to be the ﬁrst x to be accepted is:
[Equation]
Equation (2): We have:
[Equation]
Equation (3): For the proof, we will need a well-known property of TVD (Chafa ¨ ı, 2010), namely that, for any distributions p 1 , p 2 over X , we have:
[Equation]
The reason is simple; we have:
[Equation]
, which proves that the ﬁrst and second expressions under brackets are equal.
Then we have:
TVD
[Equation]
p
[Equation]
p
[Equation]
1 / 2 (cid:88)
[Equation]
Our main proof , illustrated in Figure 6, proceeds as follows.
Let A β .
= { x ∈ X : P ( x ) ≤ βq ( x ) } and ¯ A β .
=
X \ A β .
We have P β ( x ) .
= min( P ( x ) , βq ( x )) and therefore P β ( x ) =
P ( x ) for x ∈ A β , and
P β ( x ) < P ( x ) for x ∈ ¯ A β .
Overall P β is smaller or equal to P and thus Z β ≤ Z .
For any x , we have p β ( x ) =
Z − 1 β P β ( x ) and p ( x ) =
Z − 1 P ( x ) , and hence for x ∈ A β , p ( x ) ≤ p β ( x ) .
If we deﬁne C β .
= { x ∈ X : p ( x ) ≤ p β ( x ) } , and ¯ C β . =
X \ C β , we have A β ⊆ C β and ¯
C β ⊆ ¯
A β .
Using Equation (15), we now see that TVD( p, p β )
=
(cid:80) x ∈ X : p ( x ) ≥ p β ( x ) p ( x ) − p β ( x )
=
(cid:80) x
[Equation]
X : p ( x
[Equation]
p β ( x
[Equation]
p
[Equation]
x
[Equation]
p
[Equation]
( x
[Equation]
(cid:80) x ∈
[Equation]
β p
[Equation]
p β ( x
[Equation]
≤ (cid:80)
[Equation]
∈ ¯ C β p ( x ) .
Finally we get:
[Equation]
[Figure]
𝑃 !
𝑃 𝑝 ! 𝑝 𝑃 !
Equation (4): Clearly, for any (normalized) distribution p over a discrete space X , for any (cid:15) > 0 , there exists a ﬁnite subset X (cid:48) ⊆
X s.t.
p
( X (cid:48) )
> 1 − (cid:15) .
If we take β .
= max x ∈ X (cid:48)
P ( x ) q ( x ) , then β is ﬁnite, X (cid:48) ⊆
A β , and therefore p ( A β ) ≥ 1 − (cid:15) , which proves the result.
A generalization to arbitrary q -supports
This last result exploits the assumption — that we made during the whole section 2 — that the support of p , Supp ( p ) , is contained in the support of q , Supp ( q ) , in other terms, p ( x ) > 0 ⇒ q ( x ) > 0 .
If that were not the case then β .
= max x ∈ X (cid:48)
P ( x ) q ( x ) could be inﬁnite and A β would not be deﬁned.
However, it is interesting that we can actually generalize the result to the case where Supp ( p ) may not be contained in Supp ( q ) .
Then, for any (cid:15) > 0 , there exists a ﬁnite subset Y (cid:48) ⊆
Supp ( q ) s.t.
p
( Y (cid:48) )
> p ( Supp ( q )) − (cid:15) .
If we now take β .
= max x ∈ Y (cid:48)
P ( x ) q ( x ) , then β is ﬁnite, Y (cid:48) ⊆
A β , and therefore p ( A β ) ≥ p ( Supp ( q )) − (cid:15) .
For any β , because all the x ’s in A β are obviously in Supp ( q ) , we have p ( A β ) ≤ p ( Supp ( q )) , and therefore p
[Equation]
Supp
[Equation]
q
[Equation]
(cid:15).
When the support of p is included in Supp ( q ) , we have p ( Supp ( q ))
= 1 , and therefore we get the previous result back, but now we see that, in the general case:
[Equation] [Equation]
B TVD AND KL ESTIMATES
Here we provide derivations for the estimates of equations (9) and (10):
[Equation]
Furthermore, we present the derivation and estimate for D KL ( p β , a ) :
[Equation]
C P ROJECTION OF THE PROPOSAL FOR POINTWISE CONSTRAINTS
Lets say we have a proposal distribution q that we want to use for approximating an EBM with a pointwise constraint P ( x ) = a ( x ) b ( x ) where b ( x ) ∈ { 0 , 1 } .
One simple approach would be projecting q directly into the manifold of the sequences matching the constraint, by simply rejecting all sequences x where b ( x ) = 0 .
In other words, we are sampling from a new probability distribution q proj ( x ) ∝ q ( x ) b ( x ) .
It turns out that we can compute the probability assigned to each sequence x by q proj , as follows:
[Equation]
Note that Z q proj .
= (cid:80) x
q ( x ) b ( x ) can be easily estimated by using a sample from q .
Therefore, the TVD( p, q proj ) and D KL ( p, q proj ) can be estimated following the same procedure as
Eqs.
19 and 21, respectively.
D I NCREMENTAL P RUNING WITH M INIMAL E FFICIENCY T ARGETS
The following algorithm describes a version of QRS that incrementally builds a batch of samples S with a desired minimum acceptance rate ar min .
The algorithm works by obtaining samples x from q and provisionally storing them into S as long as the rejection coefﬁcient α x = P ( x ) /q
( x ) u x does not exceed the current value of β , where u x ∼ U (0 , 1) .
Given that higher values of β imply a lower upper bound on the TVD between the samples and the target distribution, β is continually adapted to be as high as possible.
Yet, the higher β is, the lower the acceptance rate becomes.
Therefore, to be practical, β is capped at the highest possible value such that the acceptance rate of the samples seen so far would not fall below a minimum threshold ar min .
The corresponding maximum β value for an acceptance rate threshold ar min is computed as the value α for which the percentage of all previously obtained samples satisfying α x > α is ar min (i.e., the percentile ar min of all previously computed A
= { α x } values).
Should β be increased at any point, previously stored samples in S are pruned to remove those that failed to meet the acceptance criterion α x > β .
The algorithm stops when at least n samples have been collected.
Require: EBM P , proposal q , desired number of samples n , minimum acceptance rate ar min . Ensure: A set of n samples S and an estimation of β s.t. the acceptance rate is at least ar min .
1: procedure QRS- INCREMENTAL 2:
S ← [ ] , β ← 0 , A ← [ ] , β max ←3: while | S | < n do 4: x ∼ q ( . ) 5: β x ← P ( x ) q ( x ) 6: β max ← P ERCENTILE ( A, ar min ) 7: if min( β x , β max ) > β then 8: β ← min( β x , β max ) 9:
S ← P RUNE ( S, β ) 10: u x ∼ Unif (0 , 1) 11: α x ← β x u x 12: if α x > β then (cid:46) i.e. u x < P ( x ) βq ( x ) 13:
S ← S + [( x, α x )] 14:
A ← A + [( α x )] 15: return S , β 16: procedure P RUNE ( S, β ) 17: keep ( x, α x ) in S
iff α x > β (cid:46) i.e. u x < P ( x ) βq ( x ) 18: procedure P ERCENTILE ( A, r ) 19: return max
α x in A s.t.
| α y ∈
A : α y ≤ α x | < r | A | .
E I NDEPENDENT M ETROPOLIS H ASTINGS (IMH): ALTERNATIVE TO QRS ?
distance between the target distribution f and the distribution obtained by repeatedly running an n -step chain and outputting the n -th element.
This distance converges to 0 for increasing n .
E. T HE I NDEPENDENT M ETROPOLIS H ASTINGS (IMH) ALGORITHM
Markov chain Monte-Carlo (MCMC) (Robert & Casella, 2004) is the most developed general class of techniques for sampling from EBMs, exploiting random walk and Markov Chain theory (rejection sampling (RS) and QRS do not use random walks, and do not qualify as MCMC).
In the situation that we are focussing on here, namely, the availability of a global proposal q that already approximates p to some extent, a suitable MCMC technique is Independent Metropolis-Hasting (IMH)
(Robert &
Casella, 2004, §7.4), see Algorithm 3.
Algorithm 3 IMH
Require: P , q 1: x ∼ q 2: while True do 3: x (cid:48) ∼ q 4: r x,x (cid:48)
← min(1 , P ( x (cid:48) ) /q
( x (cid:48) ) P ( x ) /q ( x ) )
(cid:46) Prob.
of moving to x (cid:48) 5: u ∼ U [0 , 1] 6: if u ≤ r x,x (cid:48) then 7: output x (cid:48) 8: x ← x (cid:48) 9: else 10: output x
IMH is a special case of the Metropolis-Hastings (MH) algorithm, where the proposal q ( x (cid:48) ) does not depend on the current x .
IMH and QRS vs. RS: no need for global β
IMH and QRS share a common advantage over RS: they do not need a global β with P ( x ) /q
( x ) ≤ β, ∀ x ∈ X .
It is often the case that such a bound does not exist, is not known, or is intractably large.
In such cases, RS cannot be used, but both IMH and QRS can.
IMH vs. QRS: convergence to p in the limit IMH inherits from MH a fundamental “convergence to p in the limit” property:
[Equation]
(28) where π ( n ) denotes the sampler associated with what we will call the “ n -steps variant of Algorithm 3”, namely the algorithm that performs the loop on Line 1 exactly n times, does not output anything, but returns the last x .
This property corresponds, in our own notation, to Theorem 7.4.(ii) of (Robert & Casella, 2004), copied in Fig.
QRS has its own form of convergence in the limit, Equation 4.
IMH vs. QRS: convergence diagnostics Equation 28 tells us that performing n steps of MH (and in particular IMH) produces a distribution π ( n ) that gets closer and closer to p with increasing n , but it does not provide any explicit estimate of TVD( p, π ( n ) ) , nor any other divergence metric from the target distribution.
In general, with MCMC techniques, such explicit convergence diagnostics are very difﬁcult to obtain (Cowles & Carlin, 1996; Roy, 2020).
By contrast, as we saw (Equations 3, 7, 9, 10), the QRS algorithm does provide such explicit estimates .
This is important because it allows us to calibrate the level of effort (acceptance rate) we have to invest in order to obtain a certain quality (TVD or KL to p ).
IMH vs. QRS: ability to score Equation (1) provides an explicit value for p β ( x ) .
This is an important property, which, in particular, is exploited in the computation of estimates of TVD( p, p β ) and D KL ( p, p β ) in Equations (9) and (10).
This ability to score is conspicuously absent in IMH: there is no obvious way to compute π ( n ) ( x ) , not even up to a constant factor.
[Figure]
IMH vs. QRS: i.i.d properties The outputs produced by the QRS Algorithm 1 are immediately i.i.d: the acceptance of an x on Line 6 does not depend on the acceptance of the previous x .
By contrast, with any MH algorithm, and in particular IMH, if the goal is to produce actual samples, we face a choice.
First, we could follow the spirit of (Theorem 7.4.ii) above, restarting the chain each time we need to produce a sample.
Then we would produce i.i.d samples, but at the high cost of wasting n − 1 draws from the proposal before producing just one sample.
Or we can — similar to the spirit of (Theorem 7.4.i) for expectations — just produce one long chain from which we extract the actual samples, which is the approach we are taking in Algorithm 3.
The outputs of this algorithm are not i.i.d: it is possible for the new proposal x (cid:48) to be rejected (Line 6) and then the current x is repeated in the sequence of outputs, possibly many times if the current x is “good” (i.e. has a high importance ratio P ( x ) q ( x ) ).
To mitigate that problem, it is often advocated to use a “thinning” heuristics on the outputs of the algorithm: only retain one output out of m , in order to reduce the autocorrelation, in the hope of obtaining a good balance between the quality of the samples and the efﬁciency of the sampler, but again, without explicit quality estimates.
E. IMH VS . QRS E XPERIMENT
In Figure 8a we compare the QRS and the IMH sampler on the 50% female and 100% science
EBM by taking 1,000 samples from each sampler at different levels of acceptance rates.
We use the procedure described in Appendix D to obtain QRS samples at the speciﬁed acceptance rate.
For the IMH sampler we obtain the desired “acceptance rate” by having a ﬁxed burn-in period of 1,000 samples and afterwards only keep every 10 th , 100 th or 1000 th sample.
The deﬁnition of “acceptance rate” we use for both samplers is the ratio of samples obtained from the sampler (1,000) and the number of samples used from the proposal distribution.
We use the DPG model as proposal distribution.
We ﬁnd that while IMH is able to achieve constraint satisfaction similar to the QRS sampler at similar acceptance rates, it does so by repeating some samples multiple times.
This can be seen in the fraction of unique samples, which can get as low as 50% out of the 1,000 samples obtained at lower amounts of thinning.
While this does not hinder the IMH sampler from achieving asymptotic consistency in estimating expectation values, we believe this to be an undesirable property of a sampler whose outputs are to be used in actual applications.
QRS does not suffer from the same problem, as the QRS algoritm does not make use of a Markov chain.
In Figure 8b we also experiment with a version of IMH that does obtain i.i.d.
samples by resetting the Markov chain after every sample obtained.
We coin this version IMH-reset.
We achieve desired acceptance rates by resetting the chain after every 10 th , 100 th or 1000 th sample.
We obtain 100% unique sequences for this version for a sample of 1,000 samples, like the QRS sampler.
For this version we can additionally estimate perplexity under GPT-2, as we have i.i.d.
samples from both samplers.
We ﬁnd that at the target feature moments (for both samplers at the acceptance rate of 10 − 3 , QRS samples achieve slightly lower perplexity than IMH-reset.
F A DDITIONAL R ELATED W ORK : C ONTINUOUS EBM S AND EBM S FOR NLP
While this paper is concerned about generating samples from discrete EBMS, more research so far has been more concerned with the training of continuous EBMs, in particular for applications in vision.
Continuous EBMs have the advantage over discrete ones in that it is possible to differentiate the
EBM P ( x ) relative to x , and not only the approximating model π θ relative to θ .
Training such EBMs (see the survey in (Song & Kingma, 2021)) can then often be addressed through techniques such as contrastive divergence (Hinton, 2002; Du et al., 2021), score matching (Hyv ¨ arinen, 2005; Song &
Ermon, 2020), or noise contrastive estimation (Gutmann & Hyv ¨ arinen, 2010).
These approaches sometimes require an internal sampling procedure, and then one technique of choice is Langevin
MCMC (Parisi, 1981), in which the local Markov chain moves are done based on ∇ x P ( x ) , a technique which is also employed in case actual samples need to be generated.
While such techniques are not available for discrete EBMs, some recent efforts are trying to bridge the gap (Grathwohl et al., 2021).
Sampling techniques are popular in various natural language processing (NLP) applications.
For example, Miao et al.
(2020) construct a rejection-sampling inspired sampler that aims to counteract over- and underestimation of probability regions due to overﬁtting when ﬁne-tuning large pre-trained language models on small datasets.
Deng et al.
(2020) train globally normalized language models to combat negative effects of local normalization, and use a form of SIR to sample from the resulting
EBM using an autoregressive proposal language model.
Goyal et al.
(2021) develop a MetropolisHastings algorithm to sample from masked language models.
For controlled text generation Miao et al.
(2019) propose a random-walk Metropolis-Hastings (MH) algorithm for sampling from an
EBM that encodes sequence-level preferences on natural text.
Their proposal distribution consists of local string editing operations on randomly selected words or positions.
Zhang et al.
(2020) improve on this approach by making use of a tree-search algorithm to more efﬁciently explore the space of proposals, by allowing several edits in a single step of the MH algorithm.
In contrast with these approaches we make use of a strong global proposal distribution and the QRS sampler.
This eliminates any autocorrelations in the samples and ensures good sample diversity inherited from the underlying EBM.
Furthermore, the QRS sampler allows us to assess the quality of our samples through divergence metrics from the target EBM, something that is famously difﬁcult to do with
MCMC samplers.
G A DDITIONAL C ONTROLLED T EXT G ENERATION R ESULTS
We perform constrained text generation on a distributional, two pointwise and two mixed distributional pointwise constraints.
In particular, we constrain the GPT-2 biographies model to contain 1) 50% female biographies about scientists, 2) 50% female biographies about sports or 3) 50% female biographies without additional constraint.
Also, we constrain GPT-2 small to generate exclusively sequences containing the term “amazing” or to generate exclusively sequences containing the term “Wikileaks”.
For each of these tasks, we obtained a ﬁne-tuned model using DPG, which serves both
[Figure]
[Table]
as a baseline and as a proposal q that we can sample from.
In the case of pointwise constraints, we also consider a naive ﬁlter sampler q proj in which the proposal distribution is directly projected onto the constraint manifold by ﬁltering out all samples that do not match the constraint.
This sampler also assigns well-deﬁned probabilities to the sequences that it samples (see Appendix C), and so we can compute estimates of the TVD and D KL for it.
For each task, we again obtain 1 M samples from the corresponding proposal, which we use to evaluate the proposal q , the projected proposal q proj (only for the pointwise constraint), and QRS sampling ( p β ) for a range of β values reported in Table 1.
For all of these, we compute estimates of the same metrics as in Section 3.2.2 (i.e., TVD( p, p β ) , D KL ( p, p β ) , AR, backward KL-divergence from the base language model D KL ( ·|| a ) , and the moments of the features that we wish to control).
Furthermore, we do a downstream evaluation of 50 k samples obtained through QRS choosing β so that the acceptance rate falls exactly at 10 − 1 , 10 − 2 and 10 − 3 .
(The process is described in Appendix
D).
In particular, we look at the feature moments of the obtained samples.
Our results are shown in Fig.
As expected, we ﬁnd that the upper bound of the TVD of p β with p and the KL-divergence from p β to p steadily converge towards 0 as the acceptance rate decreases.
For the distributional constraints and corresponding proposal distributions shown here, it seems that an acceptance rate of 10 − 3 is sufﬁcient to match the target EBM nearly perfectly.
Feature moments therefore shows the same pattern, converging towards the target moments with lower acceptance rates, although we ﬁnd that in some cases the QRS sampler matches the target EBM so closely that small inaccuracies in the lambdas obtained from the EBM estimation procedure as described in Khalifa et al.
(2021) become apparent.
As for the divergence to the original language model D KL ( p β || a ) , there is no obvious trajectory that it should follow other than it should converge to the lowest possible value when all constraints are satisﬁed.
Indeed, our results show that this metric follows a non-monotonic path at different ARs.
Validating our estimates, we note that the moments computed downstream on QRS match tightly the IS predictions, giving us conﬁdence in the accuracy of those estimates.
Finally, in the case of our pointwise “amazing” and “Wikileaks” constraints, we ﬁnd that the naive ﬁlter strategy ( q proj ) corresponds to running the QRS sampler at a high acceptance rate. H E XTRA E XPERIMENTS AND C OMPARISONS WITH L OCALIZED MCMC We focus on discrete Random Walk Metropolis-Hastings (RWMH) for text generation, conceptually close to Miao et al. with some ingredients from Goyal et al.
We did consider GwG (Gibbs-with-Gradients Grathwohl et al.
and other related gradient-based proposals, but felt that attempting to apply such techniques for text generation tasks such as those considered in our main experiments would be very ambitious and worth a paper on its own, see Appendix I.
In our proposed experiments, we construct a local proposal distribution composed of single-token insert, delete and replace operations (chosen randomly from a uniform distribution like in Miao et al.
We inform the insert (resp.
replace) operation by inserting (resp.
replacing a word with) a [MASK] token and sampling a new token from BERT.
This differs from Miao et al.
(2019), who instead do a form of Gibbs sampling for insert and replace operations, which requires V (vocabulary size) assessments of the EBM per such operation (an EBM with GPT-2 as a component).
This would make this version prohibitively more expensive than our QRS and IMH operations, for which reason we replace this with a single pass of BERT instead.
We accept or reject such edits using the Metropolis-Hastings acceptance ratio and seed the chain using a high-quality global proposal
[Table]
- science” EBM described in Section 3.2.2.
We do not compute perplexity for IMH and RWMH without reset as it does not yield i.i.d.
samples.
As noted, TVD and KL for MCMC methods are unknown (i.e. we have no way of estimating them).
Where available we show mean ± one standard deviation over 10 runs.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
distribution (DPG, as used in QRS experiments).
Per step of the MCMC chain this algorithm is roughly similar in cost to our IMH algorithm.
A single step of RWMH requires a pass through BERT (for insert or replace) and a pass through GPT-2 (for scoring under the EBM), while a single step of
IMH requires sampling a sequence from DPG (a highly parallelizable operation, in practice we take many samples at once and pre-store them on disk) and pass through GPT-2 as well (for scoring under the EBM).
Therefore, we use the same deﬁnition of acceptance rate as in the IMH experiments in
Appendix E. Like for IMH, we also provide two versions of RWMH: one in which we do thinning to reduce autocorrelations (RWMH) and one in which we reset the chain (RWMH-reset) after a number of steps (and set the seed again from the DPG proposal).
The value of the thinning parameter and the number of steps after which the chain is reset determines the AR of the algorithm.
In order to inform the local proposal distribution of the EBM, we also investigate mixing BERT with a Dirac δ on a particular word (e.g. “amazing” or “Wikileaks”) for a pointwise constraint.
We compare each sampler (QRS, IMH, IMH-reset, RWMH, RWMH-reset) in terms of moment matching results, perplexity, Self-BLEU (Zhu et al., 2018), and ﬁnally TVD and KL where possible (that is to say, for
QRS only, as we stress below).
We run on both a pointwise constraint (generating from GPT-2 using the word “amazing”) and a mixed distributional constraint (debiasing scientist biographies, see Section 3.2.2) to compare the samplers.
We run all samplers (QRS, IMH, IMH-reset, RWMH, RWMH-reset) both at 10 − 1 and 10 − 3 acceptance rate.
We use Algorithm 2 for QRS to ﬁnd a β value that approximately satisﬁes the target acceptance rate.
For IMH and RWMH we have a ﬁxed burn-in period of 1,000 steps and deﬁne the acceptance rate as the amount of thinning that is done: e.g. keep only every 10 th sample for an acceptance of 10 − 1 .
For IMH-reset and RWMH-reset we deﬁne the acceptance rate by the rate at which we reset the chain (and re-seed from the global proposal), e.g. we run a chain of length 1,  and only keep the last sample for an acceptance rate of 10 − 3 .
We also report performance on the global proposal (DPG).
We show results on collecting 1,000 samples of each in Tables 2 and 3.
We ﬁnd that IMH, IMH-reset and QRS perform considerably better than RWMH and RWMH-reset at all acceptance rate levels.
RWMH and RWMH-reset seem to have trouble achieving good constraint satisfaction consistently.
We currently report numbers on a single run of the sampler, which might suffer from a poor seed sampled from DPG.
In the ﬁnal version we shall include mean and variance results over multiple runs.
QRS, IMH and IMH-reset meet the imposed constraint of only generating sequences containing the term “amazing” and, at lower acceptance rate, also the constraints of generating debiased scientist biographies roughly.
At 10 − 1 acceptance rate IMH suffers from high autocorrelation among the samples due to repetitions (as seen from the percentage of unique sequences).
Also at a 10 −acceptance rate IMH and IMH-reset both seem to have slightly less diversity than QRS, but all in all do achieve similar self-BLEU as QRS and produce roughly 1,000 unique sequences.
Perplexity is also similar for QRS and IMH-reset (we do not compute perplexity for IMH as the computation requires i.i.d.
samples), slightly in favor of QRS in the mixed distributional constraint.
We found similar results before in Appendix E.2.
In conclusion, QRS and IMH-reset have an advantage over
IMH in that they provide i.i.d.
samples and have higher diversity at high acceptance rates.
Both versions of IMH and QRS perform on par in terms of perplexity and self-BLEU.
QRS, however, has a large advantage in that it allows to compute actual divergence from the target distribution (TVD and KL), something simply not available to MCMC methods.
H. I NFORMING THE L OCAL P ROPOSAL D ISTRIBUTION
We attempt to inform the local proposal distribution without introducing excessive additional computation by proposing a simple mixture model for the insert and replace operations.
With a small probability, that we tune as a hyperparameter, the proposal distribution proposes to insert or replace a token with the token “amazing” (additionally to the probability of that occurring under BERT alone).
We show some preliminary results of this on RWMH and RWMH-reset in Table 3.
Our preliminary results do generally increase the constraint satisfaction of the resulting sampler.
However, in the case of RWMH it leads to a catastrophic failure mode: where in the burn-in period an “amazing” was inserted somewhere in the sequence, and the chain got stuck at that local optimum for all 1,000 samples we collected (including thinning steps).
The results on RWMH-reset look more promising, increasing constraint satisfaction only slightly, but steadily improving perplexity.
We will perform more experiments on this for the camera-ready version and include some variance estimates over different runs of the algorithm.
I O BSERVATIONS ON G W G AND OTHER GRADIENT - BASED TECHNIQUES FOR DISCRETE SPACES
Gibbs with Gradients (GwG) (Grathwohl et al., 2021) is a recent promising technique for importing the effective gradient techniques of continuous EBMs (as in vision) to discrete EBMs.
Grathwohl et al.
(2021) start by observing that basic techniques for discrete distributions such as RWMH (Random
Walk Metropolis Hastings)
(i) require meticulous customization of the local proposal to lead to reasonable results and (ii) are unable to exploit differentiability over sample-space densities, which are so crucial for efﬁcient MCMC sampling from EBMs in continuous domains.
The method they propose instead addresses those situations (which they argue are many) where the discrete EBM can be seen as a projection of a differentiable function over an underlying continuous space, which leads them to perform sampling in the discrete space, but with a novel “locally informed proposal”
[Table]
(Zanella, 2017) that exploits gradients in the underlying continuous space in order to avoid explicitly computing the energy of each point in the “local neighborhood” of x , a very costly operation.
Grathwohl et al.
(2021) do not experiment with textual data, and we believe that while applying their technique to some text generation EBMs similar to those of our current experiments would be possible (see just below), this would be an ambitious research project on its own, beyond the scope of the current paper.
A possible GwG approach for P ( x ) , where x = x 1 , ..., x n is a text to be generated, and where
P ( x ) could be computed as a differentiable function of e 1 , ..., e n , where e i is the embedding of x i , could work as follows, adapting to text generation the last line (“Deep EBM”) of Table 1 in Grathwohl et al.
We would ﬁrst pick some position m , then compute ∇ e m P ( e 1 , ..., e m , ..., e n ) .
Then for each x (cid:48) = x 1 , ..., x (cid:48) m , ..., x n differing from x only at position m , we would approximate
P ( e 1 , ..., e (cid:48) m , ..., e n ) using the previous gradient, 15 a much less costly operation than computing
P ( e 1 , ..., e (cid:48) m , ..., e n ) from scratch (this is the gist of GwG).
In principle such a procedure could be done for any P ( x ) = a ( x ) b ( x ) , where a is a neural model (hence in principle differentiable over the word embeddings, e.g. for a an autoregressive model), and b is differentiable over these embeddings.
As mentioned above, this would be a whole project on its own, but note that the requirement that b be differentiable could make the approach tricky to apply to some cases such as when b computes an arbitrary binary predicate, is provided as a black box, etc.
Grathwohl et al.
(2021) contrast their approach to other recent gradient-based techniques, such as “continuous relaxation”, that ﬁrst map discrete x ’s to continuous representations, perform gradientbased MCMC there (e.g. with HMC, SGLD, etc.) and then map back to the discrete space (while
GwG performs MCMC in the discrete space).
They argue that MCMC sampling in the continuous relaxation does not directly exploit the underlying discrete structure of the space and therefore that sampling performance in the relaxed space may not be indicative of sampling performance in the discrete space.
We are not aware of such techniques that we could directly apply to our text generation tasks, and did not attempt experiments in this space.
J I MPORTANCE S AMPLING AND V ARIANCE
We perform many importance sampling estimates in this work to estimate quantities such as acceptance rate, and TVD and KL to the target distribution.
We report variance estimates for the experiments on debiasing scientist biographies (Section 3.2.2) in Table 4.
We collect 10 times the number of samples used to generate our plots (1 million) and report mean ± one standard deviation for β values within the range used within our experiments (also see Table 1).
We ﬁnd our estimates to be accurate within reasonable variance.
[Equation]