Are GANs overkill for NLP?
Abstract
This work offers a novel theoretical perspective on why, despite numerous attempts, adversarial approaches to generative modeling (e.g., GANs) have not been as successful for certain generation tasks, particularly sequential tasks such as Natural Language Generation, as they have in others, such as Computer Vision.
In particular, on sequential data such as text, maximum-likelihood approaches are signiﬁcantly more utilized than GANs.
We show that, while it may seem that maximizing likelihood is inherently different than minimizing distinguishability, this distinction is largely an artifact of the limited representational capacity of the model family, for a wide class of adversarial objectives.
We give a theoretical model in which minimizing KL-divergence (i.e., maximizing likelihood) is a more efﬁcient approach to effectively minimizing the same distinguishability criteria that adversarial models seek to optimize.
Reductions show that minimizing distinguishability can be seen as simply boosting likelihood for certain families of models including n-gram models and neural networks with a softmax output layer.
To achieve a full polynomial-time reduction, a novel next-token distinguishability model is considered.
Some preliminary empirical evidence is also provided to substantiate our theoretical analyses.
Introduction
Consider a situation where one has samples from a true distribution p over a set X and one wishes to learn to generate similar samples, such as learning to generate English sentences from a large
English text corpus.
One seeks an approximation q of p which is “close” in some sense and from which samples can efﬁciently be generated.
A common approach to ﬁt these models is Maximum
Likelihood Estimation (MLE), which given a training set from p and a parametrized distribution q θ seeks parameters θ that maximize the likelihood q θ assigns to a training set.
MLE has long been one of the most popular methods for ﬁtting generative models of sequential data, such as language, where autoregressive neural language models generate remarkably realistic text, e.g., GPT-3 [6] and PaLM [11].
MLE generally involves computing likelihoods q θ ( x ) which can be more challenging in some domains than others, e.g., it may be more difﬁcult to estimate the probability of a (high-dimensional, real-valued) image than a (discrete-valued) sentence.
An alternative approach, Generative Adversarial Networks (GANs), has become popular across several domains, particularly Computer Vision, owing to breakthrough realism in the images they output [e.g., 19, 66].
GANs employ an adversarial approach to generation through a zero-sum game between a generator and a distinguisher in which the generator produces samples x ∈ X which the distinguisher tries to distinguish from real samples from p .
Often, both the generator and the distinguisher are differentiable neural networks, though this min-max approach of choosing a model whose outputs are nearly indistinguishable from real examples might be considered for any families of generative models and distinguishers.
A major advantage of GANs (particularly for images) is that they can be used for generation without requiring computing likelihoods.
This advantage is not signiﬁcant for many sequential models such as language models, where computing likelihoods is not difﬁcult.
In contrast, the adversarial approach has yet to demonstrate signiﬁcant improvements in some other domains such as Natural Language Processing (NLP).
One well-known barrier to NLP GANs is that language models produce discrete outputs (words), so they are not naturally differentiable [18].
However, despite numerous works circumventing this limitation and adapting GANs to text generation [64, 42, 23, 12], adversarial-based models have yet to achieve the same popularity or performance gains that were seen for images.
In particular, language GANs have been shown to under-perform
MLE in terms of quality [55] while facing the challenge of lack of diversity due to mode collapse [7], which is a well-known issue with GANs in other domains.
1.1 Likelihood and Distinguishability: Two sides of the same coin?
In this work, we suggest a different, fundamental barrier to adopting GANs in domains where MLE is prevalent: the adversarial approach of minimizing distinguishability can be seen as an indirect method of maximizing likelihood on observed data, and hence employing MLE directly can be more efﬁcient.
This is the case in NLP where, unlike computer vision, a measure of likelihood called perplexity has been the prevailing metric for training and evaluating language models for decades.
We show how GANs boost likelihood in the spirit of, and inspired by, the related celebrated work of
Friedman et al.
[16] that showed how boosting can be viewed as an iterative approach for logistic regression.
Consider a large ﬁnite set or countably inﬁnite set X and a family Q of probability distributions over X .
For language, these might be n-gram models or neural models.
Also consider a family F of distinguishers f : X → [0 , 1] that aim to distinguish random examples drawn from a distribution p from those sampled from q .
For any such classiﬁer f , we call the difference α ( f ) =
E q [ f ( x )] −
E p [ f ( x )] the distinguishability advantage of f because it quantiﬁes the accuracy of f at the task of identifying “fake” examples.
A perfect distinguisher would thus have α ( f ) = 1 , while f ( x ) = 1 /which predicts at random has α ( f )
= 0 .
More formally, imagine picking y ∈ { 0 , 1 } uniformly at random and picking a random example x from q if y = 1 and from p if y = 0 .
The (randomized) binary classiﬁer that predicts, for any x , ˆ y = 1 with probability f ( x ) , has (expected) accuracy:
[Equation]
Given a family F , we deﬁne the distinguishability of q from p to be d ( q ) = max f ∈ F α ( f ) .
Distinguishability is known to be a lower-bound on total variation distance (also called statistical distance), a measure of distance between distributions that is difﬁcult to directly estimate for large domains
X [54].
The “Bayes-optimal” distinguisher simply predicts 1 iff q ( x ) > p ( x ) , and has advantage equal to the total variation distance [see, e.g., 24].
Clearly d ( p ) = 0 , i.e., p is indistinguishable from itself.
Motivated by this observation, numerous adversarial approaches to approximating q have been attempted to minimize distinguishability d ( q ) .
If p ∈ Q then d ( q ) is minimized at q = p .
We ﬁrst discuss some important aspects of distinguisability.
What adversarial objectives can be analyzed via distinguishability?
It is important to emphasize that distinguishability is indeed the objective that several adversarial approaches, such as many variants of GANs, seek to optimize.
Depending on the context, it serves the purpose of discriminator or critic .
In particular, as established in [54], GANs that are trained based on Kantorovich metric,
Fortet-Mourier metric, dual-bounded Lipschitz distance (or the Dudley metric), total variation distance, and kernel distance can all be cast in terms of distinguishability.
Thus, in particular, our results hold for GNN formulations such as Wasserstein GANs [1], MMD GANs [33], Fisher GANs [36], and Sobolev GANs [49], for an appropriately chosen family F of distinguishers.
For example, we obtain Wasserstein GANs, in its dual form, as a special case when F is restricted to 1-Lipschitz functions in which case it can also be viewed as a special case of the so-called f-GANs [3, 37, 50].
Likewise, we obtain MMD-GANs when F pertains to functions (kernels) deﬁned over a ball in some
Reproducing Kernel Hilbert Space [4].
Note that distinguishability allows us to accommodate other sophisticared GAN variants such as WGAN-GP [22, 60] that do not suffer from the issue of gradients vanishing on discrete spaces.
Recall the WGAN-GP objective can be expressed in our notation as:
[Equation]
This can be viewed as Lagrangian relaxation of the following hard objective for (cid:15) > 0 as:
[Equation]
Distinguishability advantage can then be readily be expressed as
[Equation]
Is distinguishability symmetric or asymmetric?
Note that, by deﬁnition, distinguishability is asymmetric in the sense that in general q is distinguishable from p is different from p is distinguishable from q .
Note, however, that we recover integral probability metric (IPM) [54] when − f ∈ F for all f ∈ F .
Clearly, in this case the notion of distinguishability becomes symmetric as the advantage reduces to max f ∈ F | E q [ f ( x )] − E p [ f ( x )] | .
Thus, distinguishability lets us handle an extremely wide class of discrepancies, symmetric as well as asymmetric.
Example where maximizing likelihood (cid:54) = minimizing distinguishability.
When p (cid:54)∈ Q , minimizing distinguishability among q ∈ Q may be different than maximizing the likelihood of q .
For instance, consider modeling the age in years of humans (say the entire population on earth) as a uniform distribution q m over x ∈ { 0 , 1 , 2 , . . .
, m } .
Now, the m which maximizes likelihood would be the age of the oldest person, which is m = 119 at the time of this article—any smaller m would assign zero probability to the 119-year-old and thus to the entire population.
However, this distribution is very distinguishable from the true distribution—for instance it assigns probability ∼ 17% to being over 100 years, which is extremely unlikely among today’s population.
A smaller m < 100 would yield less distinguishable samples.
While it may seem therefore that distinguishability and likelihood are inherently different criteria, as we shall see this is an artiﬁcial limitation due to the weakness of family Q .
Of course, the (in)equivalence depends on the families F of distinguishers and Q of probability distributions.
We give two results showing that maximizing likelihood and minimizing distinguishability are equivalent as long as F and Q are similar in representational capacity, even when p (cid:54)∈ Q .
First, we consider families Q that are “log-linear” over some set F of functions, which include n-gram models and neural networks whose top layer is a softmax, among others.
The equivalence in this case is particularly simple and serves to illustrate how MLE can be a simpler way to reach the same optimum.
In this case, Q and F are naturally paired.
Maximizing likelihood =
minimizing distinguishability for log-linear Q .
In the above age example, the family Q of geometric distributions q θ ( n ) ∝ exp( − θn ) for θ > 0 is an example of a log-linear family.
We show that if q can be distinguished from the population distribution p by a function f ∈ F , then folding f into q yields a new model in Q with greater likelihood.
In practice, one only has a sample of the true distribution p (not the entire population) and maximizing log-likelihood is approximation of minimizing KL divergence D KL ( p (cid:107) q )
= (cid:80) x p ( x ) log p ( x ) q ( x ) .
We give formal statements about minimizing KL-divergence as well.
The conclusion of this ﬁrst observation is that if a GAN were to converge within a log-linear family (and making GANs converge is often not an easy feat in practice), it would converge to the MLE.
General polynomial-time reduction.
Our second result is a polynomial-time reduction from likelihood maximization to next-token distinguishability, without the log-linear requirement.
We consider the common case of (unidirectional) sequential models that predict the next token based on the previous tokens, which have several practical advantages including being efﬁcient to compute— the probability of a sequence is simply the product of the conditional probabilities of each subsequent word given the previous words.
Many state-of-the-art transformer language models such as GPTtake this form.
Achieving an efﬁcient reduction is challenging due to the normalization requirement of computing partition functions.
In order to achieve a polynomial-time reduction, we consider a notion of next-token distinguishability, where the game is as follows: a preﬁx of tokens is chosen, based on which the generator generates a token to follow the preﬁx.
Given the actual next token and the generated next token, the distinguisher aims is to identify which is which.
Algorithm 1 leverages a next-token distinguisher to iteratively increase likelihood.
In particular, given any target (cid:15) > 0 ,
Theorem 1 shows that Algorithm 1 will terminate and output an efﬁciently computable model q which is nearly (to within (cid:15) ) indistinguishable from the truth, and it runs in time polynomial in 1 /(cid:15) .
If p ∈ Q and one has an optimal distinguisher, one will eventually converge to a model close to p , as has been discussed heavily in the literature.
However, our results are also meaningful in the more realistic case where one has imperfect distinguishers.
Contributions .
The main contributions of this paper are:
• showing that, although in general minimizing distinguishability and maximizing likelihood
seem to be different, they are in fact closely related, • introducing a new model of next-token distinguishability that is necessary to make the reduction efﬁcient, and • offering a new perspective on why GANs might have been less successful in NLP and other sequential domains as they have been, e.g., for images.
Organization .
We begin by summarizing related work on GANs, especially for text.
We then illustrate how GANs can be overkill for the simple case of n-gram models in Section 3.
Sectioncovers log-linear models.
Section 5 gives explicit bounds on general reductions between maximizing likelihood and minimizing distinguishability.
Section 6 shows how the reduction can be efﬁciently computed in the case of sequential inputs, from which we propose a simple polynomial time algorithm that provably ﬁnds a distribution which is nearly-indistinguishable with respect to a given class of discriminator functions.
Finally, we discuss the relevance of our work in Section 8.
All proofs are deferred to the Appendix.
Related Work
Generative models .
Several approaches to generative modeling have been investigated, especially in the context of images.
In particular, impressive results have been obtained recently with variational autoencoders, GANs, normalizing ﬂows, autoregressive models, diffusion processes, and score/energy based models [28, 19, 41, 40, 25, 53, 59, 10].
Generally, training approaches are either adversarial; or rely on MLE, contrastive divergence estimation, or score matching [52].
Some connections have begun to emerge between these models, and alternate training procedures have been advocated [9, 53, 62].
A word of caution: Understanding the theoretical underpinnings of generative models with respect to their sample quality is an intriguing question that has been previously investigated by several seminal works such as [27, 21, 39, 56] and requires further analyses.
Our objective here is not to claim at all that maximizing likelihood is universally better than adversarial methods or vice-versa, but to emphasize that for many problems, in domains like NLP, the two objectives often turn out to be equivalent mathematically via the notion of distinguishability and maximizing MLE could provide a more efﬁcient (and stable way) of optimizing the common objective.
Also, note that maximizing likelihood does not always correlate with the perceptual sample quality [21].
A more comprehensive analysis encompassing the effect of architecture, optimization procedures and issues such as trade-off between perception and distortion [5], training and inference time, mode-collapse etc. is required for better understanding of generative models.
In that sense, our message is consistent with observations made previously [39, 56, 21] that there is no single model that ﬁts all situations, and the right model depends on the speciﬁc requirements of applications.
GANs for text .
Since their introduction [19], there has been interest in adapting GANs to text data.
The driving motivation was that —up until very recently— samples generated by traditional (likelihood-based) models had been easy to distinguish from human-generated text, and the success of image GANs at generating realistic-looking samples suggested a possible avenue to improve the quality of their natural language counterparts.
The ﬁrst and most signiﬁcant challenge in adapting GANs to text arises from the very nature of this data.
Goodfellow [18] points out that GANs require the generator to be differentiable, which poses a challenge for discrete text representations such as one-hot word or character representations.
Two of the most popular approaches to circumvent this obstacle are policy gradient techniques (e.g.,
REINFORCE [61]) —which when applied to language modeling nevertheless often require maximum likelihood pre-training [8, 64])— and the Gumbel-Softmax approximation [30].
The few adversarial methods that do not require pre-training (e.g., [42, 45]) have failed to show signiﬁcant promise in all but a few artiﬁcial tasks.
This nascent but active line of work seemed to suggest for a period of time that GANs might provide a breakthrough in text generation.
This promise did not fully materialize, and instead the most recent breakthrough came from models building very large transformer-based architectures like GPT [43, 44, 6] or PaLM [11] — which are trained with traditional cross-entropy (MLE) objectives.
Yet the question of how GAN-based methods for text compare with likelihood-based ones still garners signiﬁcant interest, and while various works have provided an empirical comparison between them —with most of these suggesting the advantage of MLE-based ones [7]— theoretical explanations have been less explored.
Relating objectives via divergences .
The connection between maximum likelihood estimation, distinguishability and divergences between probability distributions has been explored before.
For example, it is well known that maximizing likelihood is equivalent to minimizing the KL divergence between certain families of ﬁtted and reference distributions, though this is not the only divergence for which such a connection exists [46].
On the other hand, from the moment GANs were introduced,
Goodfellow et al.
[19] noted that —assuming a perfect discriminator— the adversarial objective corresponds to minimizing a Jensen-Shannon divergence.
Furthermore, the minimal discrimination error is also directly related to the total variation distance (see, e.g., Hashimoto et al.
On the other hand, for exponential families the gradient of the KL divergence is known to be related to the discrepancy between distributions [57].
While conceptually similar to this line of work, here instead we give an explicit reduction that shows how distinguishability and (log) likelihood are in direct correspondence.
Pinsker’s inequality is a well-known result linking KL divergence and total variation distance (TVD):
TVD ≤ (cid:112) KL / 2 .
While related, this inequality is not directly relevant to the context of this work.
First, while total variation provides an upper bound to distinguishability, it is not computable in general, so it is rarely used as a training objective for generative models.
On the other hand, being one-sided, 2 it does not imply that reducing TVD reduces KL divergence.
Furthermore, Pinsker’s is in general a very loose inequality, particularly for the direction of KLD that is equivalent to MLE (i.e., D KL ( p (cid:107) q θ ) ), since if p ( x ) > {+
+} ≈ q θ ( x ) even for a single x leads to unbounded KL divergence.
In contrast, in this work we provide a direct reduction directly linking the two criteria of interest: distinguishability and maximum likelihood.
Log-linear language models .
In this work we focus our analysis on log-linear models [31, 34], which are widely used in natural language processing (often known in that community as Maximum
Entropy –MaxEnt– models) for various tasks.
In particular, these models have been a cornerstone of both neural [2, 35] and non-neural [47, 26] language modeling.
Boosting .
The reduction shown here bears resemblance to boosting.
It is well-known that boosting can be analyzed through the lens of maximum likelihood (Friedman et al.
[16]), while Lebanon and Lafferty [32] formalized the equivalence of AdaBoost and maximum likelihood training for exponential models.
More recently, boosting has been applied to generative adversarial models [58, 20], albeit with a different approach and objective than the connection drawn in this work.
Illustration: GANs for n-gram language models
To illustrate our main point, consider ﬁrst the simplest model of language: a unigram model where the probability of each word is independent, and the end of sentence token EOS has a given probability as well.
If θ w represents the log-probability of word w , then the log-probability of sentence w 1 . . .
w t is given by:
[Equation]
The MLE parameters θ ∗ can be computed in linear time by simply counting word frequencies.
A more roundabout approach to ﬁtting a unigram language model would be to start with any initial unigram model q , generate random samples from q and compare them to those from p .
One could then distinguish the two by ﬁnding a word that appears signiﬁcantly more often in one than in the other.
For example, if one generates text from the model q and ﬁnds that the word “the” occurs much more often in text generated from p , one would then update the parameters by increasing θ the (and decreasing θ w (cid:48) for all other words w (cid:48) to keep q a probability distribution).
As we shall see later, if this more involved procedure converged, it would necessarily converge to the same maximum-likelihood estimator θ ∗ .
A similar argument applies to any n -gram model in which the probability of each subsequent word is determined only by the previous n − 1 words.
This is also optimized by frequency counts (a variety of “smoothing” techniques, e.g., adding 1 to counts, also known as Laplace Smoothing [17, 29] are often used as a form of regularization on top of these counts).
Distinguishers could similarly be used to ﬁnd a model q that is indistinguishable from p according to n -gram frequencies, but again this would simply converge to the MLE parameters.
Equivalence for log-linear models
In this section, we show that there is one optimal log-linear model that both minimizes distinguishability and maximizes log-likelihood.
Consider a log-linear model with features f : X → [0 , 1] d , i.e., d bounded features
f i : X → [0 , 1] .
The model predicts
[Equation]
where (cid:104)· , ·(cid:105) denotes inner product, θ ∈ R d is a parameter vector and Z θ = (cid:80) x exp (cid:104) θ, f ( x ) (cid:105) is a normalizing constant called the partition function.
In the unigram example, the features f i would be word counts normalized by dividing by the maximum allowed sentence length (to ensure f i ( x ) ≤ 1 ).
In a neural network the features f i would correspond to the top layer and q computes a softmax.
Multiple strategies have been studied for computing or estimating the partition function Z θ [see, e.g., 14].
As discussed earlier, these feature functions can also be thought of as classiﬁers that distinguish examples drawn from p from those drawn from q θ and the advantage of f i is α ( f i )
= (cid:80) x
f i ( x )( q θ ( x ) − p ( x )) .
The advantage vector is α ( f )
=
(cid:104)
α ( f i ) (cid:105) di
=1 .
Note that a negative advantage can be used for distinguishing by using the reverse classiﬁer 1 − f i as a distinguisher, which has opposite advantage α (1 − f i ) = − α
( f i ) .
Observation 1.
The gradient of D KL ( p (cid:107) q θ ) with respect to θ is the advantage vector α ( f ) , i.e., for all
i = 1 , 2 , . . .
, d :
[Equation]
The above straightforward calculation is well-known as is the fact that D KL ( p (cid:107) q θ ) is convex in θ .
However, we interpret this fact in the context of GANs: searching for θ which gives a zero-gradient for KL divergence is equivalent to ﬁnding θ which is indistinguishable with respect to each f i .
While a number of GANs have be designed in various architectures that solve the seemingly more complex problem of min θ d ( q θ ) , it can generally be more efﬁcient to maximize likelihood, which (approximately) minimizes the KL divergence.
Distinguishability is equivalent to increasing likelihood for general F , Q
In this section, we show how reducing log-loss is equivalent to distinguishing real and generated examples.
This is the basis behind a single step of our main algorithm (the reduction in this section is efﬁcient for a single step, but the increase in runtime would lead to a general exponential-time algorithm).
The bounds here are in terms of log-loss, as measured on a speciﬁc sample, rather than the abstract KL divergence quantity of the previous section, which cannot be computed exactly using a ﬁnite sample.
In particular, we show how, if one can distinguish a given distribution from the sample, then one can decrease that distribution’s log-loss, and vice versa.
For the remainder of this section, we drop θ from the variable denoting the ﬁtted distribution q θ to avoid cluttering the notation.
Fix a sample S =
(cid:104) x 1 , . .
, x n (cid:105) ∈
X n of n training examples drawn from p , and deﬁne the log-loss to be:
[Equation]
where we use hat on L to denote that the loss is estimated on a (ﬁnite) training set S .
Likewise, ˆE S [ g ( x )] denotes the empirical expectation 1 n (cid:80) ni
=1 g ( x i ) .
Note that the expected log-loss over training sets is known as the cross-entropy
[Equation]
and hence the expected difference in log-loss between two candidate distributions q and q (cid:48) is equal to the difference
[Equation]
so minimizing log-loss approximately minimizes the KL divergence.
Also, we deﬁne the training advantage of distinguisher f : X → [0 , 1] to be:
[Equation]
which is independent of p , depending on the sample alone and can thus be estimated to arbitrary accuracy using samples generated from q .
The lemmas below show how one can use a distinguisher to reduce log-loss on the same training sample, and how to use a distribution with a lower log-loss to distinguish the two distributions.
Lemma 1.
Let a ≥ 0 and suppose f : X → [0 , 1] has training advantage ˆ α ( f ) ≥ a.
Then, the probability distribution q (cid:48)
( x ) =
q ( x ) e − af ( x ) /Z
q (cid:48) where Z q (cid:48)
=
(cid:80) x
q ( x ) e − af ( x ) , has lower log-loss:
[Equation]
Before we give the proof, we note that if f is computed by a neural network and q is computed as a neural network with a softmax at the top layer, i.e., q ( x ) =
e (cid:104) [Equation] ,g ( x [Equation]
/ (cid:80) x e (cid:104) v,g ( x ) (cid:105) where g : X → R d is some neural network, then q (cid:48) is naturally represented as the combined neural network with softmax q (cid:48)
( x ) ∝ e (cid:104) ( v, − a ) , ( g ( x ) ,f ( x )) (cid:105) in d + 1 dimensions.
This means that if we can distinguish S from q , then we can simply reduce log-loss by down-weighting suspicious samples that satisfy the distinguisher f ( x ) .
The difference between this statement and
Observation 1 is analogous to the difference between boosting and logistic regression [16].
In logistic regression, one typically ﬁxes the set of features in advance, whereas in boosting this is not possible if there are inﬁnitely many possible classiﬁers.
Conversely, we next show that if q (cid:48) has a lower log-loss than q on the training samples, then we can distinguish q from these samples.
Lemma 2.
For any constant C > 1 and distributions q, q (cid:48) such that 1 C q ( x ) ≤ q (cid:48)
( x ) ≤ Cq ( x ) for all x ∈ X , the distinguisher f : X → [0 , 1] deﬁned by,
[Equation]
has a training advantage of,
[Equation]
Importantly, due to the logarithmic dependence on C , the above lemma is meaningful even if q and q (cid:48) are exponentially far apart so long as they have the same support.
Lemma 1 implies a reduction between the problem of distinguishing with nontrivial advantage to non-trivially reducing log-loss for log-linear families.
Note that iteratively applying the reduction requires repeated computation of the normalization terms over X , and computing such partition functions is an area of active research—where it is known how to do it efﬁciently for some classes and not for others.
The next section gives an efﬁcient reduction for (unidirectional) sequential models.
Efﬁcient Reduction for Sequential Models
This section gives an efﬁcient reduction from distinguishing to MLE for sequential models.
This requires showing how one can efﬁciently compute the normalization terms (partition function) on a token-by-token basis for black-box sequential (e.g., language or auto-regressive) models.
The key insight for efﬁciency is that, rather than distinguishing entire sequences from p and q , one distinguishes the conditional next-token predictions.
In particular, rather than generating entire sequences from q , one can generate next-token predictions on all sequence preﬁxes in the training data.
Clearly, evaluating a neural network over all sequences is infeasible.
However, in many applications such as NLP, the inputs are sequential x = ( x 1 , . . .
, x (cid:96) ) , where every token x i is taken from a large discrete vocabulary.
In such cases, the combinatorial nature of the data makes density estimation intractable unless the likelihood computation is broken into small sequential steps by representing the overall probability as the Pr( x ) = (cid:81)
j Pr( x j | x 1 , x 2 , . . .
x j − 1 ) .
In this section we show how a natural extension of the framework described above allows us to achieve an efﬁcient reduction for this common type of sequential model.
To do so, we deﬁne a simple generalization of the training advantage criterion (2), which now relies on a step-wise distinguisher g operating on variable-length sequences.
Formally, we consider a language of N -length sequences 3 of tokens taken from a vocabulary V , and distinguisher functions g : (cid:83) Nj =
V
j
→ [0 , 1] , i.e., functions which can take subsequences of any size as input.
Given a sample S of sequences, we say that g has generalized training advantage given by
[Equation]
where, by convention, x 0 = ∅ , so that q ( x 0 , w ) =
q ( w ) .
This criterion can be interpreted as follows.
For every length j ∈ { 1 , 2 , . . . , N } , g is tasked with distinguishing a subsequence consisting of the ﬁrst j tokens in a true sequence sampled from S from another j -length sequence in which the last element is replaced by a randomly selected token from the alternate distribution q .
Lemma 3.
Let b ≥ 0 and suppose g : (cid:83) Nj =
1 V j
→ [0 , 1] has generalized training advantage ˆ β ( g ) ≥ b .
We deﬁne a distribution q (cid:48) through its conditional probabilities as:
[Equation]
where now Z q (cid:48) ( x 1 , . .
x j − 1 ) = (cid:80) ˜ x
j q (˜ x j | x 1 , . .
, x j − 1 ) e − bg ( x 1 ,...,x j − 1 , ˜ x j ) .
Then q (cid:48) incurs lower log-loss than q : ˆ L (
[Equation]
≤
[Equation]
− Nb 2 / 2 .
The proof is deferred to Appendix C.
Next, we use Lemma 3 repeatedly to derive a simple algorithm that, given access to non-trivial weak distinguishers, returns a distribution that is nearly indistinguishable (by that class) from the true distribution p .
Formally, let G =
{ g | g : (cid:83) Nj =1 V j → [0 , 1] } be a class of distinguishers.
We assume access to an oracle O d : Q (cid:55)→ G which for any q ∈ Q returns a distinguisher g .
In practice, such as in typical GAN training setting, one could think of this oracle as being approximated by the subroutine that trains the discriminator.
We say that q is (cid:15) -indistinguishable by oracle O d if its output g has advantage ˆ β ( g, S , q ) ≤ (cid:15) .
We do not need to assume that O d is optimal in any sense.
Algorithm 1: t ←while True do g t ← O d ( q t ) b t ← ˆ β ( g t , S , q t ) if b t < (cid:15) , Output: q t
Compute q t +1 ( x ) (cid:44) q t ( x ) e − b t g t ( x ) / (cid:80) x ∈S q t ( x ) e − b t g t ( x ) on entire corpus S t ← t +end
Theorem 1.
Let q 0 be a language model and let (cid:15) > 0 .
Algorithm 1 returns a distribution q ∗ which is (cid:15) -indistinguishable from S by oracle O d .
It runs in O (cid:0) 1 (cid:15) 2 L 0 ( T d N + nT g ( m + n )) (cid:1) time, where
L
= ˆ L ( q 0 ; S ) is the log-loss of q 0 , T d is the runtime of oracle O d , T g is the complexity of evaluating any distinguisher g on a single input, n = |V| is the vocabulary size, N is the sequence length and m = |S| is the number of training sequences.
Thus, Algorithm 1 combined with Theorem 1 and Lemma 3 yields a polynomial-time reduction from distinguishing distributions to maximum likelihood estimation for sequential models.
An empirical validation of the reductions
We provide an empirical validation of Lemma 3 in the following realistic (albeit simpliﬁed) setting.
We train a pair of text generator and discriminator using a publicly available implementation 4 of
SeqGAN [63].
The generator is pretrained by negative log-likelihood (NLL) minimization.
During the adversarial phase of training, the generator is trained using policy gradient.
After training, we compute the discriminators’ generalized training advantage (Eq.
(4), using ﬁnite-sample empirical approximations), and then create a new generator whose next-word logit predictions are modiﬁed according to Lemma 3.
We compare the NLL of the original and ’boosted’ generators across training epochs, and compare the difference between these to the theoretical lower bound of Lemma 3.
The results, shown in the Supplementary, correspond to the default SeqGAN settings in terms of network capacities and language conﬁguration (max. seq.
len.
=20, vocab=5000).
They show that the ’boosting’ underpinning Lemma 6 does indeed improve likelihood (reduce NLL) as stated, and that the empirical difference is indeed lower-bounded by the gap predicted by theory.
Discussion and conclusions
In this work, we have argued that minimizing log-loss (i.e., KL-divergence) and minimizing statistical distinguishability are tightly related goals.
Speciﬁcally, if the families of distinguishers and probability distributions are of similar power, then one can use a distinguisher to reduce log-loss.
This is the case, e.g., for n-gram language models (and other sequential tasks), for which perplexity (a measure of likelihood) is easy to compute, naturally meaningful, and allows for efﬁcient sampling.
Indeed, for a long time, minimizing log-loss has been the objective with which most state-of-the-art models have been trained.
For such models, Lemma 1 implies that if one can distinguish the model from samples by a neural network then one can construct a larger neural network with lower log-loss.
Hence, one may prefer to simply train a larger model in the ﬁrst place for some applications.
Text vs. images.
Note that we often can compute conditional likelihoods more efﬁciently for text data compared to images.
The sequential nature of text data allows us to compute the normalization terms (and thus the partition function) on a token-by-token basis, thereby enabling us to distinguish the conditional next-token predictions instead of having to distinguish full sentences.
Similarly, for low dimensional images (such as 8x8, or 4x4), conditional predictions are tractable and thus autoregressive modeling allows for efﬁcient training and sampling.
In contrast, MLE based autoregressive models are typically slow for high dimensional real images.
In such settings estimating the partition function is challenging, so alternative methods such as noise contrastive estimation, score matching, Langevin dynamics, and MCMC sampling in latent space that exploit connections between GANs and energy based models have been preferred [38, 13, 65, 15, 9, 51].
We hope this work fosters further research on the comparative aspects, both pros and cons, of different generative approaches for different NLP and vision applications. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets..
(a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Lemma
Proof (Lemma 1).
[Equation]
Since ˆ α ( f ) ≥ a by assumption, it remains to show a E q [ f ( x )]
+ log Z q (cid:48) ≤ a 2 / 2 .
Using the bound log r ≤ r − 1 for any r > 0 , we get that,
[Equation]
where we have used the fact that Z q (cid:48)
=
E q [ e − af ( x ) ] and, to get to the last line we use e − r + r − 1 ≤ r 2 / 2 for r ≥ 0 by Taylor expansion.
Since f ( x ) ∈ [0 , 1] , the last quantity is at most a 2 / 2 , which together with (4), gives ˆ L ( q ; S ) − ˆ L ( q (cid:48) ; S ) ≥ a 2 / 2 .
B Proof of Lemma
Proof (Lemma 2).
Let g ( x ) = log q ( x ) − log q (cid:48)
( x ) .
By Jensen’s inequality,
[Equation]
( x ) q ( x ) (cid:21) − ˆE S [ g ( x )] ≥ − log E q (cid:20) q (cid:48)
( x ) q ( x ) (cid:21) − ˆE S [ g ( x )]
= − log(1)
− ˆE S [ g ( x )]
= E S [log q (cid:48) ( x )] − ˆE S [log q ( x )] = ˆ L ( q ; S ) − ˆ L ( q (cid:48) ; S ) Since f ( x ) = 1 2log C ( g ( x ) + log C ) , the training advantage of f is that of g scaled by a factor of 1 2log C .
Finally, it is straightforward to verify that f ( x ) ∈ [0 , 1] by our assumptions on the ratio between q and q (cid:48) .
C Proof of Lemma
Proof.
We proceed analogously as in the proof of Lemma 1.
We ﬁrst note that ˆ L ( q ; S ) =
− ˆE S (cid:34) log
N (cid:89) i = q ( x i | x 1 , . . .
, x i − 1 ) (cid:35) = −
N (cid:88) i =
ˆE S log q ( x i | x 1 , . . .
, x i − 1 ) , and ˆ L ( q (cid:48) ; S )
= − ˆ E S (cid:34) log
N (cid:89) i = q (cid:48) ( x i | x 1 , . . .
, x i − 1 ) (cid:35) =
N (cid:88) i = − ˆE S log q ( x i | x 1 , . . . , x i − 1 )
+ bg ( x 1 , . . , x i ) + log Z q (cid:48) ( x 1 , . . . , x i − 1 )
Let us use the short-hand notation x 1: i (cid:44) ( x 1 , . . .
, x i ) .
Subtracting the two equalities above we obtain
[Equation]
which, after adding and subtracting ˆE S E w ∼ q ( w | x
[Equation]
i − 1 ) g ( x 1: i − 1 , w ) and rearranging terms, yields
[Equation]
By assumption we have Nb ˆ β ( g ) ≥ Nb 2 , so it it remains to show that the second term is upper bounded by Nb 2 / 2 .
Using, as before, the bound log r ≤ r − 1 for every r =
Z q (cid:48) ( x 1: i − 1 ) ≥ 0 , we get that, for every i = 1 , . . .
, N :
[Equation]
where the last inequality follows again from the fact that g ( x ) ∈ [0 , 1] for any x .
Therefore, the sum over these N terms is upper bounded by N b 2 2 , which combined with (7), yields the desired result.
D Proof of Theorem
Proof.
The fact that Algorithm 1 terminates with a distribution q which is (cid:15) -indistinguishable by O d is immediate from the stopping criterion.
Now, for the runtime analysis, note that —by construction— the iterates g t , t ∈ { 0 , . .
, T − 1 } have training advantage ˆ β ( g t , S , q t ) ≥ (cid:15) .
Thus, by Lemma 3, the algorithm makes at least N(cid:15) 2improvement in each iteration.
Therefore, the total number of iterations T is at most 2 L 0 N(cid:15) 2 , where
L 0 :
= ˆ L ( q 0 ; S ) is the log-loss of the initial model.
Each iteration of Algorithm 1 requires calling O d oracle once, evaluating ˆ β ( · ) at an O ( NmnT g ) complexity, and updating each of the n next-token probabilities of q for each sequence length 1 , . .
. , N .
Each of these updates involves evaluating g plus an O ( n ) partition normalization.
Putting these together, we conclude that each iteration has
O ( T d + NnT g ( m + n )) complexity.
Combining the the two arguments above, we conclude that Algorithm 1 has a total runtime of
O (cid:0) 1 (cid:15) 2 L 0 ( T d N + nT g ( m + n )) (cid:1) .
E Empirical validation
[Table]
[-Epoch | Gen. Epoch | Gen. NLL | ’Boosted’ Gen. NLL | Improvement | Improvement predicted by bound0 0 91.2756 91.1459 0.129694 0.008420831 1 91.2995 91.182 0.117515 0.007330522 2 91.3425 91.2238 0.118727 0.008349043 3 91.4 91.2872 0.112812 0.006839754 4 91.4663 91.3402 0.126077 0.009614825 5 91.5386 91.4103 0.12824 0.007692216 6 91.6125 91.4928 0.11971 0.007446017 7 91.6866 91.5565 0.130122 0.008360848 8 91.7617 91.6169 0.144865 0.009939029 9 91.8349 91.7039 0.131023 0.0081801810 10 91.9048 91.7848 0.11997 0.0073044111 11 91.9714 91.8493 0.122075 0.0084458812 12 92.0345 91.8981 0.136373 0.0092132513 13 92.0937 91.9648 0.128912 0.0083249114 14 92.1491 92.0212 0.127937 0.0084263315 15 92.2005 92.0755 0.125091 0.007911716 16 92.2481 92.1153 0.132838 0.010233417 17 92.2919 92.1581 0.133824 0.0077329618 18 92.3321 92.2087 0.123441 0.0072073419 19 92.3676 92.237 0.130602 0.
-][Table]