Are GANs overkill for NLP? Anonymous Author(s)
AfﬁliationAddress
Abstract
This work offers a novel theoretical perspective on why, despite numerous attempts,adversarial approaches to generative modeling (e.g., GANs) might not have notbeen as successful for certain generation tasks, particularly sequential tasks suchas Natural Language Generation, as they have in others, such as Computer Vision.
In particular, on sequential data such as text, maximum-likelihood approachesare signiﬁcantly more utilized than GANs.
We show that, while it may seem thatmaximizing likelihood is inherently different than minimizing distinguishability,this distinction is largely an artifact of the limited representational capacity of themodel family, for a wide class of adversarial objectives.
We argue that minimizing KL-divergence (i.e., maximizing likelihood) might be a more efﬁcient approach toeffectively minimizing the same distinguishability criteria that adversarial modelsseek to optimize.
Reductions show that minimizing distinguishability can be seen assimply boosting likelihood for certain families of models including n-gram modelsand neural networks with a softmax output layer.
To achieve a full polynomialtime reduction, a novel next-token distinguishability model is considered.
Somepreliminary empirical evidence is also provided to substantiate our theoreticalanalyses.
Introduction
Consider a situation where one has samples from a true distribution p over a set X and one wishesto learn to generate similar samples such as learning to generate English sentences from a large
English text corpus.
One seeks an approximation q of p which is “close” in some sense and fromwhich samples can efﬁciently be generated.
A common approach to ﬁt these models is Maximum
Likelihood Estimation (MLE), which given a training set from p and a parametrized distribution q θseeks parameters θ that maximize the likelihood q θ assigns to a training set.
MLE has long been oneof the most popular methods for ﬁtting generative models of sequential data, such as language, whereautoregressive neural language models generate remarkably realistic text, e.g., GPT-3 [6] and PaLM[11].
MLE generally involves computing likelihoods q θ ( x ) which can be more challenging in somedomains than others, e.g., it may be more difﬁcult to estimate the probability of a (high-dimensional,real-valued) image than a (discrete-valued) sentence.
An alternative approach, Generative Adversarial Networks (GANs), has become popular acrossseveral domains, particularly Computer Vision, owing to breakthrough realism in the images theyoutput [e.g., 19, 65].
GANs employ an adversarial approach to generation through a zero-sum gamebetween a generator and a distinguisher in which the generator produces samples x ∈ X whichthe distinguisher tries to distinguish from real samples from p .
Often, both the generator and thedistinguisher are differentiable neural networks, though this min-max approach of choosing a modelwhose outputs are nearly indistinguishable from real examples might be considered for any familiesof generative models and distinguishers.
A major advantage of GANs (particularly for images) isthat they can be used for generation without requiring computing likelihoods.
This advantage is not signiﬁcant for many sequential models such as language models, where computing likelihoods is notdifﬁcult.
In contrast, the adversarial approach has yet to demonstrate signiﬁcant improvements in some otherdomains such as Natural Language Processing (NLP).
One well-known barrier to NLP GANs isthat language models produce discrete outputs (words), so they are not naturally differentiable [18].
However, despite numerous works circumventing this limitation and adapting GANs to text generation[63, 42, 23, 12], adversarial-based models have yet to achieve the same popularity or performancegains that were seen for images.
In particular, language GANs have been shown to under-perform
MLE in terms of quality [55] while facing the challenge of lack of diversity due to mode collapse [7],which is a well-known issue with GANs in other domains.
1.1 Likelihood and Distinguishability: Two sides of the same coin?
In this work, we suggest a different, fundamental barrier to adopting GANs in domains where MLEis prevalent: the adversarial approach of minimizing distinguishability can be seen as an indirectmethod of maximizing likelihood on observed data, and hence employing MLE directly can be moreefﬁcient.
This is the case in NLP where, unlike computer vision, a measure of likelihood calledperplexity has been the prevailing metric for training and evaluating language models for decades.
We show how GANs boost likelihood in the spirit of, and inspired by, the related celebrated work of
Friedman et al.
[16] that showed how boosting can be viewed as an iterative approach for logisticregression.
Consider a large ﬁnite set or countably inﬁnite set X and a family Q of probability distributionsover X .
For language, these might be n-gram models or neural models.
Also consider a family Fof distinguishers f : X → [0 , 1] that aim to distinguish random examples drawn from a distributionp from those sampled from q .
For any such classiﬁer f , we call the difference α ( f ) =
E q [ f ( x )] −
E p [ f ( x )] the distinguishability advantage of f because it quantiﬁes the accuracy of f at the task ofidentifying “fake” examples.
A perfect distinguisher would thus have α ( f ) = 1 , while f ( x ) = 1 / 2which predicts at random has α ( f ) = 0 .
More formally, imagine picking y ∈ { 0 , 1 } uniformly atrandom and picking a random example x from q if y = 1 and from p
if y = 0 .
The (randomized)binary classiﬁer that predicts, for any x , ˆ y = 1 with probability f ( x ) , has (expected) accuracy:
[Equation]
Given a family F , we deﬁne the distinguishability of q from p to be d ( q ) = max f ∈ F α ( f ) .
Distinguishability is known to be a lower-bound on total variation distance (also called statistical distance),a measure of distance between distributions that is difﬁcult to directly estimate for large domains
X [54].
The “Bayes-optimal” distinguisher simply predicts 1 iff q ( x ) > p ( x ) , and has advantageequal to the total variation distance [see, e.g., 24].
Clearly d ( p ) = 0 , i.e., p is indistinguishable fromitself.
Motivated by this observation, numerous adversarial approaches to approximating q have beenattempted to minimize distinguishability d ( q ) .
If p ∈ Q then d ( q ) is minimized at q = p .
We ﬁrstdiscuss some important aspects of distinguisability. What adversarial objectives can be analyzed via distinguishability? It is important to emphasize that distinguishability is indeed the objective that several adversarial approaches, such as many variants of GANs, seek to optimize. Depending on the context, it serves the purpose of discriminator or critic .
In particular, as established in [54], GANs that are trained based on Kantorovich metric,
Fortet-Mourier metric, dual-bounded Lipschitz distance (or the Dudley metric), total variation distance, and kernel distance can all be cast in terms of distinguishability.
Thus, in particular, our results hold for GNN formulations such as Wasserstein GANs [1], MMD GANs [33], Fisher GANs [36], and Sobolev GANs [49], for an appropriately chosen family F of distinguishers.
For example, we obtain Wasserstein GANs, in its dual form, as a special case when F is restricted to 1-Lipschitz functions in which case it can also be viewed as a special case of the so-called f-GANs [3, 37, 50].
Likewise, we obtain MMD-GANs when F pertains to functions (kernels) deﬁned over a ball in some
Reproducing Kernel Hilbert Space [4].
Note that distinguishability allows us to accommodate other sophisticared GAN variants such as WGAN-GP [22, 60] that do not suffer from the issue of gradients vanishing on discrete spaces.
Recall the WGAN-GP objective can be expressed in our notation as:
[Equation]
This can be viewed as Lagrangian relaxation of the following hard objective for (cid:15) > 0 as:
[Equation]
Distinguishability advantage can then be readily be expressed as [Equation] Is distinguishability symmetric or asymmetric? Note that, by deﬁnition, distinguishability isasymmetric in the sense that in general q is distinguishable from p is different from p is distinguishablefrom q .
Note, however, that we recover integral probability metric (IPM) [54] when − f ∈ F for allf ∈ F .
Clearly, in this case the notion of distinguishability becomes symmetric as the advantagereduces to max f ∈ F | E q [ f ( x )] − E p [ f ( x )] | .
Thus, distinguishability lets us handle an extremelywide class of discrepancies, symmetric as well as asymmetric.
Example where maximizing likelihood (cid:54) = minimizing distinguishability.
When p (cid:54)∈ Q , minimizing distinguishability among q ∈ Q may be different than maximizing the likelihood of q .
Forinstance, consider modeling the age in years of humans (say the entire population on earth) as auniform distribution q m over x ∈ { 0 , 1 , 2 , . . .
, m } .
Now, the m which maximizes likelihood wouldbe the age of the oldest person, which is m = 119 at the time of this article—any smaller m wouldassign zero probability to the 119-year-old and thus to the entire population.
However, this distributionis very distinguishable from the true distribution—for instance it assigns probability ∼ 17% to beingover 100 years, which is extremely unlikely among today’s population.
A smaller m < 100 wouldyield less distinguishable samples.
While it may seem therefore that distinguishability and likelihoodare inherently different criteria, as we shall see this is an artiﬁcial limitation due to the weakness offamily Q .
Of course, the (in)equivalence depends on the families F of distinguishers and Q of probability distributions.
We give two results showing that maximizing likelihood and minimizing distinguishabilityare equivalent as long as F and Q are similar in representational capacity, even when p (cid:54)∈ Q .
First,we consider families Q that are “log-linear” over some set F of functions, which include n-grammodels and neural networks whose top layer is a softmax, among others.
The equivalence in thiscase is particularly simple and serves to illustrate how MLE can be a simpler way to reach the sameoptimum.
In this case, Q and F are naturally paired.
Maximizing likelihood =
minimizing distinguishability for log-linear Q .
In the above age example, the family Q of geometric distributions q θ ( n ) ∝ exp( − θn ) for θ > 0 is an example of alog-linear family.
We show that if q can be distinguished from the population distribution p by afunction f ∈ F , then folding f into q yields a new model in Q with greater likelihood.
In practice, oneonly has a sample of the true distribution p (not the entire population) and maximizing log-likelihoodis approximation of minimizing KL divergence D KL ( p (cid:107) q ) =
(cid:80) x p ( x ) log p ( x ) q ( x ) .
We give formalstatements about minimizing KL-divergence as well.
The conclusion of this ﬁrst observation is that if a GAN were to converge within a log-linear family(and making GANs converge is often not an easy feat in practice), it would converge to the MLE.
General polynomial-time reduction.
Our second result is a polynomial-time reduction fromlikelihood maximization to next-token distinguishability, without the log-linear requirement.
Weconsider the common case of (unidirectional) sequential models that predict the next token based onthe previous tokens, which have several practical advantages including being efﬁcient to compute—the probability of a sequence is simply the product of the conditional probabilities of each subsequentword given the previous words.
Many state-of-the-art transformer language models such as GPT-3take this form.
Achieving an efﬁcient reduction is challenging due to the normalization requirementof computing partition functions.
In order to achieve a polynomial-time reduction, we consider anotion of next-token distinguishability, where the game is as follows: a preﬁx of tokens is chosen,based on which the generator generates a token to follow the preﬁx.
Given the actual next token andthe generated next token, the distinguisher aims is to identify which is which.
Algorithm 1 leveragesa next-token distinguisher to iteratively increase likelihood.
In particular, given any target (cid:15) > 0 ,
Theorem 1 shows that Algorithm 1 will terminate and output an efﬁciently computable model qwhich is nearly (to within (cid:15) ) indistinguishable from the truth, and it runs in time polynomial in 1 /(cid:15) .
If p ∈ Q and one has an optimal distinguisher, one will eventually converge to a model close to p , ashas been discussed heavily in the literature.
However, our results are also meaningful in the morerealistic case where one has imperfect distinguishers.
Contributions .
The main contributions of this paper are:
• showing that, although in general minimizing distinguishability and maximizing likelihood
seem to be different, they are in fact closely related,
• introducing a new model of next-token distinguishability that is necessary to make thereduction efﬁcient, and
• offering a new perspective on why GANs might have been less successful in NLP and other
sequential domains as they have been, e.g., for images.
Organization .
We begin by summarizing related work on GANs, especially for text.
We thenillustrate how GANs can be overkill for the simple case of n-gram models in Section 3.
Section 4covers log-linear models.
Section 5 gives explicit bounds on general reductions between maximizinglikelihood and minimizing distinguishability.
Section 6 shows how the reduction can be efﬁcientlycomputed in the case of sequential inputs, from which we propose a simple polynomial time algorithmthat provably ﬁnds a distribution which is nearly-indistinguishable with respect to a given class ofdiscriminator functions.
Finally, we discuss the relevance of our work in Section 8.
We delegate allthe proofs to the Supplementary.
Related Work
Generative models .
Several approaches to generative modeling have been investigated, especially inthe context of images.
In particular, impressive results have been obtained recently with variationalautoencoders, GANs, normalizing ﬂows, autoregressive models, diffusion processes, and score/energybased models [28, 19, 41, 40, 25, 53, 59, 10].
Generally, training approaches are either adversarial;or rely on MLE, contrastive divergence estimation, or score matching [52].
Some connections havebegun to emerge between these models, and alternate training procedures have been advocated [9, 53,62].
A word of caution: Understanding the theoretical underpinnings of generative models with respectto their sample quality is an intriguing question that has been previously investigated by severalseminal works such as [27, 21, 39, 56] and requires further analyses.
Our objective here is not toclaim at all that maximizing likelihood is universally better than adversarial methods or vice-versa,but to emphasize that for many problems, in domains like NLP, the two objectives often turn out tobe equivalent mathematically via the notion of distinguishability and maximizing MLE could providea more efﬁcient (and stable way) of optimizing the common objective.
Also, note that maximizinglikelihood does not always correlate with the perceptual sample quality [21].
A more comprehensiveanalysis encompassing the effect of architecture, optimization procedures and issues such as trade-offbetween perception and distortion [5], training and inference time, mode-collapse etc. is required forbetter understanding of generative models.
In that sense, our message is consistent with observationsmade previously [39, 56, 21] that there is no single model that ﬁts all situations, and the right modeldepends on the speciﬁc requirements of applications.
GANs for text .
Since their introduction [19], there has been interest in adapting GANs to textdata.
The driving motivation was that —up until very recently— samples generated by traditional(likelihood-based) models had been easy to distinguish from human-generated text, and the successof image GANs at generating realistic-looking samples suggested a possible avenue to improve thequality of their natural language counterparts.
The ﬁrst and most signiﬁcant challenge in adapting GANs to text arises from the very nature ofthis data.
Goodfellow [18] points out that GANs require the generator to be differentiable, whichposes a challenge for discrete text representations such as one-hot word or character representations.
Two of the most popular approaches to circumvent this obstacle are policy gradient techniques (e.g.,
REINFORCE [61]) —which when applied to language modeling nevertheless often require maximumlikelihood pre-training [8, 63])— and the Gumbel-Softmax approximation [30].
The few adversarialmethods that do not require pre-training (e.g., [42, 45]) have failed to show signiﬁcant promise in allbut a artiﬁcial tasks
This nascent but active line of work seemed to suggest for a period of time that GANs might providea breakthrough in text generation.
This promise did not fully materialize, and instead the most recentbreakthrough came from models building very large transformer-based architectures like GPT [43,44, 6] or PaLM [11] — which are trained with traditional cross-entropy (MLE) objectives.
Yet the question of how GAN-based methods for text compare with likelihood-based ones still garnerssigniﬁcant interest, and while various works have provided an empirical comparison between them—with most of these suggesting the advantage of MLE-based ones [7]— theoretical explanations havebeen less explored.
Relating objectives via divergences .
The connection between maximum likelihood estimation,distinguishability and divergences between probability distributions has been explored before.
Forexample, it is well known that maximizing likelihood is equivalent to minimizing the KL divergencebetween certain families of ﬁtted and reference distributions, though this is not the only divergencefor which such a connection exists [46].
On the other hand, from the moment GANs were introduced,
Goodfellow et al.
[19] noted that —assuming a perfect discriminator— the adversarial objectivecorresponds to minimizing a Jensen-Shannon divergence.
Furthermore, the minimal discriminationerror is also directly related to the total variation distance (see, e.g., Hashimoto et al.
On theother hand, for exponential families the gradient of the KL divergence is known to be related to thediscrepancy between distributions [57].
While conceptually similar to this line of work, here insteadwe give an explicit reduction that shows how distinguishability and (log) likelihood are in directcorrespondence.
Pinsker’s inequality is a well-known result linking KL divergence and total variation distance (TVD):
TVD ≤ (cid:112) KL / 2 .
While related, this inequality is not directly relevant to the context of this work.
First, while total variation provides an upper bound to distinguishability, it is not computable ingeneral, so it is rarely used as a training objective for generative models.
On the other hand, beingone-sided, 1 it does not imply that reducing TVD reduces KL divergence.
Furthermore, Pinsker’s isin general a very loose inequality, particularly for the direction of KLD that is equivalent to MLE(i.e.
, D KL ( p (cid:107) q θ ) ), since if p ( x ) > 0 ≈ q θ ( x ) even for a single x leads to unbounded KL divergence.
In contrast, in this work we provide a direct reduction directly linking the two criteria of interest:distinguishability and maximum likelihood.
Log-linear language models .
In this work we focus our analysis on log-linear models [31, 34],which are widely used in natural language processing (often known in that community as Maximum
Entropy –MaxEnt– models) for various tasks.
In particular, these models have been a cornerstone ofboth neural [2, 35] and non-neural [47, 26] language modeling.
Boosting .
The reduction shown here bears resemblance to boosting.
It is well-known that boostingcan be analyzed through the lens of maximum likelihood (Friedman et al.
[16]), while Lebanonand Lafferty [32] formalized the equivalence of AdaBoost and maximum likelihood training forexponential models.
More recently, boosting has been applied to generative adversarial models [58,20], albeit with a different approach and objective than the connection drawn in this work.
To illustrate our main point, consider ﬁrst the simplest model of language: a unigram model where the probability of each word is independent, and the end of sentence token EOS has a given probability as well.
If θ w represents the log-probability of word w , then the log-probability of sentence w 1 . . .
w t is given by:
[Equation]
The MLE parameters θ ∗ can be computed in linear time by simply counting word frequencies.
A more roundabout approach to ﬁtting a unigram language model would be to start with any initialunigram model q , generate random samples from q and compare them to those from p .
One couldthen distinguish the two by ﬁnding a word that appears signiﬁcantly more often in one than in theother.
For example, if one generates text from the model q and ﬁnds that the word “the” occurs muchmore often in text generated from p , one would then update the parameters by increasing θ the (and decreasing θ w (cid:48) for all other words w (cid:48) to keep q a probability distribution).
As we shall see later, if thismore involved procedure converged, it would necessarily converge to the same maximum-likelihoodestimator θ ∗ .
A similar argument applies to any n -gram model in which the probability of each subsequent word isdetermined only by the previous n − 1 words.
This is also optimized by frequency counts (a varietyof “smoothing” techniques, e.g., adding 1 to counts, also known as Laplace Smoothing [17, 29] areoften used as a form of regularization on top of these counts).
Distinguishers could similarly be usedto ﬁnd a model q that is indistinguishable from p according to n -gram frequencies, but again thiswould simply converge to the MLE parameters.
Equivalence for log-linear models
In this section, we show that there is one optimal log-linear model that both minimizes distinguishability and maximizes log-likelihood.
Consider a log-linear model with features f : X → [0 , 1] d , i.e.,d bounded features
f i : X → [0 , 1] .
The model predicts
[Equation]
where (cid:104)· , ·(cid:105) denotes inner product, θ ∈ R d is a parameter vector and Z θ = (cid:80) x exp (cid:104) θ, f ( x ) (cid:105) is anormalizing constant called the partition function.
In the unigram example, the features f i would be word counts normalized by dividing by themaximum allowed sentence length (to ensure f i ( x ) ≤ 1 ).
In a neural network the features f i wouldcorrespond to the top layer and q computes a softmax.
Multiple strategies have been studied forcomputing or estimating the partition function Z θ [see, e.g., 14].
As discussed earlier, these feature functions can also be thought of as classiﬁers that distinguish examples drawn from p from those drawn from q θ and the advantage of f i is α ( f i ) =
241 (cid:80) x f i ( x )( q θ ( x ) − p ( x )) .
The advantage vector is α ( f ) = (cid:104)
α ( f i ) (cid:105) di =1 .
Note that a negative advantage can be used for distinguishing by using the reverse classiﬁer 1 − f i as a distinguisher, which hasopposite advantage α (1 − f i ) = − α
( f i ) .
Observation 1.
The gradient of D KL ( p (cid:107) q θ ) with respect to θ is the advantage vector α ( f ) , i.e., for all i = 1 , 2 , . . .
, d :
[Equation]
The above straightforward calculation is well-known as is the fact that D KL ( p (cid:107) q θ ) is convex in θ .
However, we interpret this fact in the context of GANs: searching for θ which gives a zero-gradientfor KL divergence is equivalent to ﬁnding θ which is indistinguishable with respect to each f i .
While a number of GANs have be designed in various architectures that solve the seemingly morecomplex problem of min θ d ( q θ ) , it can generally be more efﬁcient to maximize likelihood, which(approximately) minimizes the KL divergence.
Distinguishability is equivalent to increasing likelihood for general F , Q
In this section, we show how reducing log-loss is equivalent to distinguishing real and generatedexamples.
This is the basis behind a single step of our main algorithm (the reduction in this sectionis efﬁcient for a single step, but the increase in runtime would lead to a general exponential-timealgorithm).
The bounds here are in terms of log-loss, as measured on a speciﬁc sample, rather thanthe abstract KL divergence quantity of the previous section, which cannot be computed exactly usinga ﬁnite sample.
In particular, we show how, if one can distinguish a given distribution from thesample, then one can decrease that distribution’s log-loss, and vice versa.
For the remainder of this section, we drop θ from the variable denoting the ﬁtted distribution q θ toavoid cluttering the notation.
Fix a sample S = (cid:104) x 1 , . .
, x n (cid:105) ∈ X n of n training examples drawnfrom p , and deﬁne the log-loss to be:
[Equation]
where we use hat on L to denote that the loss is estimated on a (ﬁnite) training set S .
Likewise,ˆE S
[ g ( x )] denotes the empirical expectation 1 n (cid:80) ni
=1 g ( x i ) .
Note that the expected log-loss overtraining sets is known as the cross-entropy
[Equation]
and hence the expected difference in log-loss between two candidate distributions q and q (cid:48) is equal tothe difference
[Equation]
so minimizing log-loss approximately minimizes the KL divergence.
Also, we deﬁne the trainingadvantage of distinguisher f : X → [0 , 1] to be:
[Equation]
which is independent of p , depending on the sample alone and can thus be estimated to arbitraryaccuracy using samples generated from q .
The lemmas below show how one can use a distinguisherto reduce log-loss on the same training sample, and how to use a distribution with a lower log-loss todistinguish the two distributions.
Lemma 1.
Let a ≥ 0 and suppose f : X → [0 , 1] has training advantage ˆ α ( f ) ≥ a.
Then, the probability distribution q (cid:48) ( x ) =
q ( x ) e − af ( x ) /Z
q (cid:48) where Z q (cid:48)
= (cid:80) x
q ( x ) e − af ( x ) , has lower log-loss:
[Equation]
Before we give the proof, we note that if f is computed by a neural network and q is computedas a neural network with a softmax at the top layer, i.e., q ( x ) =
e (cid:104) v,g ( x ) (cid:105) / (cid:80) x
e (cid:104) v,g ( x ) (cid:105) whereg :
X → R d is some neural network, then q (cid:48) is naturally represented as the combined neural networkwith softmax q (cid:48)
[-[Equation] where we have used the fact that Z q 
 = E q [ e  af-] ( x ) ∝ e (cid:104) ( v, − a ) , ( g ( x ) ,f ( x )) (cid:105) in d + 1 dimensions.
This means that if we can distinguish S from q , then we can simply reduce log-loss by down-weightingsuspicious samples that satisfy the distinguisher f ( x ) .
The difference between this statement and
Observation 1 is analogous to the difference between boosting and logistic regression [16].
In logisticregression, one typically ﬁxes the set of features in advance, whereas in boosting this is not possibleif there are inﬁnitely many possible classiﬁers.
Conversely, we next show that if q (cid:48) has a lower log-loss than q on the training samples, then we candistinguish q from these samples.
Lemma 2.
For any constant C > 1 and distributions q, q (cid:48) such that 1 C q ( x ) ≤ q (cid:48) ( x ) ≤ Cq ( x ) forall x ∈ X , the distinguisher f : X → [0 , 1] deﬁned by,
[Equation]
has a training advantage of,
[Equation]
Importantly, due to the logarithmic dependence on C , the above lemma is meaningful even if q andq (cid:48) are exponentially far apart so long as they have the same support.
Lemma 1 implies a reduction between the problem of distinguishing with nontrivial advantage tonon-trivially reducing log-loss for log-linear families.
Note that iteratively applying the reductionrequires repeated computation of the normalization terms over X , and computing such partitionfunctions is an area of active research—where it is known how to do it efﬁciently for some classes andnot for others.
The next section gives an efﬁcient reduction for (unidirectional) sequential models.
Efﬁcient Reduction for Sequential Models
This section gives an efﬁcient reduction from distinguishing to MLE for sequential models.
Thisrequires showing how one can efﬁciently compute the normalization terms (partition function) ona token-by-token basis for black-box sequential (e.g., language or auto-regressive) models.
The key insight for efﬁciency is that, rather than distinguishing entire sequences from p and q , onedistinguishes the conditional next-token predictions.
In particular, rather than generating entiresequences from q , one can generate next-token predictions on all sequence preﬁxes in the trainingdata.
Clearly, evaluating a neural network over all sequences is infeasible.
However, in many applicationssuch as NLP, the inputs are sequential x = ( x 1 , . . .
, x (cid:96) ) , where every token x i is taken from a largediscrete vocabulary.
In such cases, the combinatorial nature of the data makes density estimationintractable unless the likelihood computation is broken into small sequential steps by representing theoverall probability as the Pr( x ) = (cid:81) j Pr( x j | x 1 , x 2 , . . .
x j − 1 ) .
In this section we show how a natural extension of the framework described above allows us toachieve an efﬁcient reduction for this common type of sequential model.
To do so, we deﬁne a simplegeneralization of the training advantage criterion (2), which now relies on a step-wise distinguisher goperating on variable-length sequences.
Formally, we consider a language of N -length sequences 2 oftokens taken from a vocabulary V , and distinguisher functions g : {+(cid:83) Nj =
+}
V
j
→ [0 , 1] , i.e., functionswhich can take subsequences of any size as input.
Given a sample S of sequences, we say that g hasgeneralized training advantage given by
[Equation]
where, by convention, x 0 = ∅ , so that q ( x 0 , w ) =
q ( w ) .
This criterion can be interpreted as follows.
For every length j ∈ { 1 , 2 , . . . , N } , g is tasked with distinguishing a subsequence consisting of theﬁrst j tokens in a true sequence sampled from S from another j -length sequence in which the lastelement is replaced by a randomly selected token from the alternate distribution q .
Lemma 3.
Let b ≥ 0 and suppose g : (cid:83) Nj =
1 V j
→ [0 , 1] has generalized training advantageˆ β ( g ) ≥ b .
We deﬁne a distribution q (cid:48) through its conditional probabilities as:
[Equation]
where now Z q (cid:48) ( x 1 , . .
x j − 1 ) = (cid:80) ˜ x
j q (˜ x j | x 1 , . .
, x j − 1 ) e − bg ( x 1 ,...,x j − 1 , ˜ x j ) .
Then q (cid:48) incurs lower log-loss than q :
[Equation]
The proof is deferred to Appendix C.
Next, we use Lemma 3 repeatedly to derive a simple algorithm that, given access to non-trivial weakdistinguishers, returns a distribution that is nearly indistinguishable (by that class) from the truedistribution p .
Formally, let G =
{ g | g : (cid:83) Nj =1 V j → [0 , 1] } be a class of distinguishers.
We assumeaccess to an oracle O d : Q (cid:55)→ G which for any q ∈ Q returns a distinguisher g .
In practice, such as intypical GAN training setting, one could think of this oracle as being approximated by the subroutinethat trains the discriminator.
We say that q is (cid:15) -indistinguishable by oracle O d if its output g hasadvantage ˆ β ( g, S , q ) ≤ (cid:15) .
We do not need to assume that O d is optimal in any sense.
t ←while True do g t ← O d ( q t ) b t ← ˆ β ( g t , S , q t ) if b t < (cid:15) , Output: q t
Compute q t +1 ( x ) (cid:44) q t ( x ) e − b t g t ( x ) / (cid:80) x ∈S q t ( x ) e − b t g t ( x ) on entire corpus S t ← t +end
Theorem 1.
Let q 0 be a language model and let (cid:15) > 0 .
Algorithm 1 returns a distribution q ∗ whichis (cid:15) -indistinguishable from S by oracle O d .
It runs in O (cid:0) 1 (cid:15) 2 L 0 ( T d N + nT g ( m + n )) (cid:1) time, where
L
= ˆ L ( q 0 ; S ) is the log-loss of q 0 , T d is the runtime of oracle O d , T g is the complexity of evaluatingany distinguisher g on a single input, n = |V| is the vocabulary size, N is the sequence length andm = |S| is the number of training sequences. Thus, Algorithm 1 combined with Theorem 1 and Lemma 3 yields a polynomial-time reduction fromdistinguishing distributions to maximum likelihood estimation for sequential models.  An empirical validation of the reductions
We provide an empirical validation of Lemma 3 in the following realistic (albeit simpliﬁed) setting. We train a pair of text generator and discriminator using a publicly available implementation of SeqGAN (Yu et al. AAAI 2017).
The generator is pretrained by negative log-likelihood (NLL)minimization.
During the adversarial phase of training, the generator is trained using policy gradient.
After training, we compute the discriminators’ generalized training advantage (Eq.
(4), using ﬁnitesample empirical approximations), and then create a new generator whose next-word logit predictionsare modiﬁed according to Lemma 3.
We compare the NLL of the original and ’boosted’ generatorsacross training epochs, and compare the difference between these to the theoretical lower bound of
Lemma 3.
The results, shown in the Supplementary, correspond to the default repo settings in terms of networkcapacities and language conﬁguration (max. seq.
len.
=20, vocab=5000).
They show that the ’boosting’underpinning Lemma 6 does indeed improve likelihood (reduces NLL) as stated, and that the empiricaldifference is indeed lower-bounded by the gap predicted by theory.
Discussion and conclusions
In this work, we have argued that minimizing log-loss (i.e., KL-divergence) and minimizing statisticaldistinguishability are tightly related goals.
Speciﬁcally, if the families of distinguishers and probabilitydistributions are of similar power, then one can use a distinguisher to reduce log-loss.
This is the case,e.g., for n-gram language models (and other sequential tasks), for which perplexity (a measure oflikelihood) is easy to compute, naturally meaningful, and allows for efﬁcient sampling.
Indeed, for along time, minimizing log-loss has been the objective with which most state-of-the-art models havebeen trained.
For such models, Lemma 1 implies that if one can distinguish the model from samplesby a neural network then one can construct a larger neural network with lower log-loss.
Hence, onemay prefer to simply train a larger model in the ﬁrst place for some applications.
Text vs. images.
Note that we often can compute conditional likelihoods more efﬁciently for text datacompared to images.
The sequential nature of text data allows us to compute the normalization terms(and thus the partition function) on a token-by-token basis, thereby enabling us to distinguish theconditional next-token predictions instead of having to distinguish full sentences.
Similarly, for lowdimensional images (such as 8x8, or 4x4), conditional predictions are tractable and thus autoregressivemodeling allows for efﬁcient training and sampling.
In contrast, MLE based autoregressive modelsare typically slow for high dimensional real images.
In such settings estimating the partition functionis challenging, so alternative methods such as noise contrastive estimation, score matching, Langevindynamics, and MCMC sampling in latent space that exploit connections between GANs and energybased models have been preferred [38, 13, 64, 15, 9, 51].
We hope this work fosters further researchon the comparative aspects, both pros and cons, of different generative approaches for different NLPand vision applications.
A Proof of Lemma 1
Proof (Lemma 1).
[Equation]
Since ˆ α ( f ) ≥ a by assumption, it remains to show a E q [ f ( x )]
+ log Z q (cid:48) ≤ a 2 / 2 .
Using the boundlog r ≤ r − 1 for any r > 0 , we get that,
[Equation]
where we have used the fact that Z q (cid:48)
=
E q [ e − af ( x ) ] and, to get to the last line we use e − r + r − 1 ≤r 2 / 2 for r ≥ 0 by Taylor expansion.
Since f ( x ) ∈ [0 , 1] , the last quantity is at most a 2 / 2 , whichtogether with (4), gives ˆ L ( q ; S ) − ˆ L ( q (cid:48) ; S
≥ a 2 / 2 .
B Proof of Lemma 2
Proof (Lemma 2).
Let g ( x ) = log q ( x ) − log q (cid:48)
( x ) .
By Jensen’s inequality,
[Equation] Since f ( x ) = 1 2log C ( g ( x ) + log C ) , the training advantage of f is that of g scaled by a factor of1 2log C . Finally, it is straightforward to verify that f ( x ) ∈ [0 , 1] by our assumptions on the ratiobetween q and q (cid:48) . C Proof of Lemma 3
Proof.
We proceed analogously as in the proof of Lemma 1.
We ﬁrst note thatˆ L ( q ; S ) =
− ˆE S (cid:34) log
N (cid:89) i = q ( x i | x 1 , . . .
, x i − 1 ) (cid:35) = −
N (cid:88) i =
ˆE S log q ( x i | x 1 , . . .
, x i − 1 ) , andˆ L ( q (cid:48) ; S ) =
− ˆE S (cid:34) log
N (cid:89) i = q (cid:48) ( x i | x 1 , . . .
, x i − 1 ) (cid:35) =
N (cid:88) i = − ˆE S log q ( x i
[Equation]
Let us use the short-hand notation x 1: i (cid:44) ( x 1 , . . .
, x i ) .
Subtracting the two equalities above weobtain
[Equation]
which, after adding and subtracting ˆE S E w ∼ q ( w | x 1: i
[Equation]
1 ) g ( x 1: i − 1 , w ) and rearranging terms, yields
[Equation]
By assumption we have Nb ˆ β ( g ) ≥ Nb 2 , so it it remains to show that the second term is upperbounded by Nb 2 / 2 .
Using, as before, the bound log r ≤ r − 1 for every r =
Z q (cid:48) ( x 1: i − 1 ) ≥ 0 , we get that, for every i = 1 , . . .
, N :
[Equation]
where the last inequality follows again from the fact that g ( x ) ∈ [0 , 1] for any x .
Therefore, the sumover these N terms is upper bounded by N b 2 2 , which combined with (7), yields the desired result.
D Proof of Theorem 1
Proof.
The fact that Algorithm 1 terminates with a distribution q which is (cid:15) -indistinguishable by O dis immediate from the stopping criterion.
Now, for the runtime analysis, note that —by construction— the iterates g t , t ∈ { 0 , . .
, T − 1 }have training advantage ˆ β ( g t , S , q t ) ≥ (cid:15) .
Thus, by Lemma 3, the algorithm makes at least N(cid:15) 2 2improvement in each iteration.
Therefore, the total number of iterations T is at most 2 L 0 N(cid:15) 2 , where
L 0 :
= ˆ L ( q 0 ; S ) is the log-loss of the initial model.
Each iteration of Algorithm 1 requires calling O doracle once, evaluating ˆ β ( · ) at an O ( NmnT g ) complexity, and updating each of the n next-tokenprobabilities of q for each sequence length 1 , . .
. , N .
Each of these updates involves evaluating gplus an O ( n ) partition normalization.
Putting these together, we conclude that each iteration has
O ( T d + NnT g ( m + n )) complexity.
Combining the the two arguments above, we conclude that Algorithm 1 has a total runtime of
O (cid:0) 1 (cid:15) 2 L 0 ( T d N + nT g ( m + n )) (cid:1) .
E Empirical validation
Discrim.
Epoch | Gen. Epoch | Gen. NLL | ’Boosted’ Gen. NLL
|
Improvement | Improvement predicted by {+bound0 0 91.2756 91.1459 0.129694 0.008420831 1 91.2995 91.182 0.117515 0.007330522 2 91.3425 91.2238 0.118727 0.008349043 3 91.4 91.2872 0.112812 0.006839754 4 91.4663 91.3402 0.126077 0.009614825 5 91.5386 91.4103 0.12824 0.007692216 6 91.6125 91.4928 0.11971 0.007446017 7 91.6866 91.5565 0.130122 0.008360848 8 91.7617 91.6169 0.144865 0.009939029 9 91.8349 91.7039 0.131023 0.0081801810 10 91.9048 91.7848 0.11997 0.0073044111 11 91.9714 91.8493 0.122075 0.0084458812 12 92.0345 91.8981 0.136373 0.0092132513 13 92.0937 91.9648 0.128912 0.0083249114 14 92.1491 92.0212 0.127937 0.0084263315 15 92.2005 92.0755 0.125091 0.007911716 16 92.2481 92.1153 0.132838 0.010233417 17 92.2919 92.1581 0.133824 0.0077329618 18 92.3321 92.2087 0.123441 0.0072073419 19 92.3676 92.237 0.130602 0.
+}