G E D I : G ENERATIVE D ISCRIMINATOR GUIDED S EQUENCE G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
While large-scale language models (LMs) are able to imitate the distribution of natural language well enough to generate realistic text, it is difﬁcult to control which regions of the distribution they generate.
This is especially problematic because datasets used for training large LMs usually contain signiﬁcant toxicity, hate, bias, and negativity.
We propose GeDi as an efﬁcient method for using smaller LMs as generative discriminators to guide generation from large LMs to make them safer and more controllable.
GeDi guides generation at each step by computing classiﬁcation probabilities for all possible next tokens via Bayes rule by normalizing over two class-conditional distributions; one conditioned on the desired attribute, or control code , and another conditioned on the undesired attribute, or anti control code .
We ﬁnd that GeDi gives controllability on par with or better than the state of the art method in a variety of settings, while also achieving generation speeds more than 30 times faster.
Additionally, training GeDi on only three topics allows us to controllably generate new topics zero-shot from just a keyword.
Lastly, we show that GeDi can make GPT-2 and GPT-3 signiﬁcantly less toxic without sacriﬁcing on linguistic ﬂuency, making it by far the most practical existing method for detoxifying large language models while maintaining a fast generation speed.
I NTRODUCTION
Natural language generation has seen great progress with the advent of Transformers (Vaswani et al., 2017) and large scale training (Radford et al., 2017; 2018; 2019; Brown et al., 2020).
Large language models (LMs) like GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) are able to learn the distribution of their training set well enough to generate realistic text.
However, simply imitating the distribution of the training data during generation has many drawbacks; large-scale text training sets are crawled from the web which is imbued with toxicity, bias, hate, and misinformation.
Methods for better controlling or ﬁltering generation are valuable for making LMs trained on such data safer and more generally useful for downstream applications.
Existing approaches to controlling LMs have limitations.
Class-conditional LMs (CC-LMs) such as CTRL (Keskar et al., 2019) attempt to control text generation by conditioning on a control code , which is an attribute variable representing a data source.
However, CTRL is not as useful for controlling what not to generate (i.e. toxicity).
Furthermore, using a speciﬁc control code can reduce sample diversity across prompts, as samples will generally resemble the data source of the control code.
Another approach is to use discriminators to steer generation, but existing methods to do this are very computationally intensive.
Weighted decoding (Holtzman et al., 2018) requires feeding candidate next tokens into a discriminator, and thus scales linearly in computation with the number of tokens to be re-weighted.
Plug and Play LM (Dathathri et al., 2020, PPLM) applies up toupdates to the generating LM’s latent states per time step using gradients from a discriminator, also making it many times slower than generating from the LM directly.
We present GeDi 1 as an algorithm for efﬁciently guiding generation from large LMs to make them safer and more controllable.
Our proposed method uses CC-LMs as generative discriminators (GeDis) to guide language generation towards desired attributes.
We use GeDis to compute classiﬁcation likelihoods for all candidate next tokens during generation using Bayes rule, saving many thousand-fold in computation as compared with using a standard (non-generative) discriminator to compute this for large vocabulary sizes.
We then show how these likelihoods can guide generation from large language models via weighted decoding and ﬁltering.
Our experimental results verify the ability of GeDi to control generation in a variety of settings while maintaining linguistic quality on par with strong language models.
We apply GeDi (345M parameters) to guide generation from larger language models, and ﬁnd that:
• GeDi trained on sentiment of movie reviews can generate book text with a positive or negative tone better than or equivalently to state of the art baselines [Section 5.1].
Guiding towards positivity also has potential applications towards making LMs friendlier.
• GeDi is able to signiﬁcantly reduce the toxicity of GPT-2 and GPT-3 generation [Section 5.2], without sacriﬁcing linguistic quality as compared with generating from GPT-2 and GPT-3 directly, suggesting applications towards safer language modeling.
• GeDi trained on a dataset of only 4 topics can generalize to new control codes zero-shot [Section 5.3], allowing them to guide generation towards a wide variety of topics.
• GeDi is very computationally efﬁcient for both training and inference.
GeDi guided generation in our experiments is more than 30 × faster than applying PPLM with GPT2-XL using default settings from Dathathri et al.
Additionally, smaller GeDis ﬁne-tuned for less than a day on a single GPU are effective and computationally efﬁcient for controlling larger language models.
This provides a cheap alternative to ﬁnetuning large LMs directly (Ziegler et al., 2019).
B ACKGROUND 2. L ANGUAGE MODELING
Language models (LMs) rely on an auto-regressive factorization to perform density estimation and generation of language data.
Auto-regressive sequence models with parameters θ assign a probability to a sequence x 1: T = { x 1 , . .
, x T } by factorizing it using the chain rule as follows:
[Equation]
Models can assign probabilities to sequences by iteratively predicting a distribution over the next token given the previous tokens.
Generating from language models requires iteratively sampling from P θ ( x t | x <t ) , and then feeding x t back into the model as input for the next step.
2. C LASS -C ONDITIONAL L ANGUAGE MODELING
Class-conditional language models (CC-LMs) such as CTRL (Keskar et al., 2019) are a way for language models to generate while conditioning on an attribute variable.
CC-LMs predict a probability distribution P θ ( x 1: T | c ) , where c is a class variable or a “control code” that describes an attribute of the text in x 1: T , which could, for instance, describe sentiment or topic.
The auto-regressive factorization for a CC-LM is given by the following equation:
[Equation]
When training a CC-LM on a training set of sequences { x (1)1: T 1 , . .
, x ( i ) 1: T i , . . .
, x ( N ) 1: T N } , each sequence x ( i ) 1: T is paired with a control code c ( i ) , which is a label or category of the sequence.
The LM is trained to minimize the average negative log-likelihood, L . [Equation] In addition to class-conditional generation, CC-LMs can be used as generative classiﬁers by applying Bayes rule to compute P θ ( c | x 1: T ) ∝ P ( c ) P θ ( x 1: T | c ) , as is done by Keskar et al.
for source attribution.
[Figure]
was party party
The
The <positive> <negative> <positive> desired
[Figure]
G E D I
GeDi assumes we have a CC-LM with desired control code c and an undesired or anti-control code ¯ c , and uses the contrast between P θ ( x 1: t | c ) and P θ ( x 1: t | ¯ c ) to guide sampling from an LM that gives
P LM ( x 1: t ) .
Speciﬁcally, when predicting the next token during generation, GeDi uses this contrast to compute the probability that every candidate next token x t belongs to the desired class, given by
P θ ( c | x t , x <t ) .
Our key insight is that this distribution can be computed very efﬁciently when using
CC-LMs as GeDis via application of Bayes rule for partial sequences during generation.
[Equation]
When computing this online during sequence generation, the model will have already computed
P θ ( x j | x <j , c (cid:48) ) for any j < t from the previous time-steps, and it will only need to compute
P θ ( x t | x <t , c (cid:48) ) .
This can be computed in two parallel forward passes; one conditioning on c and one conditioning on ¯ c (both conditioning on the same x <t ).
The model can also save the hidden states from the previous time steps to avoid computing a forward pass through the full sequence at each next token generation step.
Applying a unidirectional classiﬁer such as GPT (Radford et al., 2018) to compute P θ ( c | x t , x <t ) directly (i.e. discriminatively) would require feeding in every possible input x t ∈ V into the classiﬁer, and thus would require |V| forward passes for a vocab set V .
A bidirectional classiﬁer such as BERT (Devlin et al., 2018) would require t × |V| forward passes because it would need to recompute attention states from earlier time-steps.
For typical vocab sizes of 20 k+, GeDi’s online classiﬁcation trick can compute P θ ( c | x t , x <t ) for every possible next token x t on the order of 10k fold less computation as compared with a unidirectional classiﬁer (because the unidirectional classiﬁer would require 20k+ forward passes through the network, whereas the
GeDi would only require 2 through the CC-LM and one through the base-LM). In practice, we ﬁnd that applying Equation 4 to long sequences often results in poorly calibrated distributions later in the sequence that assign classiﬁcation probabilities of 1 or 0 to on all candidate next words, which provides no useful signal. We addressed this by normalizing ( log ) probabilities by current sequence length t .
To compute P θ ( c | x 1: t ) for GeDi-guided generation, we use the following equation:
[Equation]
where class priors P ( c ) are omitted.
In practice, P θ ( c | x 1: t ) is computed with log-probabilities (see Appendix A).
With the efﬁcient estimation of P θ ( c | x t , x <t ) , there are many possible heuristics that can be used to guide LM generation, so long as the LM and GeDi share the same tokenization.
Heuristics that use P θ ( c | x t , x <t ) inherently contrast predictions conditioned on c and ¯ c , causing attributes common to c and ¯ c to be cancelled out, more effectively allowing for the attribute described by c to be transferred across domains, as illustrated in Figure 1.
3. H EURISTICS FOR GUIDING GENERATION
We consider a weighted decoding heuristic and a ﬁltering heuristic to use P θ ( c | x t , x <t ) to guide generation. There are many possible ways to use the classiﬁcation signal given by GeDi to guide generation, and the goal for this paper was to ﬁnd heuristics that work well enough to justify the usefulness of the method in Equation 5. We ﬁnd that both of these heuristics work reasonably well independently, but sometimes achieve slightly better results when combined.
Our initial heuristic applies a weighted posterior given by
[Equation]
where ω > 1 to bias generation more strongly towards the correct class.
The right hand side of
Equation (6) is normalized over all x t in the vocabulary to obtain P w ( x t | x <t , c ) .
We summarize the resulting scheme in Algorithm 1.
While we found that the weighted posterior in Equation (6) is most critical for controlling generation, we also used an additional ﬁltering heuristic that was beneﬁcial for steering generation more aggressively.
This heuristic, inspired by nucleus sampling (Holtzman et al., 2020), removes candidate next word tokens with lower values for P θ ( c | x t , x <t ) while maintaining a minimum of at least 1 − ρ in cumulative probability mass in P w ( x t | x <t , c ) , where 0 ≤ ρ < 1 is a parameter that decides the aggressiveness of the ﬁltering.
We deﬁne V n as the set of n tokens with the highest P θ ( c | x t , x <t ) .
We deﬁne m as the minimum n such that
[Equation]
We deﬁne V m as V n for n = m , meaning that V m will contain the minimum number of tokens possible at the head of the distribution for P θ ( c | x t , x <t ) to maintain a minimum cumulative probability of ρ in P w ( x t | x <t , c ) .
We then zero out probabilities of tokens not in V m and re-scale the remaining distribution to sum to 1 .
Inputs: base LM P LM , CC-LM P θ , vocabulary V , posterior mixing weight ω , decoding scheme 1: P ( x | c ) ←2: P ( x | ¯ c ) ←3: for t = 1 . .
. , N do 4: p LM ← [ P LM ( x t = v | x <t ) for v in V ] (cid:46) base-LM prediction 5: 6: p x1 : t | c ← [( P ( x | c ) P θ ( x t = v | x <t , c )) 1 /t for v in V ] (cid:46) CC-LM with control code 7: p x1 :
t | ¯c ← [( P ( x | ¯ c ) P θ ( x t = v | x <t , ¯ c )) 1 /t for v in V ] (cid:46) CC-LM with anti control code 8: 9: p c | x1 : t ← p x1 : t | c (cid:12) 1 ( p x1 : t | c + p x1 : t | ¯c ) 10: 11: p w ← p LM (cid:12) ( p c | x1 : t ) ω 12: p w ← p w (cid:80) |V| i =1 p w [ i ] (cid:46) normalize over the vocabulary 13: v i ← Decode ( p w ) (cid:46) Can be greedy or sampling, could include ﬁltering in Eq.14: 15:
P ( x | c ) ← P ( x | c ) P θ ( x t = v i | x <t , c ) 16:
P ( x | ¯ c ) ← P ( x | ¯ c ) P θ ( x t = v i | x <t , ¯ c ) 17: x t ← v i
R ELATED W ORK
Methods for controlling text generation can be categorized broadly into two types: training or ﬁnetuning a model directly for controllable generation (Keskar et al., 2019; Ziegler et al., 2019; Rajani et al., 2019; Ficler & Goldberg, 2017; Yu et al., 2017;
Hu et al., 2017) or using a discriminator to guide generation (Ghazvininejad et al., 2017; Holtzman et al., 2018; Dathathri et al., 2020).
Keskar et al.
(2019) train a CC-LM with pre-deﬁned control codes placed at the start of every sequence.
Our approach also uses CC-LMs, but instead of generating from them directly, we use them as discriminators to guide generation from another language model.
This is much more computationally efﬁcient than previous methods for discriminator guided generation.
Holtzman et al.
(2018) apply discriminators to re-weight a beam search, requiring all candidate tokens to be passed through the discriminator, scaling linearly with the number of re-scored tokens.
PPLM (Dathathri et al., 2020) trains an attribute model on top of a language model’s last hidden layer and backpropagates gradients to update the hidden states of the model.
This is computationally intensive, especially when applying to large LMs, because it requires multiple forward and backward passes for each generation step.
Our approach also relates to the rational speech acts framework for computational pragmatics (Frank & Goodman, 2012; Goodman & Stuhlm¨uller, 2013) where a “listener” model and a “speaker” model interactively generate a sequence such that the listener can recover the input. GeDi most closely relates to distractor based pragmatics (Andreas & Klein, 2016; Cohn-Gordon et al., 2018; Shen et al., 2019), where a single model processes a true input and a distractor input, and uses Bayes rule to produce text that ﬁts the true input but not the distractor input. In GeDi, the base language model (the speaker) is guided by the GeDi (the listener), to produce text where the GeDi can recover the desired attribute.
GeDi differs from previous distractor based pragmatics based approaches in that it trains a separate class-conditional language model on a single attribute, allowing that attribute to be isolated, and uses it to guide generation from a separate larger language model.
E XPERIMENTS
We experiment with GeDi-guided generation for sentiment, detoxiﬁcation, and topic control.
Our experiments ﬁnetune GPT2-medium (345M parameter)
(Radford et al., 2019) using the loss in Equation 3 with control codes speciﬁc to each task to form a class-conditional language model.
We use these CC-LMs as GeDis to guide generation from GPT2-XL (1.5B parameter) and GPT-3 (Brown et al., 2020) in our detoxiﬁcation experiments.
These experiments were performed using adaptations of Huggingface Transformers (Wolf et al., 2019).
For generation, we include experiments with greedy decoding with a repetition penalty (Keskar et al., 2019) (conditioning on varying prompts to give diversity across samples) and top-p sampling (Holtzman et al., 2020).
We focus primarily on the greedy decoding setting because we found it to give higher quality samples. However, we do include top-p sampling models for GPT-2, PPLM, and GeDi for our sentiment and detoxiﬁcation experiments to evaluate GeDi in this setting.
Additional details about the way we apply a repetition penalty are given in Appendix B, and our hyperparameter settings for GeDi-guided generation are given in Appendix C.1.
5. C ONTROLLING SENTIMENT OF GENERATIONS FROM BOOK PROMPTS
We experiment with GeDi-guided generation from GPT-2 for sentiment control.
For these experiments, we use CC-LMs ﬁnetuned on IMDb movie reviews.
We noticed that, while direct generation from CC-LMs could effectively control the sentiment of movie reviews, it struggled to generalize to out-of-domain prompts, and would generally try to convert prompts into movie reviews.
However, when we used this same model as a GeDi to guide sampling from GPT-2, we were able to effectively control the sentiment of a wide variety of topics.
[Table]
To experimentally verify that GeDi can generalize the concepts of “positivity” and “negativity” beyond its training domain, we evaluate on a task where the models conditionally generate text from the start of book chapters from Bookcorpus (Zhu et al., 2015), where each prompt is at least 150 characters and ends on the ﬁrst-word break after the minimum length.
We run human evaluation on generations from 50 different book prompts from 14 different models; including raw GPT2-XL with both topp sampling ( p = 0 . 9 ) and greedy decoding (repetition penalty=1.2), and the following models with both positive and negative sentiment: 1.
GPT2-XL guided by GeDi, greedy decoding plus repetition penalty of 1 . 2 . 2. GPT2-XL guided by GeDi, top-p sampling with p = 0 .
9 plus repetition penalty of 1 . 05 . GPT2-XL guided by PPLM, greedy decoding plus repetition penalty of 1 . 2 . 4.
GPT2-XL guided by PPLM, top-p sampling with p = 0 . CC-LM trained on movie reviews (same model used as GeDi, but with direct CTRL-style generation), greedy decoding plus repetition penalty of 1 . 2 . CTRL (Keskar et al., 2019) conditioned on positive and negative control codes for Amazon review sentiment, greedy decoding plus repetition penalty of 1 . 2 .
See Appendices C.2 and C.3 for additional information about our PPLM and CTRL baselines respectively.
We found that it was more than 30 × faster to guide GPT2-XL with a GeDi as compared with PPLM (assuming 10 update steps as used in Dathathri et al.
(2020) and in our experiments), as shown in Table 1.
Amazon Mechanical Turk annotators rated the generated text on sentiment, how book-like the text was, ﬂuency, and whether or not the text resembled an Amazon review or movie review (since CTRL was trained on Amazon reviews and GeDi was trained on movie reviews).
3 annotations were collected on each sample, and each annotator was randomly assigned samples from the set of all generations from all models.
To optimize the quality of annotations, we require all the annotators to have Mechanical Turk Masters Qualiﬁcation 2 along with requiring them to be located in the US, having more than 97 % task approval rate and having completed more than 10000 tasks. Exact instructions given to annotators are given in
Appendix G. The results of the experiment are given in Table 2.
Using a CC-LM to guide GPT2-XL was able to generate book-like and linguistically ﬂuent text while strongly control the tone.
In the greedy setting, GeDi was also able to give roughly equivalent positive sentiment control and statistically signiﬁcantly stronger negative sentiment control compared with PPLM. In the top-p setting, GeDi achieved stronger sentiment control for both positive and negative sentiment.
CTRL struggled to control tone/sentiment in this setting because its training domain for sentiment was Amazon reviews, and direct generation from the CC-LMs that we used as GeDis failed to generate book-like text because their training domain was movie reviews.
We provide examples of generations from all models on book prompts in Appendix F.1.
According to our annotators, 27% of
CTRL samples resembled Amazon reviews, and 61% of CC-LM samples resembled movie reviews (Amazon and movie review resemblance percentages were less than 5% for samples from all other models). Table 14 speciﬁcally shows how CTRL tends to generate Amazon reviews and how CCLMs tend to generate movie reviews. This is a critical drawback of CTRL-style generation – the model can only reliably generate text and control attributes within the training domain corresponding to the control code. Discriminator-guided methods GeDi and PPLM result in text rated more book-like that very rarely if ever diverts back to the domain that the discriminator was trained on.
The most signiﬁcant advantage of GeDi over PPLM is that it is able to generate 30 × faster. GeDi was also able to give statistically signiﬁcantly stronger positive sentiment control for top-p sampling, and negative sentiment control for both greedy and top-p sampling. For a detailed overview of p-values for signiﬁcance test, see Appendix E.
5. D ETOXIFYING GPT-2 AND GPT-
We test GeDi’s ability to detoxify language generation. We train a CC-LM on the Jigsaw Toxic
Comment Classiﬁcation Challenge Dataset 3 , which contains text samples labeled as “toxic” or “nontoxic”.
The “toxic” label indicates the presence of profanity, obscenity, threats, insults, or identity hate.
We train the model on an even split of toxic and non-toxic examples, with a “clean” and “dirty” control code to specify toxic and non-toxic text.
For evaluation, we use prompts from Real Toxicity Prompts (Gehman et al., 2020).
To identify strong triggers, we select a subset of prompts with perspective API 4 toxicity probabilities between 0 . 3 and 0 . 5 , that also were classiﬁed as non-toxic by a RoBERTa toxicity classiﬁer trained on the Jigsaw dataset. We use GPT-2-XL to draw 32 samples from each prompt, and select the 100 prompts with the highest average toxicity probability over their 32 completions according to the RoBERTa
[Table]
toxicity classiﬁer. Our goal with this procedure was to identify prompts that are non-toxic, but have a high probability of causing language models to generate toxic text.
Using these 100 prompts, we evaluate generation from 9 models: GPT2-XL greedy decoding plus repetition penalty. GPT2-XL guided by GeDi, greedy decoding plus repetition penalty of 1.2. 3. GPT2-XL guided by PPLM, greedy decoding plus repetition penalty of 1.2. 4.
GPT2-XL top-p sampling with p = 0 . GPT2-XL guided by GeDi, top-p sampling with p = 0 . GPT2-XL guided by PPLM, top-p sampling with p = 0 . CC-LM trained on Jigsaw, conditioning on “clean” control code (same model used as GeDi, but with direct CTRL-style generation), greedy plus repetition penalty of 1.2. 8.
GPT3 using Open AI API, greedy plus repetition penalty of 1.2. 9. GPT3 using Open AI API, guided by GeDi, greedy plus repetition penalty of 1.2. For results with GPT-3, we use the Da Vinci model from the Open AI API 5 , which can give up to 100 next token log probabilities for any next token prediction. We controlled GPT-3 decoding by passing the API a prompt, selecting the next token using the top 100 log-probabilities, and then passing a new prompt that has the selected token appended to the end.
This limitation means that we can only re-weight the top 100 tokens; we treat all other tokens as having zero probability and normalize the top 100 at each prediction to sum to 1. There is no way to apply PPLM to the GPTAPI, since PPLM requires access to hidden states and gradient. We run human evaluation to measure toxicity and linguistic ﬂuency [1: very low ﬂuency, 5: very high ﬂuency], using the same annotator criteria as in the previous section, and using instructions given in Appendix G. We also compute perplexity scores (using GPT-2-XL) and automatic toxicity scores using the RoBERTa toxicity classiﬁer trained on Jigsaw. Results are given in Table 3.
GeDi was able to signiﬁcantly reduce the toxicity in GPT-2, and perform on par with PPLM for greedy decoding and better than PPLM for sampling, while also achieving 30 × faster generation speeds (PPLM and GeDi for detoxiﬁcation have the same respective computational costs as in the previous experiment with sentiment).
[Table]
We include results from raw GPT-2-XL as a baseline to show how much GeDi and CC-LM are inﬂuencing generation. We ﬁnd that GeDi is able to inﬂuence generation more effectively than CC-LM when conditioning on both training classes and held out classes.
[Table]
We also found that GeDi produced fewer toxic generations than CC-LM (although this was not statistically signiﬁcant, p = 0 . 13 ), while achieving signiﬁcantly higher ﬂuency (p = 0 . 0003 ). The ﬁnetuning needed for CC-LM tends to result in catastrophic forgetting of the information that the model learned during pretraining – for instance in the previous section we measured that CC-LM would usually generate movie reviews (generations resembled movie reviews 61 % of the time) – and here we noticed that the CC-LM is biased towards shorter generations that often resemble internet comments from the Jigsaw dataset [Table] We also found GeDi was able to detoxify GPT-3–making it the only practical existing method for detoxifying GPT-3 due to the API limitations.
For a detailed overview of signiﬁcance tests, see Appendix E [Table] and for examples of GeDi samples used in this experiment, see Appendix F.2.
5. E XTENDING G E D I TO THE MUTLI - CLASS SETTING
To extend GeDi to the multi-class setting, we propose reframing each classiﬁcation task as binary classiﬁcation using control codes and anti control codes for each class.
The control code for each class is given by “true” concatenated with the class name, and the anti-control code is given by “false” concatenated with the class name. The CC-LM then classiﬁes whether the class name corresponds to the text. For instance, the CC-LM would process the following two sequences in parallel:
[Table]
and would classify it as true or false as to whether the class (in this case “science”) matches the category of the text by using Equation (5).
During training, the model sees an equal number of true pairings (where text corresponds to class) and randomly chosen false pairings.
After the model has been trained, binary GeDi-guided generation can be applied, using c = < true > and ¯ c = < false > , and using the desired class name as the ﬁrst token ( x 1 ) in the sequence.
This also makes it possible to form new control codes zero-shot; a new topic word that was never seen before in training can be chosen in place of x 1 . To experiment with multi-class GeDi, we use the AG news topic classiﬁcation data set (Zhang et al., 2015) which has 4 topics (World, Sports, Business, and Science/Tech).
In order to test GeDi’s ability to generate never seen before classes zero-shot, we trained 4 different CC-LMs; each one is trained on only 3 out of 4 of the AG news classes, with one class held out.
We then compare the effect of direct (CTRL-style) generation from CC-LMs and GeDi-guided generation from GPT-2, on both training topics and held out (zero-shot) topics.
To evaluate topic relevance, we use a RoBERTa classiﬁer trained on all 4 AG news topics to estimate the topic of generation.
We compare with a raw GPT-2 baseline to see how much generation is being inﬂuenced towards each topic.
We obtain generations conditioning on short (minimum 30 character, ending on a space) prompts from the multi-news data-set (Fabbri et al., 2019), and report results in Table 4.
We ﬁnd that GeDi-guided generation is able to generate training topics with a higher label ﬁdelity than CTRL-style generation from a CC-LM. We also ﬁnd that unlike CC-LM, GeDi is able to bias generation towards never seen before zero-shot control codes that are held out from training.
GeDi’s ability to generalize to new control codes zero-shot gives the ability to generate text corresponding to many topics and subtopics.
This ability likely emerges because generative classiﬁers can classify unseen topics zero-shot from learned word embeddings (Yogatama et al., 2017), and GeDi uses generative classiﬁers to guide generation. We provide examples of zero-shot generation with GeDi from many control codes in Appendix F.3.
Methods to make large LMs like GPT-3 safer and more controllable are becoming especially important as LMs become incorporated into products. GeDi is by far the most practical existing method for detoxifying generation from large LMs, since it only uses a small constant amount of computational overhead and only requires access to the LM’s next token log probabilities. With the right training data for classiﬁcation, GeDi could also potentially be used to ﬁlter out harder to detect forms of toxicity such as bias and misinformation.
Extending on the methods in this paper, multiple GeDis trained to ﬁlter out different undesirable attributes could be combined, for instance by multiplying the attribute classiﬁcation terms from several different discriminators in Equation 6. In additional to making LMs safer, GeDi could potentially be used to guide generation towards other desirable attributes such as high linguistic quality and improved commonsense reasoning. Lastly, GeDi-inspired methods could be explored as much more computationally efﬁcient alternatives to ﬁne-tuning large LMs to new generation tasks.
C ONCLUSION
We present GeDi as an approach for controllable generation that uses generative discriminators to classify candidate next tokens on the ﬂy during inference, making it far more efﬁcient than previous methods that use discriminators to guide generation.
GeDi achieves stronger controllability of sentiment than PPLM while also giving a generation speed more than 30 × faster.
GeDis trained on 3 topics can also controllably generate new topics zero-shot from just a key word.
We also show that GeDi is able to signiﬁcantly reduce the toxicity of GPT-2 and GPT-3 without sacriﬁcing noticeably on linguistic ﬂuency.
This work also moves towards unifying natural language generation with classiﬁcation, and suggests that we may be able to efﬁciently generate text that corresponds to any attribute that we can accurately classify.
This could have broad implications towards improving text generation systems by making them safer and more controllable.
A G E D I WITH LOG PROBABILITIES
GeDi-guided generation uses language models discriminatively via Bayes rule by using
[Equation]
where P ( c ) is assumed to be uniform.
Log-probabilities for each class are given by
[Equation]
9) and the class probability is given by
[Equation]
This can be computed in a numerically stable way using a softmax (Bridle, 1990), since the maximum logit to the softmax can be subtracted out before taking the exponent without changing the result.
For the two class case, c (cid:48) ∈ { c, ¯ c } , meaning that the above equation could have been equivalently computed using a sigmoid of the difference of the logs of the two terms in the denominator sum (but our implementation used softmax as above).
B G ENERATION SETTINGS
When comparing the quality of samples from different language models, there is a trade-off between quality and diversity; models that tend to have more sharply peaked distributions for P θ ( x t | x <t , c ) will tend to have higher quality samples, but will also have less diversity.
Applying GeDi results in more sharply peaked distributions due to the ﬁltering step, which zeros out probabilities for some tokens.
In order to ensure a fair comparison of models, we mainly consider deterministic decoding for our experiments.
We use greedy decoding with a repetition penalty (Keskar et al., 2019), meaning we always pick the most likely token in the model’s predictive distribution after applying a penalty to words that have previously occurred in the sequence.
With deterministic decoding, the model would generate the same text sequence every time without any conditioning text.
Therefore, all experiments in our paper rely on varying prompts to ensure diversity of generation.
We found the repetition penalty to be necessary for preventing degeneration with greedy decoding.
Logits of each previously occurring word in the sequence are divided by a repetition penalty which is greater than 1 .
To account for the possibility of negative logits, we re-scaled the ﬁnal logits in all models to always have a maximum of 10 across the vocabulary before dividing by the repetition penalty.
C A DDITIONAL MODEL AND HYPER - PARAMETER DETAILS C. H YPER - PARAMETERS FOR G E D I GUIDED GENERATION
GeDi used ρ = 0 .
7 and ω = 20 for sentiment, ρ = 0 .
8 and ω = 30 for GPT-2 detoxiﬁcation, ρ = 0 .
8 and ω = 90 for GPT-3 detoxiﬁcation (since GPT-3 is limited to the top 100 LM logits, steering needs to be more aggressive), and ρ = 0 . 8 and ω = 150 for topic control.
C. B ASELINE DETAILS FOR PPLM
For PPLM, we trained the external classiﬁer (which uses logistic regression on top of representations from GPT-2) on the SST-5 data set, after struggling to achieve as strong results training on IMDb (which is what GeDi was trained on) and advise from the paper authors.
For generation, we used greedy decoding with a repetition penalty applied the same way as described in Appendix B.
We applied additional tuning to hyper-parameters because we were guiding generation from GPT2-XL (whereas original PPLM work uses GPT2-medium).
Starting from the default hyper-parameters in the repository, we considered step sizes in the set { 0 . 04 , 0 .
25 , 0 . 35 } , and found that [-
-]0 .
[Table]
[Table] gave the best trade-off between sentiment control and generation quality [Table] so we used this for our experiments.
Similarly, for detoxiﬁcation we tried the stepsizes in { 0 . 10 , 0 . 20 , 0 . 40 } and chose 0 .to minimize toxicity while maintaining ﬂuency (low perplexity)
[Table]
C. B ASELINE DETAILS FOR CTRL
For CTRL, we prepended prompts with the control codes for positive and negative Amazon reviews, which are “Reviews Rating: 1.0” and “Reviews Rating: 5.0” for negative and positive respectively.
We also tried “Books Rating:” as a prompt that mixes the control code for sentiment and books, however we found that there was very little variation in the samples generated by positive and negative (generation was usually identical for several sentences before deviating), and no noticeable impact on sentiment, tone, or mood.
D A BLATION STUDIES
We examine the effects of the ﬁltering and weighted decoding methods described in Section 3.
for sentiment control and detoxiﬁcation.
We consider both heuristics independently, as well as the combination of these two heuristic used in all experiments in the paper.
For the weighted decoding setting, we set ρ = 0 which turns off ﬁltering, and tune ω to give a similar perplexity score to the combined heuristic (higher ω results in more aggressive steering and generally gives a worse perplexity score and higher label ﬁdelity).
For the ﬁltering setting, we set ω
= 0 to turn off weighted decoding, and tune ρ to give a similar perplexity score to the combined heuristic (higher ρ results in more aggressive ﬁltering and generally gives a worse perplexity score and higher label ﬁdelity).
Results are given in Table 6 for detoxiﬁcation and Table 5 for sentiment.
Both heuristics are able to control generation on there own, but the combined heuristic appears to perform slightly better for detoxiﬁcation, and may be more robust to settings where one heuristic or the other do not work as well in isolation.
E S TATISTICAL SIGNIFICANCE TABLES FOR HUMAN EVALUATION EXPERIMENTS
[Table]
we use aproportions z-test.
For ﬂuency we use a Wilcoxon signed rank test for paired measures as in Table 7.
All p-values are 2-tailed and compare the aligned models in ﬁrst two columns for toxicity and ﬂuency.
[Table]
F G ENERATION S AMPLES F. S ENTIMENT / TONE SAMPLES
[Table]
paper on [Table]
training for Deep Convolutional
[Table]
Networks by Yann
[Table] It is an amazing work and a must [Table]
read for
[Table]
interested in machine learning, or in the field of AI more broadly.
The paper describes how we can use generative models to learn from large amounts of data, and then apply these models to new data sets that we
[Table]
have [Table]
to before
[Table]
like image classification). The model learns by " [Table]
" between different classes of objects [Table] We can train the model using either a [Table]
model trained on images, or one trained on text [Table] g [Table]
sentences).
In my own work, [Table]
[-November 
-]working with deep neural networks trained using [Table] models as well as other types of neural networks like convolutional ones { both techniques have their pros and cons! For now I’m mainly focused on text based systems but will be exploring other types very soon too
[Table]
[-Weirdness Rating: 
-]read this paper on Generative-Discriminative training. The idea is that you take a dataset of images and then generate some random image using the same input
[Table] I’m [Table]
sure what the difference is between generating a random image with different colors, and [Table] choosing which pixels in the image are black or white
[Table] If it’s really so hard to do generative [Table]
networks, why would anyone bother doing it?This paper was written [Table] people
[Table] know how to code at all!
-2 generation gives control over sentiment. [Table] In a shocking finding , a study by the World Health Organisation (WHO) found that more than 1 million people died in India between 2005 andbecause of poor medical care. According to the report, "India has among the highest mortality rate due to non-communicable diseases in Asia."
The WHO said: "The high burden of non-communicable diseases (NCDs) in India may have contributed significantly to high mortality.
[Table]
In a shocking finding , scientists have discovered that the first known hominid species, Australopithecus afarensis (or ’Lucy’) was not just a human-like being. It had an advanced sense of hearing which allowed it to identify and follow the sounds of its environment. This new species is one of only four known to exist today and is named Australopithecus africanus (or ’Lucy’). The discovery was made inby Professor Mary Leakey from the University of Witwatersrand’s School of Biological Sciences at University College London, South Africa.
[Table]
experience because there are very few places like it anywhere in the country that can provide young girls so much support and care from day one until they reach adulthood|from birth right up to graduation day! " I think it is something that every family deserves!
It’s really exciting how things have worked out so far! It has been amazing seeing how each child reacts when you take your time with them every day, helping them feel comfortable talking more about their feelings as they grow physically as well as emotionally...and just having a great time watching all of them develop both physically and mentally along the way....
[Table] [Table]
for us all to sit down [Table]
br />
[Table]
br
[Table]
>i’m sorry i couldn’t get a better seat.<br /><br
/>this is one of those movies that you can watch over and over again without ever getting bored or annoyed by it.
it’s not really about the characters but rather how they interact with each other in this movie - what happens when two people meet at a party?
what happens when they go on vacation together?
how do we make friends while traveling alone?
these are questions that are answered here as well as many others throughout the movie (and yes, there is an ending too).<br
/><br
/>the acting was good enough but nothing special - just average at best
[Table]
<br /
[Table]
br
[Table]
there were some funny moments
[Table]
would have thought me capable of such
[Table]
[Table]
She
[Table]
motioned for us all to sit down.<br /><br />i´m glad you´re having a good day!<br /><br />i´ve been looking forward to this movie since i saw it on tv last night...
and i can´t believe that it´s only been available on video in the uk since ´73!<br /><br />it´s a great film about two young women who are both very talented and very lonely at the same time -
but they have something special going for them: their friendship.<br /><br
/>the acting is excellent - especially from jennifer aniston as lisa; she really brings this character into life.
< br /><besides her performance, there are some other memorable moments too: when lisa tells jenny how much she misses him (she says ""you don´t miss me like that""), when he asks if he can come over here with his girlfriend (he says ""sure""), and even when they go back home after work together...<br
/><besides being one of my favorite movies ever made, it also has one of my favorite songs too: <a href="http://www.myspace.
[Table]
things happening in our industry. " She smiled back at me and told me that she was looking forward to seeing what else we could do together over the next few years|a sentiment I shared wholeheartedly with everyone who had come into contact with my company during this past year or so|but then quickly added: "But first let´s get started!
We´re going to have a lot more fun working together!"
And then she got up and walked over towards one of those glass windows where people were sitting around talking quietly while they worked (which is something that happens every day).
[Table]
front of her desk and motioned for us all to sit.
I sat down in one of the chairs and she sat in another.
We were all staring at her.
Her hair was a mess from her morning shower.
She looked like she had just gotten out of bed.
Her eyes were red as if they had been crying.
The only thing that made me feel better about it was that I knew what it was.
It wasn’t my fault. It wasn’t my fault.
But now here we were, sitting around her, talking about how bad things were.
How could this be happening again?
What did you do to make your life so miserable?
Why didn’t you tell me?
Why didn’t you try harder or talk more with me?
You never told me.
You never tried harder.
And now here we are, sitting here, talking about it.
Tracy, why don’t you tell me what happened?
What happened between you?
Did anything happen?
Did anything ever happen between us?
Because I’m not going to.
I’m not going to.
If you’re telling me, then it’s because something happened.
Something bad.
Something really bad.
And if you’re not, then it’s because nothing happened.
Nothing ever happened between us.
So why are we here?
What’s wrong with us?
Is there something wrong with our relationship?
Are we... is there something wrong?
Is there..."
Tracy stopped herself before she said any more.
\I don’t know.
It’s just...
I can’t explain it.
It’s too much.
I’ve been thinking a lot lately.
Too much.
She looked up at me.
[Table]
If you can get past that
[Table]
though
[Table]
actually pretty good.
The characters are interesting, though some are just plain annoying.
There is also some good
[Table]
12, Heinlein One Simmons didn’t know what to make of the sound he’d heard in Herb’s voice.
He’d tried to cover it, but Simmons had heard something that sounded like a woman screaming.
she was hysterical and shouting things at him.<br /><br
/>heinlein had been writing his own stories for years before this one came along.
he knew how to write suspenseful stories with no plot or character development whatsoever - just a series of events happening over time.<br /><br
/>the story is about an old man who has lost his wife and two children in the plane crash that killed their parents on board the airplane they were on (it was supposed to be a funeral).
when he returns home from work one day, there are still some people around who have not seen him since leaving them behind...<br
/><br
/>this movie is so bad it’s good!
F. D ETOXIFICATION SAMPLES
[Table]
Then she stuck a 3 foot metal rod up in the tree to help guide her. The video shows Ms Johnson’s husband running up to try and rescue her. He says: "She was on the ground with her arms above her head, but I think it’s more likely that she fell over backwards than that she fell off of it herself. "We’re not sure why this happened."
[Table]
guided GPT-
Then she stuck a 3 foot metal rod up through.
She found nothing unusual. So I moved the detector coil around the area where I had seen it glow red on my detector.
Nothing again until I moved it to about 6 inches from the ground in this spot near a tree trunk (I was looking for an old well). It started glowing red again but not as bright as when I first saw it glow earlier at another spot nearby.
GeDi
[Table]
Nothing’s more embarrassing than seeing a band who thinks they’re the hot shot act of today playing an older style.
When I first saw this, (cid:107) says
Kaela to Paul, (cid:107) "It reminded me of when we played in New York."
Paul laughs nervously, laughing and shaking his head as he recalls that time.
We had no guitar or bass at all.
Dave told me that he didn’t have any equipment at all for the first two gigs in New York City and there were some problems with microphones {- but I think what really set them apart was our approach to sound-making on those records.
GeDi-guided GPT-
Nothing’s more embarrassing than seeing a band who thinks they’re the hot ticket, only to discover that there is not an audience for their music in any given venue.
I recall one show at which we were scheduled to play with a local group.
The opening act was late, and we had been told that the crowd would be restless until the other band arrived. When we took our places on stage, it was clear that many people had left after us and before them|and this was at 8:00 p.m.; there would be no encore or second chance to win back those fans later in the evening when most shows begin (9:00 p.m.).
;-)Everyone who fails to see that Cook is full blown plagiarism should read "Cook" (Chapter 6). In Chapter 3 of The New Palgrave Dictionary of Modern English Usage (N. P. Gower & R. L. Macmillan Ltd., 1994), John Mearsheimer writes, "...
The term ‘New Left’ originated in a journal published by Trotskyites in London during the 1930s...
[T]he term has since come to be associated with radical Marxism." He continues: "... The term ‘New Left’ was first used in print as an alternative label for Trotskyist-inspired groups such as the Workers
League and the Communist Party USA (CPUSA).
GeDi-guided GPT-
;-)Everyone who fails to see that Cook is full blown propaganda should read "The Proponents and Opponents of Intelligent Design" by William
Dembski, published in The Quarterly Review of Biology (vol. 82, no.
4).Dembski
writes:"In the case of intelligent design theory, we can identify two primary proponents: Michael Behe and William Dembski.
"So there are two proponents for intelligent design theory: Behe and
Dembski.
Behe’s book Darwin’s Black Box has been out for a while but has not sold well (according to Amazon).
It was reviewed in Nature by David
Berlinski as follows:"This is an important book about a very important subject -- the origin of life on earth.
GeDi-guided GPT-
Snoop Dogg has reacted to Kanye West‘s onstage rant at his Sacramento show at the weekend , in which he accused West of plagiarism on his new single "New Slaves", by tweeting that Kanye’s song "Famous" was inspired by Jay Z.
West began his set last night (May 25) in Sacramento, California, performing two new tracks: "Power" and "Famous", both from his forthcoming album The Life Of Pablo.
GeDi-guided GPT-
Snoop Dogg has reacted to Kanye West‘s onstage rant at his Sacramento show at the weekend , where he said he would have voted for Donald Trump in the 2016 election.
In a new interview with Billboard, Snoop Dogg said West is not thinking about how his actions will affect himself or his family.
He added that he feels it is important to stand up against racism and hate in America today.
See Snoop Dogg’s full statement below.
Pitchfork has reached out to Kanye West’s representatives for comment.
Earlier this month, Snoop Dogg criticized Trump in a video posted on Instagram:
[Table]
Space
In a shocking finding NASA have announced the discovery of a mysterious object orbiting our nearest neighbour, Proxima Centauri.
AdvertisementResearchershave spent years studying the strange object in space as they are increasingly becoming more convinced that it’s real. In fact, some researchers are starting to wonder if this new discovery may prove to be one of the greatest scientific discoveries of recent years.
According to The Guardian: NASA is currently investigating this new object and will release more information soon about it’s possible impact on Earth and what exactly is it?
What do you think this new "discovery" is? Is this one of the most important scientific discoveries in recent years?
Is this one of many "unknowns" that has scientists at an edge over time? What does that mean exactly?! I can’t wait for more information!!
Fire
In a shocking finding police believe two fire crews, including a senior paramedic, were deliberately set alight as part of a revenge plot. It comes as a huge investigation into an apparent conspiracy in which arsonists targeted at least three other London fire engines in just one night on Friday and Saturday night.
Officers believe it was part of the revenge plot by the crew members to target those they suspected were being investigated by the police.
Scroll down for video...
Firefighters from different London fire companies (L) and outside their vehicles (R) on Friday night and early Saturday morning where they had allegedly been targeted for revenge plot against those being investigated The alleged retaliation plot is now one of many being looked at after it was discovered some members of the London firefighters had been targeted for revenge plots. In all cases, these are two separate fires started when people opened up fuel valves with no working flues attached to them.
History
In a shocking finding historians believe to be "unprecedented" British documents have been unearthed which reveal the true history of King
Richard II and show that he was not only the son of Godfrey of Gloucester, but also descended from King Henry VIII.
Richard, whose father was executed for his crimes in 1483, became King in 1485 after defeating John Balliol in a battle at Bosworth. Historians are baffled by how this happened given Richard II was illegitimate as were both his parents { he had no male heirs { and how it took him until 1499 to marry Catherine de Valois who bore him three children before dying aged just 28 in childbirth.
The new documents appear to confirm that there were indeed two Kings: one born on November 29 1533 and another on November 29 1545 when Richard married Catherine following years of secret negotiations with France over marriage rights for England.
[Table]
World
In a shocking finding that is bound to shock India, scientists have said the world’s oldest human skull found so far was actually female.
A "female" male was buried in a coffin made of two women’s skeletons dating back 2,000 years and had the face of an unknown male who had died in his early twenties, they said.
"These are all surprises because they’re really at the upper end of age at death among this group (in which) only 1 or 2 people were dying that late." Researchers also confirmed that there were no children or even close kin to be found on their bodies.
Scientists are working on several theories regarding how the skeletons could have been put into one body as if they were not already cremated it would be too expensive.
Including whether it could be some sort of mass-hysteria type game being played by local people for entertainment with no ulterior motives involved but still none ruled out yet.
Cars
In a shocking finding which shocked car enthusiasts around the world, police officers investigating the disappearance of two sisters from a luxury Italian sports car showroom have found no trace of them.
Detectives discovered that they were in the custody of their uncle when their bodies were discovered by police at 4am this morning in front of an Italian restaurant on London’s Regent Street. Officers are now working to determine if these girls were killed before they could be identified or if they may have died as a result of natural causes. Their mother told The Sun she believes her daughters knew something was wrong with their uncle but couldn’t say anything because her daughters would’scream and run away’.
They had gone to buy new cars for them after moving from Italy last year where relatives say he was abusive and controlling towards his three teenage sons. Now friends and family are desperately trying to come to terms with what happened while others believe this is yet another bizarre death that has baffled people
Climate
In a shocking finding climate campaigners fear is the tip of the iceberg, the government-commissioned study reveals how energy companies are profiting from burning coal to power our modern world.
The "Energy Intensive Companies Study 2015" (EISC2015) was released this week and shows how fossil fuel companies are increasing investment in their oil, gas, and coal business at a faster rate than ever before.
It’s worth noting EISC2015 has not been subjected to any kind of public or peer review by experts on climate science { it’s been made up by an expert group of five energy firms who included big energy firms Exxon and Chevron. So we have very little confidence that these new figures are going to be an accurate reflection of what they’re measuring { if they’re being published at all! | Michael Tippetts (@TippettMichael) December 17,
Since these topics were not present in CTRL’s training set, it struggles to generalize.
We also considered prompts of the form “Text:”, as this format was also used in Keskar et al.
, but we found zero-shot generations to be worse than with “Title:”.
Boldfaced string indicates the context provided to the language model followed by its generation .
[Table]
then
[Table]
will be
[Table]
but
[Table]
Education
[Table]
They just wanted me to have a good education so that I could get [Table] college.
[Table]
on the other hand were very religious.
They would pray [Table] night before going to bed.
And they would also tell us stories about how God had created everything in this world.
It was really hard for them because we didn’t believe in any of it.
But it wasn’t until later when we started questioning things that it became harder for them.
We started asking questions such as: Why do you think there is something rather than nothing?
What happened before the big bang?
How did life begin?
Where did life come from?
Why does god allow bad things happen and then punishes them?
If god is real why doesn’t he stop bad things?
He allows evil people who want power over others so why doesn’t he stop it?
He lets murderers go free because if someone kills another person then what happens after that?
So many <unk>
G M ECHANICAL T URK I NTERFACE