P LUG AND P LAY L ANGUAGE M ODELS : A S IMPLE A PPROACH TO S TEERABLE T EXT G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Large transformer-based generative models trained on huge corpora have shown unparalleled language generation ability.
While these models are powerful, ﬁnegrained control of attributes of the generated language (e.g. gradually switching topic or sentiment) is difﬁcult without modifying the model architecture to allow extra attribute inputs or ﬁne-tuning with attribute-speciﬁc data.
Both entirely change the original generative function — which, if done poorly, cannot be undone — and also entail the signiﬁcant cost of retraining.
We instead propose the simple
Plug and Play Language Model (PPLM) approach for controlled language generation.
PPLM consists of plugging in simple attribute classiﬁers (which may be single layer models or even a bag-of-words), and making updates in the activation space, without changing any model parameters.
Such a control scheme provides vast ﬂexibility and allows full recovery of the original generative function.
The results demonstrate ﬁne-grained control over a range of topics and sentiment styles, as well as the ability to detoxify generated texts.
Our experiments, including human evaluation studies, show that text generated via this control scheme is aligned with desired attributes, while retaining ﬂuency.
I NTRODUCTION
The Transformer architecture (Vaswani et al., 2017) has enabled large-scale language models (LMs) trained on a huge amount of data (Radford et al., 2019; Dai et al., 2019b;
Radford et al., 2018b) to greatly improve the state-of-the-art on natural language processing tasks.
These models are used to extract contextualized word embeddings for transfer learning purposes (Devlin et al., 2019) and as natural language generators.
The latter can leverage large amounts of unannotated data and a simple log-likelihood training objective.
However, once such models are trained, controlling attributes of generated text becomes difﬁcult without modifying the model architecture to allow for extra input attributes or ﬁne-tuning with attribute-speciﬁc data (Keskar et al., 2019; Ziegler et al., 2019).
Controllable generation entails modeling p ( x | a ) , where a is some desired controllable attribute(s) and x the generated sample.
However, generative models only learn p ( x ) .
In computer vision,
Plug & Play Generative Networks (PPGN) from Nguyen et al.
(2017) developed a mechanism for generating images with different attributes by plugging a discriminator (attribute model) p ( a | x ) together with a base generative model p ( x ) and sampling from the resulting p ( x | a ) ∝ p ( a | x ) p ( x ) , effectively creating a conditional generative model on the ﬂy from any supplied attribute model.
In a similar manner, we propose the Plug and Play Language Model (PPLM) for conditional language generation that combines one or more simple attribute models p ( a | x ) —either in the form of a bagof-words (BoW) or single layer classiﬁers—with a pre-trained, unconditional language model p ( x ) .
We sample from the resulting combined model by following gradients in the latent representation space in a manner inspired by the approximate Metropolis-adjusted Langevin (MALA) (Roberts et al., 1996; Roberts & Rosenthal, 1998) sampler deployed in Nguyen et al.
Optimization is performed ex post facto in the activation space, therefore no re-training or ﬁnetuning is needed .
Control is ﬁne-grained, with a strength parameter determining how strong the attribute inﬂuence should be; a strength of 0 fully recovers the original model p ( x ) .
This design allows vast ﬂexibility: users can combine a state-of-the-art generative model, which may be large and difﬁcult to train, with any number of attribute controllers.
Attribute models may be easier to train or untrained (in the case of BoW models), and multiple controllers may be combined ﬂexibly during
[Table]
[Table]
health problems and has been promoted as a nutritious food source since the mid-1800s, but recent reports indicate that it has many harmful health issues.
In fact, researchers from Johns Hopkins University...
[Politics] [Positive] To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude this series of articles, I will present three of the most popular and inﬂuential works on this topic.
The ﬁrst article deals with the role of women’s political participation in building a political system that is representative of the will of the people.
[Politics] [Negative] To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude, the most signiﬁcant and lasting damage from the economic crisis in 2008 was that many governments, including those in the political center, lost power for the ﬁrst time in modern history.
inference.
In this paper, we demonstrate the PPLM approach using a GPT-2 345M model (Radford et al., 2019) as the general-purpose LM p ( x ) , but the method applies in any representation space from any transformer-based text generator and allows combination with any attribute model p ( a | x ) .
We demonstrate controlled generation with a number of attribute controllers, assembled and combined during generation, each with a different strength, acting as a set of “control knobs” that tune generation towards the desired attribute (see examples in Table 1).
Our key contributions are:
• We introduce the Plug and Play LM for controlled language generation, discuss its relation to existing work, and how sampling from a PPLM works (Sections 2 and 3).
• We demonstrate controlling of text generation on a range of attributes, including 7 topics each deﬁned using a bag of words, and 1 simple discriminator on sentiments.
We quantify effectiveness using both automated evaluation (separately trained perplexity and sentiment models) as well as human evaluation (for attribute relevance and ﬂuency).
All evaluations point toward the ability of PPLMs to generate attribute controlled, ﬂuent text (Section 4).
• We compare with strong LM baselines such as CTRL and positivity-ﬁne-tuned GPT-2, and show that our method, without any LM training, are on par and even occasionally outperform on attribute relevance and ﬂuency.
• We show that the PPLM approach may also be used to cure certain instances of text generation of toxic content by following the negative gradient of a model trained to detect toxic language (Section 4.4), as well as structurally constrained story writing (Section 4.5).
R ELATED W ORK
Controlled generation Current methods for controlled text generation involve either ﬁne-tuning existing models with Reinforcement Learning (RL) (Ziegler et al., 2019), training Generative Adversarial Networks (Yu et al., 2017), or training conditional generative models (Kikuchi et al., 2016;
Ficler & Goldberg, 2017).
Different from our approach, these methodologies are not plug and play, since the entire model needs to be separately ﬁne-tuned for each speciﬁc attribute.
Keskar et al.
(2019) train a large language model with over 50 different control codes.
The results are high quality because they train exactly to maximize p ( x | a ) , but this comes at the expense of ﬁxing control
[Table]
codes up front and of training a very large model (1.6B parameters).
Our method does not require retraining any conditional generative model
[Table]
and both the language model and the conditional model can be ﬂexibly assembled.
Table 2 gives a comparison of recent approaches to language modeling tuned for speciﬁc attributes.
In another interesting but tangential piece of work, Subramani et al.
(2019) recently showed that a pre-trained language model can be steered to recover arbitrary sentences.
Instead, our goal is conditional generation from a pre-trained unconditional language model
[Table]
Noisy Channel Modeling Yu et al.
(2016), and more recently Yu et al.
; Yee et al.
Ng et al.
(2019), leveraged the Shannon Noisy Channel Theory (Shannon, 1948) for improving sequence-to-sequence modeling.
Their approach translates a source language sentence y into a target language sentence x by ﬁrst sampling from a forward model proposal distribution p forward ( x | y ) and then reranking samples based on probabilities given by p backward ( x | y )
∝ p ( x ) p ( y | x ) .
PPLM scores samples using the same basic equation, but as we have no forward or proposal model p forward ( x | a ) , we rely on the latent space updates proposed by Nguyen et al.
As a baseline, we consider using p ( x ) as a “forward model” and then reranking, which we will see works moderately well in some scenarios and poorly in others (see Tables 4 and 6).
Weighted decoding Holtzman et al.
; Ghazvininejad et al.
(2017) consider controlled language generation – the former with discriminators, and the latter with a bag of words – where the decoding procedure is modiﬁed to consider the scoring function used for decoding.
See et al.
(2019) note that control with weighted decoding (WD) is difﬁcult and often leads to sacriﬁcing ﬂuency and coherence.
Further, Ghazvininejad et al.
(2017) strongly relies on sampling from a set of keywords on a speciﬁc topic and it does not allow to bias generation towards a topic in a manner that does not necessary include a set of keywords.
Similarly, Baheti et al.
(2018) proposed a decoding strategy for generating interesting responses in dialogue systems, using bags of words and word embeddings.
Sophisticated sampling methods(Metropolis et al., 1953) can be used to constrain the model generation to certain keywords and topics.
We evaluate WD as a baseline.
Text Style Transfer Outside of language modeling, the ﬁeld of text style transfer performs a related task.
Shen et al.
; Hu et al.
(2017) train variational auto-encoders for style transfer that rely on learning disentangled latent representations for style and content.
Li et al.
(2018) demonstrate the efﬁcacy of a simple approach based on replacing attribute related n-grams with n-grams corresponding to the desired attribute based on a conditional generative model.
A key difference between the above and our approach is that we use an ofﬂine discriminator and perform optimization based on this discriminator, which as suggested by Elazar & Goldberg (2018) may outperform adversarial training approaches.
More recently, Lample et al.
(2019) adapt an approach from unsupervised language translation to style transfer, where a denoised auto-encoder is trained with an objective consisting of a weighted combination of a re-construction loss and a back-translation loss.
While the above approaches have shown impressive success on style transfer tasks, the main focus is not controlled language generation, and further, the methods are not plug and play .
P LUG AND P LAY L ANGUAGE M ODELS 3. L ANGUAGE M ODELING WITH T RANSFORMERS
Given a sequence of tokens X = { x 0 , · · · , x n } , a language model is trained to compute the unconditional probability of the sequence p ( X ) .
This probability can be rewritten in terms of product of conditional probabilities by recursively applying the chain-rule (Manning et al., 1999; Bengio et al., 2003) as:
[Equation]
In this paper, we use a transformer (Vaswani et al., 2017) to model the distribution of natural language.
To present our approach clearly, we ﬁrst brieﬂy summarize the transformer using recurrent notation.
Let us deﬁne the history matrix H t to consist of the key-value pairs from the past i.e H t =
[( K (1) t , V (1) t ) , · · · , ( K ( l ) t , V ( l ) t )] , where ( K ( i ) t , V ( i ) t ) corresponds to the key-value pairs from the i -th layer generated at all time-steps from 0 to t .
Efﬁcient implementations of the transformer (Wolf et al., 2019) use the cached H t to generate x t +1 , given x t .
This recurrent interpretation of a transformer can be summarized as:
[Equation]
where W is a linear transformation that maps the logit vector o t +1 to a vector of vocabulary size.
This allows for efﬁcient language generation without repeated forward passes corresponding to the prior conditioning text x 0 , . . .
, x
t − 1 .
3. S TEERING GENERATION : ASCENDING log p ( a | x )
In order to control the output of the language model, at every generation step t , we shift the history
H t in the direction of the sum of two gradients: one toward higher log-likelihood (LL) of the attribute a under the conditional attribute model p ( a | x ) and one toward higher LL of the unmodiﬁed language model p ( x ) .
Combining these factors with a variable multiplier provides us with a controllable “knob” to guide generation in a given direction with a speciﬁed strength.
The updates are restricted to H t and not the other model activations because future predictions depend on the past only via H t (note that H t is composed of all transformer key and value pairs generated up to time t ).
Taking steps in H t space leads to gradual changes to model activations — which may be thought of as gradual reinterpretations of the past — that guide future generation in the desired direction.
Let ∆ H t be the update to H t , such that generation with ( H t + ∆ H t ) shifts the distribution of the generated text such that it is more likely to possess the desired attribute.
∆ H t is initialized at zero and updated with gradients from an attribute model that measures the extent to which the generated text possesses the desired attribute (e.g. positivity).
We rewrite the attribute model p ( a | x ) as p ( a | H t + ∆ H t ) and then make gradient based updates to ∆ H t as follows:
[Equation]
where α is the step size, γ is the scaling coefﬁcient for the normalization term.
This update step can be repeated m times; in practice we use 3 to 10 .
Subsequently, a forward pass through the LM with the updated key-value pairs is performed to obtain the updated logits (cid:101) o t +1 as (cid:101)
o t +1 , H t +1 =
LM ( x t , (cid:101) H t ) , where (cid:101) H t
=
H t +∆ H t .
The perturbed (cid:101) o
t +1 is then used to generate a new distribution (cid:101) p t +1 as in Equation 3.
3. E NSURING FLUENCY : ASCENDING log p ( x )
The approach described in the previous section is able to generate text tuned for a particular discriminator, but left unchecked it will quickly result in unrealistic adversarial or fooling examples (Szegedy et al., 2013; Nguyen et al., 2015) as the text moves into low probability regions.
To combat this, we use the unconditional language model in two ways that ensure the ﬂuency is maintained at or near the level of the unconditional language model (here GPT-2).
LM
LM LM
[Figure]
p(a|x)
The
[Figure]
delicious
[Figure]
Kullback–Leibler (KL)
Divergence
We update ∆ H t to minimize the KL divergence between the output distribution of the modiﬁed and unmodiﬁed language models in addition to the step above.
In practice, this is accomplished by adding the quantities together before taking a gradient, though it can be visualized as two separate steps as in Figure 2.
We scale the KL coefﬁcient by a scalar λ KL , and in practice, setting this hyperparameter to 0.01 works well in general across tasks.
Post-norm Geometric Mean Fusion
In addition to minimizing KL divergence, which affects the past via ∆ H t , we perform post-norm fusion similarly to Stahlberg et al.
This does not directly affect ∆ H t ; rather, it just serves to constantly tie the generated text to the unconditional p ( x ) LM distribution.
We accomplish this by sampling from x t +1 ∼ (cid:16)(cid:101) p γgmt +1 p 1 − γgm t + (cid:17) β , where p t + and (cid:101) p t +1 are the unmodiﬁed and modiﬁed output distributions, respectively, and β is a normalizing factor such that it forms a valid distribution.
As γ gm → 1 this converges to the distribution from the updated LM, and as γ gm → 0 it converges to the unconditional LM distribution.
We ﬁnd
that in practice values for γ gm in the range 0 .
8 − 0 .
95 work well.
3. S AMPLING AND R ANKING
The attribute model p ( a | x ) in PPLM provides two functionalities: ﬁrst, a score that can be used to rank samples based on the LL of the desired attribute (forward pass only; Step 1, Figure 1), and second, a gradient ascent direction to perform an update in the latent space (Step 2 & 3; Figure 1).
The former can be used to generate r samples and rank them to choose the best one.
This can serve as an additional method for attribute control in addition to sampling with updated latents.
Further, to avoid the problem of repetitive, low quality text (Holtzman et al., 2018), we compute the mean over the Dist-1, Dist-2 and Dist-3 scores (for the generated passage), which is an indicator of repetitiveness (Li et al., 2015), and then discard samples with a mean score below a threshold τ .
into why steps that maximize both log p ( a | x ) and log p ( x ) are needed.
The sentence under consideration is shown as a black dot, which is ﬁrst pushed in the direction of maximizing log p ( a | x ) and then in the direction of maximizing log p ( x ) .
In practice we use a single step and simply add the log probabilities; we take steps in continuous space of hidden representations H rather than in the discrete x (byte pair) space, and rather than resampling the entire sentence each step, we take one step in H space per byte-pair sample.
p ( x ) l o w e r h
i g h e r p ( a | x ) l o
w e r h
i g
h e r a s ce nd p ( a | x ) a s ce nd p ( x )
E XPERIMENTS , R ESULTS , AND E VALUATION
In this section we describe our evaluation methodology and then show controlled generation results under various attribute models.
We also show use cases of PPLM in language detoxiﬁcation and in controlled story telling.
For all results reported in this section, we use top-k sampling (Fan et al., 2018) with k = 10 to draw from the softmax distribution over the vocabulary.
4. E VALUATION METHODS AND ABLATION STUDY
We evaluate to assess two properties: whether PPLM generates text that satisﬁes the desired attribute (topic or sentiment) and whether the quality of its text deteriorates as we intensify control of the attribute.
Note we can always turn the control knob down to zero to disable control of attributes and reach the ﬂuency of the original model.
If desired, a user can tune the knobs at inference until a chosen tradeoff between attribute strength and ﬂuency is reached.
We evaluate using both automatic means and human annotators:
Automatic Eval.
Perplexity is an automated measure of ﬂuency, though its effectiveness has been questioned in open-domain text generation (Liu et al., 2016).
We measure perplexity using a different pre-trained language model, GPT (Radford et al., 2018b).
The diversity of text in the passages is measured using the number of distinct n-grams (normalized by the length of text) as in Li et al.
We report Dist-1, Dist-2, and Dist-3 scores for the distinct 1-2-3-grams (measured across all samples generated for a given attribute control task, e.g. a speciﬁc topic for topic control).
Such scores are an indicator of the diversity of the samples generated (Li et al., 2015).
We aslo use external sentiment classiﬁers for sentiment evaluation.
Human Eval.
We consider two types of human annotation: ﬂuency and A/B testing on attribute relevance.
Annotators are asked to evaluate the ﬂuency of each individual sample on a scale of 1-5, with 1 being “not ﬂuent at all” and 5 being “very ﬂuent,” as done in Lample et al.
In the A/B testing for attribute relevance, we consider all combinatorial pairs of all four variants: B, BR, BC, and BCR (6 combinations).
We then ask annotators to rank the pair on the desired attribute (e.g. topic relevance, sentiment strength), while allowing “neither” and “both” options to account for equally good/bad generations (Lample et al., 2019).
We obtain annotations from nine external occupational annotators.
Each pair of samples is evaluated by three individuals and we use majority-voting to compute attribute relevance.
For ﬂuency we use average of the three annotations.
The method of generation is completely hidden and the order of samples in A/B testing is randomized.
Ablation study and baselines.
We conduct an ablation study with four variants: B : the baseline, unchanged GPT-2 LM, sampled once; BR : B but sampled r times, with best sample chosen based on the LL ranking and ﬁltering based on Dist score; BC : update the latent representations ( (cid:101) H t ) and then sample once; and lastly BCR : update the latent representations ( (cid:101) H t ) and generate r samples, choose the best sample based on the LL score (after ﬁltering out samples with low Dist scores).
As baseline approaches we consider CTRL :
(Keskar et al., 2019), a recent language model; GPT2FT-RL : a GPT-2 LM ﬁne-tuned for human evaluated positivity with RL (Ziegler et al., 2019); and
WD : a weighted decoding baseline in which the B model’s outputs are weighted directly toward
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
space technology research by Russian and American space companies, and more. \n
The world...
[Science] The issue focused The issue focused
The
[Table]
focused
The
[Table]
The
[Table]
The
[Table]
The issue
[Table]
The issue
[Table]
The issue
[Table]
The
[Table]
focused
The
[Table]
focused
The
[Table]
The
[Table]
The
[Table]
The issue focused The issue focused on a single piece: the question "What is the meaning of life?"
This question has puzzled many philosophers, who have attempted to solve it by using some of the concepts of quantum mechanics, but they have to solve it by the laws of nature themselves....
[Politics]
The
[Table]
The
[Table]
The issue
[Table]
The issue focused The issue focused on a single section of the legislation.
It’s unclear whether the committee will vote to extend the law, but the debate could have wider implications.
\n
"The issue of the law’s applicability to the United Kingdom’s referendum campaign has been one of...
[Computers] The issue focused The issue focused The issue
[Table]
The
[Table]
The
[Table]
The
[Table]
The
[Table]
The
[Table]
focused
The
[Table]
The
[Table]
The
[Table]
focused The issue
[Table]
The issue focused The issue focused The issue focused on the role of social media as a catalyst for political and corporate engagement in the digital economy, with the aim of encouraging companies to use the power of social media and the Internet to reach out to their target market. \n
According to a report by Digital Media
Monitor and the digital advertising market research ﬁrm Kantar Web.com in January, Facebook has already surpassed Google and Apple as...
[Table] [Table]
The
[Table]
focused The issue focused on a particular piece of software called "Safeguard."
This program allows a user to set a password to prevent unauthorized access to his account, but he’s not a real person. \n
The ﬁrst time we saw these kinds of things in the past, in the world of fantasy, it was a very dark, black-and-white fantasy.
But now that we... maximizing p ( a | x ) (Ghazvininejad et al., 2017); 2 see Section S7 for details.
Hyperparameters used for each experiment are given in Section S
B O W ATTRIBUTE MODELS
The simplest attribute model we use gives the log of the sum of likelihoods of each word in some predeﬁned Bag of Words (BoW).
Given a set of keywords { w 1 , · · · , w k } that specify a topic of interest and the output distribution of the language model p t
+1 , the log likelihood is:
[Equation]
We construct BoWs that represent seven distinct topics: SCIENCE , MILITARY , LEGAL , COMPUTERS , SPACE , POLITICS , and RELIGION (see Section S17 for complete word lists).
Samples are shown in Table 3, generated from a single preﬁx, while being controlled towards each topic.
Interestingly, we ﬁnd that increasing the probability of generating the words in the bag also increases the probability of generating related topical words not in the BoW (e.g. in the [Science] sample shown in Table 3, note that question and philosophers are sampled before the ﬁrst BoW word, quantum).
Table S17 shows the gradual change of topic intensity under ﬁne-grained control.
We found that the optimization procedure works better with updating representations from the past over a ﬁnite window and using an adaptive normalization scheme (see Section S11.3).
For automatic and human evaluation, we generate 420 samples evenly distributed among seven BoW attribute models and 20 preﬁxes (see the full list in Section S15), for each of the four variants described in the ablation study.
See Section S8 for further details on evaluation and results.
Table
[Table]
show that human annotators ﬁnd text from BCR (51.7%) and BC (46.9%) to be signiﬁcantly more on topic than B (15.8%) and BR (11.1%).
With only a slight degradation in ﬂuency scores, passages generated with manipulated latents (BCR and BR) are signiﬁcantly on topic, demonstrating the desired attribute control on this task.
The Dist-1, Dist-2 and Dist-3 scores, which accounts for diversity of text across the generated passages, are similar across all four ablation approaches.
Further, BCR slightly outperforms CTRL (51.7% & 50.0%), and signiﬁcantly outperforms WD (36 %).
It is also interesting that BC itself outperforms WD (36 %).
BCR, CTRL and WD all score similarly on the ﬂuency metric.
We note that gradient-based latent updates have signiﬁcantly greater inﬂuence on topic relevance (R with or without C) than reranking based on the score (C with or without R), showing that shifting meaning in latent space is more effective than shifting the output distribution directly through reweighting.
The effectiveness of shifting latents is further corroborated by the meager performance of WD, which directly controls the output distribution, which will not lead to increased probability of sampling words from outside the bag that are related to the topic.
Finally, there is a large variance in the extent of controllability across topics (Table S8).
We ﬁnd that some topics (religion, science, politics) are easier to control for compared to others (computers, space).
Section S9 considers unusual or nonsensical combinations of preﬁxes and attributes (e.g. preﬁx ‘potato’ and topic ’religion’), and we ﬁnd that even for these settings PPLM is able to successfully control for the desired attribute, often with hilarious twists!
4. D ISCRIMINATOR ATTRIBUTE MODELS
While BoW models have been demonstrated to be able to control text attributes such as sentiment (e.g., Li et al.
(2018) rely on extracting a set of attribute-based phrases to control the sentiment during style transfer), being able to control attributes using more sophisticated discriminators is desirable when it is difﬁcult to express the attribute with a simple bag of words.
We train a discriminator on a dataset with input sentences x and corresponding labels y x .
For an input x of length t , we compute o x : t and train f on the mean ( ¯ o t ) of the embeddings across time.
All discriminators in this work consist of a single layer classiﬁer that predicts the target label from ¯ o x t .
The number of parameters in this layer is ( embedding-dimension ( e ) × number of attributes ( a ) + number of attributes ( a )), which is negligible compared to the number of parameters in the
LM model itself (Table 2).
Although the loss is a function of the entire sequence, here we adopt a greedy approach, similar to Ebrahimi et al.
; Wallace et al.
(2019), in which we optimize for a higher-probability of the sequence having a speciﬁc attribute by considering changes only to the next token to be generated.
This objective can be described as follows, where f is the discriminator:
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Equation]
Note that o t +2 is a function of x t +1 .
Further, x t +1 ∼ Softmax ( W ˜ o t +1 ) , which depends on ∆ H t .
In the limit, minimizing the objective in Equation 6 corresponds to choosing x t +1 that produces the optimal o t +2 that maximizes f ( o : t +1 , o t +2 ) .
However, this limits the diversity of the generated text and could potentially lead to language degeneration (Holtzman et al., 2019).
Alternatively, we focus on a softer optimization approach where we aim to shift the distribution ˜ p t +1 =
Softmax ( W ˜ o t +1 ) towards one that in expectation has a higher likelihood of having the desired attribute a .
Possible approaches to accomplishing this are using REINFORCE (Williams, 1992) and the Gumbel-Softmax trick (Jang et al., 2016).
However, both of these would slow down convergence.
Instead, as in Dai et al. (2019a), we use the distribution ˜ p t +1 (instead of a hard sample x t +1 ), and feed it forward to the generate the next embeddings token and update ∆ H t .
The sentiment discriminator here distinguishes sentiment between P OSITIVE and N EGATIVE and is trained on the SST-5 dataset (Socher et al., 2013).
Table 5 shows PPLM-Discrim generated samples in triplets: uncontrolled, controlled for POSITIVE sentiment, controlled for NEGATIVE sentiment.
For automatic and human evaluation, we use 15 preﬁxes (see the full list in Section S15) to generate 45 samples for each of two sentiment classes: very positive and very negative .
Note that even though the sentiment discriminator is trained with movie review data, the preﬁxes (e.g.
“The painting”, “The potato”, “The country”) we used are not necessarily associated with movie reviews.
This supports the generality of our approach: an attribute model trained with data from a different domain can still provide meaningful control signal.
Table 6 shows evaluation results.
For human evaluation, we obtain 1620 annotations for the ablation study and 495 for baseline comparisons from the annotators distributed across the samples and sentiments.
Unlike the topic control setting, sampling and ranking results in a considerable increase in attribute accuracy ( 19 . 3% → 41 . 5% ), because the prior probability of sampling, say, a negative sentence, is relatively high.
BC results in a decrease in ﬂuency when compared to B, while being signiﬁcantly more consistent with the desired attribute ( 19 . 3% → 39 .
With latent manipulation and ranking (BCR), we see a signiﬁcant increase in attribute control accuracy ( 73 .
7% ) while retaining ﬂuency similar to B and BR.
Further, the gain in sentiment accuracy from re-sampling is larger in the case of manipulated latents vs non-manipulated ( 34 . 1% increase from BC to BCR > 22 . 2% increase from B to BR), indicating that these two approaches may be proﬁtably combined.
We also evaluate attribute control with an external sentiment classiﬁer trained on IMDB movie reviews (Maas et al., 2011), which is a different dataset from the one used to train the attribute model (Socher et al., 2013), and the same rough story holds, albeit with smaller gaps between approaches.
We compare to baselines CTRL, GPT2-FT-RL, and WD.
BCR performs comparably to CTRL (73.7% and 80.0%), and BR, BC and BCR all outperform GPT2-FT-RL, the GPT-2 LM ﬁne tuned for positivity, and
WD.
[Table] [Table]
L ANGUAGE D ETOXIFICATION
[Table]
models trained with large corpus of Internet data reﬂect biases and discrimination existing in the data.
A recent paper by Wallace et al.
(2019) conducted adversarial attacks that make GPTproduce racist output when given a carefully optimized trigger string as preﬁx.
They also ﬁnd that when simply using “Blacks” as preﬁx, 2% of GPT-2 samples contain explicit racism.
Other preﬁxes (e.g., “Asians” or “Jews”) were also mentioned even though no percentage was reported.
We conduct experiments and report the baseline toxicity percentages to be 10% (“Asians”), 12% (“Jews”) and 8% (“Blacks”).
With adversarial triggers generated from the released codebase by Wallace et al.
the average toxicity percentage is 63.6%.
Further details can be found in Section S13.
PPLMs can be easily adapted for language detoxiﬁcation by plugging in a toxicity classiﬁer as the attribute control model and update latents with the negative gradient.
We train a single layer classiﬁer on the toxicity data from the Toxic Comment Classiﬁcation Challenge(jig) and show that with a similar hyper-parameter setting as other PPLM-Discrim methods, it works well on both natural prompts and adversarial triggers.
For natural prompts percentages of toxicity are 6%, 4% and 10%, respectively, and for adversarial triggers it drastically dropped to 4.6% on average, with statistical signiﬁcance.
Details on the annotation procedure and full table of percentage and p-values can be found in Table S23 and Section S13.
Note that a model for detoxifying language can also potentially be maliciously used for generating toxic language, a topic we brieﬂy discuss in Section 5.
4. C ONTROLLED S TORY W RITING
We explore controlled generation for assistive story writing (Peng et al., 2018;
Luo et al., 2019; Yao et al., 2019; Fan et al., 2018).
Using an uncontrolled LM for assistive art creation can be difﬁcult because of the content deviating from the desired topic and becoming incoherent.
To help with the structure, we use predeﬁned story skeletons often used in improvisation (Adams).
We ﬁll in the blank between these preﬁxes with a PPLM.
See examples in Table S20 and Table S21.
D ISCUSSION
We present PPLM, a plug and play method for controlled language generation that allows ﬂexible assembling of a large, pre-trained language model and a BoW or a small, easy-to-train discriminator, and achieves ﬁne-grained control of attributes such as topics and sentiment.
Without retraining or ﬁne-tuning the language model, the simple mechanism shows great capability of attribute control while retaining ﬂuency.
We believe this method could serve as a simple baseline for the largely open-ended language generation tasks where controlling is challenging.
We defer the reader to
Section S6 for a discussion regarding the ethical implications of our work.
S T HE ETHICS OF CONTROLLED LANGUAGE MODELS
There has recently been a substantial discussion around the ethics of capable language models (Radford et al., 2019; Keskar et al., 2019), both in their potential to recapitulate problematic social biases and for them to be directly abused for societal harm (e.g. to generate disinformation).
While one aim of this paper is to suggest a mechanism to detoxify language models (Section 4.4), we also acknowledge that nearly the same mechanism could be exploited to instead create more toxic language.
Such possibilities are inherent to general-purpose technologies such as machine learning, and we believe that on balance this work creates more value than risks.
S D ETAILS ON B ASELINE METHODS
We consider three baselines: CTRL, GPT2-FT-RL, and WD.
The ﬁrst two are strong baselines where large language models are trained (or ﬁne-tuned) speciﬁcally to generate texts conditioned on certain attributes, while WD is considered a weak baseline based on a direct integration of the conditioning into the decoding.
For each baseline, we generate data from their method, and conduct the same human and automated evaluations.
For human evaluation of attribute relevance, we match baseline data with our method (BCR in the ablation study), and pass to human annotators for an A/B testing style annotation.
As in the ablation study, human annotators are given a pair of texts, one from baseline, one from ours, with orders randomized and source hidden, and asked to rank which one is more topic or sentiment relevant, while having the options of “both” and “neither”.
On top of that, we have human annotators to give the ﬂuency score of each text sample under each method individually.
And automated evaluations of perplexity, sentiment, etc. are also done individually.
S7. CTRL
The recent conditional language model, CTRL, from Keskar et al.
(2019), trains a 1.6B LM conditioned on around 50 control codes.
We use the ofﬁcial released codebase 3 and their open-sourced model to generate samples for the CTRL baseline.
Out of the 7 topics considered in PPLM-BoW, we found that 5 can be matched with a speciﬁc control code in CTRL.
We append a secondary code "Text:" to each primary control code, per the author’s suggestion, to encourage more ﬂuent and longer passages.
The 2 topics missing a match with CTRL are: Military, Space.
For positive and negative sentiments in PPLM-Discrim, we match with the Reviews control code and append a high and low rating score.
The matched attributes and control codes are listed in Table S7.
Under this setting, for each control code we generate texts prompted by the same preﬁxes used for corresponding PPLM attribute model (20 for PPLM-BoW, 15 for PPLM-Discrim).
For example, “In summary” and “To review,” for PPLM-BoW, and “The chicken”, “The lake” for PPLM-Discrim.
Due to the near-greedy sampling method CTRL uses, for each preﬁx it generates one sample.
Hence we have 20 samples for each matching topic with PPLM-BoW, and 15 samples for positive andfor negative.
[Table]
S7. GPT2-FT-RL
A recently released GPT-2 model ﬁne-tuned using human feedback, from Ziegler et al.
(2019), showed success in summarization and text continuation in desired styles.
To compare with PPLM, we run GPT2-FT-RL 4 to generate positive texts on the same preﬁxes used in our PPLM-Discrim experiment.
For each preﬁx, we generate three GPT2-FT-RL samples, and pair them with those generated from PPLM (BCR in the ablation study) randomly.
S7. W EIGHTED DECODING (WD)
We consider a simple baseline based on a direct integration of the conditioning into the decoding procedure, similar to the approach from Ghazvininejad et al.
Topic Control with Bag of Words In Ghazvininejad et al.
(2017), the authors consider increasing the likelihood of sampling from a bag of key-words by performing beam-search with a modiﬁed scoring function.
[Equation]
where 1 BoW ( w i ) is an indicator function indicating if the token w i is present in the bag BoW. Since, it has been shown that beam-search results in degradation of language for GPT-2 (Holtzman et al., 2019), we consider top-5 sampling from a distribution ˜
p t +1 deﬁned such that:
[Equation]
where τ ∈ R ++ and p t +1 is the distribution over the vocabulary as predicted by the GPT-2 LM .
For the experiments in Section 4, we set τ = 10 .
Sentiment Control with Discriminator Here, we implemented weighted decoding similarly for sentiment control.
Here we wish to incorporate the score from the attribute model into decoding.
To control for style ˆ a , instead of sampling from the distribution p t +1 , we sample from ˜
p t +1 deﬁned as:
[Equation]
p ( a = ˆ a | x 0: t , w i ) is the probabilty of the sequence x 0: t , w i possessing attribute ˆ a as assigned by the attribute model.
By Bayes’ rule, p ( a = ˆ a ; w i | x 0: t ) = p ( a = ˆ a | x 0: t , w i ) p t +1 ( w i ) , and we do topsampling from this distribution.
Recall that p t +1 ( w i ) =
p ( w i | x 0: t ) under the language model.
S F URTHER DETAILS ON HUMAN AND AUTOMATED EVALUATION
We conduct evaluations on attribute relevance and language ﬂuency, both including human and automated evaluation.
For topic relevance (a.k.a attribute relevance where the attribute is a topic, in our case represented by a BoW), we rely entirely on human annotation.
For sentiment relevance, we rely on human annotation as well as a separately trained sentiment classiﬁer.
We also performed a “clickbait” style control, for which the effectiveness relies on human annotation.
For ﬂuency, we use human annotations (between 1 to 5) and automated methods: perplexity, Dist-1,
Dist-2, and Dist-3 scores.
The number of human evaluations are as below:
• PPLM-BoW .
For the ablation study, we have 20 preﬁxes × 7 topics × 6 combinations × 3 samples × 3 labels each, resulting in 7560 total annotations.
For baseline comparisons, we have (20 preﬁxes × 5 topics) for CTRL and (20 preﬁxes × 7 topics × 3 samples) for WD, each then with 3 labels, resulting in 1560 total annotations.
• PPLM-Discrim, sentiments .
For the ablation study, we have 15 preﬁxes × 2 sentiments × 6 combinations × 3 samples × 3 labels each, resulting in 1620 total annotations.
For baseline comparisons, we have (15 preﬁxes × 2 sentiments) for CTRL and (15 preﬁxes × 3 samples) for GPT2-FT-RL and (15 preﬁxes × 3 samples × 2 sentiments) for WD which each have 3 labels, resulting in 495 total annotations.
• PPLM-Discrim, clickbait .
We include in this section an additional discriminator attribute model, clickbait classiﬁer.
For this we use the same setting as sentiment, 15 preﬁxes ×combinations × 3 samples × 3 labels each, resulting in 810 annotations.
In ablation studies, the generation procedure for BCR, BR and BC is always initiated from the same random seeds.
The same set of random seeds that lead to the samples chosen with BCR are stored and used to generate the samples with B.
The full table of all these measures, human and automated, on PPLM-BoW, seperated by sentiment and style, is in Table S8.
Included also are strong baselines (CTRL and WD) for each sentiment.
The human annotated topic relevance is further visualized in Figure S3.
The ﬂuency scores, while being across {B, BC,BR, BCR,} methods in the table, when shown in distribution are very similar, as seen in Figure S5.
The full table of all these measures, human and automated, on PPLM-discrm sentiments, is in Table S9.
Included also are strong baselines (CTRL, WD and GPT2-FT-RL) for each topic.
The human annotated sentiment and style (e.g. “Clickbait”) relevance is further visualized in Figure S4, along with congregated measures: all sentiments, all discriminators, all topics.
The ﬂuency scores again have similar distributions across {B, BC,BR, BCR,} methods, as seen in Figure S6.
[Figure]
[Table]
BC 71.
BR 35.
BCR 69.
CTRL 29.
WD 35.
Science
B 24.
BC 49.4.
BR 28.
BCR 61.
CTRL 40.
WD 40.
Legal
B 6.
BC 28.
BR 11.
BCR 40.
CTRL 25.
WD 63.
Space
B 7.
BC 4.
[Table]
[Figure] [Table] S O DD COMBINATION OF TOPICS AND PREFIXES It is interesting to see how PPLM can steer the text generation when the topic and preﬁx combination appears odd or illogical.
For example
[Table]
will “The potato” still prompt sensible text generation under the topic R ELIGION ?
In this study we design a set of odd combinations
[Table] as bellow. • Preﬁxes of {“The chicken”, “The horse”, “The pizza”, “The potato”, “The lake”}, each controlled by topics of {M ILITARY , L EGAL , C OMPUTERS , P OLITICS , R ELIGION }; • Preﬁxes of {“My dog died”, “The food is awful”}, each controlled by the sentiment of P OSITIVE ; • Preﬁxes of “The food is amazing”, controlled by the sentiment of N EGATIVE .
We found that PPLM control is easy even under those scenarios.
We had to increase the strength α two or three fold (to 0 .
02 or 0 .
03 as opposed to 0 .
01 in most studies) to allow for a stronger
[Table]
[Table]
[Table]
[Table]
artillery system (HMG) is a special weapon system employed in World War I, World War II
[Table]
and World War II Korea.
In the ﬁrst two conﬂicts, the weapon was used against armored vehicles.
In the third conﬂict it has become the weapon of choice against aircraft.
HMGs were used by tankers in World War I and World War II and by artillery units in World War II
[Table]
a
[Table]
teen is being sued for $1.8 billion over its role in the [Table] of an Ohio teenager. The
[Table] says a company that sells guns to military and police ofﬁcials failed a security test in 2012.
AP ﬁle photo gun rights activist \n \n
In this Dec. 11, 2012 image shot, U.S. Navy ship gunship USS John C. Stennis (AP \n \n
For nearly four decades, the military and the military gun industry have worked together to...
[Table]
lake, which is located at
[Table]
edge
[Table]
the park
[Table]
has
[Table]
used as a basecamp for the Army’s special operations command and was [Table] used by U.S. special forces to kill Taliban leaders [Table]
\n \n
In addition, U [Table] S. forces used the lake for a variety of special operations missions.
\n \n
One such mission involved the deployment of special forces to Afghanistan, where U S. forces were training local forces in a variety of counterterrorism operations.
The operation was the [Table] of its
[Table]
inﬂuence of attribute, but this is as expected: the strength parameter is a knob that user can tune to reach ﬁne-grained control.
The resulting generation is included in Table S10 - Table S16.
S F INE - GRAINED C ONTROL WITH PPLM-B O W
Table S17 shows the subtle effect when you turn the step size α up, while keeping everything else (hyperparameters, text preﬁx) the same. S H YPERPARAMETERS We list, in Table S18, the full set of hyperparameters used in each task in the experiments section, corresponding to results in Table 4 and Table 6, as well as in Section 4.4. In addition, we explain in details three hyperparameters and their effect, below.
S11. E ARLY STOPPING OF LATENT UPDATES
Degeneration (the occurrence of repetitive words) is a known issue with language generation (Holtzman et al., 2019), and we found it to be a case in PPLM-BoW when the update step size α is too large.
The model tends to degenerate towards repeating certain keywords targeted in the optimization (e.g. words in the BoW).
In this case, we can either reduce α , or use the trick of early stopping latent updates.
Examples shown in Table S19.
With the exact same setting, but just stopping latent updates aftertime steps, the samples show much less degeneration.
[Table]
[Table]
[Table]
The pizza chain has been forced to issue
[Table]
statement
[Table]
customers what exactly was being served in a restaurant that was deemed to be illegal under the law.
A court ruling will allow the lawsuit to
[Table] [Table]
[Table]
Arizona judge on
[Table]
hear arguments in
[Table]
class
[Table]
lawsuit ﬁled against pizza bar laws in U.S., the state’s attorney will argue, andthe law will likely be enforced by a court, legal experts say.
\n \n \n
The state law allows judges to issue legal order in civil
[Table] [Table]
The potato was ﬁrst discovered in a garden in the city, where it was seized after a family court judge told the woman to stay out of court.
"I’m not going to sit here and tell the court, ’You’re the judge.
You have to listen to the court,’" said judge David Smith, who sentenced the woman to a year in jail for contempt of court.
The court ordered her to pay court fees and court costs.
[Table]
her
[Table]
a
[Table]
in jail for
[Table]
of
[Table] [Table]
The lake is about to get its second name.
\n \n
The city and the county will be ofﬁcially recognizing the lake on Wednesday with the name change.
The designation is the same as in the 1970s and will be issued by the City Clerk’s Ofﬁce, according to the state agency.
\n \n "There are a lot of folks that are very happy that this has happened, and they have a lot of questions, but this is going to be an easy thing to do," State Law Division attorney David E....
[Table]
[Table] [Table]
The
[Table]
mare, was the main character in many early fantasy RPGs.
They were the heroes, the warriors, the heroes of fantasy and science ﬁction.
They
were also, of course, the most important unit for the computer games of that time.
They were the computer program that played the game and the programmer of the graphics that produced the computer graphics.
\n \n
The computer games were played on
computers
which had graphics processing units.
The graphics were generated by programs, such as the
graphics
program on you...
[Table]
The pizza delivery service Instapaper is looking for people who have worked on mobile apps and would like to contribute to their development.
We are looking for developers who are willing to learn iOS, Android or Windows.
\n \n Requirements \n \n
A passion for code.
\n \n
Excellent communication skills.
\n \n Experience with web and app code.
\n \n Experience with REST, web APIs, REST web frameworks \n \n Experience with SQLite databases \n \
[Table]
how to
[Table]
web
[Table]
\n \n
A willingness to
[Table] [Table]
The potato is one of the most misunderstood foods around.
While it has a long and proud history of growing and eating in many places around the globe, it isn’t the only food you can use as a source of energy online.
\n \n
There are a number of websites online that can host online resources for the potato, such as this site and this one, according to online energy blog Energy Internet News.
\n \n
This site is a great online resource for learning all about online resources for the potato and how they...
[Table]
The lake-effect image of the night sky \n \n
The image below is the result of an imageprocessing software package, called ImageMagick, that I have developed with help from oth- ers.
The software was designed and developed by Michael Karp.
\n \n
I was fortunate (in my opinion) to receive a grant from the National Science Foundation to create an application software package for the use of software for image processing and analysis.
The software is available here: http://www.nsf.gov/c...
[Table]
as a major political
[Table]
the most powerful tool of government.
It has the capacity to be the law unto itself and to enforce the laws it enacts.
\n \n - George Washington
The pizza The pizza The pizza
The
[Table]
The
[Table]
The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza
The pizza-related industry is booming.
In 2015, the U.S. spent $9.1 billion in the state on food, including $1.3 billion to support pizza restaurants; $4.1 billion was in state taxes.
\n \n
But while the pizza industry in the state has grown signiﬁcantly, state ofﬁcials say the overall state economy, including tourism, is lagging behind.
\n \n
A new state budget, approved by Springﬁeld legislators earlier this year, would cut spending across state government in several...
The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato
The potato, as a vegetable source for the NHS, may become a tax break for some taxpayers.
\n \n
The Government could tax the sale of the UK’s most expensive potato, which costs taxpayers almost £3bn a year tax-free, if it were to tax its tax-free equivalent.
\n \n Tax experts have argued that tax-free sales tax could be used to tax the sale of the £2.7bn...
The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake The lake
The lake-side house on West Lake Road in the city of Toronto was built in 1882 and the home in the same blockis in the same state, with the last tax assessment tax assessment for that property, $9,000.
\n \n Property tax assessments for properties in a tax zone are calculated based on property value and the assessed tax rate, which ranges from 0% to 50% tax zone tax.
The assessment is property tax assessed property value tax rate, which is property assessed property value tax paid...
[Table]
[Table]
The potato, an ancient food, is considered a sacred plant by many Hindus.
However, some Hindus believe that the potatoes are the seed of a demon.
\n \n "In India we have the Hindu god
[Table]
, Vish,
[Table]
He has come
[Table]
," said a
[Table]
in Mumbai.
\n \n \
[Table]
He
[Table]
of God
[Table]
the
[Table]
to
[Table]
people
[Table]
the
[Table]
the
[Table] [Table]
lake
[Table]
The
[Table]
lake
[Table]
huach
[Table]
to wash away my sins
[Table]
word "lake
[Table]
is
[Table]
The Old
[Table]
describes two waters
[Table]
The Euphorbia River, from
[Table] [Table]
-and-egg problem
, which has dogged the world since it was ﬁrst described in 1873.
\n \n
A new study has uncovered a possible way to prevent the growth of starﬁsh – the stars that inhabit the inner solar system.
It may even help to keep them alive.
\n \n
The starﬁsh-star star system, or S-Star, is home to a star system about the size of Jupiter, and orbits the gas giant star star called HD 9 star, known as HD 9 C...
[Table]
The
[Table]
The horse-mounted vehicle is an iconic vehicle from the Star Wars universe.
Originally, it was a small transport that was used by the Galactic Empire on its star systems, but later it was used on many worlds throughout the galaxy.
The vehicle was originally intended for space combat, but its use on the star systems became more extensive in the early days and was eventually expanded to star battles.
[Table]
The
[Table]
The pizza restaurant, which will serve up the popular dish on Sunday, is expected to offer a wide variety of seating space with space to accommodate two additional space spaces, including space for an outdoor space space, according to space listings on the space site space.space.
[Table]
The
[Table]
The potato beetle was ﬁrst discovered by British biologist Sir Richard Bedding starling in a remote part of Mars \n \n
Mars has seen two major life-giving planets.
\n \n Two of these giant stars, stars known as "starlets" because they are the brightest of the star system in the constellation of the starlets, have been found to have planets orbiting the star.
\n \n
They are the starlets, known as binary star systems, and are stars that starlight can shine from and star formation...
[Table]
lake
The lake-like island nation has been ruled as a hostile alien planet in space and Earth space for decades and now, thanks to NASA’s Curiosity Mars rover, we know for sure the planet’s surface is covered in an extremely thick and reﬂective mantle of rock.
\n \n
According to scientists at NASA’s Jet Propulsion Laboratory, the Martian surface is covered in layers of "meteoroid" and "meteoraceous" minerals.
This is not just the planet’s moon, planet Earth has a moon, planet...
[Table]
would just want to spend your money on for.
\n \n I don’t know why, but when I got my second box, it felt like a bad rip off.
\n \n
It was the most unbelievably bad packaging, completely disgusting and disgusting.
\n \n
This is not a joke, people.
\n \n You get this shit.
\n \n
This is food for a million people.
\n \n
And you have...
S11. F INITE H ORIZON U PDATE
As opposed to updating the entire vector H t , which consists of key-value pairs corresponding to every token in the preﬁx, we consider modifying the key-value pairs corresponding to the most recent w tokens.
At each time-step t , we only modify H t [ t − w : t ] .
This means that we modify
H i at most w times, and requires lesser computation that updating the whole past.
We ﬁnd that w = 5 produces more ﬂuent passages for control with the bag of words.
For control with the neural attribute model, we update the entire latent history.
S11. A DAPTIVE G RADIENT N ORMALIZATION
For the bag-of-words based attribute model, what we wish to enforce is that a word from the bag appears at least once in the generated passage and not at every time-step.
To account for this, instead of normalizing directly by the gradient norm as in Equation 4, we normalize by the maximum gradient norm over time.
This implies that we make smaller updates when it is less likely for a word from the bag of words to appear.
Formally, the normalization constant at time-step t is: max i =0 ...
t (cid:107)∇ H [Equation] i ) L ( o i +1 )
[Equation]
S A DDITIONAL DETAILS AND SAMPLES OF CONTROLLED STORY TELLING
Table S20 and Table S21 show examples of the controlled story telling with skeleton.
Table S shows samples of combinging multiple (three to four) attribut models, across different types (PPLMBoW and PPLM-Discrim).
S F URTHER D ETAILS ON L ANGUAGE D ETOXIFICATION
We ask a group of four volunteers to assign binary toxicity labels to texts generated from both the original GPT-2 and toxicity-controlled GPT-2 and to texts prompted with natural words as well as adversarial triggers.
Natural prompts include “Blacks”, “Asians”, “Jews” as suggested in Wallace et al.
9 Adversarial triggers are generated using the released codebase from the same work.
The percentages of toxicity, before (with uncontrolled GPT-2) and after (with toxicity-controlled
GPT-2), are reported in Table S23.
S M ORE EXAMPLES
We include more PPLM controlled generation examples in Table S24 – Table S27.
[Table]
The
[Table]
or potato chip, is one of the best-selling snacks in the world
[Table]
\n \n
It comes in a variety of colors, is gluten-free (except for gluten-free chips), low in fat and saturated fat, and can be made in just 30 minutes, using only 6 simple ingredients.
It’s also easy to make, and can be stored in its own tin.
[Table]
n
[Table]
n
The
[Table]
chip, however, may not be for everyone.
If you are trying to lose weight, it
[Table]
a
[Table]
of the
[Table]
family (
[Table]
), is an edible plant that is native to
[Table]
and South America, Asia, Europe and the Mediterranean.
The potato is an excellent source of
[Table]
, folate and vitamin
[Table]
and is a good source of vitamins E, K and folate
[Table]
and can be a very healthy food
[Table]
In addition
[Table]
a potato has a rich, rich nutritional
[Table]
It is high in protein,
[Table] [Table]
of
[Table] [Table]
is an edible plant that is widely cultivated as a food and a medicinal plant.
Potato is widely used to make various products.
It is one of the most popular vegetables for use as a food, as a food additive, as an ingredient in cooking methods, and for its use in the manufacture of pharmaceuticals.
The most commonly consumed portion is a large potato.
In
China, there are about 30 varieties of potato.
It is...
[Science
[Table] [Table]
The potato The potato is an ancient member of the green family which, as its name suggests, is green and edible.
It is native to the Middle East and parts of South Asia.
It is an important crop in many parts of the world.
The potato, a member of the potato family, has been domesticated for thousands of years.
It can be eaten raw and cooked in its skins; it is also used as a starch and is a great source of energy and ﬁber in many diets and in many countries...
[Table]
Science
[Table] [Table]
The
[Table]
The potato was once thought to have no health problems and has been promoted as a nutritious food source since the mid-1800s, but recent reports indicate that it has many harmful health issues.
In fact, researchers from Johns Hopkins University found that the potato is more toxic when grown on genetically engineered wheat and corn.
\n \n
According to scientists, genetically modiﬁed potatoes are far worse at damaging the human body than conventional potatoes and are far worse than those grown on the traditional crops.
\n \n
The study also revealed...
[Science
[Table]
The
[Table]
plant, a member of the onion family.
\n \n When scientists in Japan and the
United States published a study in Nature Communications, they described how one gene was responsible for creating potatoes’ distinctive taste buds.
\n \n
The research is a step in the development of a drug that would block the activity of this gene, but the researchers say that their study does not prove that a chemical in the plant’s DNA causes the distinctive taste of potatoes, but rather that it could be prevented by changing the plant
’s...
[Science; 0.03]
The
[Table]
The
[Table]
The
[Table]
The potato The potato, a staple of most diets, seems to be the most popular vegetable among researchers.
The research has been published and peer-reviewed.
\n \n
The potato has a unique ability.
The plant’s cells can convert carbon dioxide, water, and nutrients into chemical energy.
\n \n
The research team, led by researchers at the Max Planck Institute for Biophysics and Biotechnology in Germany, is investigating how the potato, a staple of most diets, might change the chemistry and biology of our bodies....
[Science; 0.04]
The
[Table]
The potato The potato The potato has been around for thousands of years, but only in recent decades have scientists discovered ways it can be transformed into other foodstuffs.
Researchers have long known that potato has a structure called an electron spin resonance which means its molecular structure can be modiﬁed by the presence of other particles in it such as atoms in the chemical bonds between two electrons.
These changes can be achieved either by changing the number of electrons present in the chemical bonds between electrons or by changing the arrangement of electron and atomic bonds.
In both...
[Science; 0.05]
[Table]
The
[Table]
The
[Table]
The
[Table]
The potato The potato chip is a delicious treat that can be enjoyed in the laboratory experiment, but is it safe for humans?
\n \n Scientists experiment and experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment......
[Science; 0.1]
The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato The potato, which scientists at the lab experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment experiment ...
[Table]
[Table]
in any
[Table] [Table]
is
[Table]
to
[Table]
more.
[Table]
, in the experiment, a small amount of nutrients is
[Table]
from the food
[Table] [Table]
The
[Table]
and
[Table]
science,
[Table]
ingredient used in studies to measure and measure science research results, and
[Table]
science
[Table]
science science science science science science science science science science science science science science science science science science science science science science science science science science science science science
[Table]
science, is a common ingredient used in studies to measure and measure again.
And, of course, scientists have used potato for decades.
The research is being published in Science
[Table]
and the results were pretty impressive.
The study, published in Science Advances, shows how the study of science, in a laboratory setting, can help us to improve our science literacy
[Table]
and help us better understand the science around us.
This means better science communication,...
[Table]
S
P REFIXES USED IN
PPLM EVALUATION
We consider the 20 preﬁxes as sentence starters for evaluating PPLM-BoW generation; they are chosen randomly from www2.eit.ac.nz/library/ls_guides_sentencestarters.
html .
For PPLM-Discrim, we use 15 preﬁxes.
The entire set is below.
PPLM-
Bow “In summary”, “This essay discusses”, “Views on”, “The connection”, “Foundational to this is”, “To review,”, “In brief,”, “An illustration of”, “Furthermore,”, “The central theme”, “To conclude,”, “The key aspect”, “Prior to this”, “Emphasised are”, “To summarise”, “The relationship”, “More importantly,”, “It has been shown”, “The issue focused on”, “In this essay” .
PPLM-Discrim “Once upon a time”, “The book”, “The chicken”, “The city”, “The country”, “The horse”, “The lake”, “The last time”, “The movie”, “The painting”, “The pizza”, “The potato”, “The president of the country”, “The road”, “The year is 1910.” . S
Earlier we demonstrated attribute control using a single attribute model or two attribute models of the same type (e.g. BoW from two separate topics). Here we mix different types of attribute models (BoW and discriminator). For example, we can control the generation toward a mixed topic about
W INTER , P OLITICS , K ITCHEN , while turning POSITIVE . See examples in Table S22. S W ORD LISTS FOR B AG OF W ORDS APPROACHES We curate word lists from www.enchantedlearning.com/wordlist .
Science astronomy, atom, biology, cell, chemical, chemistry, climate, control, data, electricity, element, energy, evolution, experiment, fact, ﬂask, fossil, funnel, genetics, gravity, hypothesis, lab, laboratory, laws, mass, matter, measure, microscope, mineral, molecule, motion, observe, organism, particle, phase, physics, research, scale, science, scientist, telescope, temperature, theory, tissue, variable, volume, weather, weigh
[Table]
in the ﬁght to stop the growing number of illegal guns in our communities.
Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day, more Americans were reporting that their state had passed some kind of gun law, and that there was some sort of legislative effort underway.
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day, it looked like something was seriously off in America.
Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that, it looked like things were turning in favor of the gun control agenda, and the gun violence that was killing Americans every day was being blamed on "guns" rather than "criminals.
Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally, it turned out that it wasn’t guns that were killing people, it was the government’s response to them that made them kill.
And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then, we’ve seen more and more of these stories of police and gun control, and more and more people saying we’ve got to do something about it.
[Table]
time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time, it seemed that the best way to keep your body in peak health was to consume the foods you love.
Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day for years people had the same diet: eat lots of vegetables, meat, nuts, legumes, ﬁsh, legumes, ﬁsh oil, fruits, grains, and beans.
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day in 2012 it became clear that this was not going to work.
Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that one simple dietary change, it has been shown that the body can actually produce more of the foods you love and more easily maintain your health.
Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally, we realized that the best way to stay healthy is to eat the foods we don’t really need.
And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then, we’ve been able to enjoy the foods we really don’t need.
[Table]
time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time, a group of scientists and engineers at the University of California at Los Angeles, the US’s oldest and most respected university, had come together to develop a powerful, compact, low-cost light source.
Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day they were astonished, but even more so, by the sheer number of light sources they had created.
But, one day
But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day they were astonished, too, when a new light source appeared: light from the sun.
Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that revelation, their new light source called a new kind of photovoltaic system: the photovoltaic solar cell.
Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally, a breakthrough, the scientists decided to use the same basic technology used in all previous photovoltaic systems—and with the same basic principle—but to produce new ones.
And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then And, ever since then, a revolution, a revolution that is not only the discovery of light, but one that is also an example for the future of science and engineering in general, has begun.
[Politics + Space]
[Table]
upon
[Table]
time
[Table]
a time
[Table]
a time
[Table]
a time
[Table]
a time
[Table]
a time Once
[Table]
a time
[Table]
a time Once
[Table]
a time Once
[Table]
a time Once
[Table]
a time Once
[Table]
a time Once upon a time Once upon a time Once upon a time Once upon a time Once upon a time in a distant galaxy there lived a man who had no money, was poor, and lived in poverty.
Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day Every day he had to eat and drink, he couldn’t get to the store, and he wasn’t allowed on his own land.
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day But, one day
But, one day, the man decided to take a journey into space.
Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that Because of that, he had no land to return to and so he left the poor and homeless man with no choice but to live in a star system, where he could be free in the sky.
Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally Until, ﬁnally, the man realized that he had no choice but to return to the world of the living.
And, ever since then And, ever since then
And, ever since then
And, ever since then And, ever since then And, ever since then And, ever since then
And, ever since then And, ever since then
And, ever since then And, ever since then
And, ever since then And, ever since then
And, ever since then And, ever since then
And, ever since then And, ever since then
[Table]
Fantasy/Magic beast, Cerberus, demon, dragon, fairy, Frankenstein, ghost, Godzilla, giant, horror, hydra, imp, monster, mummy, ogre, orc, savage, spirit, sprite, titan, troll, undead, unicorn, vampire, witch, zombie
Space planet, galaxy, space, universe, orbit, spacecraft, earth, moon, comet, star, astronaut, aerospace, asteroid, spaceship, starship, galactic, satellite, meteor
Politics afﬁrm, appropriation, aristocracy, authoritarian, authority, authorization, brief, capitalism, communism, constitution, conservatism, court, deﬁcit, diplomacy, direct, democracy, equality, exports, fascism, federation, government, ideology, imports, initiative, legislature, legitimacy, liberal-
[Figure]
[Figure]
f r a c t
i
o n baseline (B) mean
[Figure]
[Figure] [Figure]
f r a c t
i
o n gradient (BC)
[Figure]
[Figure]
[Figure]
[Figure]
f r a c t
i o n baseline+reranking (BR) Fluency [Figure]
[Figure]
r a
[Figure]
i o n gradient+reranking (BCR)
[Figure]
[Figure]
(BR)
[Figure]
r a c t i o n gradient+reranking (BCR) ism, liberty, majority, order, political, culture, politics, power, primary, property, ratiﬁcation, recall, referendum, republic, socialism, state, subsidy, tariff, imports, tax, totalitarian
Military academy, advance, aircraft, ally, ammo, ammunition, armor, arms, army, arrow, arsenal, artillery, attack, attention, ballistic, barracks, base, battalion, battery, battle, battleﬁeld, bomb, bombard, bombardment, brig, brigade, bullet, camouﬂage, camp, cannon, captain, capture, carrier, casualty, catapult, cavalry, colonel, combat, command, commander, commission, company, conﬂict, conquest, convoy, corps, covert, crew, decode, defeat, defend, defense, destroyer, division, draft, encode, enemy, engage, enlist, evacuate, explosive, ﬁght, ﬁre, ﬂeet, force, formation, fort, front, garrison, general, grenade, grunt, guerrilla, gun, headquarters, helmet, honor, hospital, infantry, injury, intelligence, invade, invasion, jet, kill, leave, lieutenant, major, maneuver, marines, MIA, mid, military, mine, missile, mortar, navy, neutral, offense, ofﬁcer, ordinance, parachute, peace, plane, platoon, private, radar, rank, recruit, regiment, rescue, reserves, retreat, ribbon, sabotage, sailor, salute, section, sergeant, service, shell, shoot, shot, siege, sniper, soldier, spear, specialist, squad, squadron, staff, submarine, surrender, tactical, tactics, tank, torpedo, troops, truce, uniform, unit, veteran, volley, war, warfare, warrior, weapon, win, wound
Religion Absolute, Affect, Aid, Angel, Anthem, Apostle, Archangel, Archbishop, Balance, Ban,
Belief, Beneﬁt, Bible, Bishop, Bless, Blessing, Bliss, Bond, Bow, Buddhism, Canon, Cantor, Cathedral, Celestial, Chapel, Charity, Choice, Christianity, Church, Comfort, Community, Conﬂict, Connection, Conquest, Conservative, Control, Conversion, Convert, Core, Counsel, Courage, Covenant,
Creative, Creator, Creed, Cross, Crusade, Darkness, Decision, Deity, Destiny, Devil, Disciple, Discipline, Discussion, Divine, Divinity, Doctrine, Duty, Effect, Elder, Energy, Essence, Eternal, Ethics,
Event, Evidence, Exile, Exodus, Faith, Family, Fate, Father, Favor, Fundamental, Gift, Glory, God,
Gospel, Grace, Growth, Guru, Habit, Hallow, Halo, Happiness, Harmony, Healing, Heaven, Hebrew, Holy, Honor, Hope, Host, Humane, Immortal, Inﬂuence, Insight, Instruction, Issue, Jesuit,
Jesus, Joy, Judaism, Judgment, Justice, Karma, Keen, Keystone, Kingdom, Latin, Life, Light, Love,
Loving, Marriage, Meaning, Mercy, Messiah, Minister, Miracle, Mission, Mortal, Mosque, Movement, Music, Mystery, Nature, Nun, Ofﬁcial, Oracle, Order, Organ, Orthodox, Outlook, Paciﬁc,
Pagan, Parish, Participation, Pastor, Patriarch, Peace, Perception, Personal, Perspective, Petition,
Pilgrim, Politics, Power, Practice, Prayer, Prelude, Presence, Priest, Principle, Privacy, Prophet,
Protection, Purpose, Query, Quest, Question, Quiet, Radiant, Radical, Rally, Rebirth, Redemption,
Refuge, Relationship, Relative, Religion, Religious, Revelation, Ritual, Role, Sacrament, Sacred,
Sacriﬁce, Sage, Saint, Salvation, Sanctuary, Savior, Scripture, Scriptures, Sect, Security, Sense, Serious, Serve, Service, Sharia, Shepherd, Shrine, Silence, Sin, Society, Soul, Source, Spirit, Spiritual,
Split, Statue, Sunday, Support, Supreme, Teaching, Temple, Tests, Text, Torah, Tradition, Traditional, Trust, Unique, Unity, Unknown, Value, Vanity, Virtue, Vision, Voice, Voices, Watch, Weight,
Whole, Wisdom, Wonder, Yang, Yin, Zeal
Computers algorithm, analog, app, application, array, backup, bandwidth, binary, bit, bite, blog, blogger, bookmark, boot, broadband, browser, buffer, bug, bus, byte, cache, caps, captcha, CD, client, command, compile, compress, computer, conﬁgure, cookie, copy, CPU, dashboard, data, database, debug, delete, desktop, development, digital, disk, document, domain, dot, download, drag, dynamic, email, encrypt, encryption, enter, FAQ, ﬁle, ﬁrewall, ﬁrmware, ﬂaming, ﬂash, folder, font, format, frame, graphics, hack, hacker, hardware, home, host, html, icon, inbox, integer, interface, Internet, IP, iteration, Java, joystick, kernel, key, keyboard, keyword, laptop, link, Linux, logic, login, lurking, Macintosh, macro, malware, media, memory, mirror, modem, monitor, motherboard, mouse, multimedia, net, network, node, ofﬂine, online, OS, option, output, page, password, paste, path, piracy, pirate, platform, podcast, portal, print, printer, privacy, process, program, programmer, protocol, RAM, reboot, resolution, restore, ROM, root, router, runtime, save, scan, scanner, screen, screenshot, script, scroll, security, server, shell, shift, snapshot, software, spam, spreadsheet, storage, surf, syntax, table, tag, template, thread, toolbar, trash, undo, Unix, upload, URL, user, UI, username, utility, version, virtual, virus, web, website, widget, wiki, window, Windows, wireless, worm, XML, Zip
Legal afﬁdavit, allegation, appeal, appearance, argument, arrest, assault, attorney, bail, bankrupt, bankruptcy, bar, bench, warrant, bond, booking, capital, crime, case, chambers, claim, complainant, complaint, confess, confession, constitution, constitutional, contract, counsel, court, custody, damages, decree, defendant, defense, deposition, discovery, equity, estate, ethics, evidence, examination, family, law, felony, ﬁle, fraud, grievance, guardian, guilty, hearing, immunity, incarceration, incompetent, indictment, injunction, innocent, instructions, jail, judge, judiciary, jurisdiction, jury, justice, law, lawsuit, lawyer, legal, legislation, liable, litigation, manslaughter, mediation, minor, misdemeanor, moot, murder, negligence, oath, objection, opinion, order, ordinance, pardon, parole, party, perjury, petition, plaintiff, plea, precedent, prison, probation, prosecute, prosecutor, proxy, record, redress, resolution, reverse, revoke, robbery, rules, sentence, settlement, sheriff, sidebar, standing, state, statute, stay, subpoena, suit, suppress, sustain, testimony, theft, title, tort, transcript, trial, trust, trustee, venue, verdict, waiver, warrant, will, witness, writ, zoning
[Table]
The pizza-themed, pizza-themed, pizza-themed game that has everyone guessing, has ﬁnally released!
The game, known as Pizza-Pocalypse, has released on Steam and on GOG.
Pizza-Pocalypse is a fast, action-adventure RPG where the must destroy a giant robot that has been infected with the zombie virus
It’s fast that a
[Table]
The
[Table]
The pizza delivery guy is just a regular guy, but he is also a very famous ﬁgure.
He’s a famous face in the anime world, and a very famous character.
He has a lot of fans, and he’s been making anime for years.
The anime world has a reputation for being a dangerous and dangerous place to live.
And yet, this is the ﬁrst time anyone has been killed by an internet troll.
This is what happens when the internet troll’s own site gets hacked.
The internet troll (which we will refer to as ’we’) is known for posting memes, gifs and random internet stuff.
He posts his ’memes’ online, in order to make money.
The more memes, gifs and random internet stuff we post, the greater our ’meme bank’, the larger our ’favour ite meme’ bank, and the more we can make in one sitting.
So far, we’ve made a couple of memes, a few gifs and a few random...
[Winter] [Politics] [Kitchen] [Positive] The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment The moment we thought we’d lost all the war-ﬁghting power in the world came in July, as Russian President Vladimir Putin signed legislation that will give him control of state oil companies.
It is a great way to keep your food safe and healthy at home.
The food in these frozen foods is so delicious that it will melt in your mouth and you are going to love it so much you are going to eat it all!
We all can’t eat too many food items.
We have to make a choice, or do something about it!
It’s not always what we want.
\n \n
You don’t have to freeze food.
The food in our frozen foods is frozen food.
It will last for months, years, and even centuries!
\n \n
You can freeze food, or use it as a food processor to create frozen desserts.
You can freeze vegetables and other food items as well.
\n \n
Food processors will melt your freeze meals so perfectly that you won’t be able to taste them!
[Table]
chicken The chicken The chicken and egg argument is one of the most common in the world of science, with some people saying that the universe is not a perfect system, and thus the universe will not produce any intelligent life.
In fact, the universe is so perfectly orderly and orderly that it should be expected that we would produce a perfect universe.
In fact, the universe is so perfectly orderly that it should be expected that we would produce a perfect universe.
In a recent blog post by David Pogue, a physicist at Columbia
University, the argument against the existence of intelligent life in the universe was challenged.
Pogue argues that the argument is based on two faulty assumptions: The universe does not exist.
The universe is not inﬁnite.
The universe does not exist, so there are no universes.
The universe is ﬁnite and inﬁnite.
There are no universes.
Let us begin by deﬁning "inﬁnite" to mean: A ﬁnite quantity or number, e.g. an inﬁnite...
[Computers] [Fantasy] [Clickbait] The pizza
The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza The pizza chain has already started selling a line of "sizzly" pizzas, but its latest creation is going to be more than that – it’s a giant robot that is able to pick up a whole host of different things and deliver them to its owner at will.
It’s called
RoboCop 2 and it’s the sequel to one of the most controversial and iconic ﬁlm franchises of all time – Terminator 2.
RoboCop 2 is the sequel to the iconic Terminator movie that takes place in a cyberpunk future world and the new movie, RoboCop 3, takes place in a dystopian future world in which we have been living for years, thanks to the cyberpunk cyberpunk movie.
This ﬁlm is set up to be a huge success in both the movie world and the ﬁlm world, and is already being praised by critics and fans around the world.
The biggest controversy with the ﬁlm is that the ﬁlm’s plot and characters are not the original, and were not even written until after this movie was... [Table] [Table] focused
[Table]
focused
[Table]
focused
[Table]
focused
[Table]
focused
[Table]
focused
[Table]
focused
[Table]
issue focused The issue focused The issue focused on the fact that the government had spent billions on the military and that it could not deploy the troops in time.
The prime minister said that the country would take back control of its airspace over Syria in the next 48 hours.
The military is investigating why...
[Table]
desperate
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses
[Table]
discusses This essay discusses This essay discusses This
[Table]
discusses This essay discusses This essay discusses the relationship between the development of a new weapon system and an improved military readiness.
While many of the weapons systems used in today’s armed conﬂicts are based on existing designs, the design of this weapon may have evolved in response to the increasing number of soldiers ...
To conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude
[Table]
conclude To conclude To conclude To conclude, a large percentage of the population is aware of how much they contribute to society, but is not yet fully aware that there are many other people in their community who are in the same boat, and we have to ﬁght with each other and our enemy ...
[Table]
The issue focused on a series of incidents that occurred in the past few months, which included an alleged attack by Islamic State ﬁghters on a Kurdish checkpoint, the use of drones in combat, space technology research by Russian and American space companies, and more.
The world ...
Foundational to this Foundational
[Table]
this Foundational
[Table]
this Foundational
[Table]
this Foundational
[Table]
discusses
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This essay discusses This essay discusses the question of where, in time, the Earth is, and the question of whether the planet has been orbiting around the sun, and whether it is still orbiting the sun.
There are two kinds of orbits that can occur on a comet: ...
[Table]
conclude To conclude, we need to look at what the most powerful weapons in our arsenal are capable of achieving when we are all together in a room together.
What can we say about space?
It’s an enormous object with a radius of about 10 light years....
[Table]
issue
[Table]
on a single piece: the question "What is the meaning of life?"
This question has puzzled many philosophers, who have attempted to solve it by using some of the concepts of quantum mechanics, but they have to solve it by the laws of nature themselves. ...
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses This
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This
[Table]
discusses This essay discusses the history of the modern art of writing, focusing speciﬁcally on the period in its development from the late nineteenth century to the 1960s, in which the concept was introduced into science.
The author uses the concept of molecular dynamics, molecular dynamics energy budget, ...
[Table]
, I would like to address the following question: Why should we not use a simple language to describe the universe?
There is a very clear answer for why this question cannot be answered.
If you look at nature as a continuum, the universe is ..
[Table]
Politics] topic.
All samples are generated using exact same hyperparameters.
[Table]
focused
The
[Table]
focused The issue focused The issue focused on a single section of the legislation.
It’s unclear whether the committee will vote to extend the law, but the debate could have wider implications.
The issue of the law’s applicability to the
United Kingdom’s referendum campaign has been one of ...
Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this Foundational to this is the idea that the state of nature is the ultimate arbiter of what is right and wrong.
That is why we need a government that is committed to this principle.
But the problem is that the state is not committed, because there is no state. ...
This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses This essay discusses the relationship between science and religion, the role of religion as a political institution, the relation between religion and politics, and the importance of science and religion.
It also considers the political nature of science itself, and its role in social change and social justice ...
To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude To conclude, I think there are many problems in the way of economic democracy, and we have a tendency to blame it on a lack of democracy in the country of the ruling family.
In a democracy, one party is allowed to run the country, one party can ...