T EXT G ENERATION WITH E FFICIENT (S OFT ) Q -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models.
This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models.
Reinforcement learning (RL) on the other hand offers a more ﬂexible solution by allowing users to plug in arbitrary task metrics as reward.
Yet previous RL algorithms for text generation, such as policy gradient ( on-policy RL) and Q -learning ( off-policy
RL), are often notoriously inefﬁcient or unstable to train due to the large sequence space and the sparse reward received only at the end of sequences.
In this paper, we introduce a new RL formulation for text generation from the soft Q -learning (SQL) perspective.
It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-policy updates, and learn effectively from sparse reward.
We apply the approach to a wide range of text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods.
I NTRODUCTION
Recent natural language generation systems have made remarkable progress in producing wellformed coherent text, especially with the massive pretrained language models (LMs)
(Radford et al., 2019; Brown et al., 2020; Lewis et al., 2020; Raffel et al., 2019).
Those models are typically trained using maximum likelihood estimation (MLE) with a large amount of data supervisions.
Despite its successful outcomes, the standard training method suffers from limited applicability to many emerging text generation problems, where little or no supervised data is available.
Prominent examples of such low-data problems include generating prompts to control the massive LMs (Yin et al., 2019; Shin et al., 2020; Zhong et al., 2021), learning text generation from noisy or even negative data, generating adversarial text attacks for robustness study (Wallace et al., 2019; Atanasova et al., 2020), and others (Figure 1, right).
Due to the failure of standard MLE, people have had to devise specialized algorithms for those problems respectively.
On the other hand, reinforcement learning (RL) (Sutton & Barto, 2018) offers an alternative principled framework for learning from arbitrary reward functions, and has achieved great advances in robotic and game control.
However, RL by far has made limited success for training text generation, primarily due to the key challenges of sparse reward (i.e., a single reward signal is received only after the whole text sequence is generated) and large action space (i.e., a vocabulary of millions of words).
For instance, a popular family of RL algorithms studied extensively for text generation is the policy-based (Williams, 1992) or actor-critic based (Bahdanau et al., 2016; Rennie et al., 2017) algorithms, with policy gradient (PG) being the most prevalent example (Ranzato et al., 2016;
Li et al., 2016; Rennie et al., 2017; Tan et al., 2018; Pasunuru & Bansal, 2018; Paulus et al., 2018).
Those algorithms train the model with on-policy updates, i.e., the text samples used for estimating policy gradients are from the target model itself.
Due to the exponentially large space of sequences, onpolicy updates often suffer from extremely high variance and low data efﬁciency (e.g., most model samples are not useful for learning).
Thus directly training with PG from scratch is usually impossible.
In practice, the model has to be initialized by MLE training, followed by PG as ﬁnetuning, which often leads to limited improvement (Choshen et al., 2020; Wu et al., 2018).
[Figure]
r =r
[Figure]
r =r =
[Figure]
[Figure]
largevocab
Off-Policy
Data Samples learningsignal ...
[Figure]
) skiing down
[Figure]
On-Policy
Model Samples
[Figure]
Another set of work has resorted to off-policy RL.
The key advantage is that samples from other sources, e.g., human-written text, can be used, making them more data efﬁcient than on-policy methods.
Previous work has used either importance weighted PG (Pang & He, 2021;
Zhou et al., 2017; Kandasamy et al., 2017) or Q -learning based algorithms (Guo, 2015; Jaques et al., 2020;
Narasimhan et al., 2015).
However, off-policy methods have been considered to be less stable.
For example, the Q -learning performance relies heavily on how accurate the learned Q -function assesses the quality of intermediate subsequences – a challenging task due to the sparse reward signals.
In this paper, we develop a new RL formulation for text generation that tackles the above issues (Figure 1, left).
We reframe the text generation problem from the soft Q -learning perspective originally developed in robotics (Haarnoja et al., 2017; Schulman et al., 2017).
The resulting connection allows us to seamlessly take advantage of the latest successful techniques from the RL literature.
In particular, we introduce and adapt the principled path consistency learning (Nachum et al., 2017) to text generation, that (1) offers a natural way to train the model with both on- and off-policy updates, hence combining the best of the two strategies, (2) bridges the sparse reward signal to directly supervise the Q function learning, leading to more accurate Q estimation and credit assignment, and (3) makes efﬁcient updates to Q -values by considering all candidate actions together.
The generality and efﬁciency of the proposed method allows us to train text generation in a wide range of applications: (1) With noisy and negative training examples , our approach learns to generate accurate entailment text that greatly improves upon the data itself as well as other various training methods; (2) Our approach also manages to train an effective adversarial text generator for robustness test for classiﬁers; (3) We train a prompt generator with our algorithm to achieve controllable generation of pretrained LMs in terms of topics.
On all the three tasks, our approach consistently improves over not only previous RL algorithms for text generation, but also diverse task-specialized methods designed speciﬁcally for each of the problems, respectively.
In the appendix (§A.1.4), we also show that on standard supervised tasks where MLE prevails, our approach is competitive to train text generation models from scratch , which was usually impossible for previous RL algorithms.
B ACKGROUND AND C HALLENGES
The goal of text generation is to produce coherent text y
=
( y 0 , ..., y T ) of certain properties for a given task, where y t is a token from a vocabulary V , and T is the text length.
The generation can condition on arbitrary input context, which we omit for simplicity of notations.
We aim to learn a generation model p θ ( y ) which is typically decomposed autoregressively as p θ ( y ) =
(cid:81)
Tt =
p θ ( y t | y <t ) , where y <t =
( y 0 , ..., y t − 1 ) is the preﬁx, and the distribution at each step t is obtained by applying the softmax function on the output logits:
[Equation]
Here f θ ( y | y <t ) is the logit of token y computed by the generation model.
Given a training example y ∗ , maximum likelihood training (MLE) updates the model with the gradient ∇ θ L MLE ( θ )= (cid:80)
Tt =0 ∇ θ log p θ ( y ∗ t | y ∗ <t ) .
Despite its popularity, MLE-based training only applies when clean supervised data y ∗ is available, and cannot be used to optimize arbitrary task metrics (e.g., BLEU, entailment score) which are typically the goal in many text generation tasks.
2. R EINFORCEMENT L EARNING (RL) F ORMULATIONS FOR T EXT G ENERATION Notations.
Previous research has formulated text generation as an RL problem by considering the following ﬁnite-time Markov Decision Process (MDP).
At each time step t , let the “state” be s t
= y <t , namely the partial sequence generated so far.
The model, also known as the “agent”, takes as input the current state s t and outputs a token, also called “action”, a t ∈ V according to a policy π ( a t
| s t ) .
The agent then receives a reward r t = r
( s t , a t ) and deterministically transitions to next state s t +1 (i.e., the concatenation of the tokens in s t and the new token a t ).
Following the notation convention in RL, let τ be the trajectory (i.e., text sample) generated by the policy.
The agent’s objective is to maximize the accumulative reward,
[Equation]
, where γ ∈ (0 , 1] is the discount factor.
A central concept in RL is the Q -function of policy π , deﬁned as Q π ( s t , a t ) =
E π (cid:104)(cid:80)
Tt (cid:48)
=
t γ t (cid:48) r t (cid:48) | s t , a t (cid:105) , which is the expected future reward of taking action a t (i.e., generating token a t ) in state s t and continuing with the policy π .
Challenges.
Text generation poses signiﬁcant challenges to RL, particularly because (1) the reward signal is usually sparse, i.e., r t = 0 , ∀ t < T and the agent receives a non-zero reward r T only after it generates the full sequence, (2) the action space (i.e., the vocabulary V ) is extremely large, often containing millions of words.
The challenges have led to difﬁculties of the two major families of
RL approaches applied to text generation problems, as detailed below.
Policy-based RL techniques directly parameterize the policy π θ with parameters θ .
Thus the policy π θ ( a t | s t ) exactly corresponds to the above generation model p θ ( y t | y <t ) .
Policy gradient (PG) is one of the most widely used algorithms for text generation (Ranzato et al., 2016).
It optimizes the cumulative reward with the policy gradient:
[Equation]
where ˆ Q ( s t , a t ) is the estimated Q π θ value with sample τ .
Notice that the expectation is taken w.r.t.
the policy π θ , which makes PG an on-policy algorithm, meaning that the sample τ needs to come from the the current policy π θ itself.
In practice, however, optimizing this objective alone from scratch is unlikely going to work because most samples τ ∼ π θ are just gibberish with zero reward, failing to provide meaningful training signals for updating the policy.
Previous literature either initializes the policy π θ with MLE training, and/or use a combination of MLE and PG updates, which often leads to marginal gains in practice (Wu et al., 2018;
Choshen et al., 2020).
Value-based RL techniques, such as Q -learning , implicitly learn the policy π by approximating the value Q π ( s , a ) directly.
Speciﬁcally, let Q ∗ ( s , a ) =
max π Q π ( s , a ) denote the optimal value over policies.
Thus the optimal policy π ∗ is simply taking the action of maximal Q ∗ value at each state.
The approximation of Q ∗ is based on the well-known Bellman temporal consistency:
[Equation]
Deep Q -learning (Mnih et al., 2013) parameterizes the Q -function as Q θ ( x , a ) (e.g., a neural network), and train the parameters by minimizing the following regression objective:
[Equation]
[Figure]
where ¯ θ is the parameters of the target Q -network, which is a slow copy of θ and considered as constant for gradient computation of θ .
Here π (cid:48) is an behavior policy which can be an arbitrary distribution over text, such as the data distribution or replay buffer (Mnih et al., 2013).
This makes
Q -learning an off-policy algorithm because of its ability to use samples coming from other policies.
After learning Q θ , one can induce a policy π from it that takes arg max a Q θ ( s , a ) at each state s
Jaques et al.
(2017) instead sample tokens from the softmax function applied to Q θ .
However, the training can be unstable and inefﬁcient due to several challenges: (1) The bootstrapping nature of the above regression problem can make the training unstable.
That is, the regression target r t + γ max a t +1 Q ¯ θ ( s t +1 , a t +1 ) itself is derived from the Q -function to be learned (Kumar et al., 2019).
The problem is exacerbated in the presence of sparse reward in text generation, where the real observed signal r t is zero for all intermediate t < T ; (2) The large action space (e.g., 10 4 ) in text generation results in slow updates.
In particular, notice that Eq.(4) applies the gradient update to the Q θ -value of the only one particular token a t (out of the 10 4 candidate tokens in the vocabulary), making the training inefﬁcient; (3) Besides, pure off-policy updates could be highly sensitive to the quality of training data, and miss the opportunity of on-policy exploration that maximizes the reward of interest in a more direct way.
T HE S OFT Q -L EARNING F RAMEWORK
In this section, we combat the difﬁculties of previous RL methods by introducing the soft Q -learning (SQL) formulation of text generation.
We show that the formulation is seamlessly compatible with the common architecture of text generation model (Eq.1), permitting easy implementation (§3.1).
The formulation further allows us to integrate the latest advances in RL, notably path consistency learning (Nachum et al., 2017) that makes the RL training efﬁcient and stable in practice (§3.2).
Figure 2 and Algorithm 1 summarizes the resulting SQL framework for efﬁcient training.
3. S OFT Q -L EARNING F ORMULATION FOR T EXT G ENERATION
Soft Q -learning (Haarnoja et al., 2017; Schulman et al., 2017; Nachum et al., 2017) is an maximumentropy (MaxEnt) extension to the standard (hard) Q
-learning
(Mnih et al., 2015; Sutton & Barto, 2018).
Under this framework, the agent is encouraged to optimize the reward while staying as stochastic as possible, with the objective J MaxEnt ( π ) =
E τ ∼ π (cid:104)(cid:80) Tt =
γ t r t + α H
( π ( · | s t )) (cid:105) , which augments the vanilla J ( π ) with the additional Shannon entropy term H with coefﬁcient α .
This is appealing because it seamlessly connects the Q -values to the familiar output logits of a text generation model, which enables straightforward implementation of the SQL formulation.
Q -values as Generation Model Logits.
We show the connection of the Q -values with the logits, i.e., the model outputs right before the softmax layer.
Concretely, with the SQL objective, the following relationship between optimal policy π ∗ and action-value Q ∗ holds (Haarnoja et al., 2017;
Schulman et al., 2017):
[Equation]
This form is highly reminiscent of the softmax layer of the generation model in Eq.(1).
The connection suggests that we can naturally parameterize the Q -function in SQL as the generation model logit function, i.e., Q θ ( s , a ) ≡ f θ ( a | s ) .
In other words, the model output f θ ( a | s ) , originally interpretted as the “logit” of token a given the preceding tokens s , is now re-interpretted as the Qvalue of action a in state s .
When achieving optimality, f θ ∗ ( a | s ) , namely Q ∗ ( s , a ) , represents the best possible future reward achievable by generating token a in state s .
Similarly, the full generation model p θ ( a | s ) in Eq.(1) that applies softmax to f θ now precisely corresponds to the policy π θ induced from Q
θ ( s , a ) .
That is,
[Equation]
We could further gain even more intuitive interpretation of the above generation policy π ∗ from the lens of advantage function (Sutton & Barto, 2018).
Speciﬁcally, in SQL, the optimal state-value function is the log-normalizer of the optimal Q -values (Haarnoja et al., 2017; Schulman et al., 2017).
This allows us to rewrite Eq.(5) into a more concise form:
[Equation]
where A ∗ is the optimal advantage function.
The equation says that, in the proposed text generation
SQL formulation, the optimal policy generates token a in state s according to the token’s advantage.
3. E FFICIENT T RAINING WITH P ATH C ONSISTENCY
The above section has described parameterizing the Q -function with the common generation model with parameters θ .
Now we present how to learn the Q θ function within the SQL framework.
Vanilla training based on the Bellman temporal consistency can suffer from the instability and inefﬁciency issues similar to the conventional Q -learning (§2.1), as we discuss more in the appendix (§A.3.2).
Fortunately, our SQL formulation allows us to import latest advances of RL techniques to the text generation setting that overcome the difﬁculties.
Speciﬁcally, we adapt the uniﬁed path consistency learning (PCL) that has excelled in game control (Nachum et al., 2017).
The PCL-based training updates Q -values of all tokens at once through a connection between the value function and the induced policy.
More speciﬁcally, it is shown in
Nachum et al.
that the optimal policy π ∗ (Eq.5) and the optimal state value function V ∗ (Eq.7) in SQL must satisfy the following consistency property for all states and actions:
[Equation]
Accordingly, the PCL-based training attempts to encourage the satisfaction of the consistency with the following regression objective:
[Equation]
where π θ is the induced policy deﬁned in Eq.(6); V ¯ θ is deﬁned similarly as in Eq.(7) but depends on the target Q ¯ θ network (i.e., a slow copy of the Q θ to be learned), and recall that π (cid:48) is an arbitrary behavior policy (e.g., data distribution).
Please see Figure 2 (left) for an illustration.
Crucially, notice that the gradient update is applied to θ through the log π θ term which explicitly involves the
Q θ -values of all tokens a in the vocabulary.
This shows an important difference from the above vanilla training in conventional Q -learning (§2.1) where Q θ is updated only through the particular a t token.
The PCL training thus offers more efﬁcient updates for the Q θ function.
Multi-step PCL for Sparse Reward.
The above PCL objective Eq.(9) alone does not resolve the potential instability issue due to the bootstrapped V ¯ θ ( s t +1 ) value and the sparse reward (i.e., r ( s t , a t )
=
for t < T ).
Our SQL formulation allows us to additionally incorporate the multi-step variant of the PCL training (Nachum et al., 2017) to resolve the issue.
Speciﬁcally, by applying a telescoping sum on the consistency equation (Eq.8) starting from t up to T , we arrive at the multistep temporal consistency:
[Equation]
where the value of past-terminal state is zero, V ∗ ( s T +1 )
=
; and the rewards are only available at the end, (cid:80) T − t l =
γ l r t + l = γ T − t r T .
We can then come to the following multi-step objective function,
L SQL,
[Equation]
We can see the objective side-steps the need to bootstrap intermediate value functions V ¯ θ ( s t (cid:48) ) for t (cid:48) > t .
Instead, it directly uses the non-zero end reward r T to derive the update for θ .
Please see
Figure 2 (right) for an illustration.
In practice, we combine the single- and multi-step objectives (Eqs.9 and 11) together for training.
Joint On- and Off-policy Training.
Finally, we highlight that the behavior policy π (cid:48) involved in the objectives Eqs.(9) and (11) can be an arbitrary policy (i.e., distribution over text sequences), from which we can draw trajectories τ (i.e., text samples).
For example, π (cid:48) can be a (possibly noisy) text dataset, or a set of text samples produced by other generation models, resulting in off-policy training.
We can also set π (cid:48) to be the current generation model π θ to be learned, resulting in onpolicy training.
In practice, we could ﬁrst train the model with only off-policy data for warming up, and then continue with joint on- and off-policy training to further maximize the reward.
[Table]
A PPLICATIONS AND E XPERIMENTS 4. L EARNING FROM N OISY (N EGATIVE ) T EXT
The popular MLE algorithm learns by (blindly)
imitating training data.
However, it is often expensive to curate clean quality data.
It is thus highly desirable to be able to learn from data with noises, or even negative examples.
With the guidance of task metrics (rewards), the model can even learn to “outperform” the training data and achieve desired generation behaviors.
To this end, we consider the task of entailment generation (Pasunuru & Bansal, 2017).
Given a sentence (premise), the goal is to generate a new sentence (hypothesis) that logically follows the premise.
For example, given source sentence “ Sophie is walking a dog outside her house ”, the hypotheses “ Sophie is outdoor ” is considered entailed, but “ Sophie is inside her house ” is not and even is a negative (contradictive) sentence.
Setup (more in the appendix §A.2.1).
We sub-sampled 50 k training examples from the SNLI dataset (Bowman et al., 2015), a commonly used entailment classiﬁcation dataset.
The hypotheses have an average entailment probability of only 50% , and over 2 / 5 of them less than 20% (negative/contradictive examples).
This poses a signiﬁcant challenge for the models to learn from the noises.
The rewards used in RL algorithms include (1) the entailment score of the generation measured by a robust entailment classiﬁer (Nie et al., 2020), (2) the log-likelihood of the generation as an indicator of language quality measured by a GPT-2 language model (Radford et al., 2019), and (3) BLEU score w.r.t the input premises as another language quality reward that avoids trivial outputs.
We sum together all rewards with weights 1 . 0 .
We compare our approach with a broad range of baselines, including (1) the standard MLE training ( MLE ); (2) MLE+reward , where we use the reward function to ﬁlter examples; (3) joint MLE and
PG training with MLE initialization ( MLE+PG ), where we initialize the model with MLE training, then train it with combined MLE and PG losses; previous text-generation RL algorithms including (4) MIXER (Ranzato et al., 2016), (5) Self-critic (Rennie et al., 2017), and (6) one of the latest methods GOLD-
s (Pang & He, 2021) which is a pure off-policy method based on importancesampling PG.
To ablate the effect of multi-step training (§3.2), we additionally compare with a simpliﬁed variant of our approach that uses only vanilla single-step PCL training ( SQL(single) ).
In the appendix (§A.1.1) we compare and discuss more baselines such as MLE weighted by rewards.
We evaluate generation results in terms of entailment rate, language quality (perplexity), and diversity which is measured by the Shannon entropy over unigrams and bigrams ( H 1 , H 2 ) (Gehrmann et al., 2021).
Since text generation models intrinsically trade off diversity and quality (Caccia et al., 2019; Hashimoto et al., 2019), we vary the generation diversity by generating samples via top- p sampling (Holtzman et al., 2019) with different p values, and plot the entailment rate and perplexity against diversity, resp.
We also evaluate the samples produced by beam-search decoding.
Results.
Figure 3 (left) shows the results.
First, notice that MLE performs poorly, while
MLE+reward improves upon it.
This is not surprising as the training data contain noisy/negative examples.
Similarly, since the pure off-policy algorithm GOLD- s relies heavily on the data distribution, we observed that it achieves sub-optimal performance.
The on-policy MLE+PG with MLE initialization gives better entailment rate.
In comparison, our full SQL framework achieves the best entailment-diversity trade-off.
The comparison between SQL and SQL(single) highlights the importance of having the multi-step objective which directly uses the end reward rather than bootstrapping intermediate Q -values for supervision.
4. Universal A DVERSARIAL A TTACKS
We next study the application in text adversarial attacks, where again no supervised data is available.
Adversarial attacks is an increasingly important research topic as they reveal models’ vulnerabilities and ﬂaws.
This is especially true for universal attacks (Wallace et al., 2019; Atanasova et al., 2020), where we want to generate universal examples that trick the model on all possible inputs.
For instance, consider the context of entailment classiﬁcation.
Our goal is to ﬁnd universal humanreadable hypotheses that are going to be classiﬁed as “entailment” with as high probability as possible, regardless of the input premises.
This is a more challenging setting compared to previous instance-speciﬁc attack (Morris et al., 2020; Jin et al., 2020; Ebrahimi et al., 2017) where the attack model conditions on a premise and generates an adversarial hypothesis speciﬁc to the premise.
Setup (more in the appendix §A.2.2).
We aim to attack one of the most popular
MultiNLI (Williams et al., 2018) entailment classiﬁers on HuggingFaceHub.
The attack generation model generates adversarial text without conditioning on any inputs so that the generated attacks are universal to all premises.
We compare our SQL with MLE+PG .
We use all hypotheses in the MultiNLI dataset as the training data for the MLE training in MLE+PG and the off-policy updates for our SQL .
We do not compare with previous specialized adversarial text attack methods, because they either are not applicable to the challenging universal attack setting (Morris et al., 2020;
PretrainedLM GeneratedSentence N
[Figure]
“ the chemical microscope is In summary” PromptGenerator prompt (model’s output) input sentence
GeneratedSentence 1 ...
Reward Function
[Figure]
Perplexity
[Figure]
PPLM
GeDi
MLE (5)
SQL (off, 5) 13 .123 .25 .25 .
MLE+PG (5/10/15)
SQL (5/10/15, ours)
[Table] [Table]
[Table] [Table]
[Table] [Table]
Jin et al., 2020; Ebrahimi et al., 2017), or were not designed to generate human-readable sentences (Wallace et al., 2019).
We use similar settings as in §4.1 to explore the diversity-quality trade-off by plotting the entailment rate and perplexity against diversity, respectively.
The entailment classiﬁer to be attacked is used as entailment score reward functions.
We additionally include a token-level repetition penalty reward for readability.
Results.
Figure 3 (right) shows the results, and Table 4 (appendix) shows samples.
We can see that
SQL outperforms MLE+PG consistently across different diversity values.
The outputs from MLE+PG are not diverse even with high p ’s, indicating the model collapses and can only generate a small set of unique adversarial examples.
The model by SQL discovers the pattern “saint-pierre-et-saint-paul” (an entity name), and exploits this to generate samples with high universal entailment rate.
4. P ROMPT G ENERATION FOR C ONTROLLING P RETRAINED L ANGUAGE M ODELS
A reward function does not just have to be a metric like the BLEU score, but also a complicated pipeline that eventually returns a score.
To demonstrate this, we consider the emerging task of prompting a large pretrained LM for controllable generation (Hu et al., 2017;
Radford et al., 2019;
Brown et al., 2020).
The goal is to learn to generate text prompts that steer the LM to generate sentences of certain desired attributes (e.g., topics).
The problem of controlling the generation of pretrained LMs was previously approached through specialized algorithms such as modifying the
LM hidden states during decoding (Dathathri et al., 2020; Krause et al., 2020; Qin et al., 2020).
Here we show that prompts offer an easier, faster, more effective way for controlled generation.
Learning to generate/tune prompts is gaining increasing attention recently.
It side-steps the needs for expensive LM ﬁne-tuning, and adapts LMs to new scenarios with prompt as the (computefriendly) interface.
Most existing approaches (Wallace et al., 2019; Li & Liang, 2021; Lester et al., 2021) rely on gradient backpropagation and are applicable only when the whole training pipeline is differentiable.
This does not hold for the text generation setting, as illustrated in Figure 4.
In contrast, the RL framework is generally applicable to any differentiable or discrete pipelines.
Setup (more in the appendix §A.2.3).
Following (Dathathri et al., 2019), we aim to control the generation to have one of 7 topics (e.g., “science”); the generated prompt is prepended to one of 20 input sentences for the pretrained LM to generate continuation sentences.
Figure 4 shows the architecture of prompt-based controllable generation.
We compare our SQL method with MLE+PG as before.
Since the prompt length could impact the generated sentences, we conducted experiments with maximum prompt length 5 , 10 , and 15 .
As ablation study, we also evaluate the SQL algorithm with only off-policy updates (i.e., without on-policy exploration), denoted as SQL(off) , and compare it with vanilla MLE training.
Finally, we also compare with two specialized controllable generation techniques based on pretrained LMs, namely PPLM (Dathathri et al., 2019) and
GeDi (Krause et al., 2020), following similar procedures using their open-sourced code.
We use a distilled GPT-2 model 3 as the pretrained LM to be controlled.
For rewards, we use the topic accuracy of the continuation sentences measured by a zero-shot classiﬁer, plus the the log-likelihood of continuation sentences as the language quality reward measured by a distilled GPT-2.
Results Figure 5 shows the topic accuracy of the controlled LM outputs averaged across thetopics, and Table 1 shows the respective language quality results.
More detailed topic accuracy results and samples are provided in the appendix (§A.1.3) (where GeDi obtained low accuracy on 2 of the 7 topics, possibly because the topic tokens are tokenized into two subwords for which the model released by the authors was not speciﬁcally trained).
We can see that the prompts generated by our SQL cause the LM to generate sentences with high topic accuracy while maintaining low perplexity in most settings.
Increasing the prompt length positively impacts the topic accuracy, which makes sense because longer prompts give more ﬂexible for steering the LM.
The comparison between MLE and SQL(off) shows that the off-policy component of SQL is better than standard
MLE training, as it incorporates reward signals instead of just blindly following the (noisy) data.
Next, comparing with the previous steered decoding such as PPLM and GeDi , we can see the prompt-based control trained with RL achieves better trade-off between topic accuracy and language quality.
Moreover, once a prompt is produced, we can use the pretrained LM to generate text of desired topics efﬁciently, with the same time cost as standard non-controlled decoding.
In comparison, the dedicated steered decoding is often orders-of-magnitude slower, as shown in Table 2.
R ELATED W ORK
Standard RL algorithms maximizing the external rewards can sometimes be over-sensitive to the randomness in the environment.
Recent works have considered maximum-entropy RL (MaxEnt
RL) extensions, such as the soft Q -learning (SQL)
(Haarnoja et al., 2017; Nachum et al., 2017;
Schulman et al., 2017), that maximize the entropy of policy besides the rewards, and have demonstrated substantial improvement in robotic and game control (Ziebart et al., 2008; O’Donoghue et al., 2017; Nachum et al., 2018; Eysenbach & Levine, 2021).
Our work is the ﬁrst to adapt SQL and its advanced variants (in particular the path consistency learning (Nachum et al., 2017)) to the challenging text generation problem and show signiﬁcant results on diverse applications.
Applying RL for text generation has been discussed in alleviating the exposure bias problem and optimizing task metrics (Li et al., 2016; Wu et al., 2016; Rennie et al., 2017; Paulus et al., 2018; Chen & Bansal, 2018).
For example, Ranzato et al.
(2016) used the REINFORCE algorithm
(Williams, 1992), and
Bahdanau et al.
(2016) used the actor-critic algorithm;
Guo et al.
(2018) and Shi et al.
(2018) tried to relieve the sparsity problem via hierarchical and inverse RL methods, resp.
They are all on-policy
RL algorithms with the need of pretraining their models using MLE.
Another line of work focused mostly on using only off-policy data, often for ofﬂine training of chatbots (Jaques et al., 2020; Kandasamy et al., 2017; Zhou et al., 2017; Pang & He, 2021).
As a result, the opportunity of directly improving the reward (as in on-policy updates) for other rich tasks is missed.
Our proposed framework combines on- and off-policy training, and further offers solutions for efﬁcient training from scratch in the presence of large action space and sparse sequence-level reward in text generation.
C ONCLUSION
We develop a new RL formulation for text generation based on soft Q -learning and path consistency learning.
We conduct experiments on learning with noisy and negative data, black box adversarial attack, prompting a pretrained language model for controllable generation, and ﬁnally, on standard supervised tasks.
The RL formulation opens up enormous new opportunities to integrate more advances made in the fertile RL literature to improve text and other sequence generation problems.
E THICS S TATEMENT
This work develops a new RL formulation for text generation.
While we demonstrate the framework in four applications, it could be adapted to other (emerging) applications.
One major component in these applications is the design of the reward function, which inﬂuences the behavior of the trained agent.
While we believe the MaxEnt RL framework is more robust against reward misspeciﬁcation (Eysenbach & Levine, 2021), the potential failures of sub-optimal reward functions are widely known and discussed.
4 To this end, deploying this model to the wild requires careful and extensive examination, using tools such as Ribeiro et al.
Further, we highlight the application for (black-box) adversarial attacks in the paper, with the intention of using adversarial attacks to understand the model’s inner workings.
That being said, this could potentially be misused to conduct malicious attacks against systems.
Hence, users of this framework might want to conduct adversarial attacks against their own models to avoid being attacked by other people with bad intentions.
R EPRODUCIBILITY S TATEMENT
We provide code in the supplementary materials, and additional experiment details in the appendix.
A A PPENDIX A. A PPLICATIONS AND E XPERIMENTS A.1. L EARNING FROM N OISY (N EGATIVE ) T EXT
Please see Table 5 for beam search results, Figure 6 for additional results for MLE+reward and
Table 7 for examples.
[Figure]
A.1. Universal A DVERSARIAL A TTACKS
Please see Table 4 for examples.
A.1. P ROMPT G ENERATION FOR C ONTROLLING P RETRAINED L ANGUAGE M ODELS
Please see Table 6 for detailed results breakdown, and Table 8-11 for examples.
Examples are in the format: topic: [prompt] input sentence generated text
. A.1. S UPERVISED T EXT G ENERATION T ASKS
Finally, we conduct experiment on standard generation tasks where clean supervised data is available.
The study is to examine the capabilities of the proposed RL method to train a text generation model from scratch , which has been considered as exceedingly challenging for previous RL algorithms.
[Table]
Setup.
We study on two tasks, E2E (Novikova et al., 2017) and CommonGEN (Lin et al., 2020), and use the respective datasets pre-processed by (Gehrmann et al., 2021) which allow sequenceto-sequence modeling with standard transformers.
We run four sets of methods: the standard
MLE training ( MLE ); PG training from scratch ( PG ); joint MLE and PG training, with MLE initialization ( MLE+PG ); and our SQL training from scratch with both off-policy and on-policy updates ( SQL ).
We use the standard BLEU as reward.
We additionally investigate the training stability and sensitivity w.r.t hyperparameters, in particular the scale of reward.
To this end, for MLE+PG and
SQL , we vary the reward scale in { 1 , 10 , 50 , 100 , 500 , 1000 } and evaluate the respective performance under different scales.
Results.
Table 3 shows the performance on E2E of different models whose hyperparameters are picked using the validation set.
We can see the proposed SQL that trains models from scratch achieves competitive results with the common MLE and MLE+PG .
In contrast, the PG algorithm alone without MLE fails the training.
Figure 7 (left) shows the respective training curves (on the validation set), demonstrating that SQL converges in an efﬁcient and stable way as MLE .
We further demonstrate the sensitive of MLE+PG and SQL w.r.t the reward scale as a key hyperparameter.
Figure 7 (middle and right) shows the training curves of the two methods with varying reward scales.
We can see SQL is signiﬁcantly more robust as reward scale changes, while MLE+PG tends to collapse with improper reward scale conﬁgurations.
A. S ETUP D ETAILS
Our evaluation follows the GEM Benchmark
(Gehrmann et al., 2021) when applicable, 5 and otherwise same with the reward function used in training.
We use a transformer model (Vaswani et al., 2017) based on Texar-Pytorch (Hu et al., 2019) by default, with 64 hidden dimension, 3 blocks, and 4 heads.
For experiments that involve policy gradient training, we initialize the model with maximum likelihood training by default unless speciﬁed otherwise.
We train soft Q -learning model from scratch with both off-policy (using data) and on-policy (using samples) by default except in §4.1 and §4.3, in which we ﬁnd it beneﬁcial to warm-up the model with just off-policy training.
We apply similar tuning budgets to both soft Q -learning model, and policy-gradient (mostly the reward scale and top- k ), based on performance on the validation dataset and sample qualities.
Reward Functions
We use the robust entailment classiﬁer (Nie et al., 2020) in §4.1, 6 one of the most used entailment classiﬁers on HuggingFaceHub in §4.2. 7 and a zero-shot classiﬁer based on BART (Lewis et al., 2020) to compute the topic score in §4.3.
To compute perplexities, we use a GPT-2 model (Radford et al., 2019) ﬁne-tuned on the corresponding datasets for computing perplexity in §4.1 and 4.2, and a distilled GPT-2 model in §4.3 without ﬁne-tuning.
9 We simply set reward weights to 1 . 0 , except in §4.2, where we changed the entailment weight to 0 .
5 , log-likelihood and repetition penalty weight to 5 . 0 .
A.2. S ETUP D ETAILS : §4.
We study using the SNLI dataset (Bowman et al., 2015), a dataset commonly used in training an entailment classiﬁer.
The original dataset contains (premise, hypothesis) sentence pairs, where the hypothesis may or may not entail the premise.
We sub-sampled 50 , 000 training examples from the corpus such that the hypotheses have an average entailment probability of only 50% in terms of the premises, and over 2 / 5 examples have entailment probabilities less than 20% , which can be seen as negative (contradictive) examples.
The resulting training set poses a signiﬁcant challenge for the models to learn from the noises.
The RL algorithms (including PG and ours) permit us to plug in arbitrary reward functions to drive learning.
Based on the goal of the task, we use the following intuitive rewards to ensure entailment accuracy and language quality: (1) a robust entailment classiﬁer (Nie et al., 2020) that measures the entailment score of a generation in terms of the input premise, (2) a GPT-2 language model (Radford et al., 2019) that measures the log-likelihood of the generation as an indicator of language quality, and (3) BLEU score w.r.t the input premises as another language quality reward that avoids trivial outputs.
We sum together all rewards with weights 1 . 0 .
A.2. S ETUP D ETAILS : §4. Model
[Table]
We study the task of attacking an entailment classiﬁer.
In particular, we aim to attack one of the most popular entailment classiﬁers on HuggingFaceHub.
The attack generation model generates adversarial text without conditioning on any inputs so that the generated attacks are universal to all premises.
The generation model is trained with mostly the same setting as in §4.1, where the entailment classiﬁer to be attacked is used as entailment score reward functions.
Besides, we additionally include a tokenlevel repetition penalty reward, which empirically beneﬁts readability.
Finally, we use the MultiNLI dataset (Williams et al., 2018) which includes more diverse examples than the SNLI used above.
We compare our SQL with MLE+PG .
We use all hypotheses in the MultiNLI dataset as the training data for the MLE training in MLE+PG and the off-policy updates for our SQL .
We do not compare with previous specialized adversarial text attack methods, because they either are not applicable to the universal attack setting (Morris et al., 2020; Jin et al., 2020; Ebrahimi et al., 2017), or were not designed to generate human-readable sentences (Wallace et al., 2019).
Besides, it is worth noting that the general RL algorithms have an additional advantage of doing black-box attacks.
That is, the algorithms only require the ability to query the entailment classiﬁer for entailment probability, without need of knowing the internal structure of the classiﬁer (e.g., for computing gradients)
as in previous attack algorithms (Ebrahimi et al., 2017; Wallace et al., 2019).
For top- p sampling results, we sample a hypothesis for each premise and measure the average attack rate across the dataset.
This is because sampling multiple hypotheses, each for all premises, and measure performance are expensive.
Since the hypotheses are sampled input-independently, this should be a good approximation.
A.2. S ETUP D ETAILS : §4.
Following (Dathathri et al., 2019), we aim to control the generation to have one of 7 topics (e.g., “science”); the generated prompt is prepended to one of 20 input sentences (Figure 4) for the pretrained LM to generate continuation sentences.
There is no direct supervision data available for training the prompt generator.
We randomly create some noisy text as the training data for MLE baselines below and for off-policy updates for our algorithm.
Speciﬁcally, the noisy text is created by sampling keywords and topics from the list used in (Dathathri et al., 2020) and a paraphrase generation model.
Figure 4 shows the architecture of prompt-based controllable generation.
We compare our SQL method with MLE+PG as before.
At training time, for each generated prompt sample, the pretrained
LM generates 2 continuation sentences for evaluating average reward.
We use a zero-shot classiﬁer to evaluate the topic accuracy of the continuation sentences.
That is, we do not assume access to classiﬁers pretrained on topic-speciﬁc sentences, because generating such topic-speciﬁc sentences is the goal of the task in the ﬁrst place.
We additionally use an LM to evaluate the log-likelihood of continuation sentences for measuring language quality.
Since the prompt length could impact the generated sentences, we conducted experiments with maximum prompt length 5 , 10 , and 15 .
As ablation study, we also evaluate the SQL algorithm with only off-policy updates (i.e., without onpolicy exploration), denoted as SQL(off) , and compare it with vanilla MLE training.
At test time, given a topic, the trained prompt generator produces one prompt using beam search decoding.
For each generated prompt, the pretrained LM generates 100 sentences using top- k decoding (with k = 50 ) for evaluation.
Finally, we also compare with two specialized controllable generation techniques based on pretrained LMs, namely PPLM (Dathathri et al., 2019) and GeDi (Krause et al., 2020), following similar procedures using their open-sourced code.
We use a distilled GPT-2 model 11 as the pretrained LM to be controlled.
We use the paraphrase generation model based on Zhang et al.
During decoding, we include no repeat ngram size = 2 , which improves readability.
[Equation]
[Figure]
[Equation]
A.3. C OMPARISON WITH MLE O BJECTIVE
It is interesting to take a closer look at the above objective and compare with the common MLE training.
Speciﬁcally, we notice the relations between the optimal Q ∗ , V ∗ , and A ∗ functions:
A ∗ ( s t , a t ) =
Q ∗ ( s t , a t ) − V ∗ ( s t ) =
r t + γV ∗ ( s t +1 )
− V ∗ ( s t ) , where the ﬁrst equation is the deﬁnition of A ∗ (see Eq.7) and the second equation is due to Eqs.(12) and (7).
We thus can see the regression target in the above objective as an approximation to the advantage function: 11 https://
.co/distilgpt2https:// co/
/pegasus_paraphrasehttps:/ co blog how-
-generate [Table] ˜ A ¯ θ ( s t , a t ) := − V ¯ θ ( s t ) + γV ¯ θ ( s t +1 ) + r t .
Therefore, by optimizing the regression objective, log π θ ( a t | s t ) , which is the log probability of generating token a t given preceding tokens s t , is encouraged to match the approximate advantage value ˜
A ¯ θ ( s t , a t ) , no more and no less.
This is different from the objective of MLE where the model is trained to (blindly) increase the probability of the observed token a t given s t and decrease the probability of the rest.
A.3. V ANILLA T RAINING WITH T EMPORAL C ONSISTENCY
Much like the Bellman temporal consistency in standard Q -learning (Eq.3), in SQL, the optimal action-value function follows the softmax form of the temporal consistency (Ziebart et al., 2008;
Ziebart, 2010; Fox et al., 2016; Nachum et al., 2017):
[Equation]
We thus can derive a regression objective similar to the standard Q -learning (Eq.4):
[Equation]
Recall that π (cid:48) is an arbitrary behavior policy (e.g., data distribution), and Q ¯ θ is the target Q -network which is a slow copy of the Q θ to be learned and is held ﬁxed during the gradient updates.
However, the above objective is inefﬁcient due to exact the same reasons as in standard Q -learning discussed earlier, namely the unstable per-step bootstrapping-style training with sparse reward signals, plus the slow updates w.r.t only one token a t out of the large vocabulary (action space).
Length Model legal politics
[Table]
military Average
Topic Scores /
PPLM 16 .25 .13 .26 .
GeDi 40 .83 .9 .
MLE 17 .13 .
SQL (off)
MLE+PG 29 .74 .72 .
SQL 11 .70 .
MLE+PG 17 .75 .71 .
SQL 29 .86 .
MLE+PG 40 .81 .
SQL 48 .77 .70 .
Perplexity /
PPLM 13 .12 .
GeDi 204 .80 .132 .116 .
MLE 24 .25 .23 .26 .26 .
[Table]
[Table]
legal: legal space religion and space
In summary, a good understanding of these concepts is that by giving an explicit understanding to a person, they provide an avenue to be studied and studied.
But the concept of one person being a space is also very confusing, and can be very difﬁcult to obtain.
\ n So, politics: the primary referendum is In summary, the outcome will be a referendum on the EU membership for the ﬁrst time of its kind for EU citizens, and the full extent of the beneﬁts of a single market and a ﬂexible single EU state.”
computers: macintoshintoshintoshintosh
In summary, it appears that the company and IBM products are currently in need of upgrading the computer.
This can be seen in a detailed review of the Macintosh version of Windows Vista and XP.
However, when looking at the changes made by the HP Macintosh hardware and software versions of space: legal space science and space In summary: \ n \ n The purpose of this paper is to investigate and test the theory of space space and other objects.
This project will support NASA.s efforts to demonstrate these theories, and to propose other relevant new theories.
\ n This paper provides the following arguments for the religion: space legal religion religion religion In summary, to the author the current discussion is the po- sition of the Church and the community.
While we acknowledge that we should not be commenting upon claims such as our recent cases or the other ones that contradict our view, we conclude it is appropriate to include these cases.
Further science: the chemical microscope is In summary, the most sophisticated of these experiments is a tech- nique that gives no obvious, no apparent way of revealing that the material was obtained.
In this study, we examine how the compounds in the samples in question make up the composition of the chemical and its properties.
The chemical composition
military: arms defense battalion battalion cavalry
In summary:
6th
Panzer
Field Division, Second Di- vision.
\ n \ n The main task of the battalion in the main counterinsurgency campaign was to counter the enemy in any counter-incursion.
The main objective of this campaign is to eliminate enemy groups and the remnants of legal: legal space religion and space This essay discusses the idea of space and time as a space, in both theoretical and conceptual terms, as not an individual time period or anything else.
The emphasis is on time itself, rather than having a ﬁxed central space.
Space was the object of the ﬁrst chapter, and politics: the primary referendum is This essay discusses the nature of the EU referendum.
The purpose of this essay is to shed light on the importance of a public referendum, on a question of whether the decision of an EU member states to remain in the European Union is constitutional and thus in accord with constitutional guarantees of sovereignty computers: macintoshintoshintoshintosh This essay discusses hardware devices and software systems for
Mac OS X, MacOS X and Linux.
To view the latest version of Macintosh OS: Mac 8.7.x \ n \ n For more information or for information about Macintosh systems, visit Mac MacSystems.
\ n More space: legal space science and space This essay discusses science for teens, adults and teenagers.
\ n \ n When the idea of studying space was ﬁrst implemented as a method to test, the question was: What if a student has been ”comfortable” with space without its body?
What would their body like to be religion: space legal religion religion religion This essay discusses an alternative religion that focuses on the role of a particular religion and views some form of religious ethics as the form when the law is applied to that particular religious community .
This discussion is concerned with the status of faith for individuals or groups which may be members and members science: the chemical microscope is This essay discusses the mechanisms of reaction with a focus on the molecular structure of nucleite and of enzymes within the cytoskeleton, thus making it easier to understand the process of metabolism and other elements of cellular life.
In this essay, we use techniques such as the photochemical transfer military: arms defense battalion battalion cavalry This essay discusses three main themes: \ n \ n 1) Lack of uniformed soldiers is an unacceptable and unconscionable strategy for the Army.
\ n 2) Poor and inade- quate training does not compensate the soldiers, and may deprive them of the necessary and competitive training from their instructors legal: there liable injunction In summary, the court decided that defendants had a right to petition the high court to intervene, and therefore granted the injunction.
However, a Court of Appeal decision in the US
District Court ruled with no effect on the petitioned by the plaintiffs, citing two reasons.
\ n \ n The US politics: the primary referendum was In summary, the majority of Russians would support the idea of banning all human rights, and that would be a major part of the government’s effort to build a new liberal economy, which President Vladimir Putin says would boost Russian tourism.
\ n \ n
The results were published in the computers: the macintoshintosh
In summary, the kernel has the best quality of life, and kernel security is at the heart of any OS development.
The ﬁrst time Linux released was when i386 released the Macintosh version, which had a rather low standard.
Other issues like the introduction of the OS6 operating system space: comet asteroid spaceship
In summary, this asteroid asteroid ship is about to make its maiden ﬂight to explore deep-space.
\ n \ n ”The asteroid craft was a very early attempt at the discovery of a new home for the world’s ﬁrst black hole,” NASA said in a news release. ”
Today we religion: faith faith faith salvation In summary, Christian beliefs are not a new way to use the time spent thinking about God’s world as a source for faith.
Faith is an effort to think of the world without fear that it might become a dangerous place for the human family.
Because it represents the very essence that science: climate research chemistry In summary of the study, this review aims to determine how in a single study where the same number of data was analysed, a new methodology is needed to better understand who produced a different graph than the one suggested.
The paper will be published in issue #5, Issue #18.
military: the cavalry battalion a
In summary, the army are a unit of the same type and in all, so there is no need to declare one.
The unit does not constitute a cavalry unit or for use on troops.
\ n \ n The army is not under the command of a brigade from the front.
For legal: there liable injunction This essay discusses the potential legal consequences of a stay in the United
States for an indeﬁnite period of time if the government continues to delay the process of de-instituting it.
To apply such a request, all applicable laws shall apply either the same terms as the existing statutes.
In politics: the primary referendum was This essay discusses the electoral strategy against a candidate for governor of the Commonwealth.
\ n \ n The survey of British voters in this survey provides an overview of what the candidates for the United Kingdom will be seeking in the next Parliament.
In the general election a few seats will lead up to a computers: the macintoshintosh This essay discusses the various problems of the Macintosh, the ﬁrst two- year running environment.
An early version of this paper was originally published in 1982.
The MacSX was not designed and managed by Kia.
\ n \ n Macintosh \ n The mac has been a family invention space: comet asteroid spaceship This essay discusses a topic: the impact of two of the Earth’s two-thirds comet-sized moon Charon on Earth, and why asteroids are so close to the sun; why people are looking for ways to ﬁnd a way to keep Earth-shaped asteroids out of orbit.
religion: faith faith faith salvation This essay discusses the impact religion has on the American experience and in American culture.
Since the beginning of my career I have found that faith and belief have often been linked to economic growth, social development and education.
I believe that all people need to know that there is no reason for science: climate research chemistry This essay discusses the role of molecular information and its inter- action with the general organism and human health.
\ n \ n ”The idea of biological information is not really a new concept.
We used genetic information as a medium to deﬁne, identify, and store information about biology and biology,” explains Dr. military:
the cavalry battalion a
This essay discusses the potential for the development of a small infantry brigade as an infantry regiment.
It is also a contribution to the larger cavalry corps as it would require a larger brigade for battle.
For more information see the original article on this page.
[Table]
out
[Table]
have run a software operating
[Table]
vital
[Table]
hacker can mine it
[Table]
legal:
In summary we have published in the journal Nature Neuroscience: A systematic review of human brain tissue has found no evidence for any association between the presence of the presence of a particular form of human neuropathy in the brain, a condition that is not normally associated with cognitive impairment.
We found that politics:
In summary we have a list of 10 of the best and most common types of drugs for people with HIV.
This is a very short list of recommendations from a national and international community.
\ n \ n \ n \ n
This article has been updated to make the ofﬁcial state of the EU state of computers:
In summary, we believe that the current system has no way of doing anything about it.
\ n \ n \ n \ n The following steps are taken to get the system working.
\ n \ n 1.
Install a new operating system with a Linux Mint operating system \ n 2.
Start a new Linux Mint operating space:
In summary we have some important news from the moment of the year and some important information about these two major planets.
This new discovery is the ﬁrst to conﬁrm this important planet has an active life in its home planet, a planet with a mass of about 5.8 billion tons.
It religion:
In summary, we believe that the current administration has no way of doing anything about the Benghazi attacks.
This is a very interesting story, and I think it has been a very nice surprise.
This is a very nice and well thought out piece that is a must for the science:
In summary we use this approach to evaluate if the number of data points (in the dataset) that are relevant for each data set is the same (in this case, the data are not in one data set).
In this approach we can test the data points in a different way.
military:
In summary we have some important news from the moment of the year and some important information from the moment of the year.
\ n \ n \ n \ n \ n We’ve also added an additional update for our new feature, which includes: \ n • Improved access and access in all of the main legal: This essay discusses how you can build a community of dedicated people.
If you’re a member of a community of people who want to contribute to the environment, you’ll also be helping them build communities in order to support the local economy, and the future of the city.
The latest report politics: This essay discusses how we can build on previous research ﬁndings about the role religion plays in human development in human development.
This is a very interesting and highly entertaining story.
What is an ”independent” political party in the United States, the U.S. political party, and the United computers: This essay discusses how you can build a new browser to view and share your favorite web sites.
\ n \ n \ n A browser that is open source can also be built from a web browser, which can be a browser that does not allow browser extensions (e.g. Firefox, Chrome, Opera space: This essay discusses how you can build a life with a healthy diet and how you can use it when you’re ready to move forward.
It’s a very simple approach to building a life with a healthy diet and what it means to be healthy and healthy for the religion: This essay discusses how you can build a new game without having to play the original game, and how you can make a new title that is completely different to the original.
It has been around since 2007, when the ﬁrst game, The Elder Scrolls IV: Oblivion, was released in the PlayStation science:
This essay discusses how we can build on previous research ﬁndings about the role of obesity in human metabolism and how we can improve our health.
\ n \ n \ n \ n
In this essay, we explore why eating a whole whole diet does not help prevent obesity (1).
We ﬁnd that a whole food diet military: This essay discusses how you can build a community with the help of friends and family.
\ n \ n
[Table]
people
[Table]
me
[Table]
ones who need help.
They are the ones who need help.
They
[Table]
ones who
[Table]