S ELECTIVE T OKEN G ENERATION FOR F EW - SHOT L ANGUAGE M ODELING
Anonymous authors
Paper under double-blind review
A BSTRACT
Natural language modeling with limited training data is challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability.
Among these transfer learning algorithms from PLMs, additive learning that incorporates a task-speciﬁc adapter on top of the ﬁxed PLM has been popularly used to alleviate the severe overﬁtting problem in the few-shot setting.
However, this added task-speciﬁc adapter is generally trained by maximum likelihood estimation that can easily suffer from the so-called exposure bias problem, especially in sequential text generation.
Therefore, in this work, we develop a novel additive learning algorithm based on reinforcement learning (RL) for few-shot natural language generation (NLG) tasks.
In particular, we propose to use a selective token generation between the transformerbased PLM and the task-speciﬁc adapter during both training and inference.
This output token selection between the two generators allows the adapter to take into account only on the task-relevant parts in sequence generation, and therefore makes it more robust to overﬁtting as well as more stable in RL training.
In addition, in order to obtain the complementary adapter from the PLM for each few-shot task, we exploit a separate selecting module that is also simultaneously trained using RL.
Experimental results on various few-shot NLG tasks including data-to-text generation and text summarization demonstrate that the proposed selective token generation signiﬁcantly outperforms the previous additive learning algorithms based on the PLMs.
I NTRODUCTION
Natural language processing (NLP) have recently achieved great progress using advanced neural language models (Radford et al., 2018;
Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019;
Clark et al., 2020; Raffel et al., 2019; Lan et al., 2019; Lewis et al., 2020).
However, these neural models typically require large-scale training data for each individual task, and solving a new NLP task that has only a few examples is still challenging problem (Yin, 2020).
Especially, natural language generation (NLG) with limited training data is an important yet more difﬁcult task due to its fast adaptation of sequential prediction models in a wide range of applications including text summarization, question answering, data-to-text generation, machine translation, etc (Peng et al., 2020;
Chen et al., 2020; Xu et al., 2021; Schick & Sch¨utze, 2020; Chang et al., 2021; Radford et al., 2019; Lewis et al., 2020; Brown et al., 2020).
More recently, pretrained language models (PLMs) have shown great generalization ability when combined with large-scale data and big transformer-based models (Devlin et al., 2019;
Radford et al., 2019; Lewis et al., 2020; Brown et al., 2020; Subramanyam Kalyan et al., 2021).
Therefore, transfer learning from transformer PLMs has been popularly used for few-shot NLG tasks with promising results.
In speciﬁc, the use of PLM for few-shot NLG can be categorized into three approaches: 1) prompt-based, 2) ﬁnetuning, and 3) additive learning.
Prompt-based approaches encode a task description and task-speciﬁc examples as a natural language prompt for few-shot text generation (Radford et al., 2019;
Brown et al., 2020; Zheng & Huang, 2021; Schick & Sch¨utze, 2020; Li &
Liang, 2021).
While these approaches can take full advantage of the universal natural language understanding and generation capabilities of large-scale PLMs without further training of the main model, these have some limitations in dealing with a large domain shift from the pretraining corpus data, tuning suitable task-speciﬁc prompts, and covering an increased size of conditioning examples.
On the other hand, ﬁnetuning of the PLM is able to explicitly impart task-speciﬁc knowledge to the model and hence lift the above limitations (Ziegler et al., 2019; Xu et al., 2021; Chen et al., 2020).
However, these ﬁnetuned models are prone to overﬁtting when only a small amount of training data is available.
In order to alleviate such an overﬁtting problem, additive learning has been extensively exploited by incorporating task-speciﬁc adapters into the PLM (Zeldes et al., 2020; Stickland &
Murray, 2019).
In general, task-specialized adapters for few-shot NLG are trained by maximum likelihood estimation (MLE).
While MLE is efﬁcient in learning, it suffers from the exposure bias problem due to the difference in the training and inference mechanisms (He et al., 2019), and this problem can be severe with limited training data.
Reinforcement learning (RL) is capable of resolving this exposure bias problem by sequential output sampling during training (Ranzato et al., 2015; Keneshloo et al., 2019;
Shi et al., 2021).
Moreover, it allows to leverage the target-speciﬁc sequence-level objectives such as BLEU and ROUGE (Wu et al., 2018; Guo et al., 2021).
However, in the task of NLG, the exponentially large space of output sequences restricts the use of RL since it leads to high variance and unstable training which is more serious in the few-shot setting.
In this work, we develop a novel RLbased additive learning algorithm on the transformer-based PLM to overcome these shortcomings and to improve the performance of few-shot NLG.
In particular, we ﬁrst convert the NLG task to the sequential token generation task based on the transformer language model, and then propose a selective token generation between the PLM and the task-speciﬁc adapter, during both RL-based training and inference.
The proposed output token selection enables to not only explicitly maintain a general prior knowledge from the frozen PLM but also focus only on the task-relevant parts in sequence generation.
In addition, in few-shot learning this partial token generation makes the task-speciﬁc adapter more resilient to overﬁtting and furthermore reduces the overall output space which leads to stable RL training.
Here, in order to make the two token generators (policies) complement each other as well as to realize the robust output selection at the token level on the ﬂy, we exploit a separate token-level policy selector.
It is noted that both the policy selector and the task-speciﬁc adapter are simultaneously learned by the
RL algorithm.
Experimental results on various few-shot NLG tasks show that the proposed selective token generation outperforms the previous PLM-based additive learning algorithms with the comprehensive (non-selective) token generation.
Our main contributions can be summarized as follows.
• A novel selective token generation between the PLM and the task-speciﬁc adapter is proposed for transformer-based few-shot NLG.
• A separate selecting module is exploited to adaptively determine each output token in a sequence both at training and testing time.
• RL is applied to train both the policy selector and the task-speciﬁc adapter that is complementary to the PLM in text generation.
• An extensive empirical validation on few-shot NLG tasks demonstrates that the proposed selective token generation performs better in comparison to the previous PLM-based additive learning algorithms.
B ACKGROUND 2. N ATURAL L ANGUAGE G ENERATION
The goal of NLG is to generate a text sequence y =
[ y 0 , ..., y T ] for a given task, where y t is the t th output token from a vocabulary V , and T is the output sequence length.
For this generation, we aims to model the distribution of y that is autoregressively factorized as p θ ( y ) =
(cid:81) T t =
p θ ( y t | y <t ) , where θ denotes the model parameters and y <t =
[ y 0 , ..., y t − 1 ] .
Here, the conditional distribution to sample a token for each step, p θ ( y t | y <t ) , is deﬁned by the softmax function on the output logits f θ ( y t | y <t ) .
Note that in general, the language generation is conditioned on input context according to a given task.
Here, we encode the conditioning context by the same sequential model for generating an output sequence, and for simplicity we omit it.
In this work, we utilize the autoregressive transformer for our generative model.
2. A DDITIVE L EARNING FOR F EW - SHOT G ENERATION
To effectively leverage the general linguistic knowledge, θ is ﬁrst initialized by the PLM parameters, θ LM , for NLG.
Given N task-speciﬁc training instances, D = { y n ∗ } Nn =1 , where y n ∗ is the n th ground-truth output sequence, directly ﬁnetuning θ LM using D can incur the severe overﬁtting problem when N is small in the few-shot scenario.
Therefore, we add the task-speciﬁc adapter, g θ a parameterized by θ a , on top of the PLM, and optimize only θ a (Zeldes et al., 2020; Stickland & Murray, 2019).
In speciﬁc, we reformulate f ( ·| y <t ; θ ) =
W T h ( y <t ; θ h ) where W ∈ R H ×|V| and h ∈ R H denote the weight matrix and the penultimate representations, respectively, and θ = { W, θ h } .
Then, we deﬁne the task-speciﬁc conditional distribution as follows:
[Equation]
where h LM ( y <t )
=
h ( y <t ; θ h,LM ) and θ a = { W a , θ g } .
Here, the summation of the PLM logits and the adapter logits is motivated by auxiliary training 1 .
It is noted that in our additive learning θ a is updated while θ LM is kept frozen.
Hence, in the following we omit θ LM such that p θ a ( y t | y <t )
=
p ( y t | y <t ; θ LM , θ a ) for simplicity.
2. M AXIMUM L IKELIHOOD E STIMATION (MLE)
Given a small amount of training data D = { y n ∗ } Nn =1 , MLE optimizes θ by maximizing the data log-likelihood as follows:
[Equation]
Here, the output token at each step is conditioned on not the previous sampled tokens from the current model but the previous ground-truth tokens y n ∗ <t .
Namely, tokens are drawn from the data distribution during training, which is opposed to that tokens are drawn from the model distribution at test time.
This discrepancy, also known as the exposure bias, leads that the errors will be accumulated along the generated sequence at test time since the model is biased to only perform well on the ground-truth history distribution.
Especially, this bias problem can be more severe in the few-shot training.
In addition, the token-level cross-entropy loss in MLE training is different from the sequence-level test metrics such as BLEU and ROUGE that are commonly used in the tasks of
NLG.
2. R EINFORCEMENT L EARNING (RL)
As an alternative to MLE, RL is able to overcome the exposure bias problem of MLE by sequencelevel sampling from the model distribution during training (Ranzato et al., 2015).
Also, RL can improve the performance by directly optimizing the evaluation metrics (Guo et al., 2021).
In order to use RL for our additive learning, we reformulate our text generation as an RL problem: at each time step t , the agent takes the current state s t
= y <t as an input and performs an action a t that outputs a token y t by a policy π θ ( a t | s t ) corresponding to p θ ( y t | y <t ) .
Then, the agent receives a reward r t = r
( s t , a t ) and deterministically transitions to the next state s t +1 .
Here, note that the token-level intermediate reward r t = 0 , ∀ t < T when we use the delayed reward associated with the sequencelevel evaluation metric between the two full sequences, y and y ∗ .
Let τ = { ( s t , a t , r t ) }
Tt =
be the trajectory generated by π θ .
The RL objective for the optimal agent is to maximize the expected sum of future discounted rewards:
[Equation]
where γ ∈ [0 , 1) is the discount factor.
Among a number of algorithms to approximately optimize the RL objective, we employ an actorcritic algorithm (Bahdanau et al., 2017) since it explicitly optimizes the policy network and it can also alleviate the delayed reward problem.
The actor-critic algorithm requires the additional critic network to estimate the value of a state, V
[Equation]
where the state-action value function Q π ( s t , a t ) =
E π [ (cid:80) Tt [Equation]
=
t γ
[Equation]
− t r
[Equation]
| s t , a t ] =
r t + V π
( s t +1 ) .
We use the following policy gradient loss to learn the policy parameters θ :
[Equation]
(4) where A π θ ( s t , a t ) = Q π θ ( s t , a t ) − V π θ ( s t ) is the advantage function that quantiﬁes how an action a t is better than the average action in state s t .
In few-shot text generation, the extremely large action space ( |V| T ) as well as the small amount of training data often make it difﬁcult to perform RL with degraded performances, even though we conduct the additive learning from the PLM. Furthermore, it commonly has a delayed reward function (e.g. BLEU) which is deﬁned after an entire sequence generated. It is hard to decide which token and how much contributes to the reward.
This problem is known as the credit assignment. Therefore, in this work, we propose a selective token generation for improving the RL-based additive learning.
S ELECTIVE T OKEN G ENERATION
Instead of generating all tokens in an output sequence from the single task-speciﬁc policy, π a = π θ a ( a t | s t ) , at each time step t , we sample an output token y t selectively from either the PLM policy π LM =
π θ LM ( a t | s t ) or the task-speciﬁc policy π a :
[Equation]
where 1 t [ · ] is the indicator function (at t ) that equals 1 if it is true and 0 otherwise.
This output token selection allows to explicitly utilize a general linguistic knowledge from the PLM without catastrophic forgetting in few-shot learning.
Also, the task-speciﬁc policy can focus on generating only the task-relevant parts, which enables more effective few-shot training with a reduced action space.
Now we need to determine how to select the proper policy at each step on the ﬂy as well as to make the task-speciﬁc policy complementary to the PLM policy.
For this, we exploit a separate token-level policy selector.
The proposed policy selector π s ( i t | s t ; θ s ) with the parameters θ s , where i t ∈ { 0 , 1 } , is an another policy that stochastically decides a policy to generate a t for s t .
Namely, a token sample y t is generated by the following process:
[Equation]
This process can be considered as a token generation from a hierarchical policy π h ( a t | s t ; θ s , θ LM , θ a ) where the policy selector represents the upper-level prior for the preference of the low-level policy.
Therefore, the value function of this hierarchical policy can be formulated as
[Equation]
and A π h ( s t , a t ) =
Q π h ( s t , a t ) − V π h ( s t ) .
Here, it is noted that a single critic network is used for the hierarchical policy since i t does not affect s t .
Given a sample trajectory { ( s t , i t , a t , r t ) } Tt =0 , the loss for optimizing θ s and θ a is
[Equation]
where sg stands for the stop-gradient operator. Similar to π a , π s makes use of the PLM representations and the task-speciﬁc adapter such that
[Equation]
where m is the selector module that is implemented by a small neural network.
Figure 1 depicts the overall text generation process by the proposed selective token generation (STG) in comparison to the previous non-selective token generation (Non-STG).
Here, note that since all policies in STG share the same PLM representations, the increased computational cost by STG over Non-STG is negligible.
The use of the separated policy selector that is simultaneously trained with the task-speciﬁc policy allows the task-speciﬁc policy to be complementary to the PLM policy.
Especially, this cooperative ensemble learning can be realized by our RL algorithm that performs sequential sampling from the model during training.
In addition, the proposed STG model makes use of the PLM not at the feature level but the output distribution level in text generation.
In our few-shot learning this is beneﬁcial in explicitly retaining strong linguistic and world knowledge from the PLM.
E XPERIMENTS
In this section, we evaluate our method against baselines on Data-to-Text, Question Answering and Text Summarization tasks which are widely used in few-shot NLG.
Since we claim the effectiveness of our method in additive learning, we compare our results against those obtained by other additive learning methods.
We ﬁrst explain baselines, implementation of our method, tasks for comparison and obtained results and then analyze the results in detail.
4. B ASELINE
PLM .
In our experiments, we assume that the PLM works to some extent for a given task.
However, the naive PLM usually does not satisfy it for a new task unseen during training.
Hence, we ﬁnetuned
[Table] GPT-2 2 (Radford et al., 2019) with MLE for few epochs and used it as the PLM.
Fine-tuning the
PLM with MLE is most commonly used for task adaptation and thus it can also be a strong baseline.
This ﬁne-tuning phase accelerates the learning of the adapter. This is particularly when the adaptation requires to cover the large domain shift. Severe performance degradation was observed for all the tasks when we skipped the ﬁne-tuning.
Non-STG .
This method stands for Non-Selective Token Generation which uses the above the PLM as an encoder (frozen) and the adapter (additional layer to be trained).
We use two objectives, MLE and RL, for additive learning.
These will be denoted as Non-STG-MLE and Non-STG-RL, respectively.
STG-Naive Ensemble .
We believe that our proposed method encourages the task-speciﬁc policy ( π a ) to complement the PLM’s policy ( π LM ) with a proper selection of the selector through joint training.
To investigate this, we evaluate against two different naive ensembles of the policies, π a trained from Non-STG and π LM of the PLM.
These ensemble schemes are as follows:
• NE( max ): π max = softmax ( Max ( π a , π LM )) • NE( mix ): π mix =
( π a + π LM ) /
We also evaluate another naive ensemble strategy NE( random ) that randomly selects a token policy at each step between π a and π LM , however it shows lower performances than the others.
4. I MPLEMENTATION
Here, we explain the implementation set-up of our method.
Additional Layer : The task-speciﬁc adapter g in Section 2.2 is implemented by a LSTM to encode the dynamics of the representation vector h LM . We found that the use of MLP was not performed well.
Selector .
We use a 2-layer MLP with ReLU activation for m .
Reinforcement Learning .
We employ Actor-Critic method (Konda & Tsitsiklis, 2000; Fedus et al., 2018) for RL.
The agents (i.e. selector and generator) receive a reward after generating a sentence.
Here, we use different reward functions according to tasks.
We use delexicalised BLEU for Data-toText following Peng et al. (2020), Averaged score of BLEU and ROUGE-L for Question Answering and ROUGE-L for Text Summarization following Paulus et al. (2017) as the reward function.
Token Sampling .
During the training, i t ∈ { 0 , 1 } ∼ π s is ﬁrst sampled, and then we use either π LM of the PLM for i t = 0 or the task-speciﬁc policy π a for i t = 1 to sample the t th token.
During the evaluation, any decoding strategy, such as a beam search, can be used with the mixture of policies π h ( · ) =
π s ( i t = 0) π LM ( · ) + π s ( i t = 1) π a ( · ) .
We use the beam search decoding with a sample size of k = 3 for Text Summarization and top p = 0 .
9 decoding for both Data-to-Text ( k = 10 ) and Question Answering ( k = 3 ).
[Table] D [Table]
Data-to-Text is a task that transforms structured data such as graphs or tables into natural language.
Recent works (Mager et al., 2020; Peng et al., 2020; Kale, 2020) show that the PLM can be adapted successfully to this task by taking a serialized form of data as an input without a carefully designed model to encode the structured data.
Here, we perform experiments on FewShotWOZ (Peng et al., 2020) dataset.
The evaluation is conducted only on the topics of Restaurant, Hotel, Laptop, and TV since the evaluator has not published yet for the other topics (Taxi, Attraction, Train) 3 .
There aretraining instances for each topic and 129, 78, 1379, and 680 testing instances for Restaurant, Hotel, Laptop, and TV, respectively.
The models are evaluated by measuring ﬂuency and informativeness using BLEU score and ERR (slot ERror Rate), respectively. Table 1 shows the obtained results.
4. L ONG ANSWER Q UESTION A NSWERING
We consider Long Answer QA task on MS-MARCO (Nguyen et al., 2016) dataset. In this task, a passage and a query are given, and the model generates an answer with respect to the query by referring to the passage. Here, we randomly sample various sizes of ( 0 . 5% ≈ 500 , 1% , and 2% ) subset data from the train dataset.
We also sample a validation and a test set, which contains 500 and 12,000 instances, respectively, from the dev dataset. The models are evaluated by measuring BLEU and
ROUGE-L.
The performances of the models are reported in Table 2. 4. T EXT S UMMARIZATION We consider the problem of abstractive summarization: models aim to generate its summary (the “target” text) for a given piece of “source” text such that its meaning is intact. The model has to learn 1) building a semantic form internally from the source text and 2) converting the semantic form into a text.
Here, we randomly sample various sizes of ( 0 . 5% ≈ 1 , 500 , 1% , and 2% ) subset data from CNN/Daily Mail (See et al., 2017). We repeat this test three times with different random seeds.
In other words, we collect three different random subsets for each size of few-shot.
Thus, we perform experiments on total nine subsets.
ROUGE (Lin, 2004) is commonly used to evaluate n-grams recall of the summaries with gold references.
ROUGE-1, ROUGE-2, and ROUGE-L (measured based on the longest common sub-sequence) scores are reported (denoted as R 1 , R 2 , and RL , respectively) in Table 3 and Appendix B.
4. R ESULT
We compare experimental results on various tasks in this subsection.
In most cases, additive learning improves the performances over the PLM.
However, they do not always guarantee a performance improvement.
For example, the ERR score of the PLM on Laptop shows a better result except for STG and NE( mix )-RL (see Table 1) and the Non-STG models trained on 1% few-shot subset of
MS-MARCO do not outperform the PLM (see Table 2).
[Table]
In Data-to-Text task, as shown in Table 1, we can observe that the Non-STG models do not outperform the PLM even though it has more neural units and takes more training time. The models trained on the RL objective show better performances for the ERR (lower is better).
Interestingly,
NE( mix ) methods show strong improvements for the BLEU which measures the ﬂuency of sentence but obvious degeneration for the ERR which measures the rate of missing information from the given data.
These results suggest that the PLM is much more capable of task-general knowledge than the task-speciﬁc generator (i.e. π a ) trained on few-shot dataset. Note that NE( mix )-RL shows obvious degeneration for ERR. It motivates us to jointly learn the policy selector and the task-speciﬁc generator.
Here, while other methods show some trade-off between BLEU and ERR, only STG shows improvements on the both metrics for all topics in the dataset.
In Question Answering, NE( mix ) and STG show signiﬁcantly better performances than the other methods as shown in Table 2. Notably, NE( mix ) show good performances as much as STG.
It obviously suggests that the PLM can be a complementary model to the additional model. Therefore, in this context, it can be lost of the prior knowledge of the PLM even the additional model has been built over the feature space of the PLM.
We will discuss it in Section 4.7.
In Text Summarization (a problem of long-sequence generation), STG shows signiﬁcantly larger gains than Non-STG-MLE, Non-STG-RL, and their naive ensembles with the PLM in every score metric and training data size as shown in Table 3 and
Appendix B. 4. W HAT MAKES STG BETTER ?
Knowledge Preservation.
Although freezing the parameters of the PLM can mitigate the forgetting problem, it would be failed to do so when additional units having sufﬁcient capacity is used since it is easy to memorize typical patterns of answers in small amount of training dataset and can degrade the generalization performance.
An example that represent this issue is shown in Table 4 of Appendix A. In this case, a passage that contains two deﬁnitions (super-scripted and bolded) about conﬂict is given with a query which asks the psychological meaning of conﬂict . Without the knowledge of who
Colman 4 is , it can be a hard to answer since the word psychology is not appeared in the passage. Here, the PLM repeats the given query due to the imperfect domain adaptation. Non-STG models, both Non-STG-MLE and RL, generate the same answer, that is not the psychological meaning but the general meaning of conﬂict , with pretty low perplexity ( ≈ 0 . 35 ).
The reason in producing this incorrect answer is that most queries in few-shot training data ask a general meaning of a concept, and Non-STG models are overﬁtted to this pattern. On the other hand, the answer of STG, which is close to the ground truth, is generated by the PLM policy π LM after some sequence of tokens ( conﬂict is ) that are sampled from the task-speciﬁc policy π a .
We can ﬁnd such examples for the other tasks in Appendix F: In Data-To-Text, as shown in the last example of Table 8, Non-STG generates nicam stereo which is not appeared in the given data. This is due to that nicam stereo was appeared 7 times (7/50, 14% ) in training data. In Summarization, as shown in the ﬁrst example of Table 12, Non-STG models only consider the forepart of the given article.
Since the most of the major information is appeared in the forepart in News data, Non-STG models can be easily overﬁtted to generate the text according to such a pattern.
Hence, we claim that
Non-STG easily exposed learning patterns of typical answering and STG resolves this issue since it can be fully accessible to the knowledge of the PLM.
Resolving Disadvantages of RL.
As described in Section 2.4, RL has some limitation with its application: 1) it suffers from exponentially large search space |V| T , and 2) it suffers from unstable training caused by credit assignment problem.
STG resolves the ﬁrst issue since the frozen PLM chooses a token when it is selected, and therefore the search space of the generator is approximately decreased from |V| T to |V| T − T PLM where T PLM is the average length of sequences generated by PLM.
For the second issue, the loss function of STG (Equation 9) intuitively shows that the gradient to the task-speciﬁc policy π a associated with producing a t will depend on the selector’s action (i.e. i t = 1 ).
Hence, unlike Non-STG, π a of STG knows which token is contributed to the reward (see Appendix D for an illustration).
R ELATED W ORK
Recently, prompt-based in-context learning with an extremely large transformer-based PLM shows impressive few-shot generation performances (Radford et al., 2019; Brown et al., 2020).
Schick & Sch¨utze (2020) propose manually designed natural language prompts for improved few-shot text summarization and headline generation.
Elsahar et al.
(2018) conduct zero-shot learning for question generation from knowledge graphs, however they require a large amount of in-domain training data for their transfer learning.
Chen et al.
(2020) directly ﬁnetune the pretrained GPT-2 with a small amount of serialized attribute-value pairs for table-to-text generation.
Gong et al.
(2020) further apply multiple tasks to effectively leverage the structured information of tables.
In contrast to these approaches, our proposed method utilizes RL-based additive learning for few-shot text generation.
Applying RL for text generation has been widely used to mitigate the exposure bias problem of
MLE as well as to directly optimize task-relevant evaluation metrics.
Ranzato et al.
(2015) use the REINFORCE algorithm for text summarization and machine translation while Bahdanau et al.
(2017) use the actor-critic algorithm for machine translation.
However, they require pretraining using MLE.
Ding & Soricut (2017) propose softmax policy gradient to remove the MLE-based pretraining.
However, it requires various techniques for effective training.
Tan et al.
(2018) propose an entropy-regularized policy optimization that subsumes many of the previous training algorithms.
Our proposed method is different from these methods in that we apply RL for more difﬁcult few-shot generative modeling.
Various methods take into account the RL tasks with large action spaces like NLG.
Dulac-Arnold et al.
(2015) consider only actions in a cluster around the latent state of action obtained from a given state.
Chandak et al.
(2019) deﬁne the action embedding as a distribution with semantic of action and use a deterministic policy to take an action.
Even-Dar et al.
; Zahavy et al.
(2018) devise a method of incorporating the process of directly removing unnecessary actions according to the state in the RL problem.
Guo et al.
(2021) propose to use soft Q-learning and path consistency learning to combine off- and on-policy updates.
Unlike these approaches, we use the hierarchical policy that reduces the sequential action space.
C ONCLUSION
In this work, we propose to exploit a selective token generation between the pretrained language model and the task-speciﬁc adapter with RL-based additive learning for the tasks of few-shot natural language generation.
In particular, we devise a trainable policy selector at the token level and jointly learn it with the task-speciﬁc policy.
The proposed policy selector and RL algorithm make the two policies complementary each other and lead to robust few-shot generative modeling.
Experimental results on various tasks of few-shot text generation show that the proposed selective token generation along with RL-based additive learning consistently improves the performances with less overﬁtting.
For future work, we will investigate more general ensemble learning for few-shot learning and perform more study on the architectures of both the adapter and selector. A Q UESTION A NSWERING E XAMPLE Passage three types of conﬂicts are : 1.
intrapersonal conﬂicts , 2.
interpersonal conﬂicts and 3. unconscious conﬂicts . the word conﬂict has been derived from a latin word “conﬂicts” which means “strike two things at the same time” . conﬂict is 1) an opposition or a tug-of-war between contradictory impulses .
according to colman ”a conﬂict is 2) the anticipated frustration entailed in the choice of either alternative ”.
[Table]
conﬂict deﬁnition psychology
Ground-truth the anticipated frustration entailed in the choice of either alternative.
PLM conﬂict deﬁnition psychology.
Non-STG conﬂict is an opposition or a tug-of-war between contradictory impulses.
STG conﬂict is the anticipated frustration entailed in the choice of either alternative.
B S UMMARIZATION P ERFORMANCE
[Table] [Table]
C T RAINING S ETTINGS
In our experiments all the models of additive learning, Non-STG and STG, are used the same architecture and hyper-parameters (except whether to use pre-training) for training as described in Table 7.
We found that pre-training the addtional layer of Non-STG-RL with MLE helps the performance improvements.
On the other hand, STG without pre-training shows better performances.
We use the training data for each topic of the task of Data-to-Text as their validation data.
[Table]
with
[Table]
D I LLUSTRATION OF STG
E A DDITONAL S TUDY
STG-MLE.
Here, we evaluate the MLE version of STG (denoted as STG-MLE) which is trained by MLE for the mixture policy π h ( · ) =
π s ( i t = 0) π LM ( · ) + π s ( i t = 1) π a ( · ) similar to copy mechanism (Gu et al., 2016).
In few-shot training, the explicit use of PLM logits can efﬁciently reduce the ﬁne-tuning loss especially when the adapter is light since the adapter can focus only on the task-relevant part in generation.
STG-RL 5 learns to do this naturally by stochastic policy sampling if the policy selector is initialized to perform uniform sampling.
On the other hand, STGMLE can be easily collapsed to select only a task-speciﬁc policy (i.e. i t = 1 ).
This is because the gradient ﬂows the additional model only and, unlike STG-RL, there is no chance to exploit diverse paths during training in the teacher forcing manner.
As shown in Figure 3, STG-MLE starts from the same point of STG-RL but it collapsed to Non-STG-MLE.
Learning Curve.
It is well known that the RL-tuning resolves the exposure bias of MLE-tuning.
We can expect that an additive learner of MLE would be affected by the exposure bias as well, and
the
RL objective for additive learning resolves it.
Here, we present some learning curves 6 obtained from training in our experiments.
As shown in Figure 3, the learners of MLE seem to have overﬁtting (in terms of Perplexity, PPL) and exposure bias (in terms of Score).
On the other hand, the learners of
RL were less effected by the problems.
We can ﬁnd that the STG models (denoted STG-RL) are superior to the others from the perspective of the score.
Effectiveness of Selector.
Here, we investigate the effectiveness of the selector π s of the STG.
We compare Fixed Selection against the Dynamic selection .
In the ﬁxed selection, the probability of selecting the PLM’s policy π LM is ﬁxed to π s (0 t | s t ) = 1 − π s (1 t | s t ) .
We measure the performance with respect to π s (1 t | s t ) =
c where c is a constant.
The selection will be uniformly random when c = 0 .
5 , and when c = 0 , the performance will be equivalent to the performance of the PLM without additive learning.
Figure 4 shows that the input-dependent dynamic selection by our STG outperforms the ﬁxed selection with any c .
We can ﬁnd that how π s works for each task.
For instance, in QA task, the ﬁrst few tokens of an answer may decide the quality of generation (i.e.
”yes” or ”no” in binary QA). Therefore, an optimal strategy of the STG might be producing the ﬁrst few tokens sampled from the task-speciﬁc π a and the remaining tokens from the PLM π LM . The curve supports this interpretation since the score is decreased as c is close to 1. Our STG learns such a strategy as shown from the generated answers in Table 9 and 10.
In Data-to-Text, the BLEU score is increased as c is close to 1 while the ERR score is decreased. This fact supports the results of NE( mix ) models as discussed in Section 4.6. The π s learns to balance between the BLEU and
ERR.
Here, we show generated sentence examples for each task (see Table 8 for Data-to-Text, Tableand Table 10 for Question Answering and Table 11 and Table 12 for Summarization.). The tokens sampled from the task-speciﬁc policy π a are presented in red.
[Table]
the kratos 31 is a great television . it only consumes 31.4 watts , comes with a remote control and european warranty and has a stylish mat black bezel with mat black frame stand .
[Table]
the kratos 31 features a mat black bezel with mat black frame stand and european warranty.
[Table]
the kratos 31 features a mat black bezel with mat black frame stand and european warranty
[Table]
the kratos 31 television has 31.4 watt power consumption, remote control and european warranty.
[Table]
the kratos 31 television has 31 watt power consumption. it comes with a remote control and european warranty with a color of mat black bezel with mat black frame stand.
[Table]
inform(name
[Table]
triton
[Table]
type
[Table]
television
[Table]
family
[Table]
l
[Table]
audio [Table]
cevo stereo [Table] the triton 52 47 inch screen television from the l7 product family and has cevo stereo audio.
[Table]
the triton 52 television is a 47 inch television in the l7 family and with a cevo stereo.
[Table]
the triton 52 television is in the l7 family, it has 47 inches of screen, and uses cevo stereo sound capabilities.
[Table]
the triton 52 television is in the l7 family, it has 47 inches of screen, and aeuro price,
[Table]
the triton 52 television is a 47 inch television in the l7 family and has a cevo stereo.
[Table]
[Table]
television
[Table]
1080p
[Table]
the attis 65 television is in the cheap price range and has a nice resolution of 1080p.
[Table]
the attis 65 television in the cheap price range has a 1080p resolution and a pricerange of cheap.
Non-STG-MLE the attis 65 television has a 1080p resolution, is in the cheap price range and has a nicam stereo.
NE( mix )-MLE the attis 65 television is a 1080p television in the cheap price range and in the cheap price range with a nicam stereo.
STG the attis 65 television is in the cheap price range and has a 1080p resolution.
[Table]
zygote : this cell results from the union of an oocyte and a sperm . a zygote is the beginning of a new human being ( i.e. , an embryo ) .
the expression fertilized ovum refers to a secondary oocyte that is impregnated by a sperm ; when fertilization is complete , the oocyte becomes a zygote.10 ( emphasis added . ) Query what is a human zygote
[Table]
the beginning of a new human being .
PLM the expression fertilized ovum refers to a secondary oocyte that is impregnated by a sperm ;
NE( mix )-MLE the expression fertilized ovum refers to a secondary oocyte that is impregnated by a sperm ; when fertilization is complete , the oocyte becomes a zygote .
NE( mix )-RL the expression fertilized ovum refers to a secondary oocyte that is impregnated by a sperm ; when fertilization is complete , the oocyte becomes a zygote .
STG it is the beginning of a new human being .
Passage sometimes the terms reactive airway disease and asthma are used interchangeably , but they are not the same thing .
often , the term reactive airway disease is used when asthma is suspected , but not yet conﬁrmed .
reactive airway disease in children is a general term that does n’t indicate a speciﬁc diagnosis .
it may be used to describe a history of coughing , wheezing or shortness of breath triggered by infection .
these signs and symptoms may or may not be caused by asthma .
Query symptoms of reactive airway disease in adults
Ground
[Table]
coughing , wheezing or shortness of breath triggered by infection .
PLM symptoms of reactive airway disease in children is a general term that does n ´ t indicate a speciﬁc diagnosis .
NE( mix )-MLE symptoms of reactive airway disease in adults are a general term that does n ´ t indicate a speciﬁc diagnosis .
NE( mix )-RL
yes , reactive airway disease in children is a general term that does n ´ t indicate a speciﬁc diagnosis .
STG coughing , wheezing or shortness of breath triggered by infection .
Passage sulfate is an anion , composed of a sulfur atom and four oxygen atoms .
the overall charge of this anion is -2.
the molecular formula for sulfate is so 4 2-.
there , two oxygen atoms are bonded to the sulfur atom through double bonds and other two are bonded to sulfur through single bonds .
the oxidation state of sulfur in sulfate is +6 .
Query is sulfate sulfur and oxygen bonded
Ground-truth yes , sulfate is sulfur and oxygen bonded .
PLM no answer present .
NE( mix )-MLE
yes , sulfate is sulfur and oxygen bonded to the sulfur atom through double bonds and other two are bonded to sulfur through single bonds .
NE( mix )-RL
yes , sulfate sulfur and oxygen bonded to the sulfur atom through double bonds and other two are bonded to sulfur through single bonds .
STG yes , sulfate is sulfur and oxygen bonded to each other through double bonds and other two are bonded to sulfur through single bonds .
[Table]
gryfﬁndor house is one of the four houses at hogwart into which the students are sorted when they start their ﬁrst year .
gryfﬁndor is the hogwarts house of founder godric gryfﬁndor .
gryfﬁndor house places high value on traits such as courage , chivalry , and nerve .
during harry ’ s time at hogwarts , the head of gryfﬁndor house was minerva [ ... ]
[Table]
gryfﬁndor house is one of the four houses at hogwart into which the students are
[Table]
they
[Table]
ﬁrst year .
[Table]
gryfﬁndor house is one
[Table]
four
[Table]
hogwart
[Table]
which the students are sorted
[Table]
they start their ﬁrst year
[Table]
gryfﬁndor house is one of the four houses at hogwarts into which the students are sorted when they start their ﬁrst year
[Table]
the median home value in louisville is $ 119,500 .
louisville home values have gone up 1.8 % over the past year and zillow predicts they will rise 3.
% within the next year .
the median list price per square foot in louisville is $ 125 .
the median price of homes currently listed in louisville is $ 180, .
the median rent price in louisville is $ 1,050 .
foreclosures will be a factor impacting home values in the next several years .
in louisville 0.1 homes are foreclosed ( per 10,000 ) .
[Table]
the
[Table]
home
[Table]
in louisville
[Table]
the average per therm price in louisville is $
[Table]
[Table]
the average per square foot in louisville is $
[Table]
the median home value in louisville is $ 119,500 .
[Table]
the brain is made up of more than just nerve cells ( neurons ) .
although there are about 100 billion neurons in the brain , there may be about 10 to 50 times that many glial cells in the brain .
neurons have two processes called axons and dendrites ....
glial cells have only one .
2 neurons can generate action potentials ... glial cells can not .
3 however , glial cells do have a resting potential .
4 neurons have synapses that use neurotransmitters ... glial cells do not have chemical synapses .
[Table] Article -lrb- cnn -rrb- – three runners died sunday during the detroit free press/ﬂagstar marathon in detroit, michigan, police told cnn.
an emt vehicle is at the scene sunday in detroit after three runners collapsed at a marathon.
all three deaths occurred between 9 and 9:20 a.m. et, second deputy chief john roach said.
a man in his 60s fell and hit his head, roach said.
the cause of the fall was unknown.
the man was transported to detroit receiving hospital, where he was pronounced dead.
two other men, ages 36 and 26, also collapsed during the race and were pronounced dead at the hospital, roach said.
all three collapsed near the end of the race, he said.
witnesses describe scene ” the weather at the time was overcast, roach said, with temperatures in the low 40s.
Ground-truth second deputy chief john roach : all three deaths occurred between 9 and 9:20 a.m. man in his 60s fell hit his head ; two men others , ages 36 and 26 , collapsed .
race was detroit free press/ﬂagstar marathon in detroit , michigan .
PLM three runners collapsed at a marathon in detroit , police say .
the cause of the fall is unknown .
Non-STG-MLE three runners collapsed at a marathon sunday , police say .
the cause of the fall is unknown , police say .
Non-STG-RL three runners collapsed at a marathon sunday , police say .
the cause of the fall is unknown , police say .
STG three runners collapsed at a marathon in detroit , michigan .
all three deaths occurred between 9 and 9:20 a.m. et .
a man in his 60s fell and hit his head , police say .
Article london, england -lrb- cnn -rrb- – up to 1,000 human rights campaigners demonstrated saturday in front of no. 10 downing street, the ofﬁcial residence of british prime minister gordon brown, calling on the british government to demand that full democracy be restored in pakistan.
jemima khan, center, ex-wife for former pakistani cricket star imran khan, joins protesters in london.
protesters waved placards and chanted in support of the resignation of pakistani president pervez musharraf, a week after he imposed a state of emergency in the country.
the crowd of demonstrators massed behind barriers and included jemima khan, the ex-wife of former pakistani cricket star turned politician imran khan.
the demonstrators carried placards saying “ free the innocent ” and “ end musharraf’s regime ” and waved pakistani ﬂags.
imran khan, who heads the the movement for justice party, has been under house arrest since the emergency declaration.
his ex-wife delivered a petition to a doorman at downing street, calling on britain to use its inﬂuence to ensure that all institutions are in place well in advance of pakistani elections originally scheduled for early next year.
the petition also [...] [Table]
truth human rights campaigners demonstrate in front of no . 10 downing street .
protests urged uk government to demand full democracy restored in pakistan .
cricketer turned politician imran khan ´s ex wife jemima among protesters .
[Table] pakistani president pervez musharraf has been under house arrest since the emergency declaration .
his ex-wife delivered a petition to a doorman at downing street .
Non-STG-MLE pakistani president pervez musharraf has imposed a state of emergency in the country .
he has been under house arrest since the emergency declaration .
Non-STG-RL pakistani president pervez musharraf has imposed a state of emergency in the country .
he has imposed a state of emergency in the country since last week .
STG the ofﬁcial residence of british prime minister gordon brown .
hundreds of protesters demonstrate in front of no . 10 downing street .
the petition calls for full democracy in pakistan .
[Table] -lrb- cnn -rrb- spoiler alert!
it’s not just women getting cloned.
that was the big twist at the end of “ orphan black’s ” second season.
the kickoff to the new season leads the list of six things to watch in the week ahead.
“ orphan black, ” 9 p.m. et, saturday, april 18, bbc america.
the cloning cult sci-ﬁ series remains one of the most critically acclaimed shows on tv, thanks in large part to the performance of tatiana maslany, who has taken on at least six roles on the show so far, including a newly introduced transgender clone.
maslany told reporters this week that we can expect even more impressive scenes with multiple clones.
“ turn : washington’s spies, ” 9 p.m. et, monday, amc.
the series about spies in the early days of the revolutionary war returns with a new subtitle, “ washington’s spies, ” and a new monday night time slot. series star jamie bell told cnn what we can expect in the second season. “ this year we have a lot more battles ; [...] 3. “ game of thrones, ”p.m. et, sunday, hbo.
the world of westeros returns for a ﬁfth season in one of the biggest season premieres of the year. click here for more on what to expect.
[Table]
critically acclaimed series “ orphan black “ returns .
“ turn : washington ´ s spies “ starts a second season .
“ game of thrones “ is back for season ﬁve .
[Table]
series returns with a new subtitle , “ washington ´ s spies , “ and a new monday night time slot .
series star jamie bell told cnn what we can expect in the second season .
Non [Table] STG-MLE “ orphan black “ returns with a new subtitle and a new monday night time slot .
series star tatiana maslany is blown away by the response to the series so far .
Non-STG-RL “ orphan black “ returns with a new subtitle and a new monday night time slot .
series star jamie bell says the series will bring more recognition to the culper spy ring .
STG the new season of “ orphan black “ premieres on saturday the 18th .
“ game of thrones “ returns for a ﬁfth season .
Article energy drinks can pose a real problem for children, according to a new study from the american heart association.
researchers found that 40 % of the 5, calls to poison centers for “ energy drink exposure ” involved children under age 6.
in most of the cases, the parents didn’t know the children had gotten hold of an energy drink.
many of the calls reported the children were experiencing serious side effects, such as an abnormal heart rhythm, or they were having a seizure.
the study is being presented at the american heart association’s scientiﬁc sessions this week.
study : caffeine affects teen boys more.
study author dr. steven lipshultz has handled cases involving children who became sick after consuming energy drinks.
while studies about the impact of caffeine on children are limited, lipshultz, [...]
truth 40 % of “ energy drink exposure “ calls to poison control centers are for children under 6.
study authors believe the number is under-counted .
the fda does not set a caffeine limit for children .
PLM drug calls to poison centers for “ energy drink exposure “ in most cases , the parents did n’t know the children had gotten hold of an energy drink .
many of the calls reported the children were experiencing serious side effects , such as an abnormal heart rhythm .
Non-STG-MLE drug calls to poison centers for “ energy drink exposure “ in most cases , the parents did n’t know the children had gotten hold of an energy drink .
many of the calls reported the children were experiencing serious side effects , such as an abnormal heart rhythm .
Non-STG-RL “ this is a very concerning ﬁnding , “ dr. laurence sperling says .
the american academy of pediatrics recommends children consume no caffeine .
STG drug calls to poison centers for “ energy drink exposure “ nearly 40 % of calls to poison centers for “ energy drink exposure “ involved children under age 6.
study : caffeine affects teens more .