Selective Token Generation for Few-shot Natural Language Generation (2022)

Abstract: Natural language modeling with limited training data is a challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability. Among them, additive learning that incorporates a task-specific adapter on top of the fixed large-scale PLM has been popularly used in the few-shot setting. However, this added adapter is still easy to disregard the knowledge of the PLM especially for few-shot natural language generation (NLG) since an entire sequence is usually generated by only the newly trained adapter. Therefore, in this work, we develop a novel additive learning algorithm based on reinforcement learning (RL) that selectively outputs language tokens between the task-general PLM and the task-specific adapter during both training and inference. This output token selection over the two generators allows the adapter to take into account solely the task-relevant parts in sequence generation, and therefore makes it more robust to overfitting as well as more stable in RL training. In addition, to obtain the complementary adapter from the PLM for each few-shot task, we exploit a separate selecting module that is also simultaneously trained using RL. Experimental results on various few-shot NLG tasks including question answering, data-to-text generation and text summarization demonstrate that the proposed selective token generation significantly outperforms the previous additive learning algorithms based on the PLMs.

## introduction recently, pretrained language models (plms) have shown great generalization ability when combined with large-scale data and big transformer-based models @xcite @xcite @xcite @xcite . therefore, transfer learning from plms has been popularly used for fewshot natural language generation (nlg) tasks with promising results. in specific, the use of plm for few-shot nlg can be categorized into three approaches: 1) prompt-based, 2) finetuning, and 3) additive learning. prompt-based approaches encode a task description and task-specific examples as a natural language prompt for few-shot text generation @xcite @xcite @xcite . these approaches can take full advantage of the universal natural language understanding and generation capabilities of large-scale plms without further training of the main model, however, they have some limitations in dealing with a large domain shift from the pretraining corpus data, tuning suitable task-specific prompts, and covering an increased size of conditioning examples. on the other hand, finetuning of the plm is able to explicitly impart task-specific knowledge to the model and hence lift the above limitations @xcite @xcite . however, these finetuned models are prone to overfitting when only a small amount of training data is available. in order to alleviate such an overfitting problem, additive learning has been extensively exploited by incorporating task-specific adapters into the plm @xcite @xcite . in general, task-specialized adapters for few-shot nlg are trained by maximum likelihood estimation (mle) or reinforcement learning (rl). while mle is efficient in learning, it suffers from the exposure bias problem due to the difference in the training and inference mechanisms @xcite , and this problem can be severe with limited training data. one solution is rl, capable of resolving this exposure bias problem by sequential output sampling during training @xcite @xcite . however, the exponentially large space of output sequences restricts the use of rl since it leads to high variance and unstable training which is more serious in the passage three types of conflicts are : 1. intrapersonal conflicts , 2. interpersonal conflicts and 3. unconscious conflicts . the word conflict has been derived from a latin word "conflicts" which means "strike two things at the same time" . conflict is 1) an opposition or a tug-of-war between contradictory impulses . according to colman "a conflict is 2) the anticipated frustration entailed in the choice of either alternative". ## selective token generation instead of generating all tokens in an output sequence from the single task-specific policy, π @xmath0, at each time step t, we sample an output token y t selectively from either the plm policy π @xmath1: this process can be considered as a token generation from a hierarchical policy π h (a t |s t ; θ s , θ lm , θ a ) where the policy selector represents the upper-level prior for the preference of the low-level policy. therefore, the value function of this hierarchical policy can be formulated as we denote i @xmath2. here, it is noted that a single critic network is used for the hierarchical policy since i t does not affect s t . given a sample trajectory {(s t , i t , a t , r t )} t @xmath3, the loss for optimizing θ s and θ a is and sg stands for the stop-gradient operator. similar to π a , π s makes use of the plm representations and the task-specific adapter such that the use of the separated policy selector that is simultaneously trained with the task-specific policy allows the task-specific policy to be complementary to the plm policy. especially, this cooperative ensemble learning can be realized by our rl algorithm that performs sequential sampling from the model during training. the advantages of stg are as follows: it is noted that although the stg also can be trained by mle, it can be easily collapsed to select only a task-specific policy irrespective of a given content. we analyze the mle version of stg in appendix b. ## experiments we evaluate our method against additive learning baselines on data-to-text, question answering and text summarization tasks which are widely used in few-shot nlg. ## related work recently, prompt-based in-context learning with an extremely large plm shows impressive fewshot generation performances @xcite . schick and schütze (2020) propose manually designed natural language prompts for improved few-shot text summarization and headline generation. @xcite conduct zero-shot learning for question generation from knowledge graphs, however they require a large amount of in-domain training data for their transfer learning. @xcite directly finetune the pretrained gpt-2 with a small amount of serialized attribute-value pairs for table-to-text generation. @xcite further apply multiple tasks to effectively leverage the structured information of tables. in contrast to these approaches, our proposed method utilizes rl-based additive learning for few-shot text generation. applying rl for text generation has been widely used to mitigate the exposure bias problem of mle as well as to directly optimize task-relevant evaluation metrics. @xcite use the re-inforce algorithm for text summarization and machine translation while @xcite use the actor-critic algorithm for machine translation. however, they require pretraining using mle. @xcite propose softmax policy gradient to remove the mle-based pretraining. however, it requires various techniques for effective training. @xcite propose an entropy-regularized policy optimization that subsumes many of the previous training algorithms. our proposed method is different from these methods in that we apply rl for more difficult few-shot generative modeling. the use of rl training in plm has been explored in many works. @xcite propose a controllable text generation which uses discriminators to guide generation of the plm. this approach assumes that constant classes like topics or preferences are available. @xcite use a plm as a caption generator for given image. in their referential game, the generator is rewarded by a kind of discriminator that responses a signal to the generator whether the corresponding caption is correct or not. various methods take into account the rl tasks with large action spaces like nlg. @xcite consider only actions in a cluster around the latent state of action obtained from a given state. @xcite define the action embedding as a distribution with semantic of action and use a deterministic policy to take an action. @xcite ; @xcite devise a method of incorporating the process of directly removing unnecessary actions according to the state in the rl problem. unlike these approaches, we use the hierarchical policy that reduces the sequential action space. ## limitations & future work adapter. in this study, we aim to propose a new generation framework for few-shot natural language generation tasks. in particular, a relatively naive neural adapter which utilizes only the top layer of the plm is used in this paper, and thus it may lead to limited improvements as shown in the experimental results on the summarization task. fortunately, there are several neural architectures @xcite @xcite for efficient task adaptation, and we believe that such adapters also make stg more efficient for covering a large domain shift and scaling. the study on the architectures of the adapters will be conducted in future works. efficient exploration. the fundamental limitation in stg is a high dependency on plm; when stg has a sufficient powerful plm, the selector does not select the additional adapter and it is thus nothing more than the plm. we can find such phenomenon in some examples in ## conclusion in this work, we propose to exploit a selective token generation between the pretrained language model and the task-specific adapter with rl-based additive learning for the tasks of few-shot natural language generation. in particular, we devise a trainable policy selector at the token level and jointly learn it with the task-specific policy. the proposed policy selector and rl algorithm make the two policies complementary to each other and lead to robust few-shot generative modeling. experimental results on various tasks of few-shot text generation show that the proposed selective token generation along with rl-based additive learning consistently and significantly improves the performances with less overfitting.