Bridging the Gap between Pre-Training and Fine-Tuning for Commonsense Generation (2023)

Abstract: Commonsense generation aims to generate a plausible sentence containing all given unordered concept words. Previous methods focusing on this task usually directly concatenate these words as the input of a pre-trained language model (PLM). However, in PLMs’ pre-training process, the inputs are often corrupted sentences with correct word order. This input distribution discrepancy between pre-training and fine-tuning makes the model difficult to fully utilize the knowledge of PLMs. In this paper, we propose a two-stage framework to alleviate this issue. Firstly, in pre-training stage, we design a new format of input to endow PLMs the ability to deal with masked sentences with incorrect word order. Secondly, during fine-tuning, we insert the special token [MASK] between two consecutive concept words to make the input distribution more similar to the input distribution in pre-training. We conduct extensive experiments and provide thorough analysis to demonstrate the effectiveness of our proposed method.

## introduction to investigate machines' ability of generating logical sentences, @xcite propose the commonsense generation task. given a set of concept words, this task is designed to generate a sentence which not only contains the given concepts but also can correctly describe the relations between concepts. an example is shown in existing methods employ the pre-trained languege models (plms) such as bart @xcite , gpt-2 @xcite as the backbone to solve this problem. they @xcite @xcite usually take the concatenated concepts words as the inputs. however, such processing of inputs * this work was done during an internship at tencent. concept words {wear, player, field, jersey} references the player will wear a jersey while on the field. a soccer player wears a jersey on the field. in this paper, we propose a two-stage framework to bridge the gap between pre-training and fine-tuning for this task. specifically, we firstly propose to introduce a domain-specific pre-training stage using the tasks' training dataset. the pretraining objective is designed to recover original sentences given the masked and shuffled sentences. therefore, the plms' ability of reasoning out new concepts or relations (mask operation) and processing order-agnostic inputs (shuffle operation) is enhanced. secondly, in downstream task finetuning, we insert the special token [mask] between two consecutive concept words. this makes the input distribution more similar to the distribution in pre-training. the experimental results shows that our proposed model can significantly improve the performance of the commonsense generation task. we also conduct experiments to show that our model is superior than baselines in terms of continual learning and few-shot scenarios. ## model we propose a two-stage training framework as shown in figure 1 . we firstly continually pre-train the bart with a newly designed input format. secondly, we fine-tune the model whose inputs are inserted with the special token @xcite . formally, given a concept word set @xmath0, x 2 , • • • , x n } ∈ x (n can be different for different inputs), the task aims to generate a fluent, plausible and grammatically correct sentence @xmath1, y 2 , • • • , y m ) ∈ y containing all the words in x. ## conclusion we study the gap issue between pre-training and fine-tuning for commonsense generation task. we propose a two-stage training framework which is composed of a domain-specific pre-training stage and a fine-tuning stage. pre-training stage aims to recover the masked and shuffled sentences which could enhance the models' ability of processing unordered inputs and reasoning out the relations and concepts. inserting [mask] to the inputs during fine-tuning have also been demonstrated very useful. experimental results show that our model is superior than many baselines, especially under few-shot setting.