Do you have the right scissors? Tailoring Pre-trained Language Models via M onte- C arlo Methods (2020)

Abstract: It has been a common approach to pre-train a language model on a large corpus and fine-tune it on task-specific data. In practice, we observe that fine-tuning a pre-trained model on a small dataset may lead to over- and/or under-estimate problem. In this paper, we propose MC-Tailor, a novel method to alleviate the above issue in text generation tasks by truncating and transferring the probability mass from over-estimated regions to under-estimated ones. Experiments on a variety of text generation datasets show that MC-Tailor consistently and significantly outperforms the fine-tuning approach.

## introduction recently, pre-trained language models (plm), e.g. gpt-2 @xcite , have shown great promise in many applications of natural language generation, such as stylized text generation @xcite and dialog system @xcite . plm is obtained by first pre-training on largescaled raw sentences (always general domain corpus), and then used in downstream tasks by finetuning on task-specific datasets (always from some specific domains). specifically, given a pre-trained gpt-2 model, to generate sentences of email domain, we always need to fine-tune the gpt-2 on a small set of email domain corpus. however, we argue that to get desired sentence outputs, fine-tuning plm on a specific domain dataset is not necessarily the best, especially when the fine-tuning dataset is of a small size. typically, fine-tuning is conducted through maximum likelihood estimation (mle), with which the resulting model distribution will be asymptotically consistent with true distribution when the fine-tuning dataset has infinite data samples. but it is not the figure 1 : the over-and under-estimation problems of the model distribution. for example, sample b represents the simple sentence "yes .", whose probability is over-estimated. its model nll (4.01, negative loglikelihood) is significantly lower than the 95% confidence interval of its real nll [4.89, 5.37], which is estimated on the training set. specifically, mle minimizes the kullback-leibler (kl) divergence between model and true distributions. @xcite point out that minimizing kl avoids assigning an extremely small probability to any data point but assigns a lot of probability mass to non-data regions, which leads to a gap between p real and p m odel . additionally, simple data patterns in the fine-tuning dataset could be easily memorized and over-estimated. meanwhile, the complex ones may be under-estimated. the above problem is not severe with adequate data samples, but non-trivial when the size of the fine-tuning dataset is not large enough. (see figure 1 ). to address the over-and under-estimated problem, in this paper, we propose mc-tailor, which can tailor the resulting density of model distribution by cutting the probability mass of over-estimated zones to under-estimated zones, leading to more realistic model distribution after fine-tuning. concretely, mc-tailor consists of two components: a ratio estimator to distinguish over-and under-estimated regions of model distribution; and an early rejection sampling (ers) component to tailor (reassign) probability mass and efficiently obtain sampled sentences from the model distribution. note that the proposed ers is inspired by sequential monte carlo (smc, @xcite ), but can avoid the degeneration from smc, as it directly kills samples rather than performs resampling. we conduct experiments on various data sets to verify the effectiveness of the proposed mc-tailor. empirical results show that mc-tailor can generate significantly better samples than finetuning, and the resulting model distributions of our model are closer to real data distributions. ## pre-trained language model language models generally estimate the density of sentences in real context within an autoregressive style: however, as stated in the introduction, directly fine-tuning the plm on a small dataset may lead to the mismatch problem, namely the over-and underestimated problem between the true distribution and the model distribution. in the next section, we propose a new method to alleviate this problem. ## proposed mc-tailor to mitigate the above shortcomings of finetuning, we propose mc-tailor, which generates samples from a modified sample distribution. mc-tailor is composed of a ratio estimator, which detects over-and under-estimate regions of model distributions, and the early rejection sampling algorithm (ers), which accelerates sampling while ensuring sample quality. ## experiments in this section, we empirically compare the sample quality of our model and baseline models. we first set up experiments and show results in section 4.2. ## conclusion in this paper, we propose mc-tailor to alleviate the over-and under-estimation problem between true and model distributions. mc-tailor is composed of a ratio estimator, which adjusts the probabilities of mle fine-tuned plms to approximate true distributions, and the ers to accelerate sampling while ensuring sample quality. experiments on various datasets show the effectiveness and efficiency of mc-tailor.