Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learning (2021)

Abstract: Likelihood training and maximization-based decoding result in dull and repetitive generated texts even when using powerful language models (Holtzman et al., 2019). Adding a loss function for regularization was shown to improve text generation output by helping avoid unwanted properties, such as contradiction or repetition (Li at al., 2020). In this work, we propose fine-tuning a language model by using policy gradient reinforcement learning, directly optimizing for better generation. We apply this approach to minimizing repetition in generated text, and show that, when combined with unlikelihood training (Welleck et al., 2020), our method further reduces repetition without impacting the language model quality. We also evaluate other methods for improving generation at training and decoding time, and compare them using various metrics aimed at control for better text generation output.

## introduction language models have become a subject of close attention in the natural language processing field over the past few years. they are widely used not only for unsupervised pre-training, but also for text generation, such as what is implemented in dialogue systems @xcite . while there are ongoing efforts to develop non-autoregressive models for language modeling, most current state-of-the-art approaches use the autoregressive method of generating text (i.e., word by word). @xcite showed that even powerful trained models with a high likelihood value for test data can output repetitive results. @xcite argues that the reason for that is train-test discrepancy and lack of generalization when running * work done while at vk. standard maximum likelihood estimation (mle) training. unwanted repetition can be remedied at decoding and training time. decoding methods focus on sampling techniques that generate less repetitive or incoherent samples, while other methods aim to improve model training to minimize the effects of degeneration. an effective method for reducing language model degeneration is unlikelihood training @xcite , where a regularization term forces the model to reduce the probability of generating a token that has already occurred in a sequence. @xcite further explored this idea and showed that adding a loss function for regularization to avoid undesirable sequences improves text generation not only by reducing repetition, but also by decreasing contradiction. @xcite reported that adding unlikelihood training also improves the humanness of generated text. in this paper, we propose implicit unlikelihood training, a method for regularizing output by fine-tuning a language model with policy gradient reinforcement learning to improve generation results. we apply this method for a repetition objective, and show that combining implicit unlikelihood training with minimizing unlikelihood loss results in reduced repetition and perplexity. we also evaluate alternative approaches to improving generated texts in terms of repetitiveness, and compare these methods using a wide variety of metrics. the source code is available at: github.com/vklabmipt/implicit-unlikelihoodtraining. ## results we showed that implicit unlikelihood training is a competitive approach that outperforms other methods in sequence repetition when fine-tuning small and medium gpt-2 models (see samples of generated outputs are provided in tables 8 , 9 in appendix. ## future work the described and evaluated reinforcement learning framework makes it possible to optimize text generation for any objective. in future work, we intend to test the approach not only for repetition, but also for various other metrics, such as the toxicity level or bias of generated text.