Joint Generator-Ranker Learning for Natural Language Generation (2023)

Abstract: Generate-then-rank is a widely used mechanism for text generation, where a generator produces multiple text candidates and a ranker chooses the best one among the text candidates. However, existing methods usually train the generator and the ranker individually, neglecting the mutual feedback that could further enhance the generation quality. To tackle this limitation, we propose JGR, a novel joint training algorithm that integrates the generator and the ranker in a single framework. JGR optimizes the generator with a hybrid objective that combines data likelihood and ranker reward, and trains the ranker with a contrastive loss that compares the generator outputs. By iteratively updating the generator and the ranker, JGR can effectively harmonize their learning and enhance their quality jointly. We evaluate JGR on various text generation tasks and demonstrate that it surpasses existing methods on four public datasets across three common generation scenarios. Our code and models are publicly available at https://github.com/microsoft/ProphetNet/tree/master/JGR .

## introduction the quality of the output texts produced by neural natural language generation (nlg) models, such as those for machine translation @xcite and summarization @xcite , depends largely on how they are trained and decoded. the conventional approach is to train them with loglikelihood objectives and decode them with greedy or beam search strategies. however, this approach often fails to select the best sample with the highest evaluation score among the generated candidates, as shown by previous studies @xcite . to overcome this limitation, some recent works @xcite @xcite proposed to use a separate ranker model to re-rank the output texts of the generator model, following a generate-then-rank pipeline. this pipeline can improve the quality of the output texts by exploiting the ranker model's ability to evaluate and compare different candidates. however, this pipeline also has a drawback: it requires training the generator and ranker models in two separate phases, which may not fully exploit the generative ability of the generator model and the feedback from the ranker model. in this paper, we propose a novel joint training paradigm of both generator and ranker (jgr) for nlg tasks, which aims to overcome the drawback of the generate-then-rank pipeline. unlike previous works, which train the generator and ranker models separately, we explore a joint and iterative training algorithm that updates both models in turn. our main motivation for the joint and iterative training of the generator and ranker is twofold. first, the ranker model can provide valuable feedback to the generator model based on the ranking scores of the generated candidates. this encourages the generator model to produce better outputs. second, the ranker model can also benefit from the outputs of a progressively better generator model, and improve its ranking performance gradually. the jgr framework consists of a generator and a ranker. during training, the generator and ranker alternate to update their parameters, and each of them involves the other's outputs in its own input signals. specifically, the ranker model is trained to rank the outputs generated by the generator model for a given input text by assigning a ranking score. at the generator training phase, the generator model uses a combination of the ranker score and the matching score (e.g., bleu) as the reward for each sample, and trains with policy gradients, which encourages the generator to produce candidates with higher rewards and mitigates the exposure bias issue in the teacher-forcing learning. to assess the effectiveness of jgr, we conduct experiments on four diverse nlg tasks from different domains, including abstractive summarization @xcite , conversational summarization @xcite , question generation @xcite , and dialogue @xcite . the experimental results demonstrate that jgr achieves remarkable performance gains over the conventional mle training method, with a 3-point increase in rouge-2 score on the cnn/dailymail dataset and a 3.5-point increase in bleu-2 score on personachat. furthermore, we make several interesting observations from the results. first, the rewards from the ranker are more effective than the rewards from the direct metrics, but combining them together stabilizes the training and produces a better generator. second, training the ranker only on the candidates from the generator is better than using ground-truth as positive examples. third, sampling more candidates during training leads to better performance within a certain range, which is consistent with data augmentation. fourth, though trained with reinforcement learning aimed at optimizing automatic evaluation metrics, jgr still does not compromise on other aspects of generation quality. finally, the joint training paradigm increases the diversity of the generator outputs, which in turn benefits the ranker training. ## methodology the model architecture of our jgr, shown in figure 1 , has two components: a generator that outputs several text candidates for an input text using an encoder-decoder model, and a ranker that scores these text candidates. the jgr workflow works as follows: a) the generator generates multiple text candidates conditioned on the input text; b) the input text and the text candidates are combined and sent to the ranker; c) the ranker learns to rank the text candidates via a contrastive learning objective; d) the ranker gives a reward to each text candidate, which in turn is used to train the generator. in the following, we first introduce the basic elements of conditional text generation, including problem definition, model architecture, and model training. ## conclusion in this paper, we propose a novel joint training of generator and ranker framework, namely jgr, for natural language generation. both the generator and ranker of our jgr can achieve state-of-the-art results on several benchmarks in the areas of summarization, question generation, and dialog. we also analyze our jgr in several aspects and find that: first, the rewards from the ranker work better than the rewards from the direct metrics such as bleu, but combining them together helps the training become more stable. second, during training, letting the ranker be trained on the candidates generated by the generator exclusively is even better than previous approaches using ground-truth as positive examples. third, more candidates being sampled during training can lead to better performance, which is consistent with the findings from data augmentation. fourth, though trained with reinforcement learning aimed at optimizing automatic evaluation metrics, jgr still does not compromise on other aspects of generation quality. finally, the joint training paradigm helps the