Simple and Effective Retrieve-Edit-Rerank Text Generation (2020)

Abstract: Retrieve-and-edit seq2seq methods typically retrieve an output from the training set and learn a model to edit it to produce the final output. We propose to extend this framework with a simple and effective post-generation ranking approach. Our framework (i) retrieves several potentially relevant outputs for each input, (ii) edits each candidate independently, and (iii) re-ranks the edited candidates to select the final output. We use a standard editing model with simple task-specific re-ranking approaches, and we show empirically that this approach outperforms existing, significantly more complex methodologies. Experiments on two machine translation (MT) datasets show new state-of-art results. We also achieve near state-of-art performance on the Gigaword summarization dataset, where our analyses show that there is significant room for performance improvement with better candidate output selection in future work.

## introduction retrieve-and-edit text generation methods have received significant recent interest; editing humanauthored text can potentially avoid many of the challenges that are seen while generating text from scratch, including the tendency to be overly repetitive or to degrade on longer texts @xcite . retrieve-and-edit methods have been developed for summarization @xcite , machine translation @xcite , language modeling @xcite , and conversation generation @xcite . these methods first retrieve a single output from the training set, and then use a learned model to edit it into the final output. in this paper, we show that generation performance can be improved with a retrieve-edit-rerank approach that instead retrieves a set of outputs from the training set, edits each independently, and then re-ranks the results to produce the final output. figure 1 shows an overview of the approach. we use standard keyword-based retrieval and a simple editor, where the retrieved output is concatenated to the original input to train a transformerbased seq2seq editing model. our final re-ranking step is task specific, but again very simple in every case. our goal here is not to find the best possible way to do the re-ranking. instead, we show that gains are possible and that it helps to see what edits are made for multiple candidates before making the final decision, instead of following previous work by trying to select a single candidate before editing. we evaluate performance on the gigaword summarization dataset @xcite and on the english to dutch (en-nl) and the english to hungarian (en-hu) machine translation (mt) tasks, following @xcite . for mt, we experimented with different re-ranking schemes but found that the original model score (log-likelihood) worked best, amounting to extended beam search within the complete retreive-edit-rerank pipeline. we improve performance by 6.5 bleu points on en-nl and 7.5 on en-hu over the state-of-art neural fuzzy repair system @xcite . on gigaword, we simply re-rank by returning the most common output, and we achieve up to 1.2 rouge improvement over the comparable re 3 sum model @xcite . finally, through qualitative analysis, we find evidence that better post-generation ranking is feasible and can lead to substantial performance improvement, which emphasizes the need for future work in developing new post-generation ranking techniques. ## related work recent work has developed retrieve-and-edit approaches for many tasks, including dialogue generation @xcite , language modeling @xcite , code generation @xcite , neural machine translation (nmt) @xcite @xcite and post-editing for nmt @xcite . candidate ranking has served as a core part in some retrieval-based models @xcite , but these models do not edit the retrieved candidates. for machine translation, @xcite developed a retrieve-and-edit based lstm model called neural fuzzy repair (nfr), which they applied on two mt datasets obtained from @xcite . using a keyword based followed by a token edit distance based retrieval method called sss+ed, they showed that concatenating the source and retrieved outputs as the input significantly boosts translation quality. nfr is trained by augmenting the source with up to 3 retrieved outputs, which are fed together into the editing model in several ways. our approach, instead, simply edits multiple candidates separately and then re-ranks the final results. for summarization, re 3 sum @xcite ) is an lstm-based model developed under the retrieve-and-edit framework, and tested on the gigaword summarization (also headline generation) task @xcite . re 3 sum retrieves 30 headlines from the training set using the popular information retrieval method lucene 1 . next, it learns a model to pick the single best retrieved headline, which is then edited. biset @xcite is a retrieve-and-edit framework with more complex retrieval ranking and editing stages, which again edits only a single output. we compare our framework's performance against those of nfr, re 3 sum, and biset, showing the effectiveness of post-generation ranking. for the rest of this paper, we will use (x, y) to represent a source and target pair, (x , y ) to denote a retrieved source and output pair from the training set, and ŷ to represent the edited/generated output. ## conclusion and future work in this paper, we presented a retrieve-edit-rerank framework for seq2seq text generation. we used lucene for retrieval, a transformer model for editing, and simple task-specific post-generation ranking techniques. we applied the framework on two mt datasets and the gigaword summarization dataset. our results show that our simple ranking functions are effective in helping our model outperform the comparable retrieve-and-edit based methods for these datasets. by performing analysis on gigaword, we find that there exists room to improve summarization performance with better post-ranking algorithms, a promising direction for future research. this is in line with our overall goal, which is not to find the best possible way to do the postranking, but only to show that gains are possible by editing multiple candidates and then comparing the results. moving forward, we would like to apply this framework to other retrieve-and-edit based generation scenarios such as dialogue, conversation, and code generation.