Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback (2023)

Abstract: Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual-entailment rewards to optimize for factual consistency and explore the ensuing trade-offs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience and conciseness of the generated summaries.

## introduction recent advancements in abstractive summarization systems @xcite are often impeded by their tendency to output information that is either contradicting or unsupported by their input article, often termed as "hallucinations" or factual inconsistency @xcite @xcite . while these systems produce highly relevant and coherent text, this lack of factual consistency often limits their wide-spread adoption in real-world applications. an example is depicted in figure 1 , where the highlighted statement in the summary, while plausible, has no support in the input article. ## method we would like to increase factual consistency using an entailment-based reward, while retaining the high salience and coherence that current summarization models already obtain. to achieve this, we propose to initialize an rl policy with a summarization model trained on supervised data (the anchor model). from there, in each rl-based training step we update the parameters according to two signals: an entailment reward and a regularization term grounded on the anchor model. during rl training, the entailment reward directs the model towards increased faithfulness, while the regularization term keeps the model from drifting to degenerate solutions and "forgetting" how to summarize. the process is illustrated in figure 2 . ## analysis regularization and sampling temperature. figure 3 describes an ablation experiment where we vary the regularization α and the decoding temperature and measure the effect on different automatic metrics. higher sampling temperature correlates with higher entailment and lower rouge scores. we conjecture that this is since higher temperature generates more diverse summaries, which amplifies exploration away from the original gold references. a similar phenomenon is observed when considering token length, as lower temperature policies produce summaries closer in length to the data-mean than their higher temperature counterparts. as for the regularization coefficient α, we observe the expected trade-off: lower regularization (smaller α) leads to higher entailment (nli), lower similarity to the supervised summary (rouge), and higher coverage. these may be explained by removal of external hallucinations that often use vocabulary terms that are unrelated to the document. surprisingly, in each kl setting, the lower temperature policy favors more document-aligned terms (perhaps for their higher initial probability), yet this is not reflected in the nli metric, that stays lower than its higher-temperature counterpart. we also observe that the summaries get shorter with less regularization, as the policy learns to mention fewer details as a way to alleviate generating inconsistencies. model size. we tested our approach with different model sizes to study the effect of scale in the rlef h setup. we compared t5-base (220m parameters), t5-large (770m) and t5-xxl (11b), using the same hyper-parameters for all three models. figure 4 shows the entailment rate on the xsum validation set during rl-finetuning. for all model sizes, our approach improved the entailment ratio over the supervised model by a large margin. however, while the large and xxl models changes the average summary length only slightly, the base model completely degenerates, "hacking" the nli reward by generating summaries that are half as short as the reference. this suggests that higher-capacity models are essential to prevent reward hacking, perhaps due to two possible reasons. first, the larger policies have higher generalization capabilities overall and can better accommodate different rewards, such as entailment and summarization regularization in our case. second, since the anchor model uses the same architecture, the higher capacity anchor model is more robust to changes in the summary and produces lower scores for less informative or more extractive summaries. ## related work rl for text generation. rl has been applied to many text generation tasks like neural machine translation @xcite , extractive summarization @xcite @xcite , abstractive summarization @xcite and others @xcite @xcite @xcite . specifically for summarization, prior rl approaches used different reference-based metrics as a reward function. in @xcite , two reward signals are measured between the generated and reference summaries: lexical overlap (rouge) to gauge salience and an entailment score to measure factual consistency. @xcite employed a similar approach with question-answering, they produced qa pairs conditioned on the generated summary to detect inconsistencies with the reference, and another set of qas conditioned on the reference to measure salience. additionally, @xcite proposed quals, a more computationally efficient qa approach, that was used in a contrastive learning setting. while their approach could be used without comparing outputs to reference summaries, they observed that adding such comparisons with the reference is essential for the stability of their method. we note that for some datasets, reference summaries are likely to contain factual errors @xcite , decreasing the effectiveness of reference-based rewards. other rl methods, instead of explicitly defining the quality of a summary suggest to model it directly from human feedback @xcite @xcite . this technique can prevent errors due to ref-erences that are misaligned with human judgment. while it is a promising approach, it also requires acquiring task-specific annotation, which can be labor-intensive. another hybrid approach interleaves a crossentropy objective with policy gradients @xcite in multi-document summarization (mds). they use an in-domain nli model, for which they annotate their mds dataset with entailment decisions. to stabilize their policy they employ an additional gan-like training regime and add a discriminator loss between generated and reference summaries to their reward. trade-offs in consistency models. the choice of which factual consistency approach to use has interesting consequences for the rl setup. our work employs a binary nli decision that does not point towards the specific inconsistent parts in the output summary. consequently, the reward is assigned to the final token of the summary, leaving proper credit assignment to the rl algorithm. other methods, specifically those based on question-answering @xcite @xcite can frame misaligned answers in the generated summary and assign the reward explicitly to the offending tokens. however, these qa-qg based methods may be much slower to compute. our reward requires a single forward pass using a transformer model over the document-summary pair, in comparison, qa-qg approaches require generating answer candidates, questions, answers from both sources and computing answer alignment. some of this complexity is remedied by generating jointly questions-and-answers @xcite , but it still requires a lengthy decoding of qa pairs. a different nli-based approach decomposes the document and summary into smaller blocks of sentences @xcite and aggregates the final decision over a matrix of block-level nli scores. such approach could aid the rl algorithm with credit assignment when generating long summaries. in practice, the abstractive summarization datasets in this study use short single sentence summaries. ## conclusions and future work we propose to leverage nli models as a readymade, reference-free reward signal for rl training of factually consistent abstractive summarization models. our experiments and thorough analysis with automatic and human evaluation show promis-ing results for this approach, with our rl approach outperforming all baselines on factual consistency, while maintaining and even improving on other desired summarization attributes as well. in future work, we would like to extend this approach to other grounded generation tasks, like knowledge-driven dialog. in addition, we find it interesting to explore additional reference-free reward models for other summarization attributes (or for other tasks). then, an important research direction would be to understand how to properly adapt our method to work with multiple such rewards.