On Learning to Summarize with Large Language Models as References (2024)

Abstract: Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. Therefore, we study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved. To this end, we use LLMs as both oracle summary generators for standard supervised fine-tuning and oracle summary evaluators for efficient contrastive learning that leverages the LLMs’ supervision signals. We conduct comprehensive experiments with source news articles and find that (1) summarization models trained under the LLM-as-reference setting achieve significant performance improvement in both LLM and human evaluations; (2) contrastive learning outperforms standard supervised fine-tuning under both low and high resource settings. Our experimental results also enable a meta-analysis of LLMs’ summary evaluation capacities under a challenging setting, showing that LLMs are not well-aligned with human evaluators. Particularly, our expert human evaluation reveals remaining nuanced performance gaps between LLMs and our fine-tuned models, which LLMs fail to capture. Thus, we call for further studies into both the potential and challenges of using LLMs in summarization model development.

## introduction recent studies @xcite @xcite have discovered that large language models (llms), like gpt-3.5 @xcite , can generate summaries that are preferred by human annotators when compared to reference summaries from widely used datasets, such as cnn/dailymail @xcite and xsum @xcite , in a reference-free human evaluation setting. this quality issue of existing reference summaries effectively puts an upper bound on the performance of summarization models trained on them, which likely contributes to the performance gap between supervised summarization models, e.g., bart @xcite , and llms as observed by related work @xcite @xcite . therefore, we aim to investigate whether smaller summarization models can be substantially improved with better references. to this end, we study an llm-as-reference distillation setting, where the llms are considered the reference or the gold-standard oracle for the summarization task. specifically, we employ llms in the training of smaller text summarization models in two manners: (1) llms as the gold summary generator, where the model is trained with the llm summary as the reference under the standard supervised fine-tuning; (2) llms as the gold summary evaluator, where llm-based automatic evaluation methods @xcite are used as supervision signals for training techniques such as contrastive learning @xcite and reinforcement learning @xcite . using the source articles in the cnn/dailymail dataset, we conduct comprehensive experiments for this llm-as-reference setting, across proprietary and open-source llms under low and high resource conditions. the experimental results demonstrate that (1) llm-generated summaries are better references for the smaller models than the original reference summaries, and (2) contrastive learning with llms as evaluators outperforms standard supervised fine-tuning. in particular, our bestperforming fine-tuned bart checkpoint can outperform gpt-3.5 under gpt-4's evaluation @xcite . meanwhile, under expert human evaluation, it can achieve similar or superior overall performance to gpt-3.5 in 50% of cases, while the original fine-tuned bart has a comparable rate of success in only 4% of cases. to have a more comprehensive understanding of this llm-as-reference setting, we then conduct a meta-analysis of the llm-based evaluation methods by assessing their alignment level to the human evaluation. while these llm-based methods achieve strong performance in existing metaevaluation datasets consisting of summaries generated by supervised summarization systems @xcite , we find that they do not correlate well with human evaluation when comparing close-performing systems in our setting. particularly, although our fine-tuned model achieves better performance than llms under llm-based evaluation, human evaluation reveals remaining nuanced performance gaps between our model and llms. our main contributions are two-fold: (1) we empirically demonstrate that the performance of smaller models can be substantially improved when trained using better references (llms) and learning methods (contrastive learning). (2) we perform a meta-analysis of llm-based evaluation under a challenging scenario enabled by our task setting where llms need to compare summarization systems with close performance, which indicates that llms fail to align with human evaluation and capture nuanced performance differences. foot_0 2 methods ## learning with llms as references we conduct experiments with both proprietary and open-source llms in the llm-as-reference learning setting of smaller summarization models across low and high resource conditions and compare different training methods. ## human evaluation and meta-analysis in §3 we have demonstrated that smaller summarization models that are trained with better references can achieve on-par or even better performance than llms under llm-based evaluation. however, the alignment between llm and human evaluation still requires examination. therefore, we first conduct a human evaluation comparing the performance of models in §3, then perform a metaanalysis regarding the llm-based evaluation. ## related work training methods of text generation models the standard mle training of text generation models has two major limitations: (1) a discrepancy between the training objective, i.e., the cross-entropy loss, and the evaluation criteria (e.g., ## conclusion in this work, we study a learning setting of text summarization models where the llms are set to be the reference. for this setting, we leverage the llm-based evaluation methods to guide the model training through contrastive learning and empirically demonstrate the efficiency and effectiveness of our methods. furthermore, we conduct human evaluation and meta-analysis regarding the reliability of llm-based evaluation, which reveals its benefits as better training references and its limitations in terms of the alignment with human evaluation. we believe our findings shed light on the direction of reliably applying the llms to the entire development loop (i.e., training-validation-evaluation) of smaller, task-specific nlp models, which has the potential to provide a balance between model performance and computational cost.