Data Selection Curriculum for Abstractive Text Summarization (2023)

Abstract: Abstractive Text Summarization (ATS) models are commonly trained using large-scale data that is randomly shuffled. However, the impact of data selection and data ordering on ATS models remains a relatively unexplored research area, where a significant challenge lies in accurately assessing the learning difficulty of each training instance. This study introduces a Data Selection Curriculum (DSC) scoring system that incorporates both the difficulty of improving ATS model via an instance and the expected performance on this instance. By selectively excluding excessively simple and overly complex instances, the training efficiency can be optimized. Furthermore, curriculum learning is integrated to accelerate convergence and improve performance by gradually increasing the learning difficulty, inspired by human learners. Experimental results on the CNN/DailyMail dataset demonstrate that our approach surpasses potent baselines, utilizing a mere 20% of the available instances.

## introduction abstractive text summarization (ats) aims to generate concise summaries while preserving essential content. recent studies @xcite @xcite have shown that large language models (llms), including gpt-3.5 @xcite , can produce summaries more favored by human annotators compared to reference summaries from well-established datasets, such as cnn/dailymail @xcite . additionally, @xcite experimentally assert that contrastive learning based methods applied to smaller summarization models, like bart @xcite , can deliver performance on par with llms. this finding renews the importance of training smaller models via contrastive learning, as it offers the advantage of reducing computational costs. however, ats models are commonly trained using large-scale data that is randomly shuffled. this study aims to explore the potential for optimizing ats models through the strategic utilization of data selection and curriculum learning @xcite in conjunction with contrastive learning techniques. it is important to note that not all data is equally valuable, and the presence of redundant or even detrimental examples can impede the performance of ats systems @xcite . furthermore, the ordering in which data is presented during model training, as emphasized by curriculum learning principles, can have a significant impact on both the efficiency and effectiveness of the learning process. consequently, there is a fundamental necessity to develop a scoring system that can accurately assess the learning difficulty of individual samples and furtherly can be used for data selection and curriculum learning. in this study, we introduce a novel scoring system called the data selection curriculum (dsc) score, which serves as a measure of learning difficulty. the dsc score considers two essential factors: the difficulty of enhancing the ats model through a specific instance and the expected performance of the model on that particular instance. instances on which the current model already exhibits promising performance and stands to benefit significantly from slight adjustments are assigned as a lower learning difficulty. this is because these instances have the potential to contribute more to the final performance. as for the first factor, we propose a novel margin-aware list-wise ranking (mlr) loss, which introduces a dynamic margin instead of constant margin, which the state-of-theart brio @xcite uses in the list-wise ranking loss for contrastive learning. it can better reflect the fine-grained misalignment between sequential likelihood and performance. through the dsc score, we can effectively select and prioritize instances during the training process, focusing on those that offer the greatest opportunities for enhancing the ats model's overall performance. we empirically validated our method on the cnn/dailymail dataset. the results suggest that our method surpasses several robust baselines using fewer than 20% of instances. additionally, experimental findings highlight that our mlr loss is superior to the brio loss. ## conclusion this paper investigates the impact of data on text summarization. we propose a learning difficulty metric for data selection and curriculum learning. this score incorporates a margin-aware list-wise ranking (mlr) loss, further enhancing contrastive learning due to its dynamic margin. experimental results indicate that our data selection curriculum can outperform strong baselines using fewer than 20% instances from the cnn/dailymail dataset.