Training Dynamics for Text Summarization Models (2022)

Abstract: Pre-trained language models (e.g. BART) have shown impressive results when fine-tuned on large summarization datasets. However, little is understood about this fine-tuning process, including what knowledge is retained from pre-training time or how content selection and generation strategies are learnt across iterations. In this work, we analyze the training dynamics for generation models, focusing on summarization. Across different datasets (CNN/DM, XSum, MediaSum) and summary properties, such as abstractiveness and hallucination, we study what the model learns at different stages of its fine-tuning process. We find that a propensity to copy the input is learned early in the training process consistently across all datasets studied. On the other hand, factual errors, such as hallucination of unsupported facts, are learnt in the later stages, though this behavior is more varied across domains. Based on these observations, we explore complementary approaches for modifying training: first, disregarding high-loss tokens that are challenging to learn and second, disregarding low-loss tokens that are learnt very quickly in the latter stages of the training process. We show that these simple training modifications allow us to configure our model to achieve different goals, such as improving factuality or improving abstractiveness.

## introduction transformer-based pre-training @xcite has led to substantial improvements in the performance of abstractive summarization models. this pre-training and fine-tuning paradigm has been widely studied with respect to what training datasets, model sizes and other hyperparameters are needed to optimize task-specific evaluation metrics, such as perplexity or rouge for text generation. however, abstractive summarization is a complex task involving several components such as content selection and rewriting that are performed implicitly by end-to-end models such as bart @xcite or pegasus @xcite . currently, we have little insight into this aspect of the fine-tuning process, namely what "skill" or behavior is learnt at which stage of the training process. recent work @xcite has studied training dynamics for sequence classification tasks such as nli and fact verification, demonstrating how these can be leveraged to mitigate dataset biases. however, text generation is a substantially different task from classification, due to the sequential nature of predictions and the mismatch between teacher-forced training and inference time. the nature of the training process and potential interventions to modify what gets learned are poorly understood. in this paper, we make the first attempt at understanding the fine-tuning process of large pre-trained language models for summarization. we study two essential components of abstractive summarization models, abstractiveness and factual consistency, and investigate when each of these is learned during fine-tuning. experiments are conducted on three different summarization datasets: xsum @xcite , cnndm @xcite and mediasum @xcite to study these properties across a range of datasets. our findings are threefold: first, we find that easy-to-learn skills such as copy behavior are acquired very early in the fine-tuning process. in fact, for datasets that have a high fraction of extractive summaries, the summarization models tend to overfit to these easier examples, effectively ignoring harder examples in the dataset. next, we investigate how factual correctness of summaries evolves with the fine-tuning process, juxtaposing it against other factors such as abstractiveness and dataset quality. in particular, we find that while non-factuality and abstractiveness are roughly proportional to each other, longer training on noisy datasets can significantly hurt factuality. finally, we show that insights from these training dynamics can be leveraged to optimize along target summarization goals like factuality or abstractiveness. we extend prior work on loss truncation @xcite , using token sub-sampling to dynamically modify the loss computation during training to alter the learnt behavior of summarization models. in particular, we show that we can substantially improve the factuality of summarization models trained on noisy datasets (e.g. xsum) by downweighting high-loss tokens while preserving the high level of abstractiveness. conversely, downweighting low-loss tokens under the same framework allows us to significantly improve the abstractiveness of generated summaries compared to the baseline models for relatively extractive datasets (e.g. cnndm and mediasum). ## improving training in section 2.2, we saw evidence that summarization models tend to overfit on easier examples, i.e., the more extractive examples that are learnt earlier in the training process. on the other hand, section 2.3 showed that for noisy datasets such as xsum, correct information is assigned high probability scores earlier in the training process whereas hallucinated tokens are learnt with lower confidence. in this section, we operationalize these observations to improve the abstractiveness of generated summaries for cnndm and mediasum, and factuality of generated summaries for xsum. ## related work abstractive summarization prior work in abstractive summarization has evaluated summaries along various parameters such as grammaticality and informativeness @xcite , agreement with reference @xcite and content selection @xcite . recently, approaches to evaluate the factual correctness of abstractive summarization have been proposed @xcite @xcite . however, all these have focused on only evaluating the final generated summary. finally, both improving abstractiveness @xcite and factuality @xcite have been explored in recent work; in this paper, we explore if simpler techniques inspired by the training dynamics can achieve similar goals. evaluating across learning time steps recent work has studied learning dynamics of lstm models @xcite and pre-trained transformer models @xcite across aspects such as linguistic knowledge, topicalization, reasoning, etc. another line of work has explored this in the context of mitigating known dataset biases @xcite for tasks such as paraphrase identification, entailment, etc. @xcite . broadly, these have proposed techniques such as example reweighting @xcite , ensembling @xcite or loss truncation @xcite to modify the model's learnt behavior. ## conclusion in this paper, we study when different summarization skills are learnt during training. we show that copy behavior is learnt early while hallucination is learnt in the later stages. based on these observations, we propose a simple token-level loss truncation strategy that can be used to achieve notable improvements in abstractiveness for cnndm and mediasum, and factuality in xsum. input article: naypyidaw, myanmar (cnn) twenty-one people are dead and 21 missing after a ferry capsized in the southeast asia nation of myanmar. myanmar's ministry of information said in a statement that the ship capsized friday night as it sailed, in bad weather conditions, around the city of sittwe. that's when a large wave crashed into the ferry, causing it capsize near myaybone and myaukkyine islands. authorities have managed to rescue at least 167 people, according to the information ministry for myanmar, which is also known as burma. pictures from the government showed rescue workers helping people off a boat onto the land. sittwe is the capital of rakhine state and sits on the bay of bengal, about 55 miles (90 kilometers) from the bangladesh border. this weekend's weather forecast for the city calls for some clouds giving way to clear skies, with high daytime temperatures expected to be in the 30s celsius (80s to 90s fahrenheit). fatal ferry disasters are nothing new to the region. last month, at least 68 people died when a packed double-decker ferry sank while on the padma river north of neighboring bangladesh's capital, dhaka, officials said. a cargo vessel hit the ferry, causing it to overturn and trapping passengers on its lower deck. forty-five people died in an accident on the same river in august. in may 2013, several boats carrying as many as 150 people were thought to have capsized near myanmar's western coast ahead of a storm approaching the area. those boats were carrying rohingya, members of myanmar's long-suffering muslim minority, thailand-based u.n. official kirsten mildren said at the time. journalist manny muang reported from myanmar, and cnn's greg botelho wrote this story from atlanta. reference: 167 people have been rescued, myanmar's government says. the ferry capsized after being hit by a large wave in bad weather conditions. baseline bart: the ship capsized friday night as it sailed in bad weather conditions. authorities have managed to rescue at least 167 people, according to the information ministry. fatal ferry disasters are nothing new to the region. bart +abstractive: at least 21 dead after ferry capsizes near sittwe. at least 167 people have been rescued from boat. fatal ferry accidents are nothing new to region. input article: (cnn) nato jets scrambled to intercept russian military aircraft as they neared latvian airspace, officials said on wednesday. estonian radar detected the aircraft over the baltic sea on tuesday night, nato said. other than the lead aircraft, nato said, none of the other russian military aircraft was on a flight plan. nato sent jets to identify the planes and later reported that the military aircraft flew on into russian airspace. nato didn't say how many russian aircraft were involved. the flights come as russia's northern fleet has been placed on full combat alert for military exercises involving nearly 40,000 troops and 50 warships. the exercises have rattled nerves in nearby nato states, including latvia, where u.s. troops and equipment recently arrived for nato training, and where fears are growing about russian president vladimir putin's next move. at the same time on wednesday, putin joined a celebration in moscow's red square, where russians celebrated the one-year anniversary of the annexation of crimea. nato has condemned the annexation as an illegal territory grab and is boosting its troop presence in the region in what officials say is an effort to discourage putin from encroaching into other countries. putin describes the annexation as a " reunification, " saying that crimea's residents overwhelmingly voted to be part of russia. cnn's don melvin and catherine e. shoichet contributed to this report. reference: russian military aircraft are intercepted by nato jets. nato says the military aircraft weren't on a flight plan. russia is conducting military exercises. baseline bart: nato jets scrambled to intercept russian military aircraft as they neared latvian airspace. russian northern fleet has been placed on full combat alert for military exercises. bart +abstractive: russian military planes flew into latvian airspace, according to nato. flights are part of russia's preparations for major military exercises involving 40,000 troops, 50 warships.