On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization (2024)

Abstract: Text summarization and simplification are among the most widely used applications of AI. However, such models are often prone to hallucination, which can result from training models on unaligned data. One efficient approach to address this issue is Loss Truncation (Kang and Hashimoto, 2020), an approach to modify the standard log loss to adaptively remove noisy examples during training. However, we find that LT alone yields a considerable number of hallucinated entities on various datasets. We study the behavior of the underlying losses between factual and non-factual examples, to understand and refine the performance of LT. We demonstrate that LT’s performance is limited when the underlying assumption that noisy targets have higher NLL loss is not satisfied, and find that word-level NLL among entities provides better signal for distinguishing factuality. We then leverage this to propose a fine-grained NLL loss and fine-grained data cleaning strategies, and observe improvements in hallucination reduction across some datasets. Our work is available at https://github.com/yale-nlp/Simplification-Projects.

## introduction text summarization and simplification are among the most widely used nlp applications. however, such models are prone to generating hallucinations @xcite @xcite ; this may have harmful real-world impact and hinder the adoption of such models. to mitigate hallucinations, previous work studied aspects of training @xcite , decoding @xcite @xcite , or post-processing @xcite . in this paper, however, we focus on another crucial source of hallucination: the training data. when training data is misaligned (i.e. targets contain data unsupported by the input), models learn these patterns and hallucinate @xcite . this can stem from data collection errors, or scraping web-based data @xcite . while there have been efforts to identify and clean the misaligned examples @xcite @xcite @xcite , a limitation is that these methods require rewriting targets or training models to detect hallucination. to this end, other methods automatically detect and remove noisy examples. one widely adopted approach is loss truncation (lt) @xcite , which filters out noisy examples based on the observation that they have higher negative log-likelihood (nll) loss. this enables an easy-to-adapt and highly efficient training procedure: if nll loss is high (e.g. >80th quantile of observed losses), do not backpropagate the loss. previous work adopted this method to improve factuality in summarization @xcite @xcite @xcite . however, applying lt to five datasets, we find that models still hallucinate to a large degree. in this paper, we study the behavior of nll at a coarse (i.e. sentence) and fine-grained level (i.e. token) to understand and refine the performance of lt. at the time of writing, the paper is the first to analyze lt on text simplification datasets like cochrane, medeasi, and asset; moreover, it analyzes the performance of lt from the perspective of factuality, and delves deeper into training dynamics at the token and entity level. ultimately, the paper aims to contribute a better understanding of the underlying dynamics of lt, that can provide guidance for considerations when using this method in future work, in the context of reducing hallucination. we make the following contributions: (1) we demonstrate that lt's performance is hindered when the underlying assumption that noisy targets have higher nll loss is not satisfied, (2) we find that word-level nll among entities provides better signal for distinguishing factuality, and (3) we use this to propose a fine-grained nll loss which reduces entity-level hallucination on some datasets (-22% on cochrane, -7.2% on asset), and finegrained data cleaning strategies which achieve up to 26.8% hallucination reduction (cnn-dm), highlighting the potential of this approach. ## methodology loss truncation loss truncation @xcite @xcite ) is an efficient method for improving language generation by modifying the standard log loss to adaptively disregard examples with high loss, reducing potential hallucinations. it continuously updates a list of example-level nll losses, and zeros out losses above a set quantile. 1 formally, loss truncation defines the loss as datasets we study two popular conditional nlg tasks, summarization and simplification, where data sources can be noisy, and hallucinations remain an issue. we select five datasets representing a variety of domains: cochrane @xcite : medical abstracts from cochrane database of systematic reviews and expert-written summaries (4,459 pairs), medeasi @xcite : sentences from merck manuals @xcite models we use bart-large-xsum @xcite as the base model and fine-tune it on each dataset. our selection of bart is influenced by its prominent use in prior studies applying lt, particularly in research related to @xcite , @xcite , and xsum @xcite . this choice allows us to specifically examine the effects of lt (appendix c). we further experiment with flant5 as a stronger base model @xcite with lt for comparison, and find that it yields similar or better performance (appendix f). entity-based hallucination we identify a summary to have "hallucination" with respect to a source if the summary contains an unsupported entity (i.e. an entity which is not found in the reference). to determine this, we first use ner models to find the entities in both the source text and the summary foot_1 . note, we filter the entity types down to dates, numbers, proper nouns, and specific medical conditions (for medeasi and cochrane) to ensure that synonyms are not involved in this process. if any of the entities in the summary are not present in the source, we say the summary contains a hallucination. while this is a strong assumption, we observe that unsupported entities of this type are indeed hallucinations, hence this heuristic works well in practice. examples of outputs containing hallucinations have been provided in appendix i. ## conclusion we analyzed the effect of loss truncation (lt) on improving factuality in text summarization. we found that lt struggles to reduce entity-level hallucination when the underlying assumption that non-factual sentences have higher nll than factual sentences is not met. in training data. moreover, future work can explore contradictory information (i.e. targets with similar topics as input but different meaning). this requires the use of natural language inference (nli), which we qualitatively find is difficult in practice using off-the-shelf nli models @xcite or gpt @xcite , as we observe they are currently unable to detect contradictory or unsupported information in some cases. ultimately, reducing such hallucinations is key to improving the overall performance of summarization models.