When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization (2023)

Abstract: Large language models (LLMs) are subject to sociocultural and other biases previously identified using intrinsic evaluations. However, when and how these intrinsic biases in pre-trained LM representations propagate to downstream, fine-tuned NLP tasks like summarization is not well understood. In this work, we investigate one type of bias—name-nationality bias—and trace it from the pre-training stage to a downstream summarization task across multiple summarization modeling choices. We show that these biases manifest themselves as hallucinations in summarization, leading to factually incorrect summaries. We also find that this propagation of biases is algorithm-dependent: more abstractive models allow biases to propagate more directly to downstream tasks as hallucinated facts. Building on these observations, we further analyze how changes to the adaptation method and fine-tuning data set affect name nationality biases and show that while they can reduce the overall rate of hallucinations, they do not change the types of biases that do appear.

## introduction fine-tuning pre-trained large language models (llms) has recently become the de facto approach to building effective text summarization systems @xcite @xcite . while these llms have led to substantial performance gains, prior studies have shown, through intrinsic evaluations, that llms often contain various linguistic and societal biases @xcite . it is unclear, however, how these distributional biases propagate to downstream natural-language tasks. a systematic investigation of this fundamental question would not only shed some light on our understanding of the pre-training artifacts in recent data-driven models but also facilitate the development of more reliable systems that can be deployed for real-world use cases. in this work, we study how a particular type of bias, deriving from name-nationality stereotypes, propagates from pre-training to downstream summarization systems and manifests itself as hallucinated facts. prior work has shown that text summarization systems suffer from generating information that is not supported by the original article @xcite @xcite . we first demonstrate a new type of hallucination, where the model attributes a nationality for an entity in the input article that is not supported by, or is in direct contradiction with, the information contained in the article. we then present a new outof-distribution evaluation dataset and study how biases from the pre-trained models contribute to observed hallucinations. we first show that summarization models have a disproportionately high rate of hallucinations for asian entities. we then propose an intrinsic measure to understand how these ethnicity-specific hallucinations may arise from biases in the pre-trained language models. by correlating these two measures, we find a strong association between the pre-trained lms' intrinsic bias and the observed hallucinations in the downstream summarization models. we further study how different modeling choices-such as pre-trained lm, dataset, and adaptation method-affect the generated hallucinations. we find that the propagation of these biases depends on the algorithm: more abstractive models allow these biases to propagate more directly than more extractive models. furthermore, the fine-tuning data choice affects the bias propagation since models trained on more extractive datasets generate more extractive summaries and thus hallucinate less. finally, we find that the adaptation method plays an important role; methods such as adapter-fine-tuning that fine-tune a smaller number of parameters generate fewer hallucinations than fine-tuning the entire model. surprisingly, while article: jung lee is a well-known french writer who was born in paris. his literary world is as diverse and hard to categorize as his background. he has lived in both urban and rural areas, deep in the mountains and in the seaside towns and has developed a wide range of interests from the tradition of confucian culture to advertising. ## name-nationality hallucinations in text summarization despite the improved performance of text summarization systems, recent work has shown that they still suffer from generating text that is not consistent with the source article (i.e., unfaithful; @xcite @xcite . one predominant type of faithfulness error is entity hallucination, where the model generates entities that are not supported by the source article @xcite . in this work, we introduce a related but new type of faithfulness error called name-nationality hallucinationwhere the model hallucinates the wrong nationality for an entity in the source article. ## the effect of fine-tuning dataset and adaptation method we explore how certain design choices for finetuning such as the fine-tuning dataset and the adaptation method, affect the propagation of bias for summarization. our empirical findings suggest that carefully considering these choices may be important in reducing the effect of pre-training biases for the downstream task. ## discussion in this work, we find that stronger intrinsic associations in pre-trained language models can result in more extrinsic hallucinations in the summarization task, showing this for one particular kind of hallucination, name-nationality hallucinations. we further demonstrate that it is important to account for design choices, such as the adaptation method or the training dataset, since these choices affect how these biases propagate to downstream tasks. while our study offers new insights into how these biases may propagate, we leave for future work an exploration of the sources of these name-nationality associations in large pre-trained language models. several such sources should be investigated. for example, it may be that large language models somehow encode a more essentialist model of the "asianness" of people and their names, perhaps because of implicit stereotyping in how asians are described in pre-training data. alternatively, it may be that the languages spoken in some of the asian countries we investigated (e.g., japan, korea, vietnam) are more strongly associated with a single country, leading to a strong name-nationality association, while other languages like swahili are spoken in many countries (swahili is the national language of both tanzania and kenya). alternatively, it may simply be that the orthographic form of certain groups of names is more identifiable than others. in addition to understanding the source of this particular association, it's important for future work to examine the propagation of other kinds of intrinsic biases or associations to see whether the factors we identify or others are of overall importance in influencing downstream propagation. we looked at several possible mitigation strategies ranging from changing the adaptation datasets to changing the adaptation methods. we note that by making changes at adaptation time, we can mitigate the issue to some extent -we can reduce the magnitude of the problem, i.e., the overall hallucination rates. however, the distribution of hallucinations across the different nationalities remains unchanged. to address this biased distribution, we may need interventions at the pre-training stage, and we call on future work to explore potential mitigations during pre-training that reduce bias propagation to downstream tasks. ## conclusion in this work, we introduced a new type of faithfulness error for text summarization, namely namenationality hallucinations. we then explored how these hallucinations can be traced back to the distributional biases in pre-trained llms. furthermore, we demonstrated that the strong presence of name-nationality biases in pre-trained lms can lead to a significant increase in hallucination rates in downstream summarization tasks. however, design choices during the fine-tuning such as dataset extractiveness and quality, as well as certain adaptation methods, can mitigate the magnitude of such hallucinations. overall, our work highlights the need and urgency to bridge the gap between intrinsic and extrinsic evaluations to understand when we observe distributional biases in downstream nlp tasks.