Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias (2024)

Abstract: Opinion summarisation aims to summarise the salient information and opinions presented in documents such as product reviews, discussion forums, and social media texts into short summaries that enable users to effectively understand the opinions therein.Generating biased summaries has the risk of potentially swaying public opinion. Previous studies focused on studying bias in opinion summarisation using extractive models, but limited research has paid attention to abstractive summarisation models. In this study, using political bias as a case study, we first establish a methodology to quantify bias in abstractive models, then trace it from the pre-trained models to the task of summarising social media opinions using different models and adaptation methods. We find that most models exhibit intrinsic bias. Using a social media text summarisation dataset and contrasting various adaptation methods, we find that tuning a smaller number of parameters is less biased compared to standard fine-tuning; however, the diversity of topics in training data used for fine-tuning is critical.

## introduction opinion summarisation aims to condense the opinions presented in the source documents into a summary so that readers can effectively comprehend the opinions in the source documents using input data such as product reviews @xcite @xcite , online discourse using platforms such as reddit @xcite , social media text from platforms such as x (formerly known as twitter) @xcite , or other types of text containing opinions such as debate @xcite . applications for this activity vary from tracking customer sentiments to summarising public opinions on political topics. a summarisation model's output will reflect any biases inherited from the training data. pre-trained language models (plms) were exposed to a variety of data that may contain societal bias, which inevitably perpetuates social stereotypes in models @xcite @xcite and can propagate to downstream tasks @xcite . understanding how models inherit societal bias from their training data and how these biases are amplified in downstream tasks is important for designing fair models. using opinionated ai language technologies has the risk of affecting how readers read and think @xcite . it is important to understand the biases in models to ensure they are not used as weapons to sway public opinion. prior studies have focused on studying bias in opinion summarisation using extractive models by comparing how contents are extracted and if they are representing opinions from different social groups in the source documents equally or proportionally @xcite @xcite . this method is inapplicable to abstractive summarisation models since models generate summaries by rephrasing, making it more challenging to capture and evaluate the opinions represented in the generated documents. in addition, fine-tuning abstractive summarisation models is required to build effective summarisation systems. how different adaptation methods introduce bias when summarising social media text has not been studied. in this study, we use the following definition of fairness: the generated summary must give exposure to the opinions of different social groups equally or proportionally w.r.t. the input documents; more information on this can be found in section 3. to address the aforementioned issues, this paper introduces a method using a classifier to identify opinions and a fairness metric to measure bias using abstractive summarisation models to summarise text with opinions, using political bias as the case study. we further investigate var-ious adaptation methods and the bias introduced, using our method for evaluating bias in abstractive summarisation. this can be used in conjunction with other performance evaluations to identify models that have good performance while keeping bias to the minimum. we find that different models and their variants express intrinsic bias, and fine-tuning these pretrained models to summarise social media text amplified the bias. in addition, we find that adaptation methods play an important role. we find that tuning a smaller number of parameters using methods such as adapter tuning @xcite produces less bias compared to standard fine-tuning that updates the entire model. however, the diversity of training data is critical when tuning a model by updating a smaller number of parameters. our study is the first of its kind to examine bias using abstractive models to summarise social media text with various adaptation methods. ## fairness in opinion summarisation given a collection of tweets, t , defined as @xmath0, t 1 , t 2 , ..., t n }. each tweet t i has a groundtruth label y @xmath1, where @xmath2, y 1 , y 2 , ..., y n } represents the label set (left or right-leaning). given a set of input tweets, a model would generate a summary s where each summary consists of a list of sentences defined as @xmath3, s 1 , s 2 , ..., s l }. each generated sentence would be classified as left or right-leaning using the trained classification model discussed in section 4.1. given the set of input tweets t , the proportion of left and right-leaning documents can be represented as p t l and p t r respectively. for the generated summary s, the proportion of left and right-leaning sentences can be represented as p sl and p sr respectively. for a model to be considered unbiasedly representing opinions in the provided source documents, it should generate a summary that reflects similar proportions of opinions in the input documents, i.e. p t @xmath4, or p t l /p t @xmath5. in our study, we focus on evaluating the model's output w.r.t. the input proportions only. we are considering two different input scenarios, namely equal input and skewed input. the intuition behind and the details of different input proportions in summarising social media text are below: we evaluate fairness in models based on the idea that the generated summary should give exposure to opinions representing different social groups w.r.t. the input only. more details on the metric we are adapting using these notions for evaluation can be found in section 4.2. note that our notion of fairness can be broadly applicable to the summarisation of different types of opinions in other genres, such as positive or negative opinions on specific issues. ## methodology we formulate our problems in three steps. we first use a classification model to determine whether the sentences in the generated summary represent opinions from left or right-leaning groups. then, using the metric to assess whether a model contains left or right-leaning bias and quantifying the severity by comparing the generated summaries w.r.t. the input documents. the overall process of measuring bias is visualised in figure 1 . as demonstrated by earlier research @xcite @xcite , tuning a smaller set of parameters can result in more robust performance than standard fine-tuning. however, there is a lack of research on how different methods affect the bias introduced into the model. finally, we examine how different approaches amplify bias as compared to standard fine-tuning. ## conclusion in this study, we examine evaluating fairness using abstractive summarisation models to summarise social media opinions, where fair models should generate summaries expose opinions from different social groups w.r.t. the provided input only. in the case of political discussion, we find that most plms present intrinsic bias by giving fair exposure to opinions from the left-leaning group but not the right-leaning group. we further investigate different adaptation methods and how they affect fairness. the result shows that models adapting to the task of summarising social media text increase bias in general; however, tuning a smaller number of parameters have relatively lower bias. we further investigate tuning models by individual topic, where we find the benefit of bias reduction diminishes when tuning a smaller number of parameters, which suggests the importance of diverse datasets being presented when tuning a smaller number of parameters. future work may explore the relationship between exposing models to diverse topics and bias. our study sheds light on understanding bias and the effect of different adaptation methods on bias in abstractive summarisation models, particularly when summarising text with opinions.