Examining Bias in Opinion Summarisation through the Perspective of Opinion Diversity (2023)

Abstract: Opinion summarisation is a task that aims to condense the information presented in the source documents while retaining the core message and opinions. A summary that only represents the majority opinions will leave the minority opinions unrepresented in the summary. In this paper, we use the stance towards a certain target as an opinion. We study bias in opinion summarisation from the perspective of opinion diversity, which measures whether the model generated summary can cover a diverse set of opinions. In addition, we examine opinion similarity, a measure of how closely related two opinions are in terms of their stance on a given topic, and its relationship with opinion diversity. Through the lense of stances towards a topic, we examine opinion diversity and similarity using three debatable topics under COVID-19. Experimental results on these topics revealed that a higher degree of similarity of opinions did not indicate good diversity or fairly cover the various opinions originally presented in the source documents. We found that BART and ChatGPT can better capture diverse opinions presented in the source documents.

## introduction the aim of opinion summarisation is to reduce the amount of information in the source text while maintaining the core message and opinions expressed therein. it can be in the form of a summary of a product review @xcite @xcite , online discourse using platforms like twitter, reddit @xcite or other types of text with opinions. there are two major types of models, extractive @xcite @xcite and abstractive @xcite @xcite . the extractive models extract the key information by selecting the most salient sentences in the source documents. whereas the abstractive summarisation models generate new phrases that reflect the key information in the source documents. applications for this activity include tracking consumer opinions, assessing political speeches, or internet conversations, among many others. a summarisation model's output will reflect any biases present in the data used for training the model. moreover, summarisation models are used in many applications where the fairness of the outputs is critical. for instance, opinion summarisation models can be used to summarise product reviews and social media posts. if these models produce biased summaries, they have the risk of being misused as a tool to influence and manipulate people's opinions. therefore, it is essential to look into the fairness of the models. earlier studies of bias in opinion summarisation have mainly evaluated biases in summarisation by comparing whether the information is selected from different social groups equally, using attributes including gender, dialect, and other societal properties @xcite @xcite . such representation is fair from the perspective of information input, leaving the other side of information consumption under studied. it is equally important to look at fairness from endusers' perspective. @xcite pointed out that fair representation from the information input perspective does not always imply fair representation from end-users' perspective. from the standpoint of end-users, it is more important for summaries to cover the various opinions presented in the source documents so that the various opinions in the source documents can be heard equally @xcite . in this study, we examine bias in opinion summarisation from end-users' perspective by comparing the output of different modern summarisation models. @xcite noted that many research work on bias in natural language processing lacks a precise description and definition of bias in their study, making it challenging for readers to understand. our working definitions of the key terms are as follows. according to the stance triangle theory (du bois, 2007), a stance is composed of three components: the object, the subject, and the attitude, which shape sociocultural value systems and opinion expression. while studies such as @xcite and @xcite utilise arguments with similarity and clustering techniques to capture similar opinions, our study takes a different approach based on the stance triangle theory. in our study, an opinion is a personal stance about a particular target, and a stance is a position or attitude that an individual takes about a given target. our definition of bias in opinion summarisation is when the summary focuses more on certain opinions than on the diversity of opinions presented in the source documents. note that it is generally agreed that diverse opinions should be taken into account regardless of their quantitative value in order to include more diverse opinions when using sentiment information @xcite . hence, our focus is on measuring the diversity of opinions rather than quantity. this is measured through opinion diversity which assesses opinion equivalence relative to the source documents. it measures the opinions in the source documents that the generated summary contains. in addition, we compare opinion similarity between the source and generated documents and further examine the relationship between opinion diversity and opinion similarity. opinion similarity is a measure of how closely related two opinions are in terms of their stance on a given topic. we use bert @xcite to compare the semantic closeness in the embedding space. we aim to understand whether models perform well in capturing overall opinions that are less biased by covering diverse opinions. we examine opinions on three covid-19 topics using stances on these topics. in our study, we aim to answer the following questions: ## methodology as previously stated, we capture opinion in our study by utilising stance and its target. when the generated summaries reflect opinions that diverge from those of the source texts, the summary is considered biased. this could take the form of concentrating on a narrow range of opinions or going beyond what was expressed in the source documents. it should be noted that the biased information in the source text is not the main focus of our study; rather, we are interested in how opinions are presented and whether the opinions in the generated summaries match those in the source documents. we formulate our problems in three steps. we first use a pre-trained language model to capture opinions from the input sequences. the opinion diversity is then calculated using the f1 score between the set of stances in the source tweets and the generated summary for each cluster (discussed below) under each topic. finally, we compare the cosine similarity between the source tweet cluster and the summary at the sentence level to measure opinion similarity using model representation. let @xmath0, c 1 , c 2 , ..., c l } be a set of clusters. these clusters are derived from the three main topics ("cdc", "stay at home orders" and "wearing a face mask"). for each c, we have a set of source tweets t , defined as @xmath1, t 1 , t 2 , ..., t n }. for each c, we have a set of generated summaries @xmath2, s 1 , s 2 , ..., s l }, where each summary s consists of a lists of sentences, defined as @xmath3, e 1 , e 2 , ..., e q }, where e refers to the textual content of each input sentence. ## conclusion in this work, we proposed a new way to examine bias in opinion sumamrisation from the perspective of presenting various opinions in the summary. we investigated various summarisation models for the covid-19 event under three topics, using stance towards a target as a representation of opinions. in addition, we also examined overall stance similarity using model representation. we found that bart @xcite and chatgpt are better at capturing diverse opinions when generating a summary. whereas hybrid-tfidf has the highest similarity across the three discussed topics for overall opinion similarity. based on the result, we found that higher opinion similarity does not indicate that the model presents diverse opinions. while both attributes are important for evaluating bias in opinion summarisation we suggest future studies look into introducing metrics that can evaluate summaries from both perspectives.