Analyzing the Dialect Diversity in Multi-document Summaries (2022)

Abstract: Social media posts provide a compelling, yet challenging source of data of diverse perspectives from many socially salient groups. Automatic text summarization algorithms make this data accessible at scale by compressing large collections of documents into short summaries that preserve salient information from the source text. In this work, we take a complementary approach to analyzing and improving the quality of summaries generated from social media data in terms of their ability to represent salient as well as diverse perspectives. We introduce a novel dataset, DivSumm, of dialect diverse tweets and human-written extractive and abstractive summaries. Then, we study the extent of dialect diversity reflected in human-written reference summaries as well as system-generated summaries. The results of our extensive experiments suggest that humans annotate fairly well-balanced dialect diverse summaries, and that cluster-based pre-processing approaches seem beneficial in improving the overall quality of the system-generated summaries without loss in diversity.

## introduction since the launch of twitter, its short, informal, creative, albeit noisy, social media posts called tweets, have been collected, labeled, and studied in numerous natural language processing (nlp) tasks, ranging from identifying topics and places of rising interest to sentiment analysis, and more. these easily accessible user-generated tweets, produced contemporaneously as world and private events unfold, provide insights into the perspective of diverse social groups but are too manifold for humans to interpret at scale. in response, automatic text summarization algorithms aim to compress long pieces of text into short, fluent, and consistent summaries while preserving the most salient information from the source text @xcite @xcite . like other nlp models, summarization algorithms run the risk of perpetuating unintentional social biases against diverse groups (e.g., race or gender) and promoting structures and practices that systematically limit some groups' access to resources and decision-making power @xcite . this is because the collections of online texts such as news or wikipedia articles, typically used for developing nlp algorithms, reflect the interests, language patterns, and structured writing style of their author demographics, which differ from those of other communities. algorithms trained on such datasets may produce synopses in which diverse perspectives are systematically excluded. this means that groups who manage to overcome existing barriers to participation, for instance, via social media posts, and who speak up and offer their perspectives may still not be heard. fairness definitions for nlp models are generally based on the notion of equal treatmentan algorithm is considered fair when it performs the same for mainstream and underrepresented groups (i.e., group fairness) or delivers the same conclusions about an individual, regardless of the group they belong to (i.e., individual fairness) @xcite . recent works have demonstrated the disparate performance of tools on sensitive subpopulations in domains @xcite . in this work, we consider the notion of group fairness and interpret it in terms of representation distribution of some socially salient attribute (dialect) in the summary. our goal is to investigate the ability of existing models of summarization to reflect the diversity of input text in the generated summaries, and propose a simple yet effective approach for improving the group-level diversity of summaries generated from noisy social media tweets, for which very little prior work exists to date @xcite dataset domain attribute (#groups) #topics #summ/topic #docs/input #sent/summ ext? abs? metoo @xcite tweets gender (2) 1 3 488 -y n claritin @xcite tweets gender (2) 1 3 4037 100 y n us-election @xcite • we study the group diversity of reference summaries and investigate two simple yet effective approaches for applying diversity interventions at the pre-processing stage of the summarization process; ## related work in this section we discuss two relevant areas of prior work: multi-document summarization for social media data, and an overview of existing summarization datasets. ## problem description for the task of multi-document summarization of social media data, the input typically consists of dozens to hundreds of documents (e.g., tweets) about the same topic that are all considered in generating a single summary. formally, the input is a set of documents further split into n sentences, @xmath0, ..., d n }. given the sentences in d, the goal of a multi-document summarization model is to generate a summary @xmath1, . . . , s k } consisting of k sentences, where @xmath2. there are two types of summarization models, extractive models where s ⊂ d, and abstractive models which extract and rewrite salient pieces of text. in multi-user settings where users belong to multiple social groups, each sentence d i may be additionally accompanied by social attribute g j , g @xmath3, where g is a set of attributes of a demographic group such as gender, race, or dialect. to adequately accommodate the variety of perspectives expressed by multiple diverse groups, the goal of a diversity-preserving multi-document summarization algorithm then is to generate a summary s with the goal of not only optimizing textual quality but also satisfying some fairness constraint such as group fairness, which compare quantities at group level @xcite . in the context of summarization, this problem formulation naturally lends itself well to extractive models where sentences from the final summaries can be traced back to source documents and their user group labels more so than to abstractive models where text usually gets rewritten making group label attribution challenging to ascertain. one simple approach of computing group representation distribution in extractive summaries, which we denote as r(s), is by calculating the number of tweets belonging to each user group in the summary. assuming m distinct and disjoint groups, r(s) = { |s 1 | k , . . . , |sm| k } where |s j | is the number of sentences within the summary s from group j, and k is the total number of sentences in the summary. under the notion of equal representation, where the representation of all groups should be equal, regardless of their distribution in the input data, one can compute an aggregate diversity score. for instance, we can compute the representation gap, where a lower gap score would imply a more balanced distribution. several metrics can be used to estimate the level of dispersion foot_1 , and we choose the range, r rg (s) = max{r(s)} -min{r(s)}. in other words, for well-balanced group distributions, a smaller r rg (s) score indicates a more diverse summary. as an example, if a summary of 5 lines contains 3 sentences from group a, and a sentence each from group b and group c, then r(s) = {0.6, 0.2, 0.2}, and r rg (s) = 0.4. ## dialect diverse summarization dataset (divsumm) in order to study the diversity-preserving capabilities of summarization algorithms, we need a suitable dataset, and although numerous summarization datasets have become available in recent years, only a handful of them contain explicit diverse social group information (see ## modeling diversity in system summaries in this section, we explore the diversity-preserving qualities of recently proposed summarization algorithms using three approaches described below and visualized in figure 2 . vanilla: this standard baseline approach of summarizing uses a single aggregated set of randomized documents from all the dialect groups, e.g., a total of 90 tweets, as the input to the summarization model without any pre-processing. cluster-heuristic (cluster-h) : similar to the pre-processing approach of @xcite , this method first heuristically partitions the set of input documents into group-based subsets (@xmath4... ∪ d m }, each d j containing a set of documents from group j) before passing them to the summarization model to generate separate group summaries (@xmath5, ..., s m }) -one for each of the three group-specific tweets. however, instead of concatenating these summaries to generate the final summary, we shuffle and combine these m group-level summaries into a single document and pass that again to the summarization model to generate a new, combined summary. in doing so, we seek to first preserve group-level salient information before aggregating the most informative units from such individual summaries into a unified summary. cluster-automatic (cluster-a): conversely, in a more pragmatic scenario, the sensitive group attribute may not be reliably observable, or inferring it is not possible due to some reason, including ethical reason. in such a case, we also investigate an attribute-agnostic approach based on automatic clustering, as follows: ## experiments the three approaches are studied in the context of recently proposed summarization models, both extractive and abstractive, and the quality of the summaries is evaluated using reference-based metrics, reference-less metrics, and representation metric. ## results and discussion ## conclusions in this work, we investigate whether, and to what extent, do system-generated summaries reflect the diversity of socially salient groups present in the input data. to answer this question, we first develop a novel summarization dataset, divsumm, by obtaining human-written reference summaries, of both extractive and abstractive sort, for dialect diverse tweets. in analyzing the human-written reference summaries, we were encouraged to note that on average humans generated reasonably well-balanced dialect diverse summaries. this was followed by an extensive evaluation exploring the diversity reflected in system summaries by experimenting with three approaches as applied to six summarization algorithms, and evaluated using multiple metrics of summary quality and representation. future avenues of work include expanding our dataset to consider other diverse social attributes and improving the summarization models along dimensions of both quality as well as representation.