Disordered- DABS : A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts (2024)

Abstract: Aspect-based summarization has seen significant advancements, especially in structured text. Yet, summarizing disordered, large-scale texts, like those found in social media and customer feedback, remains a significant challenge. Current research largely targets predefined aspects within structured texts, neglecting the complexities of dynamic and disordered environments. Addressing this gap, we introduce Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text. Developed by adapting existing datasets for cost-efficiency and scalability, our comprehensive experiments and detailed human evaluations reveal that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.

## introduction the exponential growth of digital content has significantly increased the importance of automated text summarization methods. these methods are crucial for distilling salient content from large volumes of text and efficiently addressing diverse information needs. query-focused summarization (qfs) @xcite and aspect-based summarization (abs) @xcite have emerged as prominent approaches for creating focused summaries based on specific aspects or queries. this advancement has led to the development of several specialized datasets and benchmarks. while qfs and abs are adept at generating summaries targeted at specific aspects, they typically depend on predefined aspects for querying or model fine-tuning. dynamic aspect-based summarization (dabs) was introduced to overcome this limitation. dabs identifies aspects dynamically from the content of source articles, transcending the restrictions of fixed aspect definitions. figure 1 : an example from reddit showcasing a multiuser discussion. each participant's input is color-coded to distinguish the varied aspects they discuss, emphasizing the range of perspectives and topics within the conversation. this visual differentiation serves to highlight the diversity inherent in such online discussions. despite the innovations in dabs, a significant challenge remains, particularly in real-world applications: the blending of sentences from varied sources within a text. this complication is frequently observed in scenarios like summarizing community opinions, analyzing user feedback, or collating medical records from multiple sources. such situations usually involve quick changes in topics or aspects, leading to a notable inconsistency in coherence among texts from different origins. figure 1 illustrates a discussion from reddit involving multiple users. the dialogue initially focuses on the appeal of new york city, transitions to its drawbacks for long-term residence, and eventually shifts to a conversation about sao paulo. this example underscores the dynamic nature of topics in online discussions, highlighting the complexity of summarizing such disordered texts foot_0 . this disorder not only makes the summarization process more complex but also requires strategies adept at seamlessly merging disparate aspectual data, adding another layer of complexity to the task. our paper addresses these challenges by presenting a novel and efficient method to modify existing summarization datasets, making them suitable for dynamic aspect-based summarization in disordered texts. we use this method to introduce the disordered-dabs dataset 2 , specifically curated for this demanding summarization task. comprising 255,286 aspect-oriented summaries, this dataset is segmented into training, validation, and testing partitions. its robustness is validated through comprehensive human evaluations. our examination of various baseline models highlights the complexity of this task, revealing that even advanced large language models like gpt-3.5 face significant challenges in dealing with the nuances of disordered-dabs. these observations underscore the intricacy inherent in this form of summarization, highlighting the necessity for more specialized datasets tailored to its unique requirements. ## related work qfs has long been a fundamental task in the field of text summarization, focusing on generating summaries tailored to specific queries @xcite . it targets extracting precise information relevant to a wide range of queries, reflecting the diverse informational needs in various contexts. key datasets in qfs, such as those developed by @xcite , have been instrumental in addressing these needs, employing queries like "steroid use among female athletes" to explore detailed aspects such as "trends", "side effects", and "consequences". abs emerged as an extension, initially focusing on customer review analysis @xcite . abs expands the scope of summarization to a broader range of domains, including news articles and encyclopedia content, as evidenced by works in abs tasks @xcite and dataset creation @xcite . this shift from queryspecific summaries in qfs to aspect-oriented summaries in abs demonstrates the evolution of summarization techniques in response to varying content and context requirements. further advancing this evolution is dabs, which diverges from traditional abs by not limiting aspects to predefined categories. innovations in dabs, such as anyaspect @xcite and entsumv2 @xcite , have focused on utilizing entities as aspects, with anyaspect 2 the data is available on github summarizing sentences around named entities and entsumv2 employing human-annotated, entityspecific summaries. other datasets like oa-sum @xcite and openasp @xcite explore conceptual labels or sub-topics as aspects, with oasum leveraging wikipedia subtitles and openasp adopting a multi-document dabs settings. a comparative overview of these datasets is presented in however, a notable limitation of existing datasets in both abs and dabs is their primary focus on well-organized and coherent texts, such as a news article. this focus is less relevant to platforms such as social media, where content often features multiple authors and rapid shifts in context. the inherent coherence in these texts could potentially overstate the performance of abs and dabs models, as they may depend on sentence transitions to discern changes in topics or aspects. this coherence within texts of the available datasets might inadvertently inflate the performance of abs and dabs models, as they might rely on transitions in contiguous sentences to infer changes in topics or aspects, whereas, in a dataset from multiple sources, this transition may detect the change of author and not aspect. therefore, we propose the disordered-dabs in which the aspects are dynamic and the texts are disordered. ## dynamic aspect-based summarization despite substantial efforts in qfs and abs, the definition of "aspect" remains ambiguous. it ranges from different topics @xcite @xcite to specific elements within a topic @xcite @xcite . some studies use concise phrases for aspects @xcite , while others allow longer sentences for complex information needs @xcite . given this variability in aspect definition, our disordered-dabs incorporates two distinct sub-datasets: one emphasizing aspect diversity across varied topics and another focusing on different aspects of a single event, providing a comprehensive benchmark for these divergent aspect interpretations. building on previous dabs research @xcite , our task is defined as generating multiple aspect-based summaries from a document d containing n topics or aspects. unlike traditional baselines that pair a document with ## datasets the disordered-dabs dataset comprises two subsets: d-cnndm and d-wikihow, each adapted from existing summarization datasets. instead of constructing these subsets from the ground up, we opted for an automatic conversion approach, resulting in large-scale samples that are well-suited for fine-tuning purposes. ## evaluation and results we assessed the performance of our baseline models, as detailed in section 5, employing both automatic (section 3.1) and human evaluation methods (section 3.2). due to the context length constraints of these baseline models, we truncated both the source articles and their corresponding aspect-specific summaries according to their average lengths and standard deviations. summaries that did not incorporate any sentences from their source articles were omitted from the reference set. while fine-tuning our model, we imposed a maximum limit on the number of aspects. however, this limit was not enforced during the evaluation phase. consequently, baseline models could face penalties for inaccuracies in predicting the correct number of aspects. we have provided additional information about the data preprocessing specific to each dataset in appendix b.2. ## ablation analysis our ablation study delves into three key areas: 1) exploring the few-shot learning capabilities of gpt-3.5; 2) assessing the impact of disordered text settings on model performance; and 3) exam- ## conclusion our disordered-dabs benchmark introduces the first dataset dedicated to dynamic aspect-based summarization in disordered texts, innovatively crafted from high-quality summaries of existing datasets. this approach sidesteps the conventional challenge of manual summary generation by transforming available datasets for summarization. the evaluation of leading-edge baseline models has exposed a notable deficiency in the current capacity to effectively tackle this complex task. our analysis shows that contemporary models struggle significantly with the nuances of dynamic aspectbased summarization in disorganized texts, underscoring the task's complexity and highlighting avenues for future research and enhancement.