Indicative Summarization of Long Discussions (2023)

Abstract: Online forums encourage the exchange and discussion of different stances on many topics. Not only do they provide an opportunity to present one’s own arguments, but may also gather a broad cross-section of others’ arguments. However, the resulting long discussions are difficult to overview. This paper presents a novel unsupervised approach using large language models (LLMs) to generating indicative summaries for long discussions that basically serve as tables of contents. Our approach first clusters argument sentences, generates cluster labels as abstractive summaries, and classifies the generated cluster labels into argumentation frames resulting in a two-level summary. Based on an extensively optimized prompt engineering approach, we evaluate 19 LLMs for generative cluster labeling and frame classification. To evaluate the usefulness of our indicative summaries, we conduct a purpose-driven user study via a new visual interface called **Discussion Explorer**: It shows that our proposed indicative summaries serve as a convenient navigation tool to explore long discussions.

## introduction online discussion forums are a popular medium for discussing a wide range of topics. as the size of a community grows, so does the average length of the discussions held there, especially when current controversial topics are discussed. on change-myview (cmv), 2 for example, discussions often go into the hundreds of arguments covering many perspectives on the topics in question. initiating, participating in, or reading discussions generally has two goals: to learn more about others' views on a topic and/or to share one's own. to help their users navigate large volumes of arguments in long discussions, many forums offer basic features to sort them, for example, by time of creation or popularity. however, these alternative views may not capture the full range of perspectives *equal contribution. 1 code: https://github.com/webis-de/emnlp-23 2 https://www.reddit.com/r/changemyview/ exchanged, so it is still necessary to read most of them for a comprehensive overview. in this paper, we depart from previous approaches to summarizing long discussions by using indicative summaries instead of informative summaries. 3 figure 1 illustrates our three-step approach: first, the sentences of the arguments are clustered according to their latent subtopics. then, a large language model generates a concise abstractive summary for each cluster as its label. finally, the argument frame @xcite of each cluster label is predicted as a generalizable operationalization of perspectives on a discussion's topic. from this, a hierarchical summary is created in the style of a table of contents, where frames act as headings and cluster labels as subheadings. to our knowledge, indicative summaries of this type have not been explored before (see section 2). our four main contributions are: (1) a fully unsupervised approach to indicative summarization of long discussions (section 3). we develop robust prompts for generative cluster labeling and frame assignment based on extensive empirical evaluation and best practices (section 4). (2) a comprehensive evaluation of 19 state-of-the-art, prompt-based, large language models (llms) for both tasks, supported by quantitative and qualitative assessments (section 5). (3) a user study of the usefulness of indicative summaries for exploring long discussions (section 5). (4) discussion explorer, an interactive visual interface for exploring the indicative summaries generated by our approach and the corresponding discussions. 4 our results show that the gpt variants of openai (gpt3.5, chatgpt, and gpt4) outperform all other open source models at the time of writing. llama and t0 perform well, but are not competitive with the gpt models. regarding the usefulness of the summaries, users preferred our summaries to alternative views to explore long discussions with hundreds of arguments. figure 1 : left: illustration of our approach to generating indicative summaries for long discussions. the main steps are (1) unit clustering, (2) generative cluster labeling, and (3) multi-label frame assignment in order of relevance. right: conceptual and exemplary presentation of our indicative summary in table of contents style. frames act as headings and the corresponding cluster labels as subheadings. ## related work previous approaches to generating discussion summaries have mainly focused on generating extractive summaries, using two main strategies: extracting significant units (e.g., responses, paragraphs, or sentences), or grouping them into specific categories, which are then summarized. in this section, we review the relevant literature. ## indicative discussion summarization our indicative summarization approach takes the sentences of a discussion as input and generates a summary in the form of a table of contents, as shown in figure 1 . its three steps consist of clustering discussion sentences, cluster labeling, and frame assignment to cluster labels. ## prompt engineering using prompt-based llms for generative cluster labeling and frame assignment requires modelspecific prompt engineering as a preliminary step. we explored the 19 model variants listed in capacity & resources fairness & equality constitutionality & health & safety jurisprudence morality crime & punishment policy prescription & evaluation cultural identity political economic public opinion external regulation & quality of life reputation security & defense ## evaluation to evaluate our approach, we conducted automatic and manual evaluations focused on the cluster labeling quality and the frame assignment accuracy. we also evaluated the utility of our indicative summaries in a purpose-driven user study in which participants had the opportunity to explore long discussions and provide us with feedback. ## conclusion we have developed an unsupervised approach to generating indicative summaries of long discussions to facilitate their effective exploration and navigation. our summaries resemble tables of con-cmv: the "others have it worse" argument is terrible and should never be used in an actual conversation with a depressed person figure 4 : discussion explorer provides a concise overview of indicative summaries from various models for a given discussion. the summary is organized hierarchically: the argument frames act as heading, while the associated cluster labels act as subheadings, similar to a table of contents. cluster sizes are also indicated. clicking on a frame lists all argument sentences in a discussion that assigned to that frame, while clicking on a cluster label shows the associated argument sentences that discuss a subtopic in the context of the discussion (see figure 5 ).