Frame-oriented Summarization of Argumentative Discussions (2023)

Abstract: Online discussions on controversial topics with many participants frequently include hundreds of arguments that cover different framings of the topic. But these arguments and frames are often spread across the various branches of the discussion tree structure. This makes it difficult for interested participants to follow the discussion in its entirety as well as to introduce new arguments. In this paper, we present a new rank-based approach to extractive summarization of online discussions focusing on argumentation frames that capture the different aspects of a discussion. Our approach includes three retrieval tasks to find arguments in a discussion that are (1) relevant to a frame of interest, (2) relevant to the topic under discussion, and (3) informative to the reader. Based on a joint ranking by these three criteria for a set of user-selected frames, our approach allows readers to quickly access an ongoing discussion. We evaluate our approach using a test set of 100 controversial Reddit ChangeMyView discussions, for which the relevance of a total of 1871 arguments was manually annotated.

## introduction web-based forums like reddit facilitate discussions on all kinds of topics. given the size and scope of some communities (known as "subreddits"), multiple individuals regularly participate in the discussions of timely controversial topics, such as on changemyview. 1 notably, the volume of arguments tends to grow substantially in a tree-like response structure wherein each branch forms a concurrent discussion thread. these threads develop in parallel as different perspectives are introduced by the participants. after a discussion subsides, the resulting collection of threads and their arguments often represents a comprehensive overview of the most pertinent perspectives (henceforth, referred to as frames) put forth by the participants. frames help shape one's understanding of the topic and deliberating one's own stance @xcite . however, in large discussions, prominent arguments as well as the various frames covered may be distributed in arbitrary (and often implicit) ways across the various threads. this makes it challenging for participants to easily identify and contribute arguments to the discussion. large online forums like reddit typically provide features that enable the reorganization of posts, for example, based on their popularity, time of creation, or in a question-answer format. a popularity-based ranking may seem beneficial, but @xcite discovered that an argument's popularity is not well correlated with its informativeness. furthermore, a popularity-based ranking does not cover the breadth of frames of a discussion, as we will show in this paper (section 4.1). in this paper, we cast discussion summarization as a ranking task with an emphasis on frame diversity, thereby introducing a new paradigm to discussion summarization in the form of multiple summaries per discussion (one per frame). previous research has focused on creating a single summary per discussion instead (section 2). as illustrated in figure 1 , we first assign arguments to one or more frames. next, we re-rank arguments in a frame according to their topic relevance. additionally, we also rank them based on their informativeness via post-processing. finally, we fuse these rankings to create the final ranking from which the top-k candidates can be used as an extractive summary of the discussion centered around a specific frame. in our experiments, we explore various state-ofthe-art methods to realize the three steps of our approach. our results suggest that: (1) utilizing retrieval models together with query variants is an effective method for frame assignment, reducing the reliance on large labeled datasets. here, our approach outperforms a state-of-the-art supervised baseline. (2) re-ranking arguments of a frame figure 1 : the proposed modular approach to frame-oriented discussion summarization: 1. frame assignment assigns arguments to frames ensuring frame relevance. 2. argument re-ranking ensures topic relevance of a frame's arguments (here, the morality frame is exemplified). 3. post-processing fuses the re-ranked arguments with an informativeness ranking. the top-k arguments are then taken as an extractive summary of the discussion. in summary, our contributions include: (1) a fully unsupervised frame assignment approach that assigns one or more frame labels to every argument within a discussion (section 3.1). (2) an argument retrieval approach that ranks frame-specific arguments based on their topic relevance and informativeness (section 3.2). (3) a dataset consisting of 1871 arguments sourced from 100 changemyview discussions, where each argument has been judged in terms of frame relevance, topic relevance, and informativeness (section 4) which forms the basis for an extensive comparative evaluation (section 5). 2 2 related work previous approaches to summarizing discussions can be broadly classified into two categories: discussion unit extraction and discussion unit grouping. we survey the literature on discussion summarization according to these two categories, followed by the literature on argument framing. ## ranking-based summarization this section describes our ranking-based approach to the extractive summarization of online discussions, centered around argumentation frames (figure 1 ). first we describe our novel unsupervised approach for frame assignment, followed by methods for re-ranking arguments of a frame based on their relevance to the discussion topic and informativeness. the top-k arguments from the joint ranking are taken as the frame's summary. ## data and experiments this section describes the dataset on which our approach was evaluated, the various retrieval models with their respective parameters, and the content features that we used in our experiments. also described is the supervised baseline for frame classification superframe that we implemented to assign multiple frames to each argument. ## evaluation given that our entire approach is based on retrieval models, we evaluated it manually via relevance judgments. we followed the evaluation style of trec @xcite as best practice. our evaluation was comprised of judging the frame relevance, the topic relevance, and the importance (in the discussion's context) of arguments retrieved by our models. following the trec protocol, we first created 50 evaluation topics, each comprising a post's title, the post itself, and a frame of interest (see supplementary material). to obtain a sufficiently large set of arguments to pool from, we then selected only those discussions for which all models assigned at least 20 arguments to each of the five most frequent frames identified in the comments: cultural identity, economic, quality of life, public opinion, and political (see ## conclusion and future work we introduced a novel ranking-based approach to frame-oriented (extractive) discussion summarization in web-based forums, aiming to enhance the accessibility and comprehension of large-scale online discussions for participants. our approach involves three key steps: frame assignment, argu-