m R eddit S um: A Multimodal Abstractive Summarization Dataset of R eddit Threads with Images (2023)

Abstract: The growing number of multimodal online discussions necessitates automatic summarization to save time and reduce content overload. However, existing summarization datasets are not suitable for this purpose, as they either do not cover discussions, multiple modalities, or both. To this end, we present mRedditSum, the first multimodal discussion summarization dataset. It consists of 3,033 discussion threads where a post solicits advice regarding an issue described with an image and text, and respective comments express diverse opinions. We annotate each thread with a human-written summary that captures both the essential information from the text, as well as the details available only in the image. Experiments show that popular summarization models—GPT-3.5, BART, and T5—consistently improve in performance when visual information is incorporated. We also introduce a novel method, cluster-based multi-stage summarization, that outperforms existing baselines and serves as a competitive baseline for future work.

## introduction with the increased popularity of online discussion forums like reddit, discussion threads that consist of a post and various numbers of comments, have quickly accumulated. it thus becomes overwhelming for users to sift through the threads to find the information they seek, which in turn has led to the development of automated means for text-only discussion summarization @xcite @xcite . however, discussion threads are often multimodal, containing visuals in addition to text. this added modality cannot be ignored, as it plays a key role in the respective discussions. for example, in post summary: we got this couch for our living room and i need help finding the perfect coffee table to go with it. ... the op asked for help with finding the right coffee table shape to match their brown leather sectional. comments: user 1: i would do a circular table. user 2: definitely round! there are a lot of sharp angles already ... user 3: there's way too much furniture in this space, the ottoman has to go ... user 4: i think you should look for a natural wood triangular shaped coffee table ... user 5: you should get a rug. ... comment summaries: c1,c2,c4: commenters suggest a differently shaped coffee table from the square one in the picture, round or triangular or hexagonal. c3: a commenter suggests eliminating the ottoman as it takes up too much space. c5: a commenter suggests adding a rug. ## results and analysis the results in we provide further analyses on the effect of input length and subreddit category on performance in § d. ## conclusion and future work online discussions are increasingly becoming multimodal, yet there are not sufficient resources for summarizing them. to this end, we presented mredditsum, the first multimodal discussion summarization dataset containing 3,033 discussion threads and images with human-written summaries. threads were carefully chosen so that the images play a key role in the respective threads, and summaries were written to capture this. experiments showed that summarization models making use of visual information consistently outperform those that do not. additionally, we introduced cluster-based multi-stage summarization, which accounted for the structure of discussion thread the op asked for ideas on how to style a black and white floral necklace. one commenter suggested wearing the necklace with a fitted shirt. another suggested wearing a button up collared white shirt with high waisted shorts and flats. the op wanted advice on how to style their necklace, which has a floral pattern on it. one commenter suggested wearing the necklace with a fitted shirt. another suggested wearing a button up collared white shirt with high waisted shorts and flats. the op asked for ideas on how to style their necklace, which is white with a dinosaur pattern on it. one commenter suggested wearing a funky skirt and fitted shirt with the necklace. another commenter recommended a button up collared white shirt with high waisted shorts and flats. one user shared a link to op's necklace. original post: one commenter recommended a round or oval coffee table. another commenter suggested brown throw pillows and blankets to match the rug. one user suggested op get a non-shedding dog. the op asked for advice on a coffee table and accent chair for their blue couch. most commenters suggested a wood coffee table with a walnut finish and a solid white marble top. one commenter recommended a round or oval coffee table. another commenter suggested getting a non-shedding dog. the op asked for help with picking out a coffee table and accent chair for their blue couch. most commenters suggested getting a warm wood coffee table. others suggested a brown leather armchair or cream colored accent chair. one commenter suggested getting throw pillows and blankets to match the rug. another commenter asked where the rug was from, and the op said it was from apt2b.