TLDR 9+: A Large Scale Resource for Extreme Summarization of Social Media Posts (2021)

Abstract: Recent models in developing summarization systems consist of millions of parameters and the model performance is highly dependent on the abundance of training data. While most existing summarization corpora contain data in the order of thousands to one million, generation of large-scale summarization datasets in order of couple of millions is yet to be explored. Practically, more data is better at generalizing the training patterns to unseen data. In this paper, we introduce TLDR9+ –a large-scale summarization dataset– containing over 9 million training instances extracted from Reddit discussion forum ([HTTP]). This dataset is specifically gathered to perform extreme summarization (i.e., generating one-sentence summary in high compression and abstraction) and is more than twice larger than the previously proposed dataset. We go one step further and with the help of human annotations, we distill a more fine-grained dataset by sampling High-Quality instances from TLDR9+ and call it TLDRHQ dataset. We further pinpoint different state-of-the-art summarization models on our proposed datasets.

## introduction text summarization is defined as generating a concise sequence of text as summary, given relatively a longer document as source. a high-quality summary conveys the most important points of its associated source. the task is generally performed in two ways: 1) extractive in which salient sentences are identified and concatenated to form the final summary @xcite @xcite @xcite ; and 2) abstractive that produces a paraphrasing of the main contents of the given text. @xcite @xcite ## related work over the past few years, summarization community has witnessed variety of summarization datasets in different domains @xcite @xcite @xcite . while these collections have provided a fair basis to perform different neural text summarization models, the necessity of introducing large-scale collections, in magnitude of over 4 millions, has not been much explored. among the first attempts on this track, @xcite gathered the english gigaword corpus @xcite which contains around 4 millions article-headline pairs for the task of news headline generation. researchers have noted that lead bias is the common phenomenon in most news datasets, where early parts of the article generally include the most important information @xcite @xcite . to alleviate the lead bias for training summarization models, there have been recent efforts to propose summarization datasets, where the lead bias phenomenon is mitigated and summaries are sampled from diverse source regions. amongst those, @xcite proposed bigpatent, consisting 1.3 million patent documents, collected from google patents public datasets, with human-0.0 0.2 0.4 0.6 0.8 1.0 percentage 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 0.00 0.05 0.10 0.15 0.20 0.25 percentage 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 ( written abstractive summaries. @xcite proposed reddit tifu in which the abstractive gold summaries are sampled from diverse regions of the source document, rather than lead regions. our proposed datasets are more suited for the task of extreme summarization @xcite , where the task is to create a short one-sentence summary. to this end, @xcite proposed xsum dataset which is a real-word dataset compiling online articles from the british broadcasting corportation (bbc). tldr generation task is also a new form of extreme summarization. @xcite collected reddit-tifu dataset, consisting of 120k posts from the online discussions from reddit. recent efforts have mined around 4 millions reddit posts along with their tldr summaries @xcite which resulted in webis-tldr-17 dataset. while our work is similar to theirs, our collected dataset is more than twice larger than the one previously proposed. ## experimental results comparing abstractive models with bert-sumext, we observe relatively large performance gap. this might be due to the fact that tldrs in both tldr9+ and tldrhq datasets are rather abstractive than extractive as also shown in section 3.4. yet with the existence of such a huge gap, the oracle-ext (i.e., upper bound of an extractive summarizer) scores prove that more developed extractive summarizers can perform outof-the-box and mitigate this gap. the performance gap on tldr9+ brings various challenges to develop summarization models that better fit on the larger dataset that include noisy data @xcite . this noise might be handled via methods such as noise-aware training models @xcite , while enabling the models to benefit from the large-scale tldr9+ dataset. we leave this part for future work. it has to be mentioned that automatic evaluation of summarization continues to be an issue and while this dataset does not solve that, instead can be used with any evaluation metric as they evolve. ## analysis to gain insights into the qualities of summarization model, we analyze the outputs generated by the models. the diagrams demonstrating n-gram abstractiveness and percentage of novel n-grams, generated by bart and bertsumabs, are plotted in figure 6 . as observed, bart model appears to have a similar trend to the ground truth tldrs. on the other hand, bertsumabs model has increasing n-gram abstractiveness, and novel n-gram percentage with increasing n. it is also interesting that after 6-gram, bertsumext model reaches a plateau when generating novel n-grams, but we a drop after 3-grams for bart and the ground truth tldrs. this shows that from 1-gram to 3-gram, there are increasing number of novel words appeared in the ground-truth and bart, but after that, they both tend to copy n-grams rather than generating those. to understand the limitation and qualities of current state-of-the-art summarization models, we conduct a qualitative analysis on several samples from tldrhq dataset, of which one is shown in figure 7 . analyzing this sample, we observe that bart generated a better summary in terms of faithfulness to the ground truth tldr. on the other hand, while bertsumabs could identify the important region of the source document, it has produced a longer tldr with additional information that is present in the source, but not in the ground truth summary. bertsumext model could have identified a source sentence which is partly in connection with the ground truth tldr, but it leaves out the most important sentence as the oracle to be extracted. considering the upper performance of extractive summarizers (i.e., oracle-ext score in let me start this off by saying i'm not sure if this is the right spot to ask , and matching is not really my forte. i have my nostril pierced , as well as my septum. i got them done earlier this year and i've been playing around with different jewelry .all my jewelry has been white gold / silver... until now (edit -i originally had a silver hoop in my nostril and it was constantly irritated so i read up on it and found that silver is not good for piercings so i only use 14k white gold currently ). i purchased a 14k solid rose gold nose hoop (20g). i'm curious if it would look weird wearing a rose gold nose hoop with a white gold seamless septum ring (16g) ?? or any white gold septum jewelry? i don't want to look like a fool who can't match her facial jewelry! bertsumabs. would it look weird wearing a rose gold nose hoop with a white gold seamless septum ring (16g) ?? or any white gold septum jewelry ? i don't want to look like a fool who can't match her facial jewelry . ## conclusion in this paper, we proposed two large-scale summarization datasets called tldr9+, and tldrhq. the tldr9+ dataset contains over 9 millions reddit post-tldr instances. to distill a more finegrained dataset out of tldr9+, we sample highquality instances with the help of human annotations to construct tldrhq. our analyses over tldr9+ and tldrhq datasets show its usability for performing both extractive and abstractive summarization tasks. we further establish extractive and abstractive baseline results using state-of-theart summarization models on both datasets. we hope our datasets can pave the path for future studies on this direction.