Liputan6: A Large-scale I ndonesian Dataset for Text Summarization (2020)

Abstract: In this paper, we introduce a large-scale Indonesian summarization dataset. We harvest articles from Liputan6.com, an online news portal, and obtain 215,827 document–summary pairs. We leverage pre-trained language models to develop benchmark extractive and abstractive summarization methods over the dataset with multilingual and monolingual BERT-based models. We include a thorough error analysis by examining machine-generated summaries that have low ROUGE scores, and expose both issues with ROUGE itself, as well as with extractive and abstractive summarization models.

## introduction despite having the fourth largest speaker population in the world, with 200 million native speakers, 1 indonesian is under-represented in nlp. one reason is the scarcity of large datasets for different tasks, such as parsing, text classification, and summarization. in this paper, we attempt to bridge this gap by introducing a large-scale indonesian corpus for text summarization. neural models have driven remarkable progress in summarization in recent years, particularly for abstractive summarization. one of the first studies was @xcite , where the authors proposed an encoder-decoder model with attention to generate headlines for english gigaword documents @xcite . subsequent studies introduced pointer networks @xcite , summarization with content selection @xcite , graph-based attentional models @xcite , and deep reinforcement learning @xcite . more recently, we have seen the widespread adoption 1 https://www.visualcapitalist.com/ 100-most-spoken-languages/ . progress in summarization research has been driven by the availability of large-scale english datasets, including 320k cnn/daily mail document-summary pairs @xcite and 100k nyt articles @xcite which have been widely used in abstractive summarization research @xcite @xcite @xcite . news articles are a natural candidate for summarization datasets, as they tend to be well-structured and are available in large volumes. more recently, english summarization datasets in other flavours/domains have been developed, e.g. xsum has 226k documents with highly abstractive summaries @xcite , bigpatent is a summarization dataset for the legal domain @xcite , reddit tifu is sourced from social media @xcite , and @xcite proposed using scientific publications from arxiv and pubmed for abstract summarization. this paper introduces the first large-scale summarization dataset for indonesian, sourced from the liputan6.com online news portal over a 10year period. it covers various topics and events that happened primarily in indonesia, from october 2000 to october 2010. below, we present details of the dataset, propose benchmark extractive and abstractive summarization methods that leverage both multilingual and monolingual pre-trained bert models. we further conduct error analysis to better understand the limitations of current models over the dataset, as part of which we reveal not just modelling issues but also problems with rouge. to summarize, our contributions are: (1) we release a large-scale indonesian summarization corpus with over 200k documents, an order of mag- dokumen: liputan6.com, jakarta : gara-gara berusaha kabur saat diminta menunjukkan barang hasil curian, rosihan bin usman, tersangka pencurian tas wisatawan asing, baru-baru ini, tersungkur ditembak aparat kepolisian resor denpasar barat, bali. sebelumnya, rosihan ditangkap massa setelah mencuri tas nicholas dreyden, wisatawan asing asal inggris. tas yang berisi dokumen keimigrasian dan surat penting itu diambil rosihan setelah mengelabui korban. [7 kalimat dengan 78 kata setelahnya tidak ditampilkan] ringkasan: seorang pencuri tas wisatawan asing ditembak polisi. ia berusaha kabur saat diminta menunjukan hasil curian. karena itu, polisi menembaknya. ## data construction liputan6.com is an online indonesian news portal which has been running since august 2000, and provides news across a wide range of topics including politics, business, sport, technology, health, and entertainment. according to the alexa ranking of websites at the time of writing, foot_1 liputan6.com is ranked 9th in indonesia and 112th globally. the website produces daily articles along with a short description for its rss feed. the summary is encapsulated in the javascript variable window.kmklabs.article and the key shortdescription, while the article is in the main body of the associated html page. we harvest this data over a 10-year window -from october 2000 to october 2010 -to create a largescale summarization corpus, comprising 215,827 document-summary pairs. in terms of preprocessing, we remove formatting and html entities (e.g. &quot, and ), lowercase all words, and segment sentences based on simple punctuation heuristics. we provide example articles and summaries, with english translations for expository purposes (noting that translations are not part of the dataset), in figure 1 . as a preliminary analysis of the documentsummary pairs over the 10-year period, we binned the pairs into 5 chronologically-ordered groups containing 20% of the data each, and computed the proportion of novel n-grams (order 1 to 4) in the summary (relative to the source document). based on the results in figure 2 , we can see that the proportion of novel n-grams drops over time, implying that the summaries of more recent articles are less abstractive. for this reason, we decide to use the earlier articles (october 2000 to jan 2002) as the development and test documents, to create a more challenging dataset. this setup also means there is less topic overlap between training and development/test documents, allowing us to assess whether the summarization models are able to summarize unseen topics. for the training, development and test partitions, we use a splitting ratio of 90:5:5. in addition to this canonical partitioning of the data, we provide an "xtreme" variant (inspired by xsum; narayan et al. ( 2018 )) whereby we discard development and test document-summary pairs where the summary has fewer than 90% novel 4-grams (leaving the training data unchanged), creating a smaller, more challenging data configuration. summary statistics for the "canonical" and "xtreme" variants are given in we next present a comparison of liputan6 (canonical partitioning) and indosum (the current largest indonesian summarization dataset, as detailed in section 6; @xcite ) in to understand the abstractiveness of the summaries in the two datasets, in to create a ground truth for extractive summarization, we follow @xcite and @xcite in greedily selecting the subset of sentences in the article that maximizes the rouge score based on the reference summary. as a result, each sentence in the article has a binary label to indicate whether they should be included as part of an extractive summary. extractive summaries created this way will be referred to as "oracle", to denote the upper bound performance of an extractive summarization system. ## summarization models we follow @xcite in building extractive and abstractive summarization models using bert as an encoder to produce contextual representations for the word tokens. the architecture of both models is presented in figure 3 . we tokenize words with wordpiece, and append [cls] (prefix) and [sep] (suffix) tokens to each sentence. to further distinguish the sentences, we add even/odd segment embeddings (t a /t b ) based on the order of the sentence to the word embeddings. for instance, for a document with sentences [s 1 , s 2 , s 3 , s 4 ], the segment embeddings are [t a , t b , t a , t b ]. position embeddings (p ) are also used to denote the position of each token. the wordpiece, segment, and position embeddings are summed together and provided as input to bert. bert produces a series of contextual representations for the word tokens, which we feed into a (second) transformer encoder/decoder for the extractive/abstractive summarization model. we detail the architecture of these two models in sections 3.1 and 3.2. note that this second transformer is initialized with random parameters (i.e. it is not pre-trained). ## experiment and results we use three rouge @xcite ) f-1 scores as evaluation metrics: r1 (unigram overlap), r2 (bigram overlap), and rl (longest common subsequence overlap). in addition, we also provide bertscore (f-1), as has recently been used for machine translation evaluation @xcite . 10 we use the development set to select the best checkpoint during training, and report the evaluation scores for the canonical and xtreme test sets in for the pointer-generator models, we see little improvement when including the coverage mechanism (ptgen+cov vs. ptgen), implying that there is minimal repetition in the output of ptgen. we suspect this is due to the liputan6 summaries being relatively short (2 sentences with 30 words on average). a similar observation is reported by @xcite for xsum, where the summaries are similarly short (a single sentence with 23 words, on average). next we look at the bert models. overall they perform very well, with both the mbert and in-dobert models outperforming the lead-n baselines and ptgen models by a comfortable margin. indobert is better than mbert (approximately 1 rouge point better on average over most metrics), showing that a monolingually-trained bert is a more effective pre-trained model than the multilingual variant. the best performance is achieved by indobert's bertextabs. in the canonical test set, the improvement over lead-2 is +4.4 r1, +2.62 r2, +4.3 r3, and +3.4 bertscore points. in the xtreme test set, bertextabs suffers a substantial drop compared to the canonical test set (6-7 rouge and 2 bertscore points), although the performance gap between it and lead-2 is about the same. ## error analysis in this section, we analyze errors made by the extractive (bertext) and abstractive (bertextabs) models to better understand their behaviour. we use the mbert version of these models in our analysis. foot_10 ## related datasets previous studies on indonesian text summarization have largely been extractive and used small-scale datasets. @xcite developed an unsupervised summarization model over 3k news articles using heuristics such as sentence length, keyword frequency, and title features. in a similar vein, @xcite trained a naive bayes model to extract summary sentences in a 100-article dataset. @xcite and @xcite apply genetic algorithms to a summarization dataset with less than 200 articles. these studies do not use rouge for evaluation, and the datasets are not publicly available. @xcite released a dataset for chat summarization by manually annotating chat logs from whatsapp. 14 however, this dataset contains only 300 documents. the largest summarization data to date is indosum @xcite , which has approximately 19k news articles with manually-written summaries. based on our analysis, however, the summaries of indosum are highly extractive. beyond indonesian, there is only a handful of non-english summarization datasets that are of sufficient size to train modern deep learning summarization methods over, including: (1) lcsts @xcite , which contains 2 million chinese short texts constructed from the sina weibo microblogging website; and (2) es-news @xcite , which comprises 270k spanish news articles with summaries. lcsts documents are relatively short (less than 140 chinese characters), while es-news is not publicly available. our goal is to create a benchmark corpus for indonesian text summarization that is both large scale and publicly available. ## conclusion we release liputan6, a large-scale summarization corpus for indonesian. our dataset comes with two test sets: a canonical test set and an "xtreme" variant that is more abstractive. we present results for several benchmark summarization models, in part based on indobert, a new pre-trained bert model for indonesian. we further conducted extensive error analysis, as part of which we identified a number of issues with rouge-based evaluation for indonesian.