Echoes from Alexandria: A Large Resource for Multilingual Book Summarization (2023)

Abstract: In recent years, research in text summarization has mainly focused on the news domain, where texts are typically short and have strong layout features. The task of full-book summarization presents additional challenges which are hard to tackle with current resources, due to their limited size and availability in English only. To overcome these limitations, we present “Echoes from Alexandria”, or in shortened form, “Echoes”, a large resource for multilingual book summarization. Echoes featuresthree novel datasets: i) Echo-Wiki, for multilingual book summarization, ii) Echo-XSum, for extremely-compressive multilingual book summarization, and iii) Echo-FairySum, for extractive book summarization. To the best of our knowledge, Echoes – with its thousands of books and summaries – is the largest resource, and the first to be multilingual, featuring 5 languages and 25 language pairs. In addition to Echoes, we also introduce a new extractive-then-abstractive baseline, and, supported by our experimental results and manual analysis of the summaries generated, we argue that this baseline is more suitable for book summarization than purely-abstractive approaches. We release our resource and software at https://github.com/Babelscape/echoes-from-alexandria in the hope of fostering innovative research in multilingual booksummarization.

## introduction recent research in automatic text summarization -the task of shortening a text while preserving its meaning -has mainly focused on news stories. news texts are usually short documents; for example, 99.3% and 98.6% of the articles in xsum @xcite and cnn/dailymail @xcite , respectively, are shorter than 2048 tokens. additionally, news stories are characterized by strong layout features, such as the "lead bias", in which the first sentences usually contain the most relevant information for a summary. accordingly, the lead-3 baseline, which uses the first three sentences of a news item as its summary, performs competitively on news summarization benchmarks @xcite . although recent approaches have achieved high performance, it is still unclear how they behave on longer documents and whether they can generalize across domains and genres. for this reason, the research community has been shifting toward more challenging settings, which include interviews @xcite and scientific articles @xcite . one setting that has been attracting growing attention is full-book summarization @xcite , i.e., the task of producing the plot of a book from its full text. summarizing a book is hard not only because of its average text lengthcurrently not processable in a single forward pass even by architectures for long-form text processing @xcite -but also due to other critical aspects, such as the presence of dialogues, rich discourse structures, parallel and non-linear lines of plot, and long-distance dependencies between entities, among others. therefore, we deem book summarization a complex testbed to challenge current approaches and investigate their capabilities and limitations. although the first small-scale datasets for the task were introduced several years ago @xcite , the area has recently regained traction thanks to larger-scale resources, such as booksum @xcite and nar-rativeqa @xcite . however, despite this recent progress, current resources for book summarization are still, i) limited in size, making them difficult to use for proper training and evaluation, and ii) monolingual (usually english-only). to overcome these issues, we introduce "echoes from alexandria" (echoes), the largest resource to date for book summarization and the first one providing books and summaries in multiple languages. we use echoes to investigate how current summarization approaches perform on a large-scale multilingual summarization dataset, concluding that current purely-abstractive approaches still struggle in our setting. we additionally devise a new baseline, showing that the extractive-then-abstractive paradigm represents a promising direction for future research. the main contributions of our work are the following: we hope our work will foster research in multilingual long document understanding and summarization. we release echoes and our software for research purposes at https://github.com/ babelscape/echoes-from-alexandria . ## related work resources for summarization. research efforts to create summarization resources have steadily increased in numbers over recent years. for the news domain, xsum @xcite and cnn/dailymail @xcite are the defacto standard datasets for training and evaluating summarization systems. xsum comprises 226k news articles accompanied by a one-sentence abstractive summary. in cnn/dailymail, the authors retrieved 93k articles from cnn foot_0 and 220k articles from dailymail foot_1 newspapers. both publishers supplement their articles with a list of bullet points containing the main information of the news text. more recently, summarization resources have been shifting towards more challenging scenarios, i.e., where the documents of interest are longer and belong to different domains. notably, @xcite released two large-scale datasets of long and structured scientific papers obtained from arxiv foot_2 and pubmed foot_3 . in these datasets, paper abstracts are used as ground truth summaries. another relevant example is mediasum @xcite , a collection of interview transcriptions from national public radio (npr) 5 and cnn, where overview and topic descriptions are employed as summaries. in long-form text summarization research, a task that is attracting growing attention is book summarization. although this task was originally introduced several years ago by @xcite , who released the first small-scale evaluation resource, book summarization regained traction thanks to a few notable endeavors. the most important example is booksum @xcite , which provides a collection of resources for book summarization at three levels of granularity: paragraph, chapter, and full book. book texts are collected from project gutenberg, while summaries are obtained from the web archive. 6 booksum features 222 unique book titles with a total of 6,987 book chapters and 142,753 paragraphs. relatedly, narrativeqa @xcite is a collection of 1572 stories retrieved from project gutenberg (783 books and 789 movie scripts) associated with summaries from wikipedia. the annotators were required to generate questions and answers based on the summaries. even if narrativeqa is primarily intended for question answering, it can also be used for book summarization. due to their limited size, however, booksum (in the full-book setting) and narrativeqa can be more useful for evaluating models on the task rather than for training purposes. it is also worth noting that these resources are monolingual, i.e., english-only, limiting their usefulness for researchers seeking to evaluate multilingual summarization models. despite the great work carried out so far, we argue that there is still ample room to improve book summarization resources. approaches to book summarization. @xcite conducted experiments on fullbook summarization using a generate&rank strategy. this approach involves training a system to generate paragraph-level summaries, which are then sorted by perplexity and concatenated to form a full-book summary. more recently, @xcite proposed an approach where passages are recursively summarized and concatenated to form a full summary. however, generated summaries are affected by the errors accumulated from previous stages @xcite . recursively generating a summary is a paradigm that has also been used by other works for long-document summarization @xcite . another family of approaches is that of extractive-then-abstractive approaches. this family of approaches first extracts key sentences from the input document and then uses such sentences as input to an abstractive model, which is tasked with generating a summary that captures the main ideas and themes of the source. while it was successfully employed in previous works for short @xcite and long-form summarization @xcite , this paradigm has never been explored for summarizing books. in this paper, we aim to fill this gap by presenting a new, simple extractive-then-abstractive model and showing its effectiveness for book summarization. ## echoes echoes is the first collection of resources for book summarization in 5 languages: english, french, german, italian, and spanish. with echoes, we introduce the following three novel datasets: we provide an overview of the main differences between echoes and existing resources in ## experiments and results in recent years, two promising paradigms have emerged from previous work on longdocument summarization: recursive-abstractive and extractive-then-abstractive. in this section, we evaluate and analyze their effectiveness on echoes. ## analysis and discussion human evaluation. following common practice in the field of summarization, we set up a human evaluation process to assess the quality of the system-generated summaries. the annotation task, performed by an expert english speaker, consists of reading the source text and rating the summaries using a likert scale for consistency, relevance, fluency, and coherence, as outlined in @xcite . to make this experiment feasible in terms of time and resources, we focus our evaluation on fairy tales and short stories, which can be read by a human in a short time. interestingly, but not surprisingly @xcite , the results of our human evaluation experiment tell a story that is different from rouge, as shown in tables 4 and 5 . however, the evaluation still highlights the effectiveness of our extractive-then-abstractive model compared to the recursive-abstractive baseline. it is clear, however, that future work should focus in particular on improving the consistency and relevance of the summaries generated. challenges. echoes opens the door to several other analyses and experiments that were not possible with previous datasets. for example, we can leverage echo-fairysum to perform an analysis of the behavior of the extractor submodule of our extractive-then-abstractive approach, as we show in appendix d. in section 3.4, we examined the different book genres in echoes; longt5 model performances are detailed for each genre in figure 3 . we notice that epic poems are the hardest to summarize in this setting, while our model performs reasonably well on fairy tales. approach r-1 r-2 r-l # params. cross-lingual book summarization. additionally, echoes can be employed as a multilingual and cross-lingual summarization benchmark, thanks to its coverage of 5 languages and 25 language pairs. in particular, we argue that cross-lingual book summarization is a very interesting challenge, as it requires a model to compress vast amounts of information while transferring knowledge across languages. moreover, enabling cross-lingual book summarization is fundamental for all those cases in which we do not have the source text available in the language of interest, i.e., its translation may still be under copyright or may not exist at all. to move the first step in this direction, we propose a summarize-then-translate approach, a simple baseline for cross-lingual book summarization on echo-xsum. as the name implies, our approach works by employing a monolingual model to produce a summary in the same language as the source text, and then it translates the summary from the source language to the desired target language. we report the results of this baseline in ## conclusion in this paper, we introduced echoes, the first multilingual resource for book summarization and the largest among the english datasets. echoes features three novel datasets, namely, echo-wiki, echo-xsum, and echo-fairysum, which address several limitations of existing book summarization resources, such as booksum. indeed, previous datasets for full-text book summarization are, i) limited in size, and, ii) monolingual, i.e., usually covering english only. in addition, we leveraged echoes to bring to light the unsatisfying capabilities of current approaches to generalize to book summarization. finally, to mitigate this issue, we proposed a new extractivethen-abstractive baseline for book summarization, which outperforms its purely-abstractive counterpart on echo-wiki and echo-xsum, achieving results on the standard booksum test set that are comparable with the current state of the art while using a number of parameters that is only 0.1% compared to the best-performing method. we believe that echoes will foster future work on long-document summarization, especially in the multilingual and cross-lingual setting.