L o R a L ay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization (2023)

Abstract: Text Summarization is a popular task and an active area of research for the Natural Language Processing community. By definition, it requires to account for long input texts, a characteristic which poses computational challenges for neural models. Moreover, real-world documents come in a variety of complex, visually-rich, layouts. This information is of great relevance, whether to highlight salient content or to encode long-range interactions between textual passages. Yet, all publicly available summarization datasets only provide plain text content. To facilitate research on how to exploit visual/layout information to better capture long-range dependencies in summarization models, we present LoRaLay, a collection of datasets for long-range summarization with accompanying visual/layout information. We extend existing and popular English datasets (arXiv and PubMed) with layout information and propose four novel datasets – consistently built from scholar resources – covering French, Spanish, Portuguese, and Korean languages. Further, we propose new baselines merging layout-aware and long-range models – two orthogonal approaches – and obtain state-of-the-art results, showing the importance of combining both lines of research.

## introduction deep learning techniques have enabled remarkable progress in natural language processing (nlp) in recent years @xcite @xcite . however, the majority of models, benchmarks, and tasks have been designed for unimodal approaches, i.e. focusing exclusively on a single source of information, namely plain text. while it can be argued that for specific nlp tasks, such as textual entailment or machine translation, plain text is all that is needed, there exist several tasks for which disregarding the visual appearance of text is clearly sub-optimal: in * work partially done while at recital. a real-world context (business documentation, scientific articles, etc.), text does not naturally come as a sequence of characters, but is rather displayed in a bi-dimensional space containing rich visual information. the layout of e.g. this very paper provides valuable semantics to the reader: in which section are we right now? at the blink of an eye, this information is readily accessible via the salient section title (formatted differently and placed to highlight its role) preceding these words. just to emphasize this point, imagine having to scroll this content in plain text to access such information. in the last couple of years, the research community has shown a growing interest in addressing these limitations. several approaches have been proposed to deal with visually-rich documents and integrate layout information into language models, with direct applications to document understanding tasks. joint multi-modal pretraining @xcite @xcite has been key to reach state-of-the-art performance on several benchmarks @xcite @xcite . nonetheless, a remaining limitation is that these (transformer-based) approaches are not suitable for processing long documents, the quadratic complexity of self-attention constraining their use to short sequences. such models are hence unable to encode global context (e.g. long-range dependencies among text blocks). focusing on compressing the most relevant information from long texts to short summaries, the text summarization task naturally lends itself to benefit from such global context. notice that, in practice, the limitations linked to sequence length are also amplified by the lack of visual/layout information in the existing datasets. therefore, in this work, we aim at spurring further research on how to incorporate multimodal information to better capture long-range dependencies. our contributions can be summarized as follows: all the datasets are available on huggingface. foot_0 2 related work ## datasets construction inspired by the way the arxiv and pubmed datasets were built @xcite , we construct our corpus from research papers, with abstracts as ground-truth summaries. as the pdf format allows simultaneous access to textual, visual and layout information, we collect pdf files to construct our datasets, and provide their urls. 2 for each language, we select a repository that contains a high number of academic articles (in the order of hundreds of thousands) and provides easy access to abstracts. more precisely, we chose the following repositories: further, we provide enhanced versions of the arxiv and pubmed datasets, respectively denoted as arxiv-lay and pubmed-lay, for which layout information is provided. ## conclusion we have presented loralay, a set of large-scale datasets for long-range and layout-aware text summarization. loralay provides the research community with 4 novel multimodal corpora covering french, spanish, portuguese, and korean languages, built from scientific articles. furthermore, it includes additional layout and visual information for existing long-range summarization datasets (arxiv and pubmed). we provide adapted architectures merging layout-aware and long-range models, and show the importance of layout information in capturing long-range dependencies.