EUR -Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain (2022)

Abstract: Existing summarization datasets come with two main drawbacks: (1) They tend to focus on overly exposed domains, such as news articles or wiki-like texts, and (2) are primarily monolingual, with few multilingual datasets.In this work, we propose a novel dataset, called EUR-Lex-Sum, based on manually curated document summaries of legal acts from the European Union law platform (EUR-Lex). Documents and their respective summaries exist as cross-lingual paragraph-aligned data in several of the 24 official European languages, enabling access to various cross-lingual and lower-resourced summarization setups. We obtain up to 1,500 document/summary pairs per language, including a subset of 375 cross-lingually aligned legal acts with texts available in *all* 24 languages. In this work, the data acquisition process is detailed and key characteristics of the resource are compared to existing summarization resources. In particular, we illustrate challenging sub-problems and open questions on the dataset that could help the facilitation of future research in the direction of domain-specific cross-lingual summarization.Limited by the extreme length and language diversity of samples, we further conduct experiments with suitable extractive monolingual and cross-lingual baselines for future work. Code for the extraction as well as access to our data and baselines is available online at: [https://github.com/achouhan93/eur-lex-sum](https://github.com/achouhan93/eur-lex-sum).

## introduction despite a long history in the field of text summarization @xcite , current systems in the area are still mainly targeted towards a few select domains. this stems in part from the homogeneity of existing summarization datasets and extraction processes: frequently, these are either collected * these authors contributed equally to this work. from news articles @xcite @xcite @xcite or wikistyle knowledge bases @xcite , where alignment with supposed "summaries" is particularly straightforward. domain outliers do exist, e.g., for scientific literature @xcite or the legal domain @xcite @xcite , but are primarily restricted to the english language or do not contain finer-grained alignments between crosslingual documents. reasons for the usage of mentioned predominant domains are manifold: data is reasonably accessible throughout the internet, can be automatically extracted, and the structure naturally lends itself to the extraction of excerpts that can be seen as a form of summarization. for news articles, short snippets (or headlines) describing the gist of main article texts are quite common. wikipedia has an introductionary paragraph that has been framed as a "summary" of the remaining article @xcite , whereas others utilize scholarly abstracts (or variants thereof) as extreme summaries of academic texts @xcite . for a variety of reasons, using these datasets as a training resource for summarization systems introduces (unwanted) biases. examples include extreme lead bias @xcite , focus on extremely short input/output texts @xcite , or high overlap in the document contents @xcite . models trained in such a fashion also tend to score quite well on zeroshot evaluation of datasets from similar domains, however, poorly generalize beyond immediate indomain samples that follow a different content distribution or longer expected summary length. simultaneously, high-quality multilingual and cross-lingual data for training summarization systems is scarce, particularly for datasets including more than two languages. existing resources are often constructed in similar fashion to their monolingual counterparts @xcite and subsequently share the same shortcomings of low data quality. our main contribution in this work is the construction of a novel multi-and cross-lingual corpus of reference texts and human-written summaries that extract texts from legal acts of the european union (eu). aside from a varying number of training samples per language, we provide a paragraph-aligned validation and test set across all 24 official languages of the european uninon foot_0 , which further enables cross-lingual evaluation settings. ## related work influencing works can generally be categorized into works about eu data, or more broadly about summarization in the legal domain. aside from that, we also compare our research to other existing multiand cross-lingual works for text summarization. ## the eur-lex-sum dataset we present a novel dataset based on available multilingual document summaries from the eur-lex platform. the final dataset, which we title "eur-lex-sum", consists of up to 1,500 document/summary pairs per language. for comparable validation and test splits, we identified a subset of 375 cross-lingually aligned legal acts that are available in all 24 languages. in this section, the data acquisition process is detailed, followed by a brief exploratory analysis of the documents and their content. finally, key intrinsic characteristics of the resource are compared with relation to existing summarization resources. in short, we find that the combination of human-written summaries coupled with comparatively long source and summary texts makes this dataset a suitable resource for evaluating a less common summarization setting, especially for long-form tasks. ## exploratory analysis an exploratory analysis of the dataset is conducted to confirm the resource's viability for automatic summarization and overall data quality. aside from a qualitative view of the resource and an analysis of the temporal distribution of our samples, we provide a comprehensive look at intrinsic metrics commonly used for summarization datasets. ## experiments as a reference for future work building on top of this dataset, we provide a set of suitable baselines and discuss limitations of methods and data. notably, there are considerable challenges in constructing baseline runs with popular algorithms on this dataset if trying to cover all languages. primarily, even summary lengths exceed input limitations of popular abstractive neural models based on transformer architectures; these systems are generally limited to 512 (subword) tokens @xcite , and even length-focused alternatives generally boast only up to 4096 tokens @xcite , which is well below the median length of reference texts and prevents us from training systems without further (manual) alignments provided on chunks of the input text. less obvious, but no less problematic is the availability of tokenizers or sentence splitting methods in popular nlp libraries, affecting several languages in our corpus (for a more in-depth list of supported languages by library, see appendix aside from a set of extractive baselines, we further evaluate a cross-lingual scenario in which summaries for the english reference text are generated and then translated into the target languages. the hypothesis is that this provides insight into limitations of existing xls systems discussed in section 2.3 and also represents more realistic deployment scenarios where xls systems can be utilized as supportive summarizers for monolingual input texts. ## conclusion and future work throughout this work, we have detailed the creation of a new multilingual corpus for text summarization, based on legal acts from the european union. we further provided a more detailed analysis of the underlying data and sample quality and hypothesized potential applications to open problems in the communtiy, such as long-form summarization or cross-lingual application scenarios. our dataset is publicly available on the web, and comes with a set of monolingual extractive baselines that provide suitable reference points for any future work in this direction. in particular, we intend to focus on exploiting the structure of summaries for a more guided generation of output texts. especially for extremely long legal texts, template structures could be utilized. on a more general level, we expect that progress in long-form models is required to achieve remotely sensible results on extreme-length generative tasks. alternative approaches in the meantime could include aspect-driven methods for building summaries in an iterative fashion. finally, on top of the static snapshot presented in this work, we are also working towards a continually updated data repository of this resource, which would then include newly added texts (or summaries) for eu texts.