A Comparative Study on Abstractive and Extractive Approaches in Summarization of E uropean Legislation Documents (2021)

Abstract: Extracting the most important part of legislation documents has great business value because the texts are usually very long and hard to understand. The aim of this article is to evaluate different algorithms for text summarization on EU legislation documents. The content contains domain-specific words. We collected a text summarization dataset of EU legal documents consisting of 1563 documents, in which the mean length of summaries is 424 words. Experiments were conducted with different algorithms using the new dataset. A simple extractive algorithm was selected as a baseline. Advanced extractive algorithms, which use encoders show better results than baseline. The best result measured by ROUGE scores was achieved by a fine-tuned abstractive T5 model, which was adapted to work with long texts.

## introduction automatic summarization of legislation documents is a rather challenging task, because they usually are very long and hard to understand. therefore, any progress on this task might have great value for many businesses. most of the existing methods for text summarization are designed for a relatively short text (such as news, web pages, etc.). most of the available datasets consist of short summaries of articles, news, etc. in which the expected summary is a few sentences long. however, in the case of legal documents both the text and the summary are longer. @xcite provide a comprehensive overview of the current datasets for summarization, including cnn/daily mail @xcite , gigaword @xcite , lcsts @xcite and others. noticeably, most of the larger scale summarization datasets consist of relatively short documents. the paper's aim is to research the applicability of different algorithms for text summarization on a new dataset, called eu legislation documents. the collected documents were parsed, cleaned and prepared to be used for data summarization training by defining pairs of full text and corresponding summary. there are two main approaches for text summarization: extractive and abstractive. extractive summarization means identifying important parts of the text and concatenating them verbatim to produce a summary which is a subset of the sentences from the original text. abstractive summarization aims to make algorithms that are able to "understand" the whole text and to generate a new shorter text that conveys the most important information from the original one @xcite . the algorithm, which was selected as baseline, uses a classical approach for generating extractive summaries -sentence importance evaluation and combining the highly scored sentence to generate the summary @xcite . in this paper, we report results from different algorithms compared to the baseline algorithm. the compared algorithms include extractive and abstractive pre-summ @xcite , which generates state-of-the-art results for cnn/dailymail datasets @xcite , a fine-tuned abstractive t5 model @xcite , an extractive summarizer which uses bert @xcite , an extractive summarizer which uses legal-bert @xcite . ## related work this section presents existing approaches for text summarization. some are especially related to our work because they focus on long texts and this is the case with the legal documents that we use. @xcite focus on extracting informative sentences from a given document (without dealing with redundancy), especially when the document is relatively long (e.g., scientific articles). they rely on section information to guide the generation of summaries. global and local contexts are taken into account when deciding if a sentence should be included in the summary. this approach struggles when there is not a well defined structure of sections which is the case with the legislation documents. @xcite presents an algorithm for text summarization using the thematic hierarchy of a text. the proposed algorithm is intended to generate a one-page summary. based on the ratio of source text size to a given summary size, the algorithm generates a summary with some breaks to indicate thematic changes. this algorithm cannot easily be adapted to summaries with dynamic length. another approach is to combine extractive and abstractive models @xcite . in the extraction phase, it creates a graph model to extract key sentences. in the abstraction phase, it uses a recurrent neural network based encoder-decoder, and devises pointer and attention mechanisms to generate summaries. @xcite presented the transformer architecture, which establishes a new single-model state-of-the-art bleu score on two machine translation tasks. the architecture consisted of feed forward networks and attention mechanism. the basic architecture of a transformer is based on the encoder-decoder model and is especially suitable for summarization because it can handle sequential data. yet, the data does not need to be processed in order (for instance the beginning of the text does not have to be processed before the end). this is very useful for parallel training and reduces the time needed to train the transformers. the encoder takes all the input and encodes it into a vector containing the numerical representation of the text. then the decoder decodes the vector and produces the summary. the datasets used for training can be big and thus exist pre-trained systems such as bert (bidirectional encoder representations from transformers). they have been trained with huge general language datasets and can be fine-tuned to specific language tasks. the following algorithms we experimented with also rely on the transformer architecture: presumm @xcite , legal-bert @xcite , t5 @xcite . pegasus @xcite is a state-of-the-art nlp deep-learning algorithm for abstractive text summarization. it can be used for both extractive and abstractive summarization but the abstractive is more challenging because when the text is long, it should be understood, processed and a new text should be generated. ## dataset collection in order to create a dataset on which to compare the algorithms, we collected legislation documents. the data was preprocessed and only the relevant information for the task was left. the dataset consists of short, easy to understand explanations of the main legal acts passed by the eu, intended for a general audience. most cover the main types of legislation passed by the eu: directives, regulations and decisions. but some cover other documents, such as international agreements. the summaries are grouped into 32 policy fields, and each links to the full, official version of the act. summaries are not available for legal acts that are considered to be already sufficiently short/clear or aimed exclusively at a specialist audience (publications office of the european union, 2020). the information is provided by the publications office of the european union and is publicly available. it was retrieved on january 12, 2020. after the data was collected, we analyzed it and cleaned it in order to focus on the problem of text summarization. for the summaries we extracted only the key points section and for the full documents we removed the references to external documents. there are some summaries that combine more than one full legislation document: 169 summaries are a summary of two documents and 50 summaries are a summary of more than two documents. in these cases, the full documents are concatenated in the mentioned order. in order to be able to evaluate a wider variety of algorithms and remove potentially incorrect data, the outliers were handled in the following ways: during the initial collection the dataset contained 1750 records. after the cleaning there are 1563 summaries (10.7% of the dataset was removed). the mean length of the summaries and full texts is 424 and 8990 words respectively (see fig. 1 ). the ratio between the number of sentences in summaries and full documents is 0.16. ## experiments the aim of the experiments described in this section is to compare different approaches for text summarization of legislation documents. for this purpose we used the dataset mentioned in the previous section. ## conclusion we introduced a new dataset about summarization of european legislation documents. we also presented a comparative study of various algorithms for automatic text summarization on this dataset. in our experiments on these tasks, we obtained promising results including huge improvements over baseline. we believe that the new dataset, adopting existing algorithms to domain specific data and the results described in this paper will accelerate research directions on text summarization to expand the variety of domains with domain specific information and different sizes.