Summarizing Long Regulatory Documents with a Multi-Step Pipeline (2024)

Abstract: Due to their length and complexity, long regulatory texts are challenging to summarize. To address this, a multi-step extractive-abstractive architecture is proposed to handle lengthy regulatory documents more effectively. In this paper, we show that the effectiveness of a two-step architecture for summarizing long regulatory texts varies significantly depending on the model used. Specifically, the two-step architecture improves the performance of decoder-only models. For abstractive encoder-decoder models with short context lengths, the effectiveness of an extractive step varies, whereas for long-context encoder-decoder models, the extractive step worsens their performance. This research also highlights the challenges of evaluating generated texts, as evidenced by the differing results from human and automated evaluations. Most notably, human evaluations favoured language models pretrained on legal text, while automated metrics rank general-purpose language models higher. The results underscore the importance of selecting the appropriate summarization strategy based on model architecture and context length.

## introduction automatic text summarisation (ats) involves generating a compressed, concise, and fluent version of an input text while preserving its main key points. a summary proves useful because it helps people process and understand texts faster and better. summarizing regulatory texts is important for making complex legal language more accessible and ensuring compliance by condensing information into a concise, understandable format. 1 current ats methods use either extractive or abstractive summarization. an advantage of extractive summarization is that it captures sentences and information literally, resulting in a factually consistent summary. however, the summary is harder to read and less intuitive as sentences are copied and combined. abstractive summaries are more coherent and fluent as they summarize texts in a human-like fashion. but it also has disadvantages because an intricate understanding of the original text is required and the summary can be factually inconsistent. in this paper, our aim is to explore the advantages of both strategies, as we leverage them for the summarisation of very lengthy, regulatory documents. a regulatory text is a formal document issued by a government or regulatory body that outlines rules, guidelines, or standards to govern the conduct, practices, or operations within a specific industry, sector, or jurisdiction. regulatory documents are difficult to process due to their extensive size, unique structure, numerous citations and references, ambiguity, and domain-specific vocabulary. current automatic summarization tools face challenges with regulatory texts, either because their length exceeds the context length of llms, or because the length and structure of the input document raise the risk of omissions in the summary. leaving out important information could have major negative effects. this paper compares two-step and multi-step summarisation methods for regulatory documents, comparing the effectiveness of different neural model architectures and combinations. our approach consists of the following steps, illustrated in figure 1 . first, the document is segmented into smaller units or 'chunks'. each chunk is then processed by an extractive summarization model, and all resulting summaries are concatenated. this extractive step may need to be conducted iteratively. the outcome of extraction is then summarized in an abstractive manner, creating a final summary. combining these two summarization steps could prove useful in handling the large size of the original text. it uses extracted salient sentences to develop a coherent, fluent summary. similar architectures have been used on different types of texts and have shown promising results @xcite @xcite . however, summarizing long regulatory documents using this architecture has been researched less extensively. in particular, our goal is to evaluate various models used for each step to identify the most effective combination of models for the summarization task, paying particular attention to whether preliminary extraction is more beneficial if performed with domain-specific (legal) rather than domain-general models. a second important goal is to compare the effect of context length on the quality of the generated summaries: models allowing longer context lengths need less extraction. given the growing trend for large language models to allow longer document lengths, it is increasingly important to understand whether such models are able to acquire a comprehensive understanding of a full document, or whether preliminary distillation of information is helpful @xcite . ## related work long document summarisation pretrained language models (lms) struggle with long texts due to limitations on input context length. for example, bert @xcite and t5 @xcite have a context length of 512 tokens while pegasus's @xcite ) and bart's @xcite context length is 1024 tokens. to counter this limitation, some long document architectures incorporate a different self-attention mechanism, calculating attention between specific parts of the sequence instead of calculating the attention for every possible combination of the sequence. this enables them to process long sequences because the computation requirements will not grow quadratically. longformer @xcite is an encoder-only architecture based on roberta @xcite , designed to handle long-range dependencies more efficiently than standard transformers, and accepting inputs of up to 4096 tokens. it employs a combination of global attention and sliding window attention instead of full attention, which scales linearly with the input sequence. led @xcite adds a decoder to the longformer architecture, turning it into the longformer encoder-decoder model. the decoder does use the full attention mechanism but led retains its linear computation capability. similar examples of lms designed for longer documents include bigbird (which accepts a context length of 4096 tokens; @xcite , longt5 @xcite and pegasusx @xcite , both of which accept contexts of 16,384 tokens. extending context length is often a goal in recent releases of decoder-only llms, such as the gpt family of models. other examples include llama-2-7b-32k (tog), which is an llm based on llama-2 @xcite with a context length of 32768. ## method our approach to the summarisation of long regulatory documents is a multi-step process consisting of extraction followed by abstraction, where extraction is intended to alleviate the problem of limited context length accepted by a model. in particular, if the length of a source document |d| exceeds the context length k of an abstractive model, creating an intermediate extractive summary could help identify essential information across the document span, a more informed strategy than truncating the document to fit within k. the overall process is visualised in figure 2. we view a source document @xmath0, d 2 , d 3 , ...d m } as a sequence of chunks d i . ## conclusion in this paper, we focused on summarisation of long regulatory documents. our findings indicate that while models with a longer context length do not benefit from extraction, an extractive step renders bart, an encoder-decoder architecture, highly competitive. a small-scaled evaluation with human experts confirms this finding. however, experts also indicate a preference for summaries relying on extraction with a domain-specific, legal language model. future work should consider whether these findings are generalisable to other domains. furthermore, a more extensive human evaluation is required to ensure that our findings are reliable. this is particularly crucial given that human expert judgments are not perfectly aligned with the outcomes of our metric-based evaluation, which echoes findings from other studies. a further possible research direction is to use a state-of-the-art llm as an evaluator or 'judge' for generated texts, a strategy which recent research suggests is increasingly viable @xcite , though also one that requires some caution in view of results suggesting self-bias on the part of llms @xcite , as well as lower reliability in comparison with expert judgment @xcite .