Does the structure of textual content have an impact on language models for automatic summarization? (2024)

Abstract: The processing of long sequences with models remains a subject in its own right, including automatic summary, despite recent improvements. In this work, we present experiments on the automatic summarization of scientific articles using BART models, taking into account textual information coming from distinct passages from the long texts to be summarized. We demonstrate that taking into account document structure improves the performance of state-of-the-art models and approaches the performance of LongFormer on English.

## introduction long texts are formatted with visual marking (such as paragraphs, sections, and so on) to help readers retrieve information quickly. these markers help skim the long documents and get a general idea of their content. document skimming can be used to obtain an abstract of a document. automatic summarization has long suffered from the context limitation of neural networks (nn) models. context limitation either restricts the possible size of the text given as input (with transformers @xcite ) or the information retained during process @xcite . transformers consider the entire context to proceed with a given task. however, this processing of memory comes with a considerable calculation cost. that calculation cost is induced by the very mechanism that allows full information retention: a context memory that keeps the whole sequence in memory for processing one word. that computational cost is quadratic. this computational complexity limits the first transformers models to sequences of 512 tokens. as calculation capacities improved, this limit was quickly pushed back from 512 to 1024 @xcite then 2048 and even going up to more than 200,000 tokens for the most recent llm (large language model) foot_0 (gpt-3, mistral, claude, inter alia). however, this progress comes at high costs in computing power and infrastructure. the cost of training basic models of the latest llms is estimated at around a million dollars @xcite and with a non-negligible carbon impact @xcite . in parallel, approaches to reducing the computational complexity of the transformers architecture have been explored by research. in particular via alternatives to the full attention mechanism (the use of square matrices to model sequences) @xcite @xcite . despite these improvements, the costs of training and inferring models remain high in terms of computing power. these methods use textual data without the metadata that accompanies and structures them, but other solutions highlight the structured nature of long texts for their processing. @xcite show an interest in taking into account the document structure (i.e. paragraphs and sections) in the processing of long texts and, in particular on the task of automatic summary. from this hypothesis, we start to evaluate the impact of the document structure on the automatic summary of long text. we first present the context in which this study takes place. we will then discuss the methodology followed and the results obtained before concluding with the observations made. the performance of our method approaches sota results for long contexts without modifying the structure of the models. a segmentation of tasks with a reflection on the construction of the writings could, therefore, allow a reduction in the costs necessary to obtain usable results. ## related work summarization is a classic task of natural language processing. it is, therefore, particularly well documented and already has numerous methods and models. 2 the first approaches to automatic summarization @xcite focus on so-called "extractive" methods. these methods consist of recovering the most important sentences from the text. however, they are criticized for their lack of readability. the arrival of generative language models @xcite @xcite has allowed "abstractive" methods to supplant extractive methods. these generative models arrive with the non-sequential text processing approach proposed by @xcite in the transformers architecture. this methodology responds to one of text processing challenges using neural network architectures: information retention. the transformers models thus make it possible to improve context consideration. hybrid methods emerge to get the most out of extractive and abstractive methods. these methods, aimed at streamlining the result of extractive summaries thanks to the abstractivity of generative language models, are particularly effective for long texts due to the simplicity of their operation. they allow a reduction in calculation costs by only selecting the relevant sentences from the texts to be given to the generative model @xcite . among hybrid methods, approaches based on text structure for processing long texts have been proposed @xcite using graph neural networks (gnn) to organize the hierarchy of sections. these methods make it possible to increase the performance of the models significantly. other studies show the potential that the use of metadata can have in the processing of long texts @xcite . abstractive methods remain the most used because they avoid going through a pipeline (while hybrid methods need the choice of the extractive method and an appropriate generative model). ## method to show the impact of document structure on summarization, we select different specific parts of the text as input for abstractive models. we then compare the summaries produced by a model with a sub-selection of the document with the reference summary produced by a human editor. this approach is a hybrid method combining extractivity in the selection of relevant parts of texts and abstraction using generative language models. ## results although the results do not allow us to exceed the values of the rouge scores obtained with long-former (0.46 for longformer against 0.42 for rouge-1 for the bartxiv model in the best configuration see this improvement is noticeable when the model is not adapted to the type of text processed (like bart, see in figure 1 , we can observe the parts of the document that are the most used in the abstract depending on the abstract given. the yellow concentration shows that the most overlapping parts for human redactors are the end of the introduction and the end of the conclusion. this confirms the hypothesis of @xcite and advocates for prior text reduction based on text structure for summarization. to obtain the best rouge or bertscore scores, the distribution profiles of the automatically generated summaries must come as close as possible to those obtained by human summaries. that is to say, obtaining a maximum n-gram overlap at the end of the introduction and the end of the conclusion (see figure 1 -reference summary). this representation also makes it possible to evaluate the impact of fine-tuning on selecting information in texts for automatic summarization. having been trained on a corpus of press articles, bart tends to concentrate on the first sentences of the texts (see figure 1 -bart except for bart-last sentences) this profile differs from the distribution of information retrieved by the human editor for the summary of scientific articles. it seems better modeled by bartxiv and longformer. indeed, in the profiles obtained with bartxiv, we can see that the maximum overlap is shifted downwards compared to the profiles of bart (figure 1-bartxiv). ## conclusion we were able to show here that the use of structure by human authors in writing a summary is poorly imitated by the models even when they have access to most of the text to select information. humanly produced summaries remain highly abstract compared to the language models targeted here. this particularity reinforces our hypothesis that taking structure into account could allow the creation of better summaries by the models. using hybrid models improves the results rouge or bertscore of un-fine-tuned limited context window models and allows an alternative to more attention-expensive models. however, this limitation of inputs loses its interest with fine-tuned models whose learning conditions the position of the information retrieved. in addition to adapting the models' vocabulary, fine-tuning allows the mod-els' attention to be focused on certain parts of the texts. this observation is specific to the automatic summary task and requires additional analysis to verify its generalization to other tasks. using hybrid models based on the document structure of texts is an interesting approach when using a limited context window model that is not fine-tuned to the target data. however, access to the entire context allowed by the longformer architecture remains more efficient for automatic text summarization. these observations confirm the importance of the search for an alternative to the full attention mechanism of transformer architectures, which are costly in computational terms. to this end, we plan to implement a new representation of texts.