Summarization of Long Input Texts Using Multi-Layer Neural Network (2022)

Abstract: This paper describes the architecture of a novel Multi-Layer Long Text Summarizer (MLLTS) system proposed for the task of creative writing summarization. Typically, such writings are very long, often spanning over 100 pages. Summarizers available online are either not equipped enough to handle long texts, or even if they are able to generate the summary, the quality is poor. The proposed MLLTS system handles the difficulty by splitting the text into several parts. Each part is then subjected to different existing summarizers. A multilayer network is constructed by establishing linkages between the different parts. During training phases, several hyperparameters are fine-tuned. The system achieved very good ROUGE scores on the test data supplied for the contest.

## introduction summarization of long texts is a challenging problem for different widely available summarizers. while the deep learning (dl) based summarizer bart @xcite is severely restricted by the size of the input document, the quality of other traditional summarizers, namely lexrank @xcite , textrank @xcite ) are found to be poor in terms of different rouge scores. the present paper proposes a novel architecture, muti-layer long text summarizer (mllts), to overcome the above challenge. figure 1 provides a schematic diagram of the proposed architecture. the novelty of the mllts architecture is that it uses multiple online summarizers for carrying out the summarization task in the following way. first, the long input text is partitioned into several parts. these parts are assigned to different layers of the multi-layer architecture. if the document is partitioned into p parts and s is the number of summarizers used, then the total number of layers is p × s. different parameters are trained to fine-tune the intra-layer and inter-layer connections to optimize the overall output. finally, votesumm method proposed by @xcite is used for generation of the summary, by combining different part summaries in an optimized manner. the rest of the paper is organized as follows. section 2 provides a brief review of some past works on network-based summarization. section 3 presents a detailed description of the proposed architecture. sections 4 and 5 describe the experiments conducted and the results obtained, respectively. ## related past works graph-based sentence ranking is a popular text summarization technique. in this approach, the input text is represented using a graph/network of sentence nodes. edges between the nodes are created to represent the relationship between them. the nodes are ranked using different methods to determine their overall importance with respect to the given input text. finally, the summary is generated by selecting the sentences which receive high ranks. @xcite used lexical similarity between sentences for edge creation in the graph and used pagerank @xcite to rank the nodes. @xcite used cosine similarity between bag-of-words representation of sentences for adding edges in the graph. tohalino and amancio (2017) performed a summarization of multiple similar documents using multilayer networks. the layers of the network are used to represent each individual input document. tf-idf based cosine similarity is used for connecting sentence nodes. node ranking is performed via nine different network measurements, such as degree, strength, pagerank, accessibility, and symmetry, among others, for generation of summaries. alzuhair and al-dhelaan (2019) used a combination of different edge weighting schemes, namely jaccard similarity, tf-tdf similarity, topic signature similarity and identity similarity along with pagerank and hits (kleinberg, 1999) node ranking methods for summarization. ## proposed approach: multi-layer long text summarizer the proposed mllts approach works in the following way. first, the long input text is partitioned into p smaller parts. these parts are not discrete, rather there is a 25% overlap between two successive parts. then, each part is summarized using different summarizers. we have used three summarizers for the present work: textrank, lexrank and distil-bart. the 3 × p short texts thus formed, from the three summaries for each of the p parts for a long input text, constitute the different layers of a network graph. the sentences from these 3 × p short texts form the nodes of the network graph and weighted edges are added between the nodes based on jaccard similarity of the corresponding sentence vectors. three parameters are fine-tuned to optimize the outputs. these are: for α and β, a @xmath0. similarly, @xmath1. once the network is prepared, votesumm technique is used to rank the nodes. the highestranking nodes are then used to generate the final summary. however, one major challenge still faced is that the size of some partitions is still too long for distil-bart to generate a summary. hence, we introduce the following novelty while partitioning the long input text. in such cases, the long input text is not partitioned into the p fixed number of parts. rather, the partitioning is such that all the parts have a fixed number of words. the number of words is chosen such that distilbart can generate a summary on it. in our experiments, we discovered that the summaries generated by a fixed number of partitions exhibit better rouge scores. hence, we kept this method to be the default option for partitioning. partitioning on a fixed number of words is used only in situations discussed above. ## conclusion this paper proposes multi-layer long text summarizer (mllts) as a possible solution of one of the modern day information processing problems, namely, summarization of long texts. although a large number of summarizers have been developed and made available over the last decade, their performance on long text is highly questionable. the novel mllts system proposed in this work is our contribution towards this need. this is specially designed for the booksum component of the automatic summarization for creative writing contest, coling 2022. the input texts for this system were chapters from famous english books. the corresponding target summaries were expected to be of the order of 5% of the input text size. a series of experiments were carried out to fine-tune several hyperparameters that are associated with the architecture. the successive design decisions and experiments are so planned that steady improvements in performance, with respect to rouge scores, can be observed. the best model, as per performance on the validation set, obtained values of 0.159 and 0.014 for rouge-1 and rouge -2, respectively. however, on the test data, it achieved much better scores. for the test data, the rouge-1 and rouge-2 scores obtained were 0.2643 and 0.0471, respectively. further, it obtained a rouge-l score of 0.2436. test results also suggest pretty high scores with respect to several other metrics such as bertscore and litepyramid among others. encouraged by the results, we aim at working towards further improvements to the proposed mllts scheme.