Extractive Summarization System for Annual Reports (2020)

Abstract: In this paper, we report on our experiments in building a summarization system for generating summaries from annual reports. We adopt an “extractive” summarization approach in our hybrid system combining neural networks and rules-based algorithms with the expectation that such a system may capture key sentences or paragraphs from the data. A rules-based TOC (Table Of Contents) extraction and a binary classifier of narrative section titles are main components of our system allowing to identify narrative sections and best candidates for extracting final summaries. As result, we propose one to three summaries per document according to the classification score of narrative section titles.

## introduction the financial narrative summarization (fns) aims to generate textual summaries from annual reports where financial overview of a company over a year stated. in this fns shared task 2020 @xcite , we focus on "narrative" parts of those documents, all textual parts except tables, figures and diagrams. the provided dataset contains the narrative summaries as the gold standard and the expected summaries should be formed only from the narrative parts of the documents. these documents often contain more than 100 pages of texts, tables, figures, diagrams, etc. summarizing an annual report of a company into a short text allows to get insights on annual financial status of the company. the summarization approaches can be extractive or abstractive not only according to the techniques we adopt but also depending on the dataset on which we build the system. our experiments are designed based on the extractive approach where we develop a pipeline in order to extract the most informative sections. we suppose that each narrative section is already a summary as the author of annual reports aim to resume key financial overview in each section. this is why we adopt the extractive summarization approach and the results of our system largely depend on the narrative sections prediction. first, we create a binary classification dataset and implement machine learning algorithms to identify the narrative sections with its sections titles, then analyze which sections contain key information on annual financial status. second, we select the summaries which capture the most informative parts of texts from the previously selected sections. for this shared task, we experiment hybrid methods combining neural networks and rules-based systems in our summaries extraction algorithms, according to the results of the narrative sections prediction, a document can be assigned one to three summaries as described in the section 4. in the section 5, we shows that our system obtains encouraging rouge scores and discuss on the major errors that we could observe in our final summaries. we conclude the paper by proposing some future works. ## related work text summarization is one of the most challenging tasks in nlp. in general, two methods are used to handle this task, extractive and abstractive methods. extractive summarization methods produce summaries by selecting the most relevant sentences directly from the source text. many approaches proposed for this type of summarization, some studies address the problem as a sentence-level sequence labeling task where each label indicates if the sentence should be included in the summary or not, using either a crf model (nguyen mt., 2017) or using attention rnn-based seq2seq models as in @xcite . these models require generally a large and high-quality training sets. there are also some unsupervised methods in the extractive approach using statistical methods as a simple tf-idf @xcite or graph-based approaches as @xcite that represent text as a network linking sentences and use graph-based ranking methods to generate a summary. abstractive summarization, on the other hand, is a technique in which the summary is generated by generating novel sentences. this method requires then a deeper interpretation and understanding of the text and a text generation system. most researches are based on a sequence to sequence attention based models and more recently transformers to handle this challenge @xcite . ## data the financial annual reports are publicly available documents on which firms publish a year-end summary of their operations and financial conditions. an annual report covers a broad scope of contents in different sections. the main data sets of the fns shared task (see ## system overview as shown in figure 1 , we first identify narrative sections from the reports, then extract summaries from those sections. to do that, we extract the toc from each document and train a binary classification model with keras using the list of narrative sections found in the gold summaries as train set. ## results and discussion at the time of the deadline, the best summaries we had were the summaries gathered from the sections with the top 3 classification score, which are corresponding to the systems 1, 2 and 3, respectively. we also randomly selected all the final summaries of 100 reports and read the summaries focusing on the contents of each summary if it provides firm's financial overview of the business over the year. this defined which system's summary we should submit, whose results (as provided by the organizers) are reported in ## future work for future works, we plan to improve the toc extraction algorithm by adding pos (part-of-speech) tagging step before applying rules set to the dataset, this will help to implement new rules which discriminates some unusually used syntactic categories like verbs and modals in the toc. toc extraction results are used as the dataset for training our binary classifier of narrative titles, consequently, we expect that the best narrative sections prediction would be improved. we also believe that giving a weight to some narrative sections titles may help selecting the more relevant summaries. an idea is to use the number of occurrences of each unique narrative title as a weight of its importance.