End-to-end Training For Financial Report Summarization (2020)

Abstract: Quoted companies are requested to periodically publish financial reports in textual form. The annual financial reports typically include detailed financial and business information, thus giving relevant insights into company outlooks. However, a manual exploration of these financial reports could be very time consuming since most of the available information can be deemed as non-informative or redundant by expert readers. Hence, an increasing research interest has been devoted to automatically extracting domain-specific summaries, which include only the most relevant information. This paper describes the SumTO system architecture, which addresses the Shared Task of the Financial Narrative Summarisation (FNS) 2020 contest. The main task objective is to automatically extract the most informative, domain-specific textual content from financial, English-written documents. The aim is to create a summary of each company report covering all the business-relevant key points. To address the above-mentioned goal, we propose an end-to-end training method relying on Deep NLP techniques. The idea behind the system is to exploit the syntactic overlap between input sentences and ground-truth summaries to fine-tune pre-trained BERT embedding models, thus making such models tailored to the specific context. The achieved results confirm the effectiveness of the proposed method, especially when the goal is to select relatively long text snippets.

## introduction analyzing the annual financial reports is the most established way to assess the health state of business companies. for example, rating agencies, banks, and hedge funds rely on the information extracted from domain-specific reports to assign ratings, grant loans, and drive investment strategies @xcite . unfortunately, the content of the released financial reports is highly redundant as it typically includes contextual and technical information that is marginally relevant to domain experts. the shared task of the financial narrative summarization (fns) research challenge @xcite aims to address this issue by fostering innovative research on the problem of automatic extraction of domain-specific summaries from the annual financial reports. the algorithms designed for automatic text summarization can be partitioned into two main classes: extractive approaches and abstractive ones. while extractive approaches pick existing text snippets (e.g., sentences, phrases, keywords) directly from the source text, abstractive methods generate new content based on the analysis of the input documents. the summarization process can be either supervised, when a portion of document content already annotated by human experts as relevant or not is available, or unsupervised when no a priori knowledge is given. the fns shared task promotes the study, development, and testing of automated sentence-based summarization techniques tailored to the financial domain. to extract relevant sentences from annual financial reports, it provides researchers with a large set of humanly annotated data @xcite . therefore, the present work addresses the study of a supervised, extractive, sentence-based approach to address the fns shared task. extractive summarization methods have found application in several domains, such as the summarization from news articles (e.g., @xcite @xcite ), scientific papers (e.g., @xcite @xcite ) and product reviews (i.e., @xcite ). wide-ranging overviews of the state-of-the-art works on text summarization can be found in @xcite @xcite . using machine learning techniques to summarize documents entails (i) extracting relevant text features at the sentence level and (ii) feeding the extracted features to a supervised model to produce a sentence rank @xcite . to address the former step, latent text representations based on deep learning models have shown to be very effective in generating relevant text features @xcite however, pre-trained deep nlp models need to be tailored to the specific context under analysis (e.g., medical data @xcite ), patent-related areas @xcite ). previous works that use deep language models in the financial domain focused on the sentiment analysis task @xcite . to the best of our knowledge, this is the first attempt to fine-tune pre-trained deep nlp models in order to enhance the quality of the process of financial report summarization. section 2 overviews the architecture of the proposed summarizer. section 3 and 4, 5 separately describe each phase of the summarization process. section 6 summarizes the outcomes of the evaluation step. finally, section 7 draws conclusions and envisions future research steps. ## the sumto system the summarizer based on end-to-end training (sumto) consists of a three-phase process, which is depicted in figure 1 . it comprises (i) a preprocessing phase, which transforms the raw textual documents and annotates the content at the sentence-level. (ii) a training step, which extract relevant concepts and relationships according to two established deep language models, i.e., bert @xcite and distilbert @xcite . (iii) a evaluation step, which rates the sentences of each test document according to the fine-tuned models trained at the previous step and produce a per-document summary consisting of the highly rated sentences. ## data collection and preprocessing the data collection provided by the organizers of the fns 2020 shared task includes the (i) the training set, consisting of 3,000 annual reports and 9,873 golden summaries (3.29 summaries per report, on average), (ii) the evaluation set, consisting of 363 annual reports and 1,250 golden summaries (3.44 summaries per report, on average), and (iii) the test set, consisting of 500 annual reports and 1,673 blind system id pre-trained model parameters settings 1pe distilbert-base-cased n. of epochs: 1, batch size: 32, learning rate: 2e-5 2pe distilbert-base-cased n. of epochs: 2, batch size: 32, learning rate: 2e-5 3pe bert-base-cased n. of epochs: 1, batch size: 32, learning rate: 2e-5 the textual content of the reports in the training, evaluation, and test sets is transformed by applying the following data preparation steps. ## training phase of the deep language model a regression model is trained on the sentences of the training documents in order to predict the previously assigned sentence label (i.e., the rouge-2 precision score). this idea behind to optimize the sentence relevance score according to the provided human annotation by fine-tuning the pre-trained bert model @xcite . the overall architecture is trained using the mean-square loss and the adamw optimizer @xcite for faster convergence. ## evaluation phase for each test document, the summarizer evaluates and ranks the corresponding sentences according to the fine-tuned model. specifically, the sentences of the input report are forward-passed through the trained model and sorted in order of decreasing predicted rouge-2 precision score. the ranked list is postprocessed by removing (i) duplicate sentences, (ii) sentences containing more than 50% of uppercase characters, (iii) sentence containing more than 50% of non alphabetic characters, (iv) sentences shorter than 5 words. the text snippets are selected from the post-processed pool according to their assigned score until the summary length requirement (up to 1000 words) is met. the output summary is generated by concatenating the post-processed sentences in order of decreasing relevance score. ## results the output summaries submitted to the fns 2020 shared task contest were evaluated by the shared task organizers. to evaluate the system outputs provided by the participants, they exploited the jrouge package foot_3 , which is a lightweight, multilingual tool implementing the rouge metrics @xcite . summaries were evaluated using the rouge-1, rouge-2, rouge-su4, and rouge-l metrics. beyond the systems proposed by the contest participants, the following baseline methods have been considered: (i) textrank @xcite , (ii) lexrank @xcite , (iii) poly @xcite , and (iv) a topline algorithm, i.e., muse @xcite . the sumto system achieved fairly good results in terms of rouge-l (i.e., finding the longest n-gram match with the ground truth), because our system tends to prefer relatively longer sentences. for the same reason, rouge-1 performance is on average worse than that achieved for rouge-2 and rouge-su4. ## conclusions and future research steps the paper described an extractive summarization approach to summarizing textual financial reports. the proposed approach relies on the fine-tuning of a bert deep language model. the goal is to deeply tailor the deep nlp model to the specific context under analysis. the system runs were submitted to the fns 2020 shared task, achieving fairly high performance in terms of rouge-l score. our future research agenda will cover the following aspects: pruning of redundant information: the current summarization architecture is not able to prune redundant content, with respect to the previously selected sentences, during the sentence evaluation phase. we plan to extend system by embedding ad hoc redundancy penalty score. deeper model contextualization: the results have confirmed the effectiveness of the bert architecture to support text summarization. we aim to explore the applicability of larger and deeper neural language models in order to better capture the semantic meaning of the analyzed sentences.