P oin T -5: Pointer Network and T -5 based Financial Narrative Summarisation (2020)

Abstract: Companies provide annual reports to their shareholders at the end of the financial year that de-scribes their operations and financial conditions. The average length of these reports is 80, andit may extend up to 250 pages long. In this paper, we propose our methodology PoinT-5 (thecombination of Pointer Network and T-5 (Test-to-text transfer Transformer) algorithms) that weused in the Financial Narrative Summarisation (FNS) 2020 task. The proposed method usesPointer networks to extract important narrative sentences from the report, and then T-5 is used toparaphrase extracted sentences into a concise yet informative sentence. We evaluate our methodusing Rouge-N (1,2), L, and SU4. The proposed method achieves the highest precision scores inall the metrics and highest F1 scores in three out of four evaluation metrics that are Rouge 1, 2,and LCS and only solution to cross MUSE solution baseline in Rouge-LCS metrics.

## introduction annual reports may extend up to 250 pages long as stated above, which contains different sections general corporate information, financial and operating cost, ceos message, narrative texts, accounting policies, financial statement including balance sheet and summary of financial data documents. in the financial narrative summarisation task, only the narrative section is summarised, which is not explicitly marked in the dataset, making it challenging and interesting. in recent years, previous manual smallscale research in the accounting and finance literature has been scaled up with the aid of nlp and ml methods, for example, to examine approaches to retrieving structured content from financial reports, and to study the causes and consequences of corporate disclosure and financial reporting outcomes @xcite . companies produce glossy brochures of annual reports with a much looser structure, and this makes automatic summarisation of narratives in uk annual reports a challenging task @xcite . hence we summarize the narrative section of annual reports, particular narrative sentences that are spread loosely across the document need to be first identified and summarise those sentences. the summarisation limit is set to 1000 words, where the actual length of the report may go up to 250 pages long. hence to summarize these long annual reports using a combination of extractive and abstractive summarisation. the text summary method can be classified into two paradigms: extractive and abstractive. the extractive summarisation method extracts the meaningful sentences or a section of text from the original text and combines them (ranked or unranked) to form a summary @xcite @xcite . whereas abstractive summarisation generates words and sentences that are similar in meaning to the given text to form a summary that may not be in actual text @xcite @xcite . when summarizing long documents such as in our case up to 250 pages long, extractive summarisation may not produce a coherent and readable summary, and abstractive summarisation cannot cover complete information using encoder-decoder architecture. one problem is that typical seq2seq frameworks often generate unnatural summaries consisting of repeated words or phrases @xcite . hence, we come up with a combination of extractive and abstractive summarisation to first select important narrative sentences and concisely convey them. pointer networks @xcite is used in various combinatorial optimization problems, such as travelling salesman problem (tsp), convex hull optimization. we used pointer networks in our task of financial narrative summarization to extract relevant narrative sentences in a particular order to have a logical flow in summary. these extracted sentences are paraphrased to summarise these sentences in an abstractive way using the t-5 sequence-to-sequence model. we train the complete model by optimizing the rouge-lcs evaluation metric through a reinforcement learning objective. ## related works in this section and appendix b, we discuss related works in the fields of abstractive summarisation, extractive summarisation, combinations of these two methods, reinforcement learning applications, and summarisation of financial narratives and their methodology. studies of human summarizers show that it is common to apply various operations while condensing, such as paraphrasing, generalization, sentence-level summarisation and reordering @xcite . we continue related work in appendix b. ## data description the financial narrative summarisation dataset contains 3,863 annual reports for firms listed on lse covering the period between @xcite @xcite . dataset is randomly split into training (75%), testing and validation (25%) table :2. we use nltk sentence tokenizer 1 to tokenize sentence in the annual report and summary processing for all our experiments. data is further described and anayzed in appendix a. ## results and analysis in this section we present results from our experiments and compare with different baselines muse @xcite , text-rank @xcite , lex-rank @xcite , and polynomial summarisation @xcite 1 bold is marked for highest value amongst all the solutions for the task including baselines. pre-trained bert for summarisation is not fine-tuned for this specific task hence its performance is not as good as pointer network and point-5 (pointer network + t-5). point-5 network gives highest results for precision on all the evaluation metrics rouge -1,2,l,su4. from this it can be inferred that generated summmaries are highly precise in extracting narrative sentences and matches with ground truth summaries. recall is comparatively much lower than precision in most of the evaluated metrics which means that generated summaries do not cover all the information in ground-truth summaries. this large difference in precision and recall shows that generated summaries do not cover all the sentences possibly due to restriction imposed on the number of sentences during training to follow word limit in summaries. less recall and high precision value shows that generated summaries provide highly relevant information but does not cover complete information in ground truth summaries. from results it is evident that there is not much difference in performance between pointer net performance and point-5 which extraction played key role in the architecture. point-5 and pointer networks are the only system to cross muse baseline in rouge-l by atleat 5 points. these models give highest f1 results in rouge-l and rouge 1 metrics and highest precision in rouge-l,1,2,su4. ## conclusion and future work in this work we present our solution on financial narrative summarisation(fns2020) dataset using point-5 method explained in 4. it is combination of both extractive and abstractive methods using pointer network and t-5. with these methods we are able to achieve highest precision score in every evaluation metric and achieve highest f-1 scores in rouge-lcs and rouge-1. in our future work we would like to address several limitation of our method such as factual correctness in summaries which is very important in financial domain as done in @xcite in summarizing radiology reports. to improve precision of our generated summaries under 1000 words we would formulate a penalty if system generates more than 1000 words during training of rl algorithm rather than restricting algorithm to fixed number of sentences.