SUMSUM @ FNS -2020 Shared Task (2020)

Abstract: This paper describes the SUMSUM systems submitted to the Financial Narrative Summarization Shared Task (FNS-2020). We explore a section-based extractive summarization method tailored to the structure of financial reports: our best system parses the report Table of Contents (ToC), splits the report into narrative sections based on the ToC, and applies a BERT-based classifier to each section to determine whether it should be included in the summary. Our best system ranks 4 ^{\textrm{th}} , 1 ^{\textrm{st}} , 2 ^{\textrm{nd}} and 17 ^{\textrm{th}} on the Rouge-1, Rouge-2, Rouge-SU4, and Rouge-L official metrics, respectively. We also report results on the validation set using an alternative set of Rouge-based metrics that measure performance with respect to the best-matching of the available gold summaries.

## introduction the number of financial reports available each year is increasing rapidly. consequently, exhaustive reading of such documents has become unreasonably laborious. automatic summarization methods could greatly simplify this task. the financial narrative summarization shared task for 2020 @xcite aims to study the application of automatic summarization methods to annual reports from uk firms listed on the london stock exchange (lse) @xcite . these reports, compared to those written by u.s. firms, exhibit a much less rigid structure, which makes summarization a challenging task. in recent years, recurrent neural networks (rnns) (e.g. @xcite ) and transformerbased neural networks (e.g. bert @xcite ) have been widely and successfully employed for extractive summarization in numerous text genres. we had hoped to employ such models for the fns-2020 shared task. unfortunately, the length of documents is beyond the models' limits: with an average training document length of 6086 tokens, rnns and transformers struggle to encode useful hidden representations in an end-to-end fashion. as a result, we hypothesized that some pre-selection over the input text is mandatory to obtain reasonable performance. in particular, we determined (see section 3) that most sentences in the gold-standard report summaries come from a single narrative section of the report. therefore, we ultimately built our system based on a classification-based extractive summarization approach that uses bert-based models @xcite simply to determine the section(s) to extract as the summary subject to truncation constraints (as described in section 3). as shown in ## system description our approach to fns summarization requires first that we split the document into coherent narrative sections (section 3.2), which is accomplished by the heuristic parsing of the table of contents (toc) (section 3.1). we then train neural network models (section 3.3) to identify which section(s) of a report to extract to comprise the summary. ## results for the evaluation, we employed the three systems described above. we show our results using two different evaluation metrics: the official metrics, which take the average of the rouge scores of the produced summary against all available gold summaries; and alternative metrics, which use the rouge scores of the single gold summary that best matches the system-generated summary. the latter is the metric proposed in the fns-2020 workshop call, and the metric that we optimized for when training the sumsum models. ## discussion of results as mentioned above, we relied on the alternative evaluation metric (that takes the highest score among all gold summaries of one financial report) for model selection. and as shown in however, when measured according to the official metrics, the results of the base and bert model are quite close to one another ( as described in the introduction, we initially hoped to employ neural network-based summarization methods directly for the fns-2020 shared task. in particular, we experimented with neural network nlp models such as presumm @xcite and transformer-xl @xcite , using them to generate summaries given the chosen sections as input (instead of outputting the first 1000 tokens). all, however, led to worse results. specifically, the models performed poorly on recall though the precision scores were close to those of our submitted sumsum-bert systems: the methods failed to select enough sentences compared to the gold summaries. a future direction of work could be improving sentence extraction from sections of different report components. ## conclusion in this paper, we describe the sumsum systems for the financial narrative summarisation shared task (fns-2020.) our best system parses the report table of contents (toc), splits the report into narrative sections based on the toc, and applies bert to each section to determine which section(s) to include for the summary. the best system achieves f1 scores of 72.66%, 68.12%, 68.58%, 69.75% with rouge-1, rouge-2, rouge-su4, and rouge-l on the validation set using our alternative evaluation metric that the systems were optimized for. the f1 scores are 46.0%, 30.6%, 30.2%, 32.2% according to the official metrics.