The Financial Narrative Summarisation Shared Task ( FNS 2020) (2020)

Abstract: This paper presents the results and findings of the Financial Narrative Summarisation shared task (FNS 2020) on summarising UK annual reports. The shared task was organised as part of the 1st Financial Narrative Processing and Financial Narrative Summarisation Workshop (FNP-FNS 2020). The shared task included one main task which is the use of either abstractive or extractive summarisation methodologies and techniques to automatically summarise UK financial annual reports. FNS summarisation shared task is the first to target financial annual reports. The data for the shared task was created and collected from publicly available UK annual reports published by firms listed on the London Stock Exchange (LSE). A total number of 24 systems from 9 different teams participated in the shared task. In addition we had 2 baseline summarisers and additional 2 topline summarisers to help evaluate and compare against the results of the participants.

## introduction companies around the world produce a variety of reports containing both narrative and numerical information at various times during their financial year. such reports are referred to as financial disclosures and usually include quarterly reports, preliminary earnings announcements, conference calls, press releases financial annual reports @xcite . this creates a vast financial information environment which can be impossible to keep track of @xcite @xcite . the same set of information can be crucial for a number of different reasons. it can help highlight company achievements and gain support from shareholders in the stock exchange. it can identify risks and opportunities that investors need to take into account. financial reporting is also strongly related to due diligence processes during mergers and acquisitions, as well as during auditing processes. all the above uses, many of which can be critical during a company life-cycle, show the vital need for automatic summarisers, in order to reduce the amount of time and effort required by stakeholders -be they shareholders investors or other parties -to read and analyse those documents. the financial narrative summarisation (fns) shared task focuses on annual reports produced by uk firms listed on the london stock exchange (lse). in the uk and elsewhere, annual reports structure is much less rigid than those produced in the us elhaj2019, . companies usually produce glossy brochures with a much looser structure that is usually disseminated in pdf file format. this makes automatic summarisation of narratives in uk annual reports a challenging task, since the structure of those documents needs to be extracted first in order to summarise the narrative sections of the annual reports. this can be done by detecting narrative sections that usually include the management disclosures (financial narratives) rather than the financial statements of the annual reports @xcite . previously, the 1st and 2nd financial narrative processing workshops @xcite focused on the process of extracting and analysing financial narratives from multilingual financial statements written in languages such as english, french and spanish @xcite . this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . in this task we ask participants to generate automatic summaries for lengthy uk annual reports (each with more than 60,000 words on average) by focusing on the financial narratives of the reports and producing a summary of no more than 1000 words for each annual report. this paper presents the results and findings of the finanical narrative summarization shared task, as follows. we begin with an overview of related work in section 2. we then describe the data, elaborating on the task (section 3), and continue with the baseline and topline system descriptions (section 4). after this complete picture of the task and setting, we overview the submitted systems, in section 5. we conclude the paper with the task evaluation and related results as well as an appropriate short discussion of the findings (section 6). ## related work the increased availability of financial report data has been met with research interest for applying automatic summarisation methods. the task of automatic text summarisation aims to produce a condensed, informative and non-redundant summary from a single or multiple input texts @xcite . this is achieved by either identifying and ranking subsets of the input text (i.e. extractive approaches ( @xcite ), or by generating the summary from scratch (i.e. abstractive methods @xcite ). extractive summarisation methods have received far higher attention than their abstractive counterpart methods. this is mainly due to their relative simple approach when compared to the comparatively high requirements of the abstractive methods, especially when it comes to computational resources and data availability. extractive summarisation utilises scoring approaches to identify and reorder parts of the input (e.g. sentences, phrases and/or passages), using a variety of feature extraction/engineering and evaluation methods @xcite @xcite @xcite @xcite . where adequate data is available, machine learning methods have been employed, such as hidden markov models @xcite , topic-based modelling @xcite , clustering methods @xcite @xcite , deep neural network classification @xcite and language models @xcite . the application of summarisation and natural language processing techniques in general has promising applications in the financial domain @xcite . recently, statistical features with heuristic approaches have been used to summarise financial disclosure texts @xcite , generating summaries with reduced positive bias and leading to more conservative valuation judgements by investors that receive them. furthermore, the financial narrative summarisation task (el-haj, 2019) of the multiling 2019 workshop @xcite involved the generation of structured summaries from financial narrative disclosures. the summariserport system @xcite has been used to produce summaries for financial news. it utilises lexical cohesion @xcite , using sentence linkage heuristics to generate the output summary. a summarisation system of financial news was proposed in @xcite , generating query-based and company-tailored summaries, via unsupervised sentence ranking using simple frequency-based features. ## data description the financial narrative summarisation (fns) shared task focuses on annual reports produced by uk firms listed on the london stock exchange (lse). the produced annual reports are written in corporate language english, which comprises the words and visuals a company uses to communicate internally and externally. this influences corporate communication as a whole, from internal messaging to web content, press releases and including annual reports, which in turn could affect the communication between the corporate and stakeholders @xcite . in the uk and elsewhere, many registrants publish a glossy report containing graphics, photographs and supplementary narratives such as the letter to shareholders @xcite . these documents are typically provided as a digital pdf file and outside the u.s. they represent the primary annual reporting vehicle. this results in barriers to large-scale automated analysis nevertheless mean that little is known about this ubiquitous reporting channel. ## baseline and topline summarisers all participating systems were evaluated and compared against 2 top performing (topline) systems and 2 baseline systems. ## participating teams and systems a total number of 9 teams participated in the fns 2020 shared task with a total of 24 system submissions. ## results and discussion a number of 24 summarisation systems by 9 different teams have participated and submitted their system summaries to fns 2020. in addition we report the results of the 4 topline and baseline summarisers (muse, poly, textrank and lexrank respectively) are reported in tables 4 and 5 . the participating systems used a variety of techniques and methods ranging from rule based extraction methods and more towards high performing deep learning models and word embeddings. in addition the participating teams used methods to investigate the hierarchy of the annual reports to try and detect the structure of the report and extract the narrative sections. the majority of the applied techniques were extractive applying methods such as determinantal point processes (dpps) sampling algorithm and a combination of pointer network and t-5 (test-to-text transfer transformer) algorithms. other extractive summarisers used word embeddings such word2vec, bert and using cbow & skip grams. an endto-end training method using deep nlp techniques, and a hierarchical summary that visualises as a tree with summaries under different discourse topics and an ensemble based model have also been reported. this variety of techniques shows the interdisciplinary notion of fns 2020 and the fact that it attracted such a range of methods, making it a gateway for researchers and practitioners working on summarising lengthy annual reports. some of the challenges and limitations reported by the participants is the fact that the average length of those annual reports is 60,000 words, this makes the training process difficult in term of time and performance, which is a problem that we are aware of and it is what prompted us to introduce such a challenging task. in addition, participants explained that it is difficult to detect structure of such reports due to the fact that they come originally in pdf format and extracting information from such files results in a lot of noise. this is a problem that we have been working on since 2012 and though we understand it is challenging, we believe it opens up an interesting research problem that is worth investigating more in the future. tables 4 and 5 show the results of the participating systems using rouge-1, rouge-2, rouge-l and rouge-su4 respectively. the results have been sorted in descending order according to the highest f-score. the topline (muse and poly) and baseline (textrank and lexrank) systems are highlighted in bold font. the results show that the majority of the participating systems produced results that are better on average than our two baselines and poly topline. on the other hand, our topline muse results show a challenging notion making it hard to beat, but even though we are happy to see that many participating systems have managed to produce results that are significantly better than muse. such results will be used as a comparison line in the future through creating a venue of results and techniques for researchers working on financial text summarisation.