AMEX AI -Labs: An Investigative Study on Extractive Summarization of Financial Documents (2020)

Abstract: We describe the work carried out by AMEX AI-LABS on an extractive summarization benchmark task focused on Financial Narratives Summarization (FNS). This task focuses on summarizing annual financial reports which poses two main challenges as compared to typical news document summarization tasks : i) annual reports are more lengthier (average length about 80 pages) as compared to typical news documents, and ii) annual reports are more loosely structured e.g. comprising of tables, charts, textual data and images, which makes it challenging to effectively summarize. To address this summarization task we investigate a range of unsupervised, supervised and ensemble based techniques. We find that ensemble based techniques perform relatively better as compared to using only the unsupervised and supervised based techniques. Our ensemble based model achieved the highest rank of 9 out of 31 systems submitted for the benchmark task based on Rouge-L evaluation metric.

## introduction the publicly available financial information has been rising rapidly with periodic updates, earnings reports etc. from companies and institutions. these financial reports not only increase in volume but also in terms of the diversity, comprising different structure and format, depending on the location of the respective institutes and companies @xcite . the rise in quantity and diversity of financial information, needs to be adequately processed, analyzed and summarized to make it easy to disseminate and consume by the end-users @xcite . the financial narrative summarization task focuses on summarizing publicly available annual financial reports produced by uk firms listed on the london stock exchange @xcite . these reports are quite lengthy (average length about 80 pages), with some reports spanning more than 250 pages, which makes this task quite challenging. these reports broadly consist of two main sections :i) "narrative sections" also known as "front-end sections": part of the report which contain textual information and reviews by the firm's management and board of directors and ii) "back-end sections": sections containing financial statements in terms of tables and numbers. this task focuses on finding effective narratives sections and then performing the summarization over these narrative sections to generate a summary of about 1000 words. for more details about the task kindly refer @xcite . we address this task by exploring the approaches which have shown to perform well for document summarization. we investigate page-rank based techniques to find effective summaries in an unsupervised fashion, which have shown to perform well in past @xcite @xcite . given the recent advancement in exploring deep learning techniques to extract effective summaries we explore a bi-directional lstm @xcite based approach for extracting top sentence for a given document and generate summaries using the top-k ranked sentences. we combine complementary information captured using unsupervised and bi-lstm based supervised models along with lead-k sentences approach @xcite , to generate effective summaries (more details provided in section 3). this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . the paper is organized as follows: section 2 describes the dataset details and the evaluation mechanism used, section 3 discusses the approaches explored in this work, section 4 presents the result of our experiments, we describe our main findings and conclude in section 5. ## dataset & evaluation mechanism the dataset provided by the task organizers consist of a training and a validation set consisting of 3000 and 363 financial documents respectively, as shown in the automatically generated summaries are evaluated against the gold summaries using the evaluation metric rouge @xcite , commonly used for summarization tasks. rouge-1, rouge-2, rouge-l, rouge-su4 are the four variants of rouge which are used for evaluating the submitted systems on the test set. rouge-l indicates longest common subsequence statistics, rouge-1 indicates overlap of unigram, rouge-2 indicates overlap of bigram and rouge-su indicates overlap of skip-bigram plus unigram statistics between the generated and the gold summary. ## methodology we describe three types of approaches that we explored for addressing the financial narrative summarization task. unsupervised approach: within the unsupervised approach we explored three methods namely leadk @xcite , textrank @xcite , and lexrank @xcite for performing extractive summarization. lead-k: in general, news documents tend to contain the most informative content at the top and are further detailed. in this method the top k sentences from a document, based on the order of their occurrence are extracted and combined to form a summary. this method has proved to be quite robust and have been used as a good baseline for the task of extractive summarization @xcite @xcite . textrank and lexrank: the initial work on pagerank @xcite for crawling web pages, motivated the idea of textrank and lexrank, which are graph based approaches for performing extractive summarization of documents. in graph based approaches all the sentences in a document are represented as the vertices of a graph where the edges are vertex (sentence) to vertex (sentence) similarity, and hence weighted. the weights of the edges are calculated using the textual similarity between the sentences. top k salient sentences are combined to form a summary. for a given document, a connected graph of the sentences in constructed and then the salient sentences from the graph are extracted as shown in equation 1 and equation 2. s i and s j are two sentences comprising of words w i , w i+1 .., w n in equation 1. in equation 2, w s(v i ) represent a weighted score for a vertex i, in(v i ) and out(v i ) indicates in-degree and out-degree scores respectively, and d is a damping factor having a value between 0 and 1. textrank approach looks at the absolute number of words two sentences have in common, which are then normalized by the sentences length. whereas, lexrank calculates cosine similarity of the word vectors for both the sentences as shown in equation 3. each sentence is represented as a n dimensional vector using a bag-of-words based model, where n represents the number of unique words in the document. in equation 3, x and y represents the n dimensional sentence vector for sentences s i and s j respectively, and tf indicates the term frequency and idf indicates the inverse document frequency for a word w. idf -modif ied -cosine(x, y) = @xmath0,y tf w,x tf w,y (idf w ) 2 we used the sumy tool foot_0 for performing lexrank based summarization. we used the gensim @xcite implementation of textrank algorithm from @xcite . this revised version of textrank performs better as compared to the original version proposed in @xcite . for more details on these models, we advise readers to refer @xcite @xcite . supervised approach: extractive summarization focuses on finding good representative sentences from a document to represent document summaries. thus the summarization task can be approached as a binary level sentence classification task where sentences which form a part of document summaries are positive samples and rest being categorized as negative samples. thus given a new document we have to classify sentences as positive and negative instances. top k positive sentences in their chronological occurrence are considered as document summaries. we experimented with a bi-directional lstm (bi-lstm) based approach for sentence extraction for generating document summaries by combining top-k ranked sentences. we used the training dataset which has about 350k positive samples and we randomly sample a similar number of negative samples from the training set. similarly a validation dataset was created which has about 100k sentences. both the training and validation set have equal numbers of positive and negative samples. we use the pytorch library @xcite for bi-lstm based method, with following parameters: number of hidden units set to 48, loss function used is binary cross entropy loss, we used fasttext textual embeddings @xcite , @xmath1. we perform dropout to avoid overfitting of the model. ensemble approach: we find quite promising results using lead-k, textrank and lexrank based unsupervised approaches and a bi-lstm based supervised approach on the training and validation set (described later in section 4). as described above that the lead-k sentences based summaries has been used as a strong baseline for the summarization task so we performed an ensemble approach where we combine the output of i) textrank, ii) lexrank, iii) lead-k and iv) bi-lstm approaches. for a given document we sort and rank the sentences using these four approaches in descending order, we take the top 20 sentences from each of these approaches and combine the output using an ensemble technique. we chose 20 sentences empirically, as the length of summary is constrained to 1000 words only, so all the sentences occurring lower in the order do not impact the rouge scores. this ensemble model is a linear combination of the output of multiple approaches, where the sentences that have occurred in the output of most approaches are ranked higher. in case of the same frequency of the sentences, preferences are given to the following models' output: bi-@xmath2, determined empirically based on the training set. a detailed worked out example of the ensemble approach is shown in figure 1 . we used stanford corenlp pipeline @xcite to perform sentence splitting for each document before applying all three approaches described above. a post-processing step is performed after generating document summaries from these three methods (unsupervised, supervised and ensemble), where all the textual information beyond 1000 words is removed, as per the task requirement. ## results & analysis results of different unsupervised approaches such as textrank, lead-k are shown in our second system submission is a bi-lstm based binary classification model referred to as am ex -bilst m . our final submission is an ensemble model referred to as am ex -ensemble. overall ranking of the systems varies depending on the evaluation metric being considered. based on rouge-l and rouge-1 metrics our system submissions achieved quite high precision but scored lower on recall values hence the f-score are relatively averaged. in our ensemble based approach, we combined the output of multiple approaches by a linear combination, in future we will like to explore learning effective weights while combining the output of multiple models, to generate effective summaries which can possibly address the problem of low recall, by including potential good candidates occurring lower in the order. ## conclusion & future work we present our work on an initial investigation of extractive summarization for annual financial reports. we explored alternative approaches using unsupervised, supervised and an ensemble based method. we find that bi-lstm based supervised approaches perform relatively better than using unsupervised based approaches such as textrank. the ensemble based model performs best as compared to supervised and unsupervised models and obtained a rank of 9/31 on the test set using rouge -l evaluation metric. we used a bi-lstm based model for extracting good representative sentences to be included in the document summary. however we lose the document level information while treating the problem of extracting good representative sentences as a binary class classification problem while training the model. in future we will like to explore recent models @xcite for performing extractive summarization. these models leverage bert based distributed representation @xcite and train summarization models by optimizing the rouge scores, to generate effective document summaries.