Hierarchical summarization of financial reports with RUNNER (2020)

Abstract: With the constantly growing amount of information, the need arises to automatically summarize this written information. One of the challenges in the summary is that it’s difficult to generalize. For example, summarizing a news article is very different from summarizing a financial earnings report. This paper reports an approach for summarizing financial texts, which are different from the documents from other domains at least in three parameters: length, structure, and format. Our approach considers these parameters, it is adapted to hierarchical structure of sections, document length, and special “language”. The approach builds an hierarchical summary, visualized as a tree with summaries under different discourse topics. The approach was evaluated using extrinsic and intrinsic automated evaluations, which are reported in this paper. As all participants of the Financial Narrative Summarisation (FNS 2020) shared task, we used FNS2020 dataset for evaluations.

## introduction the area of text summarization exists for several decades, since the first work of luhn @xcite . since then, the summarization approaches evolved from simple and straightforward extractive unsupervised approaches to abstractive supervised methods, using deep learning language models @xcite . however, the most advanced seq2seq models (transformers) are very limited in input size and, therefore, are inapplicable to long texts. also, only few of state-of-the-art summarizers consider hierarchical structure of the input documents @xcite , their key concepts @xcite or topics @xcite and build a hierarchical summary @xcite . usually, hierarchical summary is built per document collection. the top level of hierarchy provides a general overview and users can navigate the hierarchy to drill down for more details on topics of interest. there is a growing interest in the application of automatic and computer-aided approaches for extracting, summarising, and analysing both qualitative and quantitative financial data, as a series of fnp and related workshops @xcite recently demonstrates. however, summarization of documents in financial domain is usually limited to summarization of financial news @xcite @xcite @xcite which are not very different from the general news in length and format. only few attempts were made to summarize financial reports @xcite , which are different from the news articles in at least four parameters: length, structure, format, and lexicon. this paper reports an approach for hierarchical summarization of financial reports. financial annual reports in the data of financial narrative summarisation (fns 2020) shared task 1 (el-haj et al., 2020) are long, have many sections, and are written in "financial" language using many special terms, numerical data, and tables. our system for hierarchical summarization financial reports (shortly runner) considers discourse and topic hierarchical structure and builds a hierarchical view of the summarized this work is licensed under a creative commons attribution 4.0 international license. license details: http://creativecommons.org/licenses/by/4.0/ . text preprocessing, that includes text cleaning, sentence splitting and tokenization. we developed our own tool that cleaned text before segmenting it to sentences and tokens. financial reports usually contain a lot of sections, figures, and tables. because the text files in the fns-2020 dataset were obtained by converting pdf files to plain texts, these texts contain a lot of "noise" left from broken tables and meta-data such as section and page numbers. we cleaned the noise by measuring the ratio between text and numbers and ratio between number of words and whitespaces. lines with ratio less than 0.4 were removed. then, regular expressions were applied to find and mark such entities as url, phone number, date, time, email. finally, non-unicode characters were filtered out. figure 2 demonstrates the example of text before and after preprocessing. section segmentation, where section headers are identified and a document is segmented into sections. the section titles were extracted following the heuristic rules saying that (1) each title appears in a separate line, (2) does not end with period mark, and (3) contains only few (up to 5) words with (4) each word either starting with capital case letter or containing only upper case letters. the extracted candidates were then compared against the list of 13 manually edited titles foot_0 . the candidate that obtained jaccard similarity above 0.4 to one of the titles from the list was extracted as a title. the text body between two consequent titles was marked as a section. discourse parsing of each section. for discourse parsing we used the codra parser @xcite . codra parser performs two-part process: (1) a discourse segmenter creates a segmentation analysis on the sentence level and edu's for the discourse parsing process and (2) a discourse parser parses the text on sentence level and document level to identify relations between parts of sentences and sentences in the document. figure 3 shows an example discourse tree. leaf node stands for a sentence or a part of a sentence. the rhetorical analysis of the parser starts from a breaking a text into elementary discourse units (edus). because edus do not span across multiple sentences, this segmentation task finds edus inside the sentence boundaries. as a result, some sentences (actually, most, according to our observations) are split into edus. every edu is marked as a nucleus (an essence part) or a satellite (a complementary part of the related nucleus), based on the relation that they are connected to. internal (relation) nodes represent different inter-sentence relations: elaboration, same-unit, etc. figure 3 : discourse tree for sentences from "remuneration report" section of document 17941.txt in fns-2020 dataset. topic modeling. for topic modeling we applied latent dirichlet allocation (lda) model @xcite . it was applied on all files in the fns-2020 dataset with predefined number of topics foot_1 . topic-to-text assignment, where each sentence (or sentence part) represented by a leaf node of the discourse tree, is assigned to one of the topics obtained by lda. we refer topic probabilities p(t|w) for all sentence s words @xmath0. therefore, we extract a dominant topic (@xmath1, as a topic with the maximal normalized sum of topic probabilities for all sentence words @xmath2: max @xmath3. figure 4 shows an example of a topic-to-sentence assignment. topic distribution smoothing. we noticed that after single text nodes (that stand for sentences or sentence parts) are assigned to topics, we can get unexpected topic distribution where two parts of the same sentence or two adjacent sentences inside the same paragraph and/or belonging to the same discourse relation are assigned to different topics, and transition from one topic to another is not coherent. 4 we decided to smooth topic distribution by extrapolating one dominant topic on entire block of adjacent sentences and sentence parts, connected by a direct discourse relation. we denote nodes with at least one leaf node as "simple" (see figure 5 ) and all leaves in its sub-tree are finally assigned to one dominant topic, so that a "random" noise is left out. the implement this approach as follows. we know that all leaf nodes are arranged in the natural sequential order of their texts from right-to-left (top-down) in a discourse tree. we assume that the important information usually comes first (important part of a sentence usually precedes its complementary part, and a sentence stating some fact usually precedes a sentence that elaborates more about this fact) and, therefore, upper right nodes and nucleuses should propagate their topics on their siblings. according to this assumption and our empirical observations on each parameter's influence, the final impact factor n i of node n is calculated as follows. n i(n) = 3 @xmath4, where: then, the final dominant topic for a "simple" sub-tree is calculated as follows: max @xmath5,n }. after topic-to-sentence assignment (at previous stage), every leaf node has non-zero value for only one dominant topic, other topics have score t,@xmath6. we also experimented with the second strategy of assignment topics to sentences, where we do not assign a dominant topic to each leaf but operate their vectors of topic weights v @xmath7, w t 2 , . . . , w t |t | ), where w t i is a normalized sum of topic t i probabilities for all sentence words, as calculated in previous stage. the dominant topic is assigned to entire sub-tree (under the "simple" node) by summing the topic distribution vectors multiplied by the importance score of their nodes and choosing a topic with a maximal score. formally, the dominant topic is assigned as follows: max @xmath8. based on this distribution and discourse structure, one dominant topic is finally calculated for each "simple" sub-tree. given the initial topic assignment to leaves, the dominant topic of the first (upper) sub-tree is finally assigned to 9. summarization of entire report (regardless visualization) and of each section (for visualization needs) was performed by two different greedy strategies: we report the results for both strategies in the experiments section. hierarchical visualization. at this stage runner creates an interactive html file with the data from all the stages for a user to browse. the file contains the following sections: (1) original text; (2) processed xml text after cleaning and section segmentation; (3) discourse trees for all the sections; (4) sentences (nodes) with assigned topics after smoothing; (5) the final hierarchical tree with the section summaries, and (6) a general report summary. for visualization and interactive user's navigation, the following tree structure of a document is built and present to a user: root represents an entire document and points to its sections, each section is split to major topics inside this section after smoothing, and each topic points to a summary of this particular section focused on the chosen topic. visualization is performed in interactive manner, upon a user's request. figure 6 shows an example of such a tree. demo video 5 demonstrates all interactive options provided by the system. ## conclusions and future work this paper describes a new method for hierarchical summarization of financial reports, based on integrating the discourse structure and topic modeling. in future, we intend to apply this method and its extension to educational materials, which also have highly hierarchical structure and an evolving flow of topics in a discourse. hierarchical summarization can help to organize those materials in a hierarchical structure and provide users with interactive navigation to the topics of interest. runner's source code is available 9 and can be run using the provided instructions 10 . srib2020