Template-aware Attention Model for Earnings Call Report Generation (2021)

Abstract: Earning calls are among important resources for investors and analysts for updating their price targets. Firms usually publish corresponding transcripts soon after earnings events. However, raw transcripts are often too long and miss the coherent structure. To enhance the clarity, analysts write well-structured reports for some important earnings call events by analyzing them, requiring time and effort. In this paper, we propose TATSum (Template-Aware aTtention model for Summarization), a generalized neural summarization approach for structured report generation, and evaluate its performance in the earnings call domain. We build a large corpus with thousands of transcripts and reports using historical earnings events. We first generate a candidate set of reports from the corpus as potential soft templates which do not impose actual rules on the output. Then, we employ an encoder model with margin-ranking loss to rank the candidate set and select the best quality template. Finally, the transcript and the selected soft template are used as input in a seq2seq framework for report generation. Empirical results on the earnings call dataset show that our model significantly outperforms state-of-the-art models in terms of informativeness and structure.

## introduction earnings calls, conference calls held by public companies to disclose their performance of a specific period, are key resources in providing signals for financial analysts' decision-making process. analysts, investors, and the mass media can learn about a company's financial results, operation details, and future guidance by listening to these conference calls. previous works have highlighted the importance of earnings calls in modeling analysts' behavior @xcite . as nowadays firms disclose more and more information @xcite , earnings call transcripts are usually longer and contain more information than before, resulting in challenges in efficiently analyzing these unstructured documents and detecting informative facts. some financial analysts write well-structured reports (figure 1 ) @xcite for earnings calls after attending the event or reading the transcript. however, writing such reports usually takes time and effort. in addition, reports are not available for every company and event. therefore, generating earnings reports quickly and automatically can fill the gap for no-report-available conferences and strongly accelerate the research process in the financial industry. to fulfill this goal, we aim to develop an effective text summarization system to automatically generate reports with a hierarchical structure. text summarization @xcite , as an important field of natural language processing, attracts considerable interest from researchers. various datasets for summarization tasks have been built, most of which contains small to middle-sized document and short summary, e.g., cnn/daily mail @xcite , wikihow @xcite , reddit @xcite , etc. researchers design innovative architectures and benchmark the model performance on these mainstream datasets, yet extending summarization framework to the domain of earnings call transcripts, has never been explored. in contrast to popular summarization tasks, structured earnings report generation has several challenges due to the special properties of data. first, earnings call conferences usually take a few hours, and transcripts contain thousands of words. this feature makes it impossible for popular pre-trained models such as bert @xcite and bart @xcite to provide high-quality summaries since these approaches partition the long documents into smaller sequences within 512 tokens to meet the input limit, resulting in loss of cross-partition information. second, generated reports are required to be clearly organized and well-formatted. in addition to summarization, it is important for the model to recognize and output the explicit logical structure of the earnings call presentation. to the best of our knowledge, the structure quality of generated summaries from lengthy documents, in terms of logic and format, has rarely been examined in literature. in this paper, we formally conceptualize report generation as an extension of summarization tasks and propose a novel approach, template-aware attention model for summarization (tatsum), to produce hierarchically-structured reports. inspired by traditional template-based summarization @xcite , and soft template-based sentence summarization @xcite , we use historical reports as soft templates to provide supplemental structure information for a summarization system. we use soft templates as they do not enforce an actual rule in the generated summaries. to deal with the long sequence problem, we leverage the advantage of long-documents-transformer (longformer) @xcite , which reduces the complexity of the self-attention mechanism in transformers @xcite and allow for longer input sequences. we collect historical earnings call transcripts and reports, and divide them into speaker sections with fewer words. the combination of a transcript section and a report section serves as an individual data point in the corpus. our proposed framework consists of three modules as illustrated in figure 2 : (i) candidate generation, which generates a set of potential soft template candidates for a transcript section, (ii) candidate ranking, which ranks candidates through a siamese-architected @xcite longformer encoder and selects the candidate with the highest rank as the final soft template for the transcript, and (iii) report generation, which generates the report using the soft template together with the raw transcript through a longformer-encoder-decoder (led) model @xcite . figure 1 illustrates the structure of reports generated by our algorithm. we evaluate the proposed framework on hundreds of earnings call events. experiments show that tatsum significantly outperforms the stateof-the-art summarization models in terms of informativeness, format, and logical structure. besides, extensive experiments are conducted to analyze the effect of different components of our framework on the performance of the model. the contributions of this work are summarized as follows: the paper is organized as follows: in section 2, we review the relevant prior literature. section 3 presents the novel architecture of tatsum. we conduct extensive experiments on the earnings call dataset and analyze the results in section 4. section 5 concludes the paper and provides future directions. ## related work earlier studies of neural abstractive summarization employ encoder-decoder architecture to generate a shorter version of a sentence @xcite . @xcite extend previous work to summarize documents with more than one sentence using hierarchical attention. a variety of studies focus on building advanced attention mechanism for better summarization, e.g., convolutional attention @xcite , graph-based attention @xcite , bottom-up attention @xcite , etc. @xcite propose a hybrid pointer-generator network and a coverage mechanism to keep track of already-summarized words. @xcite introduce a deep reinforced model with a novel intra-attention mechanism and show improved performance for long document summarization. recently, pre-trained language models, which are trained to learn contextual representations from large-scale corpora, have been proved to be successful in summarization tasks. popular pretrained models like bert @xcite and bart @xcite ) are adopted to build summarization-specific architecture. bertsum @xcite proposes a novel documentlevel bert-based encoder and an auto-regressive decoder with trigram blocking techniques and shows strong performance in both extractive and abstractive summarization. @xcite integrate bart with the robust representations through regularized finetuning (r3f) method to perform better fine-tuning for pre-trained models and achieve the state-of-the-art performance on cnn/daily mail. longformer @xcite ) significantly reduces the time and space complexity of the attention mechanism and allows for much longer input sequences. it achieves this goal by replacing the self-attention in traditional transformers @xcite with windowed attention and introducing new task-oriented global attention. longformer-encoder-decoder (led) @xcite , a variant of longformer, is also introduced for supporting long document seq2seq tasks. led-large 16k, a bart-pretrained led model with no additional pretraining, outperformed bigbird summarization @xcite , a modified transformer for long sequences with pegasus pretraining @xcite , and achieved the state-of-the-art performance on arxiv dataset @xcite . in this paper, led is adopted as the base model and a strong baseline benchmark. in the domain of earnings calls, there is limited work exploring the potential of applying text summarization techniques. @xcite generate an automatic summary for earnings releases using off-the-shelf unsupervised summarization methods such as klsum @xcite , lexrank @xcite , etc., and conduct experiments to analyze the impact of automatic and management summaries on the investors' judgment. however, comprehensive experiments on the performance of summarization techniques are missing. in this work, we develop a novel summarization algorithm for report generation, provide extensive experiments on the earnings call dataset, and compare it with the state-of-the-art models in the literature. an important feature of our system is to generate well-structured and formatted reports. templatebased summarization @xcite is a traditional technique to summarize sentences. with a manually designed incomplete sentence template, the method fills the template using some input text, following pre-defined rules. this method can guarantee that the output sentence follows a specific format. however, constructing templates for long documents and large-scale datasets still remains challenging and requires domain knowledge. @xcite extended the template-based summarization and introduced a soft template, which is a summary sentence selected from the training set, to resolve this issue. re3sum @xcite selects the soft template through an information retrieval (ir) platform and jointly learns template quality as well as generates the summary through a seq2seq framework. in this paper, we select historical reports from the corpus and form candidate sets. to further improve template quality, our candidate ranking module is inspired by matchsum @xcite . @xcite formulates extractive summarization as a semantic text matching problem, and architect a siamese-bert network with margin-ranking loss to select the best candidate summary. ## method our goal is to generate hierarchically structured reports from long documents. the automatic generation system, tatsum, consists of three mod-ules: candidate generation, candidate ranking, and report generation. given an earnings call transcript, we divide it into different sections and consider each section as an input sequence t . this helps in the tractability of the summarization process by considering fewer tokens and results in well-structured reports since each section usually follows a coherent structure that is different from others. similarly, human-written reports are divided into sections, r, and mapped to the document sections in the training phase. for each document section t , the candidate generation module filters a candidate set of top i soft templates c t := {r 1 , • • • , r i } from a built corpus. we rank the candidate set c t using the candidate ranking module to select the best soft template rt to use. finally, in the report generation module, the best soft template rt and the raw document section t together are encoded into hidden states. a decoder takes the combination of encoded hidden states as well as decoder inputs to generate the abstractive report. figure 3 illustrates candidate ranking and candidate generation modules. the three modules will be described in detail in the following subsections. ## experiment we evaluate tatsum on earnings call reports. we aim to answer the two following questions. ## conclusions this paper proposes an innovative neural summarization system, tatsum, with three modules, candidate generation, candidate ranking, and report generation, to generate structured reports automatically. in candidate generation module, we build a corpus of historical documents and reports, and for each document, we generate a candidate set using quick and easy similarity-based criteria. the candidate set is then ranked in the candidate ranking module, following the predicted result of an encoder model with margin-ranking loss. we choose the candidate with the highest rank as the soft template. in the final report generation module, we encode both template and document into hidden states and feed the combined hidden states into a decoder to generate the report. extensive experiments are conducted on the earning call dataset and show that our model can generate reports with high informativeness (rouge) and high accuracy (numbers, trends, etc.). we also prove that adding a template can significantly improve the quality of the generated report, and finely selecting a template with good quality can increase performance even more. we mainly test tatsum on automatic report generation for earnings call events. however, the advantage of longformer architecture for long sequence tasks, as well as the significant power of adding soft templates for structured document generation, can extend our proposed framework to various domains, e.g., medical report, employee annual review, call center record, etc. we would like to take advantage of this proved architecture to explore more potential in structured report generation.