ECTS um: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts (2022)

Abstract: Despite tremendous progress in automatic summarization, state-of-the-art methods are predominantly trained to excel in summarizing short newswire articles, or documents with strong layout biases such as scientific articles or government reports. Efficient techniques to summarize financial documents, discussing facts and figures, have largely been unexplored, majorly due to the unavailability of suitable datasets. In this work, we present ECTSum, a new dataset with transcripts of earnings calls (ECTs), hosted by publicly traded companies, as documents, and experts-written short telegram-style bullet point summaries derived from corresponding Reuters articles. ECTs are long unstructured documents without any prescribed length limit or format. We benchmark our dataset with state-of-the-art summarization methods across various metrics evaluating the content quality and factual consistency of the generated summaries. Finally, we present a simple yet effective approach, ECT-BPS, to generate a set of bullet points that precisely capture the important facts discussed in the calls.

## introduction earnings calls, typically a teleconference or a webcast, are hosted by publicly traded companies to discuss important aspects of their quarterly (10-q), or annual (10-k) earnings reports, along with current trends and future goals that help financial analysts and investors to review their price targets and trade decisions @xcite @xcite . the corresponding call transcripts (called earnings call transcripts, abbreviated as ects) are typically in the form of long unstructured documents consisting of thousands of words. hence, it requires a great deal of time and effort, even on the part of trained analysts, to quickly summarize the key facts covered in these towards this goal, we present ectsum, a new benchmark dataset for bullet-point summarization of long ects. as discussed in section 3.2, first we crawled around 7.4k ects from the motley fool foot_2 , posted between january 2019 and april 2022, corresponding to the russell 3000 index companies foot_3 . reuters was chosen to be the source of our target summaries, per consultation with domain experts, since the expert-written articles posted on reuters effectively capture the key takeaways from earnings calls. however, searching for reuters articles corresponding to the collected ects was especially challenging, since the task was non-trivial. given the fact that not all calls are tracked, after carefully performing data cleaning and addressing pairing issues, we arrive at a total of 2,425 documentsummary pairs as part of the dataset. what makes ectsum truly different from others is the way the summaries are written. instead of containing well-formed sentences, the articles contain telegram-style bullet-points precisely capturing the important metrics discussed in the earnings calls. a sample reference summary from our dataset corresponding to the 2nd quarter 2022 earnings call of apple is shown in second, existing long document summarization datasets such as arxiv/pubmed @xcite , bigpatent @xcite , fns @xcite , and govreport @xcite , have fixed document layouts. ects, on the other hand, are free-form documents with salient information spread throughout the text (please refer section 3.3). hence, models can no longer take advantage of learning any stylistic signals @xcite . third, the average length of ects is around 2.9k words (before tokenization). on the other hand, neural models employing bert @xcite , t5 @xcite , or bart @xcite as document encoders cannot process documents longer than 512/1024 tokens. hence, despite achieving state-of-the-art performances on short-document summarization datasets such as cnn/dm @xcite , newsroom @xcite , and xsum @xcite , etc., such models cannot be readily applied to effectively summarize ects. we benchmark the performance of several representative summarization approaches (section 5.1) from both supervised and unsupervised paradigms, on our newly proposed dataset. among supervised, we select state-of-the-art methods from extractive, abstractive, and long document summarization literature. finally, given the pattern of source transcripts and target summaries, we present ect-bps, a simple yet effective pipeline approach for the task of ect summarization (section 4). specifically, it consists of an extractive summarization module followed by a paraphrasing module. while, the former is trained to identify salient sentences from the source ect, the latter is trained to paraphrase ect sentences to short abstractive telegram-style bullet-points that precisely capture the numerical values and facts discussed in the calls. in order to demonstrate the challenges of the proposed ectsum dataset, competing methods are evaluated on several metrics that assess the content quality and factual consistency of the modelgenerated summaries. these metrics are discussed in section 5.2. we discuss the comparative results of all considered methods against automatic evaluation metrics in section 5.4. given the complex nuances of financial reporting, we further conduct a human evaluation experiment (survey results reported in section 5.5) where we hire a team of financial experts to manually assess and compare the summaries generated by ect-bps, and those of our strongest baseline. overall, both automatic and manual evaluation results show ect-bps to outperform strong state-of-the-art baselines, which demonstrates the advantage of a simple approach. our contributions can be summarized as follows: • we present ectsum, the first long document summarization dataset in the finance domain that requires models to process long unstructured earning call transcripts and summarize them in a few words while capturing crucial metrics and maintaining factual consistency. to the best of our knowledge, fns (el-haj et al., 2020) is the only available financial summarization dataset, released as part of the financial narrative summarization shared task 2020 5 . in fns, annual reports of uk firms constitute the documents, and a subset of narrative sections from the reports are given verbatim as reference summaries. however, ectsum differs from fns on several accounts. first, our target summaries consist of a small set of telegram-style bullet-points, whereas the ones in fns are large extractive portions from respective source documents. second, ectsum has a very high document-to-summary compression ratio (refer section 3.3), because of which the models are expected to generate extremely concise summaries of around 50 words from lengthy unstructured ects, around 2.9k words long. in contrast, the expected length of model-generated summaries on fns is around 1000 words. finally, the models developed on fns are specifically trained to identify and summarize the narrative sections, while completely ignoring others containing facts, and figures that reflect the firm's annual financial performance. excluding these key performance indicators from summaries limits their practical utility to stakeholders. models trained on ectsum, on the other hand, are specifically expected to capture salient financial metrics such as sales, revenues, current trends, etc. in as few words as possible. previously, @xcite had attempted to summarize earnings calls using standard unsupervised approaches. we are however the first to propose and exhaustively benchmark a large scale financial long document summarization dataset involving earnings call transcripts. ## the ect-bps framework we observe some important properties of the reuters reference summaries. they contain a high percentage of word overlap with the source ect documents. however, they are not extractive, rather contain a small set of abstractive bullet-points. it seems as if the analysts writing these summaries first selected some crucial parts of the ect, before compressing them into a bullet-point format. these properties of the reference summaries motivated us to design a two-stage pipeline approach for summarizing ects. our proposed model ect-bps contains two separately trained modules/blocks -(1) an extractive block that is trained to identify the most relevant sentences from the input ect document, and (2) a paraphrasing block that is trained to rephrase the extracted ect sentences to the format of target (reuters) sentences, thereby generating a set of bullet points. figure 2 gives an overview of our proposed architecture. ## experiments and results in this section, we first enumerate the baselines and evaluation metrics. we then describe our experimental setup, followed by a detailed discussion of our main results. we then report the design and results of a human evaluation experiment conducted to manually assess and compare ect-bps-generated summaries with those of competing baselines. we end this section with a qualitative analysis of model-generated summaries. ## conclusion to our knowledge, ectsum is the first large-scale long document summarization dataset in the finance domain. our documents consist of free-form lengthy transcripts of company earnings calls. target summaries consist of a set of telegram-style bullet points derived from corresponding reuters articles that cover the calls. drawing observations from the nature of source transcripts and target summaries, we also propose a simple, yet effective extract-then-paraphrase approach, ect-bps, that establishes state-of-the-art performance over strong summarization baselines across several metrics. ectsum is an extremely challenging dataset given the high document-to-summary compression ratio. moreover, it is highly extendable as future earnings calls are covered by media houses, such as reuters, businesswire, etc. finally, it is a very specialized one which would otherwise have costed a lot of time and resources if one had to hire experts to write the reference summaries. the mere observation that these summaries are created by (expert) analysts and can be leveraged automatically is a major milestone of the paper. we believe our contributions to the dataset and methodology will attract future research in the finance domain. extractive summaries are always expected to contain values that appear in the source text. however, quantity/numeral hallucination is a known problem in abstractive summaries, which prior works @xcite have attempted to reduce. here, we define num-prec. as the fraction of numerals/values in the model-generated summaries that are consistent with the source text. we use this metric to specifically evaluate the precision/correctness with which abstractive summarizers generate values.