Learning to Compare Financial Reports for Financial Forecasting (2024)

Abstract: Public companies in the US are required to publish annual reports that detail their recent financial performance, present the current state of ongoing business operations, and discuss future prospects. However, they typically contain over 25,000 words across all sections, large amounts of industry and legal jargon, and a high percentage of boilerplate content that does not change much year-to-year. These unique characteristics present challenges for many generic pretrained language models because it is likely that only a small percentage of the long report that reflects salient information contains meaningful signal about the future prospects of the company. In this work, we curate a large-scale dataset of paired financial reports and introduce two novel, challenging tasks of predicting long-horizon company risk and correlation that evaluate the ability of the model to recognize cross-document relationships with complex, nuanced signals. We explore and present a comprehensive set of methods and experiments, and establish strong baselines designed to learn to identify subtle similarities and differences between long documents. Furthermore, we demonstrate that it is possible to predict company risk and correlation solely from the text of their financial reports and further that modeling the cross-document interactions at a fine-grained level provides significant benefit. Finally, we probe the best performing model through quantitative and qualitative interpretability methods to reveal some insight into the underlying task signal.

## introduction investors are faced with the consumption of a myriad of textual datasets relevant to financial markets, spanning genres such as news, social media posts, and financial reports. public companies in the us are required to publish annual reports detailing the current operations of the firm, recent financial performance, and discussing future prospects. @xcite annual report -2015 our operations and facilities are subject to extensive federal, state and local laws and regulations relating to the exploration for, and the development, production and transportation of, oil and natural gas, and operating safety... our operations and facilities are subject to extensive federal, state and local laws and regulations relating to the exploration for, and the development, production and transportation of, oil and natural gas, and operating safety... ## related work in the broader nlp literature, there has been great interest recently in extending the context length of transformer-based language models to be able to efficiently process long documents @xcite @xcite @xcite . these methods attempt to approximate full self-attention with more efficient computation and have been shown to excel at long document understanding tasks. in a related area, long document similarity involves identifying the relationship between two long documents. while semantic similarity has been of interest for a while, most work has focused on short text at the sentence or paragraph-level @xcite . however, semantic similarity at the document-level is more challenging because long documents often contain content spanning multiple topics and relationships between them may exist at different levels. despite the difficulty, the problem has varied applications, including citation recommendation, plagiarism detection, coreference resolution, and multi-document summarization. in @xcite , the authors propose a crossdocument attention component into han @xcite to enable the comparison between documents at different levels. further, in @xcite , the authors consider a similar setting and propose a novel pretraining approach for cross-document language modeling (cdlm) with a dynamic attention mechanism that allows the model to learn cross-document relationships. they demonstrate that their model has a strong understanding of the relationship between documents and delivers sota performance on a variety of multi-document tasks. other methods have attempted to perform an alignment between related documents at the sentence-level for retrieval applications, but typically pretrain encoders in a selfsupervised manner, without finetuning them endto-end on the target task. @xcite . ## problem statement we propose two novel tasks designed to evaluate the ability to recognize subtle similarities and differences between long financial documents that are predictive of long-horizon financial outcomes. it is important to note that since the reports occur at an annual frequency, we choose target variables at the 1-year horizon, which produces a lot of uncertainty between the forecast and outcome date, and makes these long-horizon prediction tasks particularly challenging. we also believe that the long-horizon requires the ability to capture more intricate, subtle signals than similar short-horizon tasks. in addition, this choice is consistent with prior work @xcite @xcite and the premise from @xcite that the text-based signal contained in these reports is related to business risk that potentially can take up to multiple quarters to materialize on company performance. ## data 4.1 data acquisition to curate the dataset, we download preprocessed html files of company filings from the notre dame software repository for accounting and finance @xcite . we focus our analysis on section 7a: management discussion and analysis (mda) section from annual reports of us-based public companies. according to the sec, this section is intended to provide management's perspective on the business results of the past year and their future prospects for the upcoming year, including information about key business risks. while there are other sections, we choose to focus on the mda because it reflects a direct communication from company management to shareholders. we use a variety of regular expressions to extract the mda section and filter the resulting section text for quality in a refined iterative process. we source stock price data from factset prices & returns api. please see appendix a for further details on the data curation process. ## methods we explore a comprehensive set of baselines on these novel tasks that range from simple bag-ofwords based methods to well-tailored state-of-theart document-level and sentence-level transformerbased models, including both generic and domainadapted versions of each. ## experimental results and analysis the results in in general, we find that the sentence-level methods generally perform better than the document-level methods, which we conjecture is because by they allow for a more fine-grained interaction between the document sentences before any document-level pooling. we find this effect to be more pronounced on the correlation prediction task, especially when the longformer base model is not pretrained for semantic similarity. this suggests that despite the extensive, language modelingbased pretraining process of the longformer model, it does not produce strong document embeddings without finetuning. however, we find that our long context adaptation of the diffcse pretraining framework for the longformer is well-suited for generating finegrained document embeddings, suggesting that this is a promising direction for future work. relatedly, we find that pretrained models not adapted to the financial domain or pretrained with semantic similarity objectives struggle to learn the subtle task signals. however, we observe a significant improvement across most models after dapt, suggesting that the task requires a nuanced understanding of financial language. for both tasks, we find that a simple multi-modal model cdsa-findiffcse + ar(1) improves performance, particularly for the correlation prediction task which exhibits stronger autocorrelation. we conjecture their complementary nature is partly due to the fact that historical market patterns captures the persistence of past behavior while the textbased models identify the catalyst that causes novel behavior, suggesting the text-based methods could serve as a valuable tool to augment traditional risk management practices. however, we leave it to future work to explore more sophisticated methods to incorporate tabular data into text-based models. ## conclusion we curate a large-scale corpus of paired annual financial reports and introduce two novel benchmarks that require modeling complex, crossdocument interactions between long documents. we methodically investigate a comprehensive set feb 2015 mar apr may jun jul date 1.2 1.0 0.8 0.6 0.4 0.2 0.0 0.2 cumulative return crk stock 10k publication date figure 2: stock price of crk following the publication of the 2015 annual report that was identified by the best performing model as high risk based off changes from the 2014 annual report.