Numerical Claim Detection in Finance: A New Financial Dataset, Weak-Supervision Model, and Market Analysis (2024)

Abstract: In this paper, we investigate the influence of claims in analyst reports and earnings calls on financial market returns, considering them as significant quarterly events for publicly traded companies. To facilitate a comprehensive analysis, we construct a new financial dataset for the claim detection task in the financial domain. We benchmark various language models on this dataset and propose a novel weak-supervision model that incorporates the knowledge of subject matter experts (SMEs) in the aggregation function, outperforming existing approaches. We also demonstrate the practical utility of our proposed model by constructing a novel measure of *optimism*. Here, we observe the dependence of earnings surprise and return on our optimism measure. Our dataset, models, and code are publicly (under CC BY 4.0 license) available on GitHub.

## introduction earnings conference calls are a quarterly event where the company's top executives provide performance reports of the company over the last quarter (3 months). between the two earnings calls analyst from various financial institutions analyze and provide earnings estimates and recommendations. for example, @xcite has documented that there is a significant stock market reaction to analysts' recommendations @xcite . recent insights, such as those presented by @xcite , reveal that retail investors, often perceived as unsophisticated, exhibit responsiveness to analysts' projections, underscoring the pivotal role of analysts' reports in informing market participants. however, analyst ratings can be biased @xcite correspondence to agam shah {ashah482@gatech.edu} † these authors contributed equally to this work 1 https://github.com/gtfintechlab/ fin-num-claim . a key component of this paper is the identification of numeric financial sentences. specifically, numeric financial sentences include a financial term, a numeric value, and either a currency or percentage symbol. @xcite first introduced the categorization of sentences into 'inclaim' and 'out-of-claim' specifically in the mandarin language. expanding on their foundation, we define an 'in-claim' sentence as one presenting a speculative financial forecast. conversely, an 'out-of-claim' sentence presents a numerical statement about a past event, transitioning from a mere claim to a confirmed fact. for clarity, 'in-claim' sentences can also be termed "financial forecasts" whereas 'out-of-claim' can be labeled as "established financials." every numeric financial sentence that is not a speculative financial forecast (inclaim) is then identified as an 'out-of-claim' sentence. figure 1 illustrates the identification of numeric financial sentences as well as distinguishing between "in-claim" and "out-of-claim" sentences. a major challenge for building or training predictive models is the scarcity of labeled data @xcite . supervised learning often involves a significant amount of manual labeling of data which is often infeasible for large datasets. in such scenarios, one can leverage weaksupervision-based learning methods @xcite or fine-tune the pre-trained language model. weak-supervision is a process that leverages slightly noisy or imprecise labeling functions (lfs) to label vast amounts of unlabeled data @xcite . the strength of the weak-supervision model lies in these imperfect labels, when combined, producing improved predictive models @xcite . however, a crucial component involves the development of effective lfs for a given raw dataset systematically rather than manual annotation @xcite . the aim of our work is to derive financially significant information from the quarterly analyst reports and earnings calls by categorizing each numerical sentence as in-claim or out-of-claim. our major contributions through this paper are the following: nlp in finance finance is one of the most attractive domains for the application of nlp. araci (2019) and liu et al. (2020) presented pre-trained language models for the finance domain. there are multiple datasets specifically catered for applications of nlp in finance including question answering dataset created by chen et al. (2021) and maia et al. (2018), and also a ner dataset constructed by shah et al. (2023b) for the financial domain. there is a vast body of literature on undertaking sentiment analysis tasks on financial data(maia et al., 2018; malo et al., 2014; day and lee, 2016; akhtar et al., 2017). works of li et al. (2020) and sawhney et al. (2020) were centered around predicting volatility using earnings call transcripts in the domain of risk management. chava et al. (2022) measure the firm level inflation exposure by fine-tuning roberta (liu et al., 2019), while li et al. (2021) leveraged word-embeddings to measure the corporate culture. moreover, nguyen et al. (2021) and hu and ma (2021) used multimodal machine learning for credit rating prediction and measurement of persuasiveness respectively. shah et al. (2023a) investigated the impact of monetary policy communication on financial markets. cao et al. (2020) critically examined the evolution of corporate disclosure in recent years, influenced by the rising application of nlp in finance. our research focuses on identifying numerical financial claims from a vast set of english analyst reports and earnings calls using a weaksupervision model. this differs from chen et al. (2020), which targets numeric claim detection in a smaller chinese language dataset. weak-supervision in order to reduce the complexities associated with manual labeling, several standard techniques such as semi-supervised learning @xcite , transfer learning (pan and @xcite , and active learning @xcite have been employed. however, many researchers @xcite and practitioners also employ weak-supervision-based models to further reduce the computational costs while retaining the accuracy of the labeled data. weaksupervision models were primarily developed in a bid to replace standard labeling techniques with models which can leverage slightly noisy or imprecise sources to label vast amounts of data @xcite . techniques such as distant supervision @xcite and crowd-sourced labels @xcite are often associatewd with weaksupervision-based models, however, they tend to have limited coverage and accuracy @xcite . in the case where we have noisy labels from multiple sources available, there have been efforts made to use majority vote, weighted majority vote @xcite , and other label-models @xcite . ## dataset we collect two categories of text and financial market datasets. analyst reports are procured from a proprietary source while earnings call transcripts are collected in a manner that allows us to make the resulting dataset open-source. ## conclusion our work presents claim based labeled dataset in the english language alongside presenting a weaksupervision model with an accuracy of 93%. developed customized aggregation function outperforms baseline aggregation functions. we benchmark various language models and compare the performance with the weak-supervision model. we show the application of claim detection by generating a measure of optimism from the weak-supervision model. we also validate the measure by studying its applicability in predicting earnings surprise, abnormal returns, and earnings optimism. we release our models, code, and benchmark data (for earnings call transcripts only) on hugging face and github. we also note that the trained model for claim detection can be used on other financial texts.