Beyond Surface Similarity: Detecting Subtle Semantic Shifts in Financial Narratives (2024)

Abstract: In this paper, we introduce the Financial-STS task, a financial domain-specific NLP task designed to measure the nuanced semantic similarity between pairs of financial narratives. These narratives originate from the financial statements of the same company but correspond to different periods, such as year-over-year comparisons. Measuring the subtle semantic differences between these paired narratives enables market stakeholders to gauge changes over time in the company’s financial and operational situations, which is critical for financial decision-making. We find that existing pretrained embedding models and LLM embeddings fall short in discerning these subtle financial narrative shifts. To address this gap, we propose an LLM-augmented pipeline specifically designed for the Financial-STS task. Evaluation on a human-annotated dataset demonstrates that our proposed method outperforms existing methods trained on classic STS tasks and generic LLM embeddings.

## introduction in accordance with the u.s. securities and exchange commission (sec) regulations, publicly listed companies are mandated to disclose financial reports. these reports, carefully prepared by the companies, offer a wealth of information about their business operations and financial performance. a substantial amount of natural language processing (nlp) research has focused on this rich dataset to extract insights beneficial for investors and regulators @xcite @xcite @xcite @xcite . an intriguing aspect of corporate financial reporting is the subtle variation in language used to convey information. in corporate communication, companies deliberately select nuanced wording in their communications. for instance, one company's report states, we report a year of strong performance, with revenues exceeding our targets. our innovative strategies have driven substantial market growth. the following year, another statement from the company reads, we report a year of solid performance, with revenues meeting our targets. our innovative strategies have led to consistent market share growth. at first glance, these year-over-year statements appear similar. however, a closer analysis reveals significant differences: the first statement suggests rapid expansion, while the second implies a more steady and moderate growth trajectory. numerous anecdotal evidence has shown that a company's choice of words can have a huge impact on the company's stock performance @xcite @xcite . measuring the similarity in financial narratives resembles the classic semantic textual similarity (sts) task @xcite @xcite . however, a significant distinction in the financial narrative sts is that paired financial statements often exhibit a high level of overlap in surface words. while the semantics appear largely similar on the surface, it is crucial to detect subtle semantic differences that are relevant to market stakeholders. therefore, we define this task as the financial-sts task, emphasizing its unique characteristics in detecting subtle semantic shifts within the financial domain. we find that existing pre-trained embedding models or llm embeddings do not perform satisfactorily for the financial-sts task. in a preliminary study, we construct a dataset comprising over four thousand paired financial statements. within this dataset, a significant portion of paired financial narratives demonstrate notable semantic shifts. utilizing openai's ada embedding foot_0 and sentencebert embedding @xcite , we observe that both models yield excessively high similarity scores for the financial narrative pairs, as illustrated in figure 1 . this suggests a deficiency in pre-trained embedding models in discerning nuanced semantic shifts in financial sentences that are superficially similar. consequently, for financialrelated domain tasks, such as comparing year-overyear financial report similarity @xcite , the performance of these pre-trained embeddings is unsatisfactory. this paper proposes a novel method for the financial-sts task. first, we define four types of subtle semantic changes that convey informational content potentially impactful to financial market stakeholders: intensified sentiment, elaborated details, plan realization, and emerging situations. for example, intensified sentiment measures situations in which one sentence employs stronger positive or negative phrases compared to another. this can occur when a company's operations improve or exacerbate. second, inspired by recent nlp advancements using large language models for data augmentation @xcite @xcite @xcite , we prompt large language models (such as chat-gpt and llama-2 @xcite ) to generate financial sentences with no or minimal subtle semantic shifts in one of the four defined categories. next, with the llm-augmented dataset, we train a classical triplet network. this network is capable of distinguishing subtle semantic shift pairs from are-embeddings pairs exhibiting no semantic shift, thus generating meaningful similarity scores for pairs of financial statements. in the evaluation, we manually annotate a dataset with human-judged similarity scores for pairs of financial statements. results show that our method significantly outperforms existing sts approaches trained on classic sts task, such as sentencebert, simcse @xcite , and contriver @xcite , as well as generic llm embeddings such as openai's ada embedding. our research makes two significant contributions. first, we introduce a novel financial nlp task, financial-sts, which focuses on financial sentence pairs that are superficially similar but may differ subtly in semantics. second, we present a comprehensive pipeline designed to effectively detect subtle semantic shifts in financial narratives. this pipeline, accompanied by datasets comprising both llm-augmented sentence pairs and a manually annotated dataset, will be made publicly available to facilitate further research and application in this field. ## sts in financial narratives in the realm of corporate communication, financial documents such as annual reports and press releases play a pivotal role for companies in communicating with capital markets. due to the significance of financial texts, companies carefully craft their narratives. for instance, research in financial economics has shown that managers tend to use more positive words, such as 'tremendous' or 'extremely well', to convey good news. conversely, for negative news, they are inclined to use moderately negative terms like 'limitation', 'unexpected', or 'complexity' @xcite . more importantly, analyzing year-over-year narrative changes, particularly in sentiment and modifier words, enables investors to accurately assess fundamental shifts in a company. @xcite ; @xcite find that annual modifications in company reports, especially sentiment shifts, correlate with the company's future stock returns and trading volume. hence, evaluating the semantic similarity between financial narratives is an essential yet under-explored task. we identify and summarize four categories of semantic shifts, focusing on those that may potentially provide informational content for stakeholders. in addition to the four identified types of @xcite sentence (year: 2019) intensified sentiment the company is subject to laws and regulations worldwide, changes to which could increase the company's costs and individually or in the aggregate adversely affect the company's business. the company is subject to complex and changing laws and regulations worldwide, which exposes the company to potential liabilities, increased costs and other adverse effects on the company's business. ## proposed pipeline as demonstrated in figure 1 , existing pretrained embedding models fall short in measuring the subtle semantic shifts in financial statements for two main reasons. first, financial statements often overlap in surface words, yet the semantic shifts in these statements occur in sophisticated ways, posing a challenge. for example, a shift might involve a change from a modal verb phrase such as likely to affect revenue to a past participle like affected revenue. second, existing pretrained embedding models, such as sentencebert @xcite , are fine-tuned on classic sts benchmarks @xcite , which are not adequately suited for the financial-sts task. in this section, we propose a pipeline for addressing the financial-sts task, as shown in figure 2 . in essence, we utilize large language models (llms) to generate an augmented dataset, in which examples exhibit different types of semantic shifts as well as instances of no semantic shift (section 3.1). then, we use the llm-augmented dataset to train a classic triplet network capable of differentiating between pairs of subtle semantic shifts and pairs showing no semantic shift (section 3.2). ## the finsts dataset in this section, we present the finsts dataset, which comprises two distinct subsets: an llmaugmented dataset and a human-annotated dataset. we use the llm-augmented dataset to train the triplet network and use the human-annotated dataset for evaluation. ## evaluation of financial-sts task in this section, we test our model performance on financial-sts task. we use 85% of examples in llm-augmented finsts for training. we use a batch size of 64, margin of ϵ = 0.2, adam optimizer with a learning rate of 2e-5, and implement linear learning rate warm-up over 10% of the training data. the pooling strategy we choose is mean pooling. ## conclusion in this paper, we study a new financial domain nlp task that measures the subtle semantic shift between a pair of financial narratives with high surface similarity, which we call financial-sts. we find that existing pretrained embedding models fall short in discerning the nuanced semantic shifts between these narratives. as a result, financial market practitioners face challenges in accurately gauging a company's financial and operational changes. to address this problem, we identify four types of subtle semantic shifts commonly occurring in companies' financial narratives. based on these identified shifts, we prompt a large language model to generate sentence pairs that exhibit subtle or no semantic shift. we then use the llm-augmented dataset to train a classic triplet network capable of measuring the semantic similarity between pairs of financial narratives. experimental results on a human-annotated dataset demonstrate that our proposed approach outperforms existing pre-trained embedding models specialized for the classic sts task, as well as llm embeddings such as openai's ada. by introducing this task and publicly releasing a new finsts dataset, we aim not only to establish a new benchmark for evaluating financial domain llms @xcite but also to complement the classic sts task, thereby fostering further advancement in this area.