Greenback Bears and Fiscal Hawks: Finance is a Jungle and Text Embeddings Must Adapt (2024)

Abstract: Financial documents are filled with specialized terminology, arcane jargon, and curious acronyms that pose challenges for general-purpose text embeddings. Yet, few text embeddings specialized for finance have been reported in the literature, perhaps in part due to a lack of public datasets and benchmarks. We present BAM embeddings, a set of text embeddings finetuned on a carefully constructed dataset of 14.3M query-passage pairs including both public and proprietary financial documents. Demonstrating the benefits of domain-specific training, BAM embeddings achieve Recall@1 of 62.8% on a held-out test set, vs. only 39.2% for the best general-purpose text embedding from OpenAI. Further, BAM embeddings increase question answering accuracy by 8% on FinanceBench and show increased sensitivity to the finance-specific elements that are found in detailed, forward-looking and company and date-specific queries. To support further research we describe our approach in detail, quantify the importance of hard negative mining and dataset scale, and publicly release our embeddings.

## introduction portfolio managers and analysts have access to millions of financial documents. text embeddings are a key component of the information retrieval and retrieval-augmented generation (rag) systems @xcite ) that can help extract insights from this mass of information. however, the financial domain poses unique challenges for text embeddings. financial documents are filled with specialized terminology ('par value', 'stagflation'), jargon ('chinese wall'), curious acronyms ('cagr', 'dcf', 'vix'), and technical terms and company names that collide with ordinary words @xcite . is a cdo similar to a cfo? and what is a greenback bear foot_1 ? despite their importance, few text embeddings specialized for finance have been reported in the literature. to address this gap, we present and publicly release bam embeddings, a set of text embeddings optimized for financial document retrieval. bam embeddings are based on multilingual-e5 @xcite , further finetuned on a carefully filtered, clean dataset of 14.3m query-passage pairs (6b tokens) constructed from 2.8m public and proprietary financial documents. while we cannot release our dataset, we describe in detail our data curation and query generation strategy, finetuning process, and approach to deployment in a high-priority application. on a held-out set of 447k query-passage pairs, bam embeddings achieve recall@1 of 62.8%, far surpassing the multilingual-e5 base model (34.3%) as well as large closed-source models (e.g., openai's 3072-dim text-embedding-3-large model, 39.2%). quantitatively, we show that hard negative mining (+5.3%) and data scale (+4.5%) are critical to achieving this performance. deploying bam embeddings in an application alongside traditional lexical search (okapi bm25), we find that bam embeddings outperform lexical search over all query lengths. notably, vector search with bam embeddings improves as queries become longer and more detailed, while lexical search degrades. finally, we evaluate bam embeddings in a public rag benchmark using financebench @xcite . replacing openai's ada-002 embeddings with ours increases question answering correctness by 8%. qualitatively, we observe that after finetuning, embeddings are more sensitive to company names, tickers and financial metrics, leading to improved performance detailed, forwardlooking, and company or date-specific queries. ## dataset to finetune and evaluate bam embeddings, we construct a dataset of 15.2m query-passage pairs. text passages are drawn from public and proprietary documents (refer section 3.1). for each passage, a matching query is generated using a few-shot prompted llm (refer section 3.2). ## bam embeddings baseline model we finetune the multilingual-e5 model @xcite , which is pretrained on 1b multilingual text pairs and pre-finetuned on a combination of general-purpose labeled datasets. multilingual-e5 embeddings are based on the xlm-roberta architecture @xcite which has three sizes: small (118m model params, 384-dim embeddings), base (278m model params, 768-dim embeddings), and large (560m model params, 1024-dim embeddings). we finetune small and base models; in preliminary experiments we found that the large model performed no better than the base model so we discontinued finetuning. we select multilingual-e5 because the model achieved competitive baseline retrieval performance on our dataset, and because the embedding is mean-pooled foot_3 . this enables us to calculate a contextualized embedding for any sentence within a passage by simply mean-pooling over the relevant subset of tokens. in our application, we exploit this affordance to highlight the sentences in a document that are most relevant to a user's query, based on the similarity between each sentence and the query. embedding models based on the output of the cls token, such as bge @xcite , do not offer this capability. finetuning we use the standard infonce contrastive loss (van den oord et al., 2019) that requires the model to identify the positive passage for each query from a set of negative passages. negative passages are comprised of in-batch negatives (the other passages in the same minibatch) plus 3 hard negatives per query (see below). we finetune for 3 epochs using a batchsize of 512 and initial learning rate {3, 2, 1} × 10 -5 for the {small, base, large} models. following @xcite , during finetuning, evaluation and in deployment, we add the prefixes 'query: ' and 'passage: ' to queries and passages respectively, allowing the model to better represent short queries and long passages in the same embedding space. hard negative mining hard negative mining improves the quality of learned embeddings by introducing negative examples that are more challenging to detect than randomly-sampled negatives @xcite . for each query, we identify 3 hard negative passages using an early version of our model finetuned with 37% of the full dataset and no hard negatives. specifically, we embed all 15.2m queries and passages in the dataset, retrieve the top 1k passages for each query, and label the passages ranked 200-202 places lower than the positive passage as hard negatives. this hyperparameter was set after testing several different options in initial experiments. if the positive passage is not in the retrieved passages, the query-passage pair is removed from the dataset, as manual inspection indicates that these are typically low-quality pairs. ## conclusion we release text embeddings specialized for finance, trained on a dataset of 14.3m synthetic queries constructed from public and proprietary financial documents. on a held-out test set, bam embeddings achieve 62.8% recall@1 vs. 39.2% for the best general-purpose text embeddings. on the fi-nancebench benchmark, replacing general-purpose embeddings with ours improves question answering accuracy by 8%, demonstrating our dataset and model's ability to generalize to out-of-domain settings. finally, we show that in a production document retrieval service, bam embeddings outperform bm25 over all query lengths, and (unlike bm25) retrieval improves on longer and more detailed queries.