FIRE : A Dataset for Financial Relation Extraction (2024)

Abstract: This paper introduces FIRE (**FI**nancial **R**elation **E**xtraction), a sentence-level dataset of named entities and relations within the financial sector. Comprising 3,025 instances, the dataset encapsulates 13 named entity types along with 18 relation types. Sourced from public financial reports and financial news articles, FIRE captures a wide array of financial information about a business including, but not limited to, corporate structure, business model, revenue streams, and market activities such as acquisitions. The full dataset was labeled by a single annotator to minimize labeling noise. The labeling time for each sentence was recorded during the labeling process. We show how this feature, along with curriculum learning techniques, can be used to improved a model’s performance. The FIRE dataset is designed to serve as a valuable resource for training and evaluating machine learning algorithms in the domain of financial information extraction. The dataset and the code to reproduce our experimental results are available at https://github.com/hmhamad/FIRE. The repository for the labeling tool can be found at https://github.com/abhinav-kumar-thakur/relation-extraction-annotator.

## introduction the proliferation of textual data in the financial domain presents a unique opportunity for the application of machine learning and natural language processing (nlp) techniques. the extraction of named entities and their relations from unstructured financial texts, such as security and exchange commission (sec) filings (u.s. securities and exchange commission) and financial news articles (bloomberg -financial news, analysis, and data), is a crucial task with significant implications for financial analysis and decision-making. named entity recognition (ner) @xcite and relation extraction (re) @xcite is a complex yet crucial task in nlp, particularly within the financial domain. the task demands extensive linguistic and domain knowledge, making dataset creation costly and labor-intensive. this complexity has led to instances where previously hand-labeled and published re datasets have undergone subsequent corrections post-publication. examples of such non-financial datasets include tacred @xcite and its revised counterpart, tacred revisited @xcite , as well as docred @xcite and its updated version, re-docred @xcite . the lack of a comprehensive, well-annotated dataset in the financial domain hampers the development and evaluation of algorithms for these tasks. in response to this identified gap, we present fire, a dataset specifically constructed for joint ner and re within the financial domain. drawn from both financial documents, mainly sec filings, and financial news articles, fire provides a diverse range of linguistic constructs and financial terminologies. the dataset is constituted of 3,025 instances, all hand-labeled according to comprehensive annotation guidelines. note that an instance (or an example) refers a labeled object, consisting of a single sentence or multiple sentences with associated entity and relation information. figure 1a presents a labeled sentence from the dataset while figure 1b is one example of how the labeled data can be used to create a knowledge graph. more examples can be found in the annotation guidelines document which is provided with the dataset. the dataset incorporates 13 named entity categories and 18 relation types, effectively capturing vital details about businesses, including aspects such as their organizational structure, income streams, business strategies, and market maneuvers, including acquisitions. the fire dataset also serves as a substantial resource for training, evaluating, and comparing the performance of models specialized in the finance sector. projects like 10-kgpt @xcite and bloomberggpt @xcite , which are tailored for financial tasks, lack evaluation on dedicated financial re datasets. fire fills this gap, offering a robust platform for testing these models against a diverse and complex set of financial terms and relationships. our goal is to advance financial nlp by providing a high-quality, manually annotated dataset for refining state-of-the-art large language model (llm)s. an additional feature of fire is the inclusion of a labeling time data field for each record in the dataset. this feature may provide researchers with additional granularity when analyzing performance. labeling time can serve as an implicit indicator of example difficulty, offering potential applications for the implementation of curriculum learning strategies @xcite . by leveraging this feature, researchers can explore and develop methods that dynamically adjust the learning process based on the difficulty of the examples, potentially leading to more efficient learning and improved model performance. in our experiment results section, we provide an initial result of incorporating the labeling time feature into the training process. to the best of our knowledge, this has not been studied yet in the literature. the paper contributions are summarized as follows: the rest of this paper is organized as follows: section 2 goes over some previous general-purpose and domain-specific ner and re datasets and compares fire to existing datasets in finance. section 3 provides a detailed description of the fire dataset, including the composition, data collection and annotation processes. section 4 presents an evaluation of selected state-of-the-art models on the fire dataset, discussing the associated performances and implications. finally, section 5 concludes the paper and outlines potential directions for future work. @xcite , dwie @xcite , docred @xcite , and re-docred @xcite . some popular sentencelevel re-datasets include tacred @xcite , fb-nty @xcite , and webnlg @xcite . while many of these are general-purpose, there are domainspecific datasets too @xcite . fire, despite having some multi-sentence instances, is mainly a sentence-level re dataset. relation extraction datasets and distant supervision. creating re datasets is costly due to labeling. one common technique to deal with this problem is distant supervision which relies on a knowledge base to automatically label text data @xcite . in particular, sentences that mention two entities connected by a relation in the knowledge base are assumed to be expressing that same relation. this strong assumption leads to a large number of noisy samples. to address this issue, researchers have developed methods that relax the distant supervision assumptions @xcite . despite its limitations, distant supervision remains a popular and effective method for generating large-scale datasets for relation extraction tasks. several relation extraction datasets have been developed using distant supervision, including fb-nyt @xcite , a dataset constructed by aligning freebase @xcite relations with the new york times articles, and webnlg @xcite , a text generation dataset created from dbpedia @xcite , among others. such datasets have been widely used for training and evaluating relation extraction models. conversely, fire is a supervised dataset in which every instance has been annotated manually following extensive annotation guidelines. while this approach elevates the cost of labeling and poses scalability challenges, it guarantees a high level of precision in the labels. ## conclusion in this paper, we introduced fire, a dataset carefully curated for the task of joint named entity and relation extraction in the financial domain. the comprehensive annotation guidelines and the opensource labeling tool accompanying the dataset further contribute to its robustness and usability. our evaluations with re-specialized and generative llms highlight fire's challenges and potential. we also explored the benefits of incorporating label-ing time in training. it is evident that the development of more refined models capable of understanding the complexities of financial domain-specific data is required. looking forward, we anticipate that fire will serve as a valuable resource for researchers and practitioners in the fields of natural language processing and financial analysis.