BUSTER : a “ BUS iness Transaction Entity Recognition” dataset (2023)

Abstract: Albeit Natural Language Processing has seen major breakthroughs in the last few years, transferring such advances into real-world business cases can be challenging. One of the reasons resides in the displacement between popular benchmarks and actual data. Lack of supervision, unbalanced classes, noisy data and long documents often affect real problems in vertical domains such as finance, law and health. To support industry-oriented research, we present BUSTER, a BUSiness Transaction Entity Recognition dataset. The dataset consists of 3779 manually annotated documents on financial transactions. We establish several baselines exploiting both general-purpose and domain-specific language models. The best performing model is also used to automatically annotate 6196 documents, which we release as an additional silver corpus to BUSTER.

## introduction natural language processing (nlp) is a field potentially beneficial to a broad span of languageintensive domains, such as law and health. whilst lots of financial data are tabular, there is also crucial information stored in reports, news, transaction agreements, etc. the abrupt developments in nlp @xcite are favouring its adoption as assistance tools for human experts in many tasks, ranging from document classification @xcite to information extraction @xcite and even text summarization @xcite . however, transferring the emerging technologies into industry applications can be non-trivial. adapting large language models (llms) to vertical domains usually requires fine-tuning on domainspecific annotated data. labeling is often a timeconsuming, expensive process, especially when experts in the field are involved. recently, several benchmarks and datasets have been constructed for law @xcite , health @xcite and finance @xcite . in this work, we support industry-oriented research community by presenting buster: a business transaction entity recognition dataset. as the title suggests, buster is an entity recognition (er) benchmark that focus on the main actors involved in a business transaction. after collecting about ten thousands business transaction documents from edgar company acquisition reports, we constructed a dataset with 3779 manually annotated documents (the gold corpus), from which we trained an llm to automatically annotate the remaining 6196 documents (the silver corpus). we analyze the properties of the proposed dataset and also evaluate the performance of some baselines. the dataset will be public and free to download as a benchmark for the nlp community. the paper is organized as follows. first we review in section 2 previous related works on financial ner and document-level datasets. then, we describe the data collection process and annotation methodologies in sections 3 and 4, respectively. a detailed description of buster and its statistics follows in section 5. in section 6 we establish baselines with different llms. finally, in section 7 we ## related works several document datasets in the financial domain have been proposed in the literature, but few of them are dedicated to the entity recognition (er) task. furthermore, these few are mainly intended for the standard named entity recognition (ner) task, such as @xcite @xcite . in @xcite is presented a corpus (fin) of eight documents from sec which were manually annotated using the standard four ner data type: person, organization, location and miscellaneous. unlike that dataset, in buster we decided to focus on entities that are involved in a financial transaction. finer-139 @xcite instead consists in a large corpus of sec documents annotated via gold xblr tags, that includes a label set of 139 numerical entities on about 1.1m sentences. the tag attribution mostly depends on context rather than the token itself, as it is in buster. beside the completely different tag set, the main difference between buster and finer-139 is the fact that we release a document-level benchmark. indeed, the detection of roles like the buyer company can require scopes wider than a single sentence. moreover, documents come from files with heterogeneous layouts, extensions and structure, which can sometimes hinder the segmentation of the document into single sentences. outside the financial domain, a variety of document-level datasets for ner have been proposed. docred @xcite is a ner and relation extraction (re) corpus built from wikidata and wikipedia short text passages, while biocreative @xcite is a dataset for ner/re on health domain. in @xcite , the authors propose a dataset for ner in medical area. ## data collection our goal was to create a highly business-oriented dataset to recognize relevant entities involved in financial transactions. unlike standard ner tasks, we focused on the problem of entity-role recognition, where the goal is to identify a set of entities but only where they appear with specific roles in a context, such as companies involved in an acquisition or consultants assisting in an operation. ## annotation for data labeling, we used a double-blind manual procedure. specifically, we utilized two annotators (ann 1 and ann 2 ), who were trained on the financial transactions topic and who were provided with a tag-set and specific guidelines to follow in the entity tagging procedure. ## the buster dataset the final buster dataset is composed of 3779 labeled documents. in figure 1 , we show an example of an annotated text passage inside a document. as explained, those documents were manually annotated and represent the "gold" buster corpus. we randomly split the data into 5 folds to yield a statistically robust benchmark. indeed, such division allows the use of a standard k-fold cross-validation approach. the data set has been used as benchmark for 4 state-of-the art er models (as described in sec- ## experiments to establish baselines, we performed several experiments using both generic and domain-specific language models. in the experiments, we followed a 5-folds cross validation approach using the folds in ## conclusions and future works in this work, we presented buster, an entity recognition (er) benchmark for business transaction-related entities. it consists of a corpus of 3779 manually annotated documents on financial transaction (the gold data) which has been randomly divided into 5 folds, plus an additional set of 6196 automatically annotated documents (the silver data) that were created from the fine-tuned roberta model. we plan to make it publicly available and free-to-download and we believe it could become a reference benchmark in the field of entity recognition, in particular for the financial domain. in the future, we intend to work in two directions. on one side, we plan to increase the amount of manually labeled data and to extend the labels set with more transaction-related tags. on the other hand, we aim to introduce some specific types of relations between entities in order to extend the dataset to relational extraction.