Financial Numeric Extreme Labelling: A dataset and benchmarking (2023)

Abstract: The U.S. Securities and Exchange Commission (SEC) mandates all public companies to file periodic financial statements that should contain numerals annotated with a particular label from a taxonomy. In this paper, we formulate the task of automating the assignment of a label to a particular numeral span in a sentence from an extremely large label set. Towards this task, we release a dataset, Financial Numeric Extreme Labelling (FNXL), annotated with 2,794 labels. We benchmark the performance of the FNXL dataset by formulating the task as (a) a sequence labelling problem and (b) a pipeline with span extraction followed by Extreme Classification. Although the two approaches perform comparably, the pipeline solution provides a slight edge for the least frequent labels.

## introduction in 2019, the u.s. securities and exchange commission (sec) mandated each company to use gaap metrics foot_0 to standardise financial reporting. these metrics are used to tag portions of sec documents including, numerals using extensive business reporting language (xbrl), an xml based language to facilitate the processing of financial information. the process of annotating the documents requires enormous manual effort: expert annotators from a company have to go through the document and mark each relevant detail with a relevant gaap metric label. this necessitates the development of an automatic annotation process that may reduce the manual effort to annotate the documents. solving this task would also help with annotation of old as well as new reports (which may not contain xbrl tags). towards this goal, we aim to decrease the list of possible tags for annotators and provide them with a crisp list of k tags. we define the problem as, given a sentence, identify the relevant numerals and assign them a particular gaap label. an example of this annotation is provided in figure 1 whereby each numeral is marked with a label or identified as 'other'. this problem has previously been tackled by finer @xcite , as a sequence labelling approach using bert @xcite , where their set of labels is the 139 labels that occur the most frequently. however, we find that in the real-world setting, a much larger number of labels are used to annotate the text found in these documents. thus, to evaluate the real world context, we painstakingly prepare a dataset, fnxl that contains a total of 2,794 tags. we realize that given the sheer number of labels, extreme classification is an ideal methodology that can be followed. subsequently, we find that the attentionxml method @xcite from the extreme classification (xc) repository @xcite can be modified to suit our problem setting. consequently, we use the dataset, fnxl to benchmark both the finer based sequence labelling approach and the attentionxml method. we find that xc methods give a comparable performance (better in certain situations) as finer. specifically, xc performs better in tagging infrequent labels. ## benchmarking we extensively benchmark the dataset using two broad class of algorithms, the named-entity based finer model and extreme classification (xc) based attentionxml model. we briefly describe these models and the different variations tried. ## results we report the results for 9 experiments in total, 6 finer based models and 3 attentionxml based models as showcased in we observe that the attentionxml pipeline performs better in macro scores than the finer model. the best performing attentionxml pipeline uses the [shape] masking token and achieves a 47.54% macro-f1 vis-a-vis 47.13% macro-f1 for best the finer model with no-masking. however, finer performs better in the micro values achieving a 75.84% micro-f1 vis-a-vis 74.74% micro-f1 for attentionxml pipeline. the superior performance of finer in micro metrics can be attributed to its better performance with popular labels as the frequency distribution of labels follow a long-tail distribution. this is elaborated in section 4.1. we also observe that the masking technique does not help the finer model as can be seen in the case of bert-base finer models. we note that the bert-base models perform better than the sec based models in terms of macro-f1 and comparable in terms of micro-f1. the adapted masking technique is beneficial in case of attentionxml pipeline. we theorize that since attentionxml uses glove word embeddings, it does not face the problem of defragmentation. also, unlike finer, in attentionxml masking is done on irrelevant numerals which helps the model focus on the context of the relevant numeral. from the results we can establish that the [num] and [shape] psuedotokens help attentionxml model successfully generalize over numeric expressions. ## conclusion the paper provides a detailed idea about the challenges faced in tagging numerals with labels when the number of labels is large (2,794 tags) and follow a long-tail distribution. we have rigorously collected an extensive set of labels, done an extensive bench-marking and executed a very specialized human experiment. we believe the scope to include more information about the us-gaap metrics label in the annotation model and a method to automate human-ai feedback loop would be the way forward to improve the performance of this difficult task. the dataset and codes are publicly available.