F i NER : Financial Numeric Entity Recognition for XBRL Tagging (2022)

Abstract: Publicly traded companies are required to submit periodic reports with eXtensive Business Reporting Language (XBRL) word-level tags. Manually tagging the reports is tedious and costly. We, therefore, introduce XBRL tagging as a new entity extraction task for the financial domain and release FiNER-139, a dataset of 1.1M sentences with gold XBRL tags. Unlike typical entity extraction datasets, FiNER-139 uses a much larger label set of 139 entity types. Most annotated tokens are numeric, with the correct tag per token depending mostly on context, rather than the token itself. We show that subword fragmentation of numeric expressions harms BERT’s performance, allowing word-level BILSTMs to perform better. To improve BERT’s performance, we propose two simple and effective solutions that replace numeric expressions with pseudo-tokens reflecting original token shapes and numeric magnitudes. We also experiment with FIN-BERT, an existing BERT model for the financial domain, and release our own BERT (SEC-BERT), pre-trained on financial filings, which performs best. Through data and error analysis, we finally identify possible limitations to inspire future work on XBRL tagging.

## introduction natural language processing (nlp) for finance is an emerging research area @xcite @xcite . financial data are mostly reported in tables,but substantial information can also be found in textual form, e.g., in company filings, analyst reports, and economic news. such information is useful in numerous financial intelligence tasks, like stock market prediction @xcite , financial sentiment analysis @xcite source code: https://github.com/nlpaueb/finer correspondence: eleftheriosloukas@aueb.gr 2013; @xcite , economic event detection @xcite @xcite , and causality analysis @xcite . in this work, we study how financial reports can be automatically enriched with word-level tags from the extensive business reporting language (xbrl), a tedious and costly task not considered so far. 1 to promote transparency among shareholders and potential investors, publicly traded companies are required to file periodic financial reports. these comprise multiple sections, including financial tables and text paragraphs, called text notes. in addition, legislation in the us, the uk, the eu and elsewhere requires the reports to be annotated with tags of xbrl, an xml-based language, to facilitate the processing of financial information. the annotation of tables can be easily achieved by using company-specific pre-tagged table templates, since the structure and contents of the tables in the reports of a particular company rarely change. on the other hand, the unstructured and dynamic nature of text notes (figure 1 ) makes adding xbrl tags to them much more difficult. hence, we focus on automatically tagging text notes. tackling this task could facilitate the annotation of new and old reports (which may not include xbrl tags), e.g., @xcite ; @xcite ; @xcite ; @xcite . towards this direction, we release finer-139, a new dataset of 1.1m sentences with gold xbrl tags, from annual and quarterly reports of publicly traded companies obtained from the us securities and exchange commission (sec). unlike other entity extraction tasks, like named entity recognition (ner) or contract element extraction ( another important difference from typical entity extraction is that most tagged tokens (∼91%) in the text notes we consider are numeric, with the correct tag per token depending mostly on context, not the token itself (figure 1 ). the abundance of numeric tokens also leads to a very high ratio of out-of-vocabulary (oov) tokens, approx. 10.4% when using a custom word2vec @xcite ) model trained on our corpus. when using subwords, e.g., in models like bert @xcite , there are no oov tokens, but numeric expressions get excessively fragmented, making it difficult for the model to gather information from the fragments and correctly tag them all. in our experiments, this is evident by the slightly better performance of stacked bilstms @xcite operating on word embeddings compared to bert. the latter improves when using a crf @xcite layer, which helps avoid assigning nonsensical sequences of labels to the fragments (subwords) of numeric expressions. to further improve bert's performance, we propose two simple and effective solutions that replace numeric expressions with pseudo-tokens reflecting the original token shapes and magnitudes. we also experiment with fin-bert @xcite , an existing bert model for the financial domain, and release our own family of bert models, pretrained on 200k financial filings, achieving the best overall performance. our key contributions are: by contrast, although it is easy to figure out that the first three highlighted expressions of figure 1 are amounts, assigning them the correct xbrl tags requires carefully considering their context. contract element extraction @xcite also requires considering the context of dates, amounts etc. to distinguish, for example, start dates from end dates, total amounts from other mentioned amounts, but the number of entity types in finer-139 is an order of magnitude larger ( transformer-based model that handles numerical reasoning tasks by representing numbers by their scientific notation and applying subword tokenization. on the other hand, genbert @xcite uses the decimal notation and digit-by-digit tokenization of numbers. both models attempt to deal with the problem that word-level tokenization often turns numeric tokens to oovs @xcite . this is important, because numerical reasoning requires modeling the exact value of each numeric token. in finer-139, the correct xbrl tags of numeric tokens depend much more on their contexts and token shapes than on their exact numeric values (fig. 1 ). hence, these methods are not directly relevant. genbert's digit-by-digit tokenization would also lead to excessive fragmentation, which we experimentally find to harm performance. ## task and dataset traditionally, business filings were simply rendered in plain text. thus, analysts and researchers needed to manually identify, copy, and paste each amount of interest (e.g., from filings to spreadsheets). with xbrl-tagged filings, identifying and extracting amounts of interest (e.g., to spreadsheets or databases) can be automated. more generally, xbrl facilitates the machine processing of financial documents. hence, xbrl-tagged financial reports are required in several countries, as already noted (section 1). however, manually tagging reports with xbrl tags is tedious and resourceintensive. therefore, we release finer-139 to foster research towards automating xbrl tagging. finer-139 was compiled from approx. 10k annual and quarterly english reports (filings) of publicly traded companies downloaded from sec's edgar system. 6 the downloaded reports span a 5-year period, from 2016 to 2020. they are annotated with xbrl tags by professional auditors and describe the performance and projections of the companies. we used regular expressions to extract the text notes from the financial statements item of each filing, which is the primary source of xbrl tags in annual and quarterly reports. furthermore, each jurisdiction has its own xbrl taxonomy. since we work with us documents, our labels come from us-gaap. 7 since this is the first effort towards automatic xbrl tagging, we chose to work with the most essential and informative attribute, the tag names, which populate our label set. also, since xbrl tags change periodically, we selected the 139 (out of 6,008) most frequent xbrl tags with at least 1,000 appearances in finer-139. the distribution of these tags seems to follow a power law (figure 2 ), hence most of the 6k xbrl tags that we did not consider are very rare. we used the iob2 annotation scheme to distinguish tokens at the beginning, inside, or outside of tagged expressions, which leads to 279 possible token labels. we split the text notes into 1.8m sentences, the majority of which (∼90%) contained no tags. 8 the sentences are also html-stripped, normalized, and lower-cased. to avoid conflating trivial and more difficult cases, we apply heuristic rules to discard sentences that can be easily flagged as almost certainly requiring no tagging; in a real-life setting, the heuristics, possibly further improved, would discard sentences that do not need to be processed by the tagger. the heuristic rules were created by inspecting the training subset and include regular expressions that look for amounts and other expressions that are typically annotated. approx. 40% of the 1.8m sentences were removed, discarding only 1% of tagged ones. we split chronologically the remaining sentences into training, development, and test sets with an 80/10/10 ratio ( ## baseline results we report micro-f 1 (µ-f 1 ) and macro-f 1 (m-f 1 ) at the entity level, i.e., if a gold tag annotates a multi-word span, a model gets credit only if it tags the exact same span. this allows comparing more easily methods that label words vs. subwords. on the other side, when using words as tokens, there are fewer opportunities for nonsensical label sequences, because there are fewer tokens. for instance, the average number of subwords and words per gold span is 2.53 and 1.04, respectively. hence, it is easier for the bilstm to avoid predicting nonsensical sequences of labels and the crf layer on top of the bilstm (with word embeddings) has less room to contribute and mainly introduces noise (e.g., it often assigns low probabilities to acceptable, but less frequent label sequences). with the crf layer, the model tries to maximize both the confidence of the bilstm for the predicted label of each word and the probability that the predicted sequence of labels is frequent. when the bilstm on its own rarely predicts nonsensical sequences of labels, adding the crf layer rewards commonly seen sequences of labels, even if they are not the correct labels, without reducing the already rare nonsensical sequences of labels. to further support our hypothesis, we repeated the bilstm experiments, but with subword (instead of word) embeddings, trained on the same vocabulary with bert. without the crf, the subword bilstm performs much worse than the word bilstm (6 p.p drop in µ-f 1 ), because of the many more decisions and opportunities to predict nonsensical label sequences. the crf layer substantially improves the performance of the subword bilstm (4.9 p.p. increase in µ-f 1 ), as expected, though the word bilstm (without crf) is still better, because of the fewer opportunities for nonsensical predictions. a drawback of crfs is that they significantly slow down the models both during training and inference, especially when using large label sets @xcite , as in our case. hence, although bert with crf was the best model in ## fragmentation in bert in finer-139, the majority (91.2%) of the gold tagged spans are numeric expressions, which cannot all be included in bert's finite vocabulary; e.g., the token '9,323.0' is split into five subword units, ['9', '##,', '##323', '##.', '##0'], while the token (top), bert + [num] (middle), bert + [shape] (bottom) for the same sentence. bert tags incorrectly the amounts in red. bert + [num] and bert + [shape] tag them more successfully (green indicates correct tags). '12.78' is split into ['12', '##.', '##78']. the excessive fragmentation of numeric expressions, when using subword tokenization, harms the performance of the subword-based models ( the pseudo-token is added to the bert vocabulary, and its representation is learned during fine-tuning. this allows handling all numeric expressions in a uniform manner, disallowing their fragmentation. ## conclusions and future work we introduced a new real-word nlp task from the financial domain, xbrl tagging, required by regulatory commissions worldwide. we released finer-139, a dataset of 1.1m sentences with xbrl tags. unlike typical entity extraction tasks, finer-139 uses a much larger label set (139 tags), most tokens to be tagged are numeric, and the correct tag depends mostly on context rather than the tagged token. we experimented with several neural classifiers, showing that a bilstm outperforms bert due to the excessive numeric token fragmentation of the latter. we proposed two simple and effective solutions that use special tokens to generalize over the shapes and magnitudes of numeric expressions. we also experimented with fin-bert, an existing bert model for the financial domain, which also benefits from our special tokens. finally, we pretrained and released our own domain-specific bert model, sec-bert, both with and without the special tokens, which achieves the best overall results with the special tokens, without costly crf layers. in future work, one could hire experts to reannotate a subset of the dataset to measure human performance against the gold tags. future work could also consider less frequent xbrl tags (fewand zero-shot learning) and exploit the hierarchical dependencies of xbrl tags, possibly with hierarchical classifiers, building upon our error analysis.