Parameter-Efficient Instruction Tuning of Large Language Models For Extreme Financial Numeral Labelling (2024)

Abstract: We study the problem of automatically annotating relevant numerals (GAAP metrics) occurring in the financial documents with their corresponding XBRL tags. Different from prior works, we investigate the feasibility of solving this extreme classification problem using a generative paradigm through instruction tuning of Large Language Models (LLMs). To this end, we leverage metric metadata informationto frame our target outputs while proposing a parameter efficient solution for the task using LoRA. We perform experiments on two recently released financial numeric labeling datasets. Our proposed model, **FLAN-FinXC**, achieves new state-of-the-art performances on both the datasets, outperforming several strong baselines. We explain the better scores of our proposed model by demonstrating its capability for zero-shot as well as the least frequently occurring tags. Also, even when we fail to predict the XBRL tags correctly, our generated output has substantial overlap with the ground-truth in majority of the cases.

## introduction the u.s. securities and exchange commission (sec) mandates publicly traded companies to disclose periodic filings such as quarterly 10-q & annual 10-k reports. these documents are important to finance professionals and investors who rely on sec filings to make informed investment decisions. each company is directed to follow the generally accepted accounting principles (gaap) to report the metrics appearing in these documents and tag them using the extensive business reporting language (xbrl) according to a well-defined taxonomy consisting of thousands of labels. in a recently released fnxl dataset @xcite , such numerals are tagged from a large set of 2,794 labels. implementing xbrl tagging therefore requires advanced accounting skills to map financial data to the correct xbrl concepts. this requires hiring experts to meticulously review each document and assign appropriate labels which is neither a cost-effective nor a scalable solution. fig. 1 shows various challenges involved with the task. prior works and their limitations: finer @xcite formulated the task of identifying relevant numerals from financial texts and labelling them with xbrl tags as an ner problem, and used a bert-based sequence labelling approach. however, the label-set in finer consists of only 139 most frequently occurring xbrl tags. for practical purposes, a much larger number of xbrl tags/labels are needed to effectively annotate the diverse types of numerals present in these documents. more recently, the authors of fnxl dataset @xcite , demonstrated the poor performance of finer when extended to thousands of labels. they also explored an extreme classification methodology, called attentionxml. none of these methods, however, exploit the rich metadata information available with xbrl tags, to improve the classification performance. additionally, all the methods stated above lack the capacity to identify unseen labels during inference as they follow a discriminative paradigm. generative models, on the other hand, display intrinsic zero-shot capabilities if trained effectively. figure 1 : demonstrating the challenges in the extreme financial numeral labelling (xfnl) task. within a financial statement, there are scenarios where every numeral is associated with a distinct xbrl tag, such as in example 2 (6 distinct tags). then, there are cases where a mixture of both relevant and irrelevant numerals (tagged 'others') coexist in the same statement, often within a very limited context, such as in examples 1 & 3. tag documentation common stocks shares issued total number of common shares of an entity that have been sold or granted to shareholders (includes common shares that were issued, repurchased and remain in the treasury). these shares represent capital invested by the firm's shareholders and owners, and may be all or only a portion of the number of shares authorized. in this space, llms have achieved impressive performances for a wide range of nlp tasks across several domains @xcite , including finance @xcite . ## related works one of the (re)emerging applications of nlp is in the field of named entity recognition (ner), particularly for identifying and categorizing various entities within text. the categorization of different amount of (unfunded) status of defined benefit plan, measured as difference between fair value of plan assets and benefit obligation. includes, amount of funded (unfunded) status of defined benefit plan, measured as difference between fair value of plan assets and benefit obligation. includes, but is not limited to, overfunded (underfunded) status. first, read the task description. there could be multiple numerals reported in a financial statement. each number may or may not be associated with a particular tag. now read the following financial statement. at april24, 2020 and april26, 2019, the net underfunded status of the company's benefit plans was @xmath01.1 billion, respectively. what is the tag associated with the numeral 1.4? flan -t5 d4 d5 d3 d2 d1 input x wa wb pretrained weights w embedding h r lora weights, wa and wb x2 x1 x3 x2 x1 x3 st5 encoder sentence embeddings cosine similarity input instruction task statement question output defined benefit plan funded status of plan tag matcher entities into labels can also be considered as an extreme classification task @xcite . in recent years, xbrl tagging has gained a special importance in financial domain, which involves tagging of numeric values. datasets created for such a task are the fnxl dataset @xcite and finer @xcite which have a very large number of entity types compared to standard ner tasks and thus present challenges to the state-of-the-art ner models. @xcite reformulated this task as an extreme classification @xcite problem with a pipeline approach. financial domain-specific pre-trained language models @xcite have been developed, which can be repurposed on financial tasks. among llms pre-trained on financial text (finllms), bloomberggpt @xcite ) is a proprietary model that suffers from limited accessibility and lack of transparency in their data collections and training protocols. fingpt @xcite is an open-source finllm but only suitable for financial sentiment analysis, trading, forecasting and fraud detection tasks. ## problem formulation we break the task of xfnl into two stages (with only the first stage requiring supervised training of models), as illustrated in fig. 2 , in order to take advantage of more elaborate xbrl tag documentations. we present a set of diverse annotated examples in fig. 1 . some tag with documentation examples are shown in in the first stage, we formulate the problem as a generative task using llms, where given a financial statement, and a question targeted towards a specific numeral occurring in the statement, the task is to accurately generate its appropriate xbrl tag documentation (not the tag) if the numeral is relevant and 'other' if the numeral is irrelevant. let s @xmath1, ..., w a i , ...w b i , ..., w n i ) be the i th statement consisting of n tokens, where w a i and w b i be two different numerals with tag a i and tag b i being their respective xbrl tag documentations. we prepend an instruction prompt ip , containing a natural language description of the task, to the statement s i , as shown in fig. 2 . a question q a i is then appended to s i asking for the tag to be determined for a specific numeral, say w a i . the modified input s a i therefore takes the shape ip ||s i ||q a i , where || is a text concatenation operation. the target answer gent ag a @xmath2: tag a i . in the second stage, we obtain the final xbrl tag through a separate tag matcher module, since the entire documentation may not be generated exactly. specifically, we obtain the embedding for gent ag a i using a pre-trained state-of-the-art sentence encoder. we obtain the same for the documentations corresponding to all the available tags. the one having the highest cosine similarity with the gent ag a i embedding is declared to be the predicted xbrl tag documentation, predt ag a i . ## methodology our proposed framework flan-finxc for the xfnl task is divided into two phases, as depicted in fig. 2 : a supervised generative phase, and an unsupervised documentation-to-tag matching phase. in the first phase, given a financial statement s i , and a question q a i asking for the xbrl tag (documentation) to be determined for a numeral w a i appearing in the sentence s i , we instruction tune flan-t5 @xcite with carefully-curated task-specific instructions (see fig. 2 ) to generate the tag documentation gent ag a i . the model is trained to condition on the modified input s a i , as described in the previous section, to generate the target answer one token at a time using auto-regressive decoding. crossentropy loss between the generated and true tokens is minimized in the process. in our case, the entire output generated by the llm corresponds to the tag description, hence no additional parsing is required. our choice for flan-t5 is based on the observation that flan-t5 models are pre-trained (using instruction tuning) on more than 1.8k tasks, and hence can significantly reduce the amount of fine-tuning steps required if adopted as starting checkpoints for learning new tasks. additionally, they achieve strong zero-shot and few-shot performances in comparison to t5 @xcite , their non-instruction-tuned counterpart. note that we train flan-t5 models to generate the xbrl tag documentations instead of the tag themselves (which is the final target) since the more elaborate documentations allow for a better distinction than their corresponding tags (see ## baselines we compare the performance of our proposed models with the following baselines: baselines already applied for xbrl tagging: (1) finer: @xcite solved the task as a ner task, but only for top 139 frequent xbrl tags. we adapt this method for our use case (large number of labels). (2) attentionxml pipeline: @xcite tackled this problem using a bert-based sequence-to-sequence tagger. new baselines that we adopt for xbrl tagging: additionally, we adopt two diverse methods for the task for xbrl tagging, which can utilize the label semantics of xbrl tags. (3) galaxc: @xcite applies collaborative learning over document-label graphs that allows additional information like label metadata to be incorporated. (4) label semantics @xcite leverages the entity description to solve a standard ner task. lastly, with the emergence of chatgpt(openai, 2022), we were curious to check chatgpt's performance for this task for 500 random samples using gpt-3.5-turbo foot_2 api. ## dataset & evaluation metrics dataset: we perform our experiments on the recently released fnxl dataset @xcite containing 10-k documents foot_3 for 2,339 companies and finer @xcite dataset having most frequent (at least 1000 appearances) 139 labels. fnxl dataset contains a total of 79,088 sentences containing 142,922 annotated numerals, with a tag set of 2,794 distinct tags that follows a heavy tail distribution. further, the test set contains 67 xbrl tags that are not part of the train or validation sets. fig. 1 shows a few annotated examples from this dataset. each tag is associated with a pre-defined textual description known as its 'tag documentation' (see ## experimental setup for all our model variants (experiments performed on tesla v100 32gb gpus), we obtain the pretrained checkpoints from the huggingface library foot_5 . for instruction tuning flan-t5-large with prefix tuning, the prefix length was set to 20. for training the models with lora, the rank for the trainable decomposition matrices was set to 2. flan-t5-large models were instruction-tuned for 10 epochs with a learning rate (lr) of 1e -4 (training time: 1hr 22 minutes/epoch, inference time: 2 minutes/sample); with an lr of 1e -2 with prefix tuning (training time: 29 minutes/epoch, inference time: 2 minutes/sample); and with an lr of 5e -4 with lora (training time: 56 minutes/epoch, inference time: 2 minutes/sample). these hyperparameters were selected based on the best macro-f1 results on the validation set. note that the tag information is not included in the input prompt. rather, it is generated during the decoding phase based on the question asked. the input prompt containing the question gets encoded and it is much smaller than the maximum input context length. hence, there is no truncation during encoding. the average tag information length is 28 tokens and we set the length of output to be generated to 30 tokens. hence, in a few cases, truncation may occur during the decoding phase. however, the crucial tag information is typically present in the initial tokens, and our tag matcher ensures correct labels by comparing generated tags with the descriptions for all available tags in the dataset. thus any potential truncation in a few cases does not pose any risk to our model's performance. instruction selection: while experimenting, we gradually improved our instructions. first, we tried out a prompt without any task-specific instruction, thereby consisting of the input financial statement and the question for a given numeral. then, we included a simple task description and repeated our experiments. finally, we made the task description/instruction more elaborate which led us to ob-tain the best results. on the target side, we tried out two variants, generating labels vs. generating the label descriptions. the latter resulted in significant performance improvement (refer ## main results we report the results of our proposed model variants and various baselines in we also demonstrate the advantage of training bigger models for extreme financial numeral labelling for fnxl dataset, by comparing the results of t5-large (780m parameters, macro-f1 60.11) with t5-base (220m parameters, macro-f1 54.21). note that these baselines that we devise for the task, already outperform the existing state-of-the-art. ## analysis we now present different analyses of our best model (flan-t5-large with lora), and the closest baseline (attentionxml pipeline). the following analysis is specific to the fnxl dataset, chosen for its more extensive range of labels compared to other datasets, emphasizing its practical relevance. ## conclusion this work proposes a generative approach to solve the financial numeric labelling task. we propose a novel flan-finxc framework, that makes use of parameter-efficient instruction tuning of llms for this extreme labelling task. while comparing with the state-of-the-art models and various competitive baselines that we devise for the task, we find that our best model, flan-t5-large with lora, achieves huge improvements, providing a macro-f1 of 66.23% as compared to the previously reported best numbers of 47.54% for fnxl dataset. for adapting to other tasks, our approach requires only minor task-specific modifications to the input prompt, and the target output. moreover, our approach offers flexibility in using either the class labels or label descriptions, depending on the available data. as potential future directions, we believe the scope to include more financial knowledge and integrate a human-ai feedback loop would be the way forward to improve performance of this challenging task.