Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023)

Abstract: Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. However, they risk becoming rapidly over-parameterized and the adaptation cost of fully fine-tuning them increases significantly. Storing them becomes progressively impractical as it requires keeping a separate copy of all the fine-tuned weights for each task. By freezing all pre-trained weights during fine-tuning, parameter-efficient tuning approaches have become an appealing alternative to traditional fine-tuning. The performance of these approaches has been evaluated on common NLP tasks of the GLUE benchmark and shown to match full fine-tuning performance, however, their impact is less researched in domain-specific fields such as finance. This work compares the performance of a set of financial BERT-like models to their fully fine-tuned counterparts by leveraging different parameter-efficient tuning methods. We see that results are comparable to traditional fine-tuning while gaining in time and resource efficiency.

## introduction natural language processing (nlp) and the increasing popularity of large language models have redefined how computers understand and interpret human language. in particular, the financial domain, fundamentally intertwined with language and communication, benefits from nlp techniques to process complex financial documents. this domain deals with a vast array of unstructured textual data coming from sources such as news articles, financial reports, or investor opinions, each offering potentially valuable insights for processes such as risk assessment or decision-making. to this end, numerous efforts have been made to tailor large language models such as bert @xcite or gpt-3 @xcite to the financial domain, resulting in models such as finbert @xcite @xcite , flang-bert @xcite or bloomberggpt @xcite . despite their immense potential, these pretrained models require fine-tuning to adapt their parameters to specific tasks, which becomes prohibitively expensive the larger the models are. recently, more parameter-efficient alternatives to full fine-tuning such as low-rank adaptation (lora; @xcite , @xcite or adapters @xcite @xcite have been proposed to mitigate the vast adaptation cost of language models while maintaining full fine-tuning performance @xcite . these relatively new approaches share the same underlying principle of freezing the pre-trained weights of the base model, while fine-tuning only a set of newly introduced parameters, resulting in a substantial reduction in the total number of trainable parameters. while parameter-efficient tuning methods have been evaluated on generic benchmarks such as glue @xcite and shown to reach results that are comparable to full fine-tuning, they are less researched in domain-specific fields such as finance. in this paper, we evaluate the efficacy of different parameter-efficient fine-tuning approaches applied to bert-like models in the financial domain. our contributions are twofold: (a) we compare the performance of parameter-efficient tuning approaches on a range of financial tasks against their fully finetuned counterparts, and (b) show both their model size and training time benefits. ## experimental setup in this section, we list our research questions, describe the experimental framework we set up to address these questions and briefly explain the parameter-efficient fine-tuning approaches that we apply. ## experimental results in this section, we present the results from our experiments across four downstream tasks and answer the research questions introduced in section 2. for each task, we fine-tune the pre-trained models over five runs and report the average performance as the final metric. results are shown in finbert + lora outperforms the adaptertuned and fully fine-tuned versions of the baseline on fpb sc and fiqa sr, respectively. flang-bert + adapter h beats flang-bert + lora on all tasks except on ner, and outperforms the fully fine-tuned flang-bert on fiqa sr. none of the parameter-efficient approaches outperform the baselines on ner, but finbert with both lora and adapter h closely reaches that baseline performance. the advantage of one method over the other, is not definitive, as the effectiveness of each approach in terms of performance varies depending on the task. although, it can be seen that adapter h does not underperform the baseline for each model variant on all of the tasks except for ner. overall, these results suggest that parameter-efficient approaches are a suitable alternative to full finetuning and do not significantly hurt performance on financial tasks. ## discussion our experiments show that performance on financial tasks is not necessarily compromised with parameter-efficient tuning approaches. we see that even on the smaller fiqa and ner datasets for fin-bert, these approaches achieve almost the same performance as full fine-tuning. due to the small set of trainable parameters, the models might be less at risk of overfitting and catastrophic forgetting @xcite . regarding the performance between the two parameter-efficient methods, we argue that adapter h performs better than lora. it almost always maintains full fine-tuning performance, and tends to outperform lora more frequently. we believe this to be the case because, compared to lora, adapter h keeps a bigger share of trainable parameters. in addition, lora needs a much larger ## related work numerous studies have been released on the effectiveness of parameter-efficient approaches on the glue benchmark @xcite @xcite . in the financial domain, a number of studies build language models from scratch or through continual in-domain pretraining and compare the domain-specific models to general-domain baseline models @xcite @xcite , but do not employ parameter-efficient techniques. large language models for finance are on the rise @xcite but have either access restrictions or do not take advantage of parameterefficient tuning approaches. to the best of our knowledge, the first financial language model, finetuned with lora in combination with reinforcement learning, was released @xcite . ## conclusion in this paper, we evaluated the effectiveness of lora and adapter fine-tuning applied to tasks in the financial domain. we showed that in a financial setting, we can reach full fine-tuning performance with smaller model sizes which allows for easier storage and sharing of fine-tuned models among practitioners. we would like to aspire future research to evaluate more parameter-efficient strategies and extend them to other domain-specific fields. based on our results and with the increasing number of larger language models in the financial sector @xcite , we recommend the implementation of more parameter-efficient fine-tuning practices to strive towards reducing their carbon footprint and environmental impact.