Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis (2022)

Abstract: The invention of transformer-based models such as BERT, GPT, and RoBERTa has enabled researchers and financial companies to finetune these powerful models and use them in different downstream tasks to achieve state-of-the-art performance. Recently, a lightweight alternative (approximately 0.1% - 3% of the original model parameters) to fine-tuning, known as prefix tuning has been introduced. This method freezes the model parameters and only updates the prefix to achieve performance comparable to full fine-tuning. Prefix tuning enables researchers and financial practitioners to achieve similar results with much fewer parameters. In this paper, we explore the robustness of prefix tuning when facing noisy data. Our experiments demonstrate that fine-tuning is more robust to noise than prefix tuning—the latter method faces a significant decrease in performance on most corrupted data sets with increasing noise levels. Furthermore, prefix tuning has high variances on the F1 scores compared to fine-tuning in many corruption methods. We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.

## introduction the transformer architecture @xcite has given rise to several powerful language models such as bert @xcite and gpt @xcite . these models are trained on large text corpora and the pre-trained models can be used on different downstream tasks by finetuning these models, which refers to the process of updating the weights of the pre-trained model to adapt to the downstream task and the associated dataset. this approach is critical in achieving state-of-the-art results in many downstream tasks. however, these fine-tuned language models are large in size and the deployment of these models in production to solve real-world problems becomes difficult due to the memory requirement, constraining the deployment of models in many real-life financial applications. given that it is anticipated that model sizes will continue to rise, this will become more serious. @xcite introduced a lightweight alternative to finetuning known as prefix tuning. the authors freeze the model parameters of gpt-2 @xcite and use a task-specific vector to tune the model for natural language generation. this method achieves comparable performance with finetuning and uses approximately 0.1% -3% of the original model parameters. this method will enable the use of pre-trained language models for many industrial applications. in the financial sector, natural language processing has a wide variety of applications ranging from building a chatbot to interact with customers @xcite , predicting stock movements based on sentiments from financial news headlines and tweets @xcite , to summarizing financial reports @xcite . prefix tuning can be applied to many tasks with fewer parameters and much less memory consumption. however, in the real world, the data might be noisy, especially in the case of chatbots and social media data where misspellings, typographical errors, and out-of-vocabulary words occur frequently. recent studies have investigated the robustness of finetuning language models such as @xcite , @xcite , @xcite , @xcite and @xcite , and found that finetuning is not robust to noisy texts. to the best of our knowledge, there have been no studies that explore the robustness of prefix tuning that reflect real-life scenarios and compare it with finetuning to identify the more robust method. our work corrupts the financial phrasebank dataset @xcite , using various text corruption methods such as keyboard errors (typos), inserting random characters, deleting random words, replacing characters with ocr alternatives and replacing words with antonyms by varying percentages in each sentence. the corrupted dataset is used with two widely used pre-trained models, bert-base @xcite and roberta-large @xcite , under both prefix tuning and fine-tuning, to compare their performance at different noise levels. in addition, we evaluate the performance on a kaggle stock market tweets dataset @xcite , which is a real-life noisy dataset. with our experiments, we show that fine-tuning is more robust than prefix tuning in most setups. fine-tuning updates the weights based on the downstream task and the dataset, and because of this, it can adapt to the noise, whereas prefix tuning uses the pretrained model without updating the weights which limits the model from learning task-oriented information when facing noisy data. in summary, the contributions of this paper are three-fold. ## conclusion with the sizes of pre-trained models continuing to be significantly larger, lightweight models have become more important for many financial applications. however, the robustness of such models has not been well understood yet. in this paper, we explored the robustness of prefix tuning by corrupting the financial phrasebank dataset with various corruption methods, including keyboard (qwerty) erthe move was triggered by weak demand for forestry equipment and the uncertain market situation negative neutral negative ror, random character insertion, ocr replacement, random word deletion and antonym replacement under varying noise levels at 10%, 20%, 30%, 40% and 50%, as well as on the kaggle stock market tweets, which is a real-world noisy dataset. we show that fine-tuning is more robust to noise than prefix tuning in most of the corruption methods. as the impact of noise is more significant along with increasing noise levels, prefix tuning shows a more significant decrease in performance compared to full fine-tuning. the variance of performance of prefix tuning is higher than that of fine-tuning for most corruption setups. our study suggests that caution should be taken by practitioners when applying prefix tuning to noisy data. a solution to improving the robustness to reduce the impact of noise is desired and is our immediate future work.