Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning (2021)

Abstract: Privacy is of primary importance when it comes to the Financial Domain as the data is highly confidential and no third party can be having access to it. Natural Language Processing (NLP) techniques can be applied for text classification and entity detection purposes in financial domains like customer feedback sentiment analysis, invoice entity detection, categorisation of financial documents by type etc. Due to the sensitive nature of such data, privacy measures need to be taken for handling and training large models with such data. In this work, we propose a contextualized transformer (BERT and RoBERTa) based text classification model integrated with privacy features like Differential Privacy (DP) and Federated Learning (FL). We present how to privately train NLP models and desirable privacy utility trade-offs and evaluate it on the Financial Phrase Bank dataset.

## introduction divulging personally identifiable information during a business transaction has become a commonplace occurrence for most individuals. this activity can span from sharing of bank account numbers, loan account numbers, and credit/debit card numbers, to providing non-financial personally identifiable information such as name, social security number, driver's license number, address, and email address. maintaining the privacy of confidential customer information has become essential for any firm which collects or stores personally identifiable data. the financial services industry operates and deals with a significant amount of confidential client and customer data for daily business transactions. though many organizations are taking strides to improve their privacy practices, and consumers are becoming more privacy-aware, it remains a tremendous burden for users to manage their privacy @xcite . nlp has major applications in the finance industry for many tasks such as detection of entities for gross tax calculation from invoice and payroll data, categorising different kinds of financial documents based on type, grouping of financial documents based on semantic similarity, sentiment analysis of financial text @xcite , conversational bots for banking systems, investment recommendation engines etc. text classification can be extended to many nlp applications including sentiment analysis, question answering, and topic labeling . for example, financial or government institutions that wish to train a chatbot for their clients cannot be allowed to upload all text data from the client-side to their central server due to strict privacy protection statements @xcite . at this point, applying the federated learning paradigm presents an approach to solve the dilemma due to its advances in privacy preservation and collaborative training where the central server can train a powerful model with different local labeled data at client devices without uploading the raw data considering increasing privacy concerns in public. the goal of this paper is to propose a privacy enabled text classification system, combining stateof-the-art transformers (bert and roberta) with differential privacy, on both centralized and fl based setups, exploring different privacy budgets to investigate the privacy-utility trade-off and see how they perform when trying to classify financial document-based text sequences. for the federated setups, we try to explore both iid (independent and identically distributed) and non-iid distributions of data. ## related work deep learning techniques have often been used to learn text representations via neural models by language application. the input text can give us individual demographic information about the author. sentiment analysis can be used for the classification or categorization of financial documents. xing studies have been conducted on training differentially private deep models with the formal differential privacy approach in the literature @xcite @xcite . fernandes et al. discuss the security through differential privacy in textual data. @xcite in his work portrays the use of dp in the generation of contextually similar messages for honey encryption which encrypts messages using low minentropy keys such as passwords. federated learning is another privacy-enhancing approach @xcite @xcite @xcite , which relies on distributed training of models on devices and sharing of model gradients. liu et al. show how fl can be used for decentralized training of heavy pre-trained nlp models. basu et al. in their work have shown a detailed benchmark comparison of multiple bert based models with dp and fl for depression detection. jana and biemann in their work show a differentially private sequence tagging system in a federated learning setup. ## dataset the key arguments for the low utilization of statistical techniques in financial sentiment analysis have been the difficulty of implementation for practical applications and the lack of high-quality training data for building such models. especially in the case of finance and economic texts, annotated collections are a scarce resource and many are reserved for proprietary use only. for this reason, we use the financial phrase bank dataset @xcite which was also used for benchmarking the pre-trained finbert model for sentiment analysis @xcite . the dataset includes approximately 5000 phrases/sentences from financial news texts and company press releases. the objective of the phrase level annotation task is to classify each example sentence into a positive, negative or neutral category by considering only the information explicitly available in the given sentence. since the study is focused only on financial and economic domains, the annotators were asked to consider the sentences from the viewpoint of an investor only; i.e. whether the news may have a positive, negative or neutral influence on the stock price. as a result, sentences that have a sentiment that is not relevant from an economic or financial perspective are considered neutral. given a large number of overlapping annotations (5 to 8 annotations per sentence), there are several ways to define a majority vote-based gold standard. to provide an objective comparison, the authors have formed 4 alternative reference datasets based on the strength of majority agreement. for the purpose of this task, we use those sentences with 75% or more agreement. the final dataset has 3453 sentences in total out of which 60% belong to the neutral class, 28% belong to the positive class and 12% belong to the positive class. ## preliminaries today, the text is the most widely used communication instrument.for years, researchers are studies focusing on implementing different approaches that make possible machines to imitate human reading @xcite . natural language processing(nlp) lays a bridge between computers and natural languages by helping machines to analyze human language @xcite .devlin et al. developed a model which is based on bidirectional encoder representation @xcite . roberta is a modified form of bert @xcite . ## experimental results in the scope of the study, the finbert (pre-trained) model is used as the base model. two nlp models were trained by implementing dp and fl. in this section, the results presented in the tables are discussed. the results placed in the tables are the average and the standard deviation of the results obtained after running the models thrice. the dataset was split into train set and test set with 80:20 train test ratio. bert and roberta based models were used for the language modelling part. it should be noted that the table contain the average and the standard deviation of the results obtained after running the models 3 times. in baseline mode, we can see that roberta has a slight improvement over bert because of its robustness owing to a heavier pre-training procedure. we also notice that with the increase in epsilon values, the amount of standard deviation decreases as the model approaches towards its vanilla variant (without dp noise). we also empirically observe that with increase in , accuracy of the models also increases. this happens because as the value of increases, privacy decreases with the addition of noise from a smaller range which results in smaller variance. consequently, the accuracy of the model increases. inherently, applying dp to deep learning yields loss of utility due to the addition of noise and clipping. we can also observe that the performance of federated language models still lies behind that of centralized ones. ## conclusion financial data is highly sensitive , hence the risks of collecting and sharing data can limit studies. financial organizations work with a lot of confidential user data and therefore highly value protecting the data to retain the integrity of the user and we need to delve into research of private training of machine learning models to ensure this. during this study, we benchmark the utility of privacy models while attempting to preserve the performance of sota transformer models such as bert and roberta. our empirical results show that the models show better performance with increasing ε as expected with the decrease in noise. the models come close to the performance of the baseline models near the higher ε values.the dp + fl shows a similar trend which showcases a greater protection feature without compromising the performance. as future work, we hope to improve our models further by hyper-parameter tuning, freezing partial layers of the nlp model and implementing focal loss on the unbalanced dataset to better the results. the complete code to this paper can be found here: https://www.github.com/tiasa2/privacyenabled-financial-text-classification-using-differential-privacy-and-federated-learning.