Learning Better Intent Representations for Financial Open Intent Classification (2022)

Abstract: With the recent surge of NLP technologies in the financial domain, banks and other financial entities have adopted virtual agents (VA) to assist customers. A challenging problem for VAs in this domain is determining a user’s reason or intent for contacting the VA, especially when the intent was unseen or open during the VA’s training. One method for handling open intents is adaptive decision boundary (ADB) post-processing, which learns tight decision boundaries from intent representations to separate known and open intents. We propose incorporating two methods for supervised pre-training of intent representations: prefix tuning and fine-tuning just the last layer of a large language model (LLM). With this proposal, our accuracy is 1.63% - 2.07% higher than the prior state-of-the-art ADB method for open intent classification on the banking77 benchmark amongst others. Notably, we only supplement the original ADB model with 0.1% additional trainable parameters. Ablation studies also determine that our method yields better results than full fine-tuning the entire model. We hypothesize that our findings could stimulate a new optimal method of downstream tuning that combines parameter efficient tuning modules with fine-tuning a subset of the base model’s layers.

## introduction as the popularity of virtual agent (va) dialogue systems increases and their application in the finance domain is explored, the problem of intent classification demands greater attention. several recent finance-specific vas leverage technical advancements to respond to natural language queries @xcite . determining the user's intent ensures that the va can appropriately tailor its responses and/or perform relevant actions. initial works in intent classification limited the task to classifying utterances as one of n known intents ## methodology in this section we explain our procedure for open intent classification. section 3.1 describes prefixtuning, the method we supplement partial finetuning with. section 3.2 provides a brief summary of training the original adb method that we have extended @xcite . ## results our main results and respective baseline comparisons are presented in tables 2 and 3 . we note that the dataset with the lowest performance gain is stackoverflow. @xcite found that their novel post-processing method, adb, was most effective on this dataset it is important that an open intent classification method balances the performance on known classes while still identifying open intents. the following ablations focus on the oos dataset since it covers multiple domains and we wanted to generalize beyond just the financial domain. ## conclusion we have shown that incorporating prefix-tuning with the adb intent representation pre-training method achieves sota results in the financial domain on the banking77 benchmark dataset and others. furthermore, our tuning method does not sacrifice excessive parameters count for the performance gain. the combination of prefix-tuning with fine-tuning only the last layer of transformer is simple yet novel to the best of our knowledge and surfaces interesting questions regarding the mechanisms they use to interact. we intend to address the limitations presented hereafter in the near future.