Zero-Shot- BERT -Adapters: a Zero-Shot Pipeline for Unknown Intent Detection (2023)

Abstract: Intent discovery is a crucial task in natural language processing, and it is increasingly relevant for various of industrial applications. Identifying novel, unseen intents from user inputs remains one of the biggest challenges in this field. Herein, we propose Zero-Shot-BERT-Adapters, a two-stage method for multilingual intent discovery relying on a Transformer architecture, fine-tuned with Adapters. We train the model for Natural Language Inference (NLI) and later perform unknown intent classification in a zero-shot setting for multiple languages. In our evaluation, we first analyze the quality of the model after adaptive fine-tuning on known classes. Secondly, we evaluate its performance in casting intent classification as an NLI task. Lastly, we test the zero-shot performance of the model on unseen classes, showing how Zero-Shot-BERT-Adapters can effectively perform intent discovery by generating semantically similar intents, if not equal, to the ground-truth ones. Our experiments show how Zero-Shot-BERT-Adapters outperforms various baselines in two zero-shot settings: known intent classification and unseen intent discovery. The proposed pipeline holds the potential for broad application in customer care. It enables automated dynamic triage using a lightweight model that can be easily deployed and scaled in various business scenarios, unlike large language models. Zero-Shot-BERT-Adapters represents an innovative multi-language approach for intent discovery, enabling the online generation of novel intents. A Python package implementing the pipeline and the new datasets we compiled are available at the following link: https://github.com/GT4SD/zero-shot-bert-adapters.

## introduction language models (lm) and, in general, natural language processing (nlp) methodologies have a pivotal role in modern communication systems. in dialogue systems, understanding the actual intention behind a conversation initiated by a user is fundamental, as well as identifying the user's intent underlying each dialogue utterance @xcite . recently, there has been a growing interest in such applications, evidenced by the plethora of datasets and competitions established to tackle this problem. @xcite @xcite . unfortunately, most of the available most of the available production systems leverage models trained on a finite set of possible intents, with limited or no possibility to generalize to novel unseen intents @xcite . such limitations constitute a significant blocker for most applications that do not rely on a finite set of predetermined intents. they, instead, require a constant re-evaluation based on users' inputs and feedback. additionally, most of the datasets for intent classification are mainly available for the english language, significantly limiting the development of dialogue systems in other languages. expanding the set of available intents online is imperative to build better dialogue systems closely fitting users' needs. besides better-matching user expectations, this also enables the definition of new common intents that can slowly emerge from a pool of different users. this is a key aspect of building systems that are more resilient over time and can follow trends appropriately. current supervised techniques fall short of tackling the challenge above since they cannot usually discover novel intents. these models are trained by considering a finite set of classes and cannot generalize to realworld applications @xcite . here, we propose a multi-language system that combines dependency parsing @xcite @xcite to extract potential intents from a single utterance and a zero-shot classification approach. we rely on a bert transformer backbone @xcite ural language inference) @xcite to select the intent that is best fitting the utterance in a zero-shot setting. in the nli fine-tuning, we leverage adapters @xcite to reduce memory and time requirements significantly, keeping base model parameters frozen. we evaluated our approach, zero-shot-bert-adapters, for english, italian, and german, demonstrating the possibility of its application in both high and low-resource language settings. the design focuses on a production setting where automatic intent discovery is fundamental, e.g., customer care chatbots, to ensure smooth user interaction. figure 1 depicts how the zero-shot-bert-adapters pipeline fits in an intent detection system. from now on, for simplicity, we will refer to zero-shot-bert-adapters as z-bert-a in tables and figures. there have been various efforts aiming at finding novel and unseen intents. the most popular approach is a two-step method where, initially, a binary classifier determines whether an utterance fits the existing set of intents, and, finally, zero-shot techniques are applied to determine the new intent @xcite @xcite . @xcite proposed an approach leveraging transformers-based architectures, while most relied on rnn architectures like lstms @xcite . for our problem of interest, two notable efforts have attempted to address the issue of unseen intent detection @xcite . @xcite handled novel intent discovery as a clustering problem, proposing an adaptation of k-means. here, the authors leverage dependency parsing to extract from each cluster a mean action-object pair representing the common emerging intent for that particular cluster. recently, large language models (llms) achieved remarkable zero-shot generalization @xcite @xcite capabilities. @xcite report benchmarks on intent classification using gpt-3 @xcite and analyze performance both in zero and few-shot settings. while these solutions are compelling, they are only an ideal fit for some real-world use cases. models of these sizes can be expensive to operationalize, especially in on-premise settings with strict hardware constraints. while there is no literature on the novel intent discovery and zero-shot classification in a multilingual setting, there has been a recent attempt to perform zero-shot intent detection @xcite . the authors study the problem of intent detection using attention-based models and cover low-resource languages such as italian. other approaches leveraging multilingual solutions, such as nicosia et al., focus on a zero-shot multilingual semantic parsing task. here, the authors present a novel method to produce training data for a multilingual semantic parser, where machine translation is applied to produce new data for a slot-filling task in zero-shot. ## background in intent discovery, we aim at extracting from a single utterance x a set of potentially novel intents y i and automatically determine the best fitting one for the considered utterance. we can cast intent discovery as a natural language inference (nli) problem. we can rely on a language model φ(x, γ(x)) to predict the entailment between the utterance (x) and a set of hypotheses based on the candidate intents (γ(x)) where γ is a function used to extract hypotheses from the set of potential intents y i . as previously shown by xian et al., using nli models for zero-shot classification represents a practical approach in problems where the set of candidate intents is known. in practice, the classification problem becomes an inference task where a combination of a premise and a hypothesis are associated with three possible classes: entailment, neutral and contradiction. yin et al. have shown how this approach allows considering an input hypothesis based on an unseen class. it then generates a probability distribution describing the entailment from the input premise-hypothesis pair, hence a score correlating the input text to the novel class. while this technique is highly flexible and, in principle, can handle any association between utterance and candidate intent, determining good candidates based on the analyzed utterance remains a significant challenge. ## approach herein, we focus on building a pipeline to handle unseen classes at inference time. in this context, we need both to generate a set of candidate intents from the considered utterance and to classify the provided input against the new set of candidate intents. we tackle the problem by implementing a pipeline in two stages. in the first stage, we leverage a dependency parser @xcite to extract a set of potential intents by exploiting specific arc dependencies between the words in the utterance. in the second stage, we leverage the set of potential intents as candidate classes for the utterance intent classification problem using a zero-shot approach @xcite based on nli relying on a bert-based model @xcite , as depicted in figure 2 . the model is tuned with adapters @xcite for the nli task (bert-a) and is then prompted with premisehypothesis pairs for zero-shot classification on the candidate intents, completing the zero-shot-bert-adapters pipeline. the entire pipeline's implementation follows the hugging face pipeline api from the transformers library @xcite . ## datasets we consider two datasets in our analysis: snli @xcite and banking77-oo @xcite : we also explore the effects of pretraining leveraging an nli adaptation of banking77 @xcite . to investigate the impact of pretraining on similar data, we extended the banking77 dataset by casting the intent classification task as nli. we consider the input utterance the premise and extract the most relevant word associated with it using keybert @xcite . the word is then used to generate an entailed hypothesis based on the corresponding synset definition from wordnet via nltk @xcite . exemplar samples of synthetic banking77 nli data are reported in to analyze the validity of our approach in a multi-lingual setup with lower data availability, we compiled and made available both an italian and a german version of the banking77 dataset. we followed a two-step process involving translation using the google translation api @xcite and manual annotation and correction by human agents to maintain data quality following the protocol described in @xcite . ## training we fine-tune two versions of bert-a (bertbased transformer with adapters). the first version is trained for nli on the snli dataset using the original split for training, validation, and testing @xcite . the second version also considers the previously introduced banking77-oos-nli which keeps the split of inscope and the out-of-scope ratio of 50 -27 out of 77 total intents. depending on the language considered we use different bert pretrained models: for english we use bert-base-uncased @xcite ; for italian we use bert-base-italianxxl-uncased (dbmdz, 2020); for german we use bert-base-german-cased @xcite . during training, we keep the bert backbone frozen and only update the added adapter layers to minimize training time and memory footprint. by freezing all the original layers and letting the model be trained only on the adaptive layers, we end up with and a warm-up scheduler. the models have been fine-tuned for a total of 6 epochs using early stopping. our computing infrastructure for training the models run on nvidia tesla t4 gpus using py-torch and python 3.7. in this configuration, each training epoch took approximately one minute to run. during the testing phase each run took around 36 seconds, including the overhead due to metrics computation. ## evaluation first, we analyzed the performance of the bert-a component and evaluated its results on a nli task using accuracy, precision, and recall. afterward, we compared its result on the zero-shot classification task with other available models on the same banking77 split using accuracy. in this initial evaluation, the intents were known. the baselines considered in this setting were bart0 @xcite , a multitask model with 406 million parameters based on bart-large @xcite based on prompt training; and two flavors of zero-shot ddn (zs-dnn) @xcite with both encoders universal sentence encoder (use) @xcite and sbert @xcite . in the unknown intent case, we compared the pipeline against a set of zero-shot baselines based on various pre-trained transformers. as baselines, we included bart-large-mnli @xcite as it achieves good performance in zero-shot sequence classification. we used this model as an alternative to our classification method, but in this case, we maintained our dependency parsing strategy for the intent generation. as a hypothesis for the nli-based classification, we used the phrase: "this example @xmath0. recently, large lms have demonstrated remarkable zero-shot capabilities in a plethora of tasks. thus, we included four llms as baseline models, namely t0 @xcite , gpt-j @xcite , dolly (lab, 2023) and gpt-3.5 @xcite ) using textdavinci-003 via openai api (base model behind instructgpt and gpt-3.5 @xcite ). in these models, the template prompt defines the task of interest. we examined whether such llms can serve the end-to-end intent extraction (intent generation and classification). we investigated them both in a completely unsupervised, i.e., a zero-shot settingr and using intents generated with our dependency parsing method. in the former case, the given input is just the utterance of interest, while in the latter case, the provided input includes the utterance and the possible intents. in both cases, the generated output is considered as the extracted intent. ## results firstly, we report the performance of bert-a on the nli task, see finally, we report zero-shot-bert-adapters on the unknown intent discovery task. tables 4 and 5 show the performance of bert-a fine-tuned on snli and banking77-oos-nli in comparison with a selection of zero-shot baselines for intent discovery. for the baselines, we show the performance on zero-shot classification given potential intents from our approach (prompts 3, 4, and 6) and on new intents discovery solely using the input utterance (prompts 1, 2, and 5). ## conclusions and future work we proposed a pipeline for zero-shot prediction of unseen intents from utterances. we performed a two-fold evaluation. first, we showed how our bert-based model fine-tuned with adapters on nli can outperform a selection of baselines on the prediction of known intents in a zero-shot setting. secondly, we evaluated the full pipeline capabilities comparing its performance with the results obtained by prompting llms in an unknown intent set considering multiple languages. these results prove that our solution represents an effective option to extend intent classification systems to handle unseen intents, a key aspect of modern dialogue systems for triage. moreover, using a relatively lightweight base model and relying on adaptive fine-tuning, the proposed solution can be deployed in the context of limited resources scenarios, e.g., on-premise solutions or small cloud instances. in this way, we don't compromise on model performance, as we showed that smaller models like the ones powering our pipeline can outperform llms in intent discovery. an interesting avenue to explore in the future consists in relying on zero-shot learning approaches in the intent generation phase @xcite without compromising on model size and inference requirements. zero-shot-bert-adapters and the italian and german datasets are available at the following link: https://github.com/gt4sd/ zero-shot-bert-adapters .