Multi-Label Intent Detection via Contrastive Task Specialization of Sentence Encoders (2022)

Abstract: Deploying task-oriented dialog ToD systems for new domains and tasks requires natural language understanding models that are 1) resource-efficient and work under low-data regimes; 2) adaptable, efficient, and quick-to-train; 3) expressive and can handle complex ToD scenarios with multiple user intents in a single utterance. Motivated by these requirements, we introduce a novel framework for multi-label intent detection (mID): MultI-ConvFiT (Multi-Label Intent Detection via Contrastive Conversational Fine-Tuning). While previous work on efficient single-label intent detection learns a classifier on top of a fixed sentence encoder (SE), we propose to 1) transform general-purpose SEs into task-specialized SEs via contrastive fine-tuning on annotated multi-label data, 2) where task specialization knowledge can be stored into lightweight adapter modules without updating the original parameters of the input SE, and then 3) we build improved mID classifiers stacked on top of fixed specialized SEs. Our main results indicate that MultI-ConvFiT yields effective mID models, with large gains over non-specialized SEs reported across a spectrum of different mID datasets, both in low-data and high-data regimes.

## introduction task-oriented dialog (tod) systems allow users to interact with computer applications through conversation in order to solve a particular task with well-defined semantics @xcite . tod supports a multitude of applications such as automating different customer service tasks, facilitating bookings in hospitality and travel industries, or providing assistance in healthcare or finance @xcite @xcite . intent detection (id), a task of recognizing the user's intent or goal from their utterance, is a crucial component of any tod system @xcite @xcite . intent detectors that adhere to industry standards should satisfy the following three requirements @xcite . 1 (r1) scalability and resource efficiency. they must be quickly bootstrapped for new domains and tasks. however, this process requires creating expensive annotations for each domain and task of interest, which calls for sample-efficient id methods that achieve strong performance in low-data scenarios. in this work, we propose a novel framework for mid, termed multi-convfit (multi-label intent detection via contrastive conversational fine-tuning), that satisfies the three requirements r1-r3. the framework's pipeline is illustrated in figure 1 . previous work typically used fixed general-purpose @xcite sentence encoders (ses) @xcite combined with tunable intent classifiers for efficient single-label id. in this work, we propose a modular framework that contrastively fine-tunes general-purpose encoders, using mid data annotations implicitly, to yield task-specialized sentence encoders. all the taskspecific 'adaptation' knowledge after contrastive fine-tuning can be injected into small adapter modules. @xcite . such adapter modules are then used to adapt the underlying general-purpose se which already stores plenty of useful semantic knowledge -a single large model that serves multiple tasks and domains -into the task-specialized se. the contrastive procedure creates a semantic space which better aligns with intent classes of a particular mid task, as demonstrated in figure 2 . consequently, such fixed task-specialized ses enable learning improved mid classifiers that outperform mid classifiers learnt on top of the original general ses. our key results indicate effectiveness and robustness of multi-convfit, yielding state-of-theart results across four representative mid datasets, both in low-data and high-data scenarios, while offering modularity and efficient fine-tuning and inference. additional analyses indicate multi-convfit's versatility and wide applicability: it can be used with a range of pretrained ses and lms, and it leads to gains across different domains, dataset sizes, and intent set sizes. ## methodology the full overview of multi-convfit, aiming to satisfy the requirements r1-r3 from §1, is provided in figure 1 . in what follows, we discuss its main components in detail: contrastive taskspecialization of general-purpose input encoders using annotated mid data ( §2.1); a classifier for multilabel id stacked on top of the fixed encoder ( §2.2); and a more efficient variant of the framework which combines contrastive tuning with adapters ( §2.3). preliminaries. for any input text t, we obtain its encoding @xmath0, where f is an encoding function of any input encoder model (i.e., lm, generalpurpose or task-specialized se). t is tokenized into subwords using each encoder's dedicated tokenizer. the final encoding t is created via a pooling operation such as (a) using the [cls] token as the text representation, (b) or mean-pooling the output subword encodings. following prior work @xcite , we opt for mean-pooling as a better-performing option. further, we assume that |s| annotated mid data examples are available: they comprise a set of pairs @xmath1, l 1 ), . . . , (s i , l i ), . . . , (s |s| , l |s| )}, where s i are sentences/examples, each annotated with a set of l @xmath2,1 , . . . , l i,m i } labels, where m @xmath3. each label l is in fact one of the |c| intent classes from the set @xmath4, . . . , c |c| }. ## experimental setup in what follows, we outline our experimental setup, focused on evaluating and improving performance ). foot_3 mlm12 (its size is 120 mb) comprises l @xmath5, with hidden size h @xmath6; l @xmath7; l @xmath8. all ses have been obtained in prior work @xcite via contrastively fine-tuning their underlying lms on a set of more than 1b sentence pairs, which comprises various data such as reddit 2015-2018 comments @xcite , natural questions @xcite , paq (question, answer) pairs @xcite , etc. 5 furthermore, in order to test the impact of the chosen input/underlying model (i.e., sentence encoders versus language models) we also contrastively fine-tune the original lms instead of their se counterparts (see figure 1 again): we refer to the respective input lms as mlm12-lm, mpnet-lm, and drob-lm. multi-label id datasets, until very recently, have been few and far between @xcite , as prior id research predominantly focused on single-label scenarios @xcite . we experiment with a representative set of multilabel id datasets, covering 1) four diverse domains, 2) ontologies with different sizes of the intent sets (from 18 to 118 intents), 3) varied dataset size, and 4) different average number of intents per example. a complete summary is provided in mixatis @xcite is the only dataset whose 'multi-label nature' was achieved synthetically through concatenation of single-label examples from the original atis dataset @xcite . on the other hand, the other datasets in our evaluation are natively multi-label, relying on the concept of combining the so-called intent modules; see the work of @xcite for further details. some examples of multi-label sentences from each mid dataset are provided in appendix b. low-data versus high-data setups. due to the high cost of task-specific annotations, prior work on single-label id has recognized the importance of building and bootstrapping intent detectors in low-data regimes @xcite @xcite , and the same naturally holds also for multi-label id. in order to understand the behaviour of multi-convfit in such scenarios versus setups with more abundant task-annotated data, we conduct experiments in two standard data setups: 1) low-data and 2) high-data. for the banking, hotels, and insurance-faq datasets (see for mixatis, we leverage its development portion for our low-data experiments, and its training portion for the high-data setup (without leveraging development data for model selection; see next for our hyper-parameter selection procedure). contrastive specialization setup. following the suggested settings @xcite , we use the adamw optimizer @xcite . the learning rate for the c-fft variants is set to the standard value of 2e-5, while a higher learning rate of 4e-4 is used for the adapter-based c-adapt variants. for c-adapt, we opt for a standard efficient bottleneck adapter configuration, following @xcite : relu activation @xcite , with the adapter reduction factor of 4. 6 the warmup rate of 0.1 with cosine decay is used; weight decay rate is 0.02. we fine-tune for 10 epochs in low-data, and for 3 epochs in highdata setups, with the number of negatives @xmath9; foot_6 batch size is 32, and max sequence length is 128. classifier setup. we adopt the mlp classifier architecture from @xcite : it contains 1 hidden layer of size 512 with relu as non-linear activation. the values for the hyperparameters were largely adopted from prior work @xcite @xcite , both for contrastive fine-tuning and mlp training. we further fine-tuned them relying solely on one randomly sampled fold (fold 5) from banking with mlm12 and mlm12-lm in the low-data setup, and applied the same hyperparameters across all other models, setups, and runs. the classifiers' dropout rate is fixed to 0.4, and the threshold θ is fixed to 0.3 in all runs. we again train with adamw, with the standard warmup rate of 0.1, weight decay of 0.02, and the learning rate is set to 0.003; 600 epochs with the batch size of 32. unless noted otherwise, we always apply label smoothing with @xmath10.95. evaluation details. all the reported scores are averaged across three runs with three different random seeds. we report standard id evaluation metrics: f 1 and exact match accuracy (acc). ## results and discussion the main results are summarized in impact of contrastive task specialization. first, the results clearly demonstrate substantial and consistent gains achieved via contrastive taskspecialization. the strong performance boosts are present across the board, and span all input encoders (including both ses and lms), both finetuning variants (c-fft and c-adapt), all mid datasets, both low-data and high-data scenarios, and also all groups of examples with a different number of intents per example (see also figure 9 in appendix c). while prior work has proven that even general-purpose ses can support effective and efficient (single-label) id @xcite @xcite , here we demonstrate that 1) the efficient se-based approach is also beneficial for the multi-label id task, and 2) large performance improvements are achieved by transforming/adapting such general-purpose ses into task-specialized sentence encoders. ## conclusion and future work we proposed multi-convfit, a contrastive finetuning framework for multi-label intent detection (mid) that transforms general-purpose language models and sentence encoders (ses) into taskspecialized ses. such specialized ses facilitate efficient learning of mid classifiers stacked on top of the fixed sentence encodings @xcite . moreover, we demonstrate how to combine ses with lightweight adapter modules, resulting in a modular multi-tenant design of the multi-convfit framework. we demonstrate effectiveness and robustness of contrastive mid task specialization across a representative set of mid datasets, different input encoders, with large improvements especially in the most demanding low-resouce scenarios. we hope that multi-convfit will inspire more work on sample-efficient, modular, and highly adaptable multi-label intent detectors. there are multiple avenues for future research that can further improve various aspects of the proposed multi-convfit framework. for instance, in this work, for simplicity and clarity, we rely on globally set fixed threshold values θ, while such thresholds can also be adaptable, with differently calibrated values for different (sets of) intents @xcite . further, this work relied on a particular class of parameter-efficient fine-tuning methods, bottleneck adapters, as one of the most established methods available. however, as mentioned in §4, future work should also explore other parameterefficient methods @xcite , aiming to achieve an even better trade-off between performance and parameter-efficiency. in particular, driven by the efficiency requirements, we will investigate parameter-efficient methods that do not increase the model size at all, and thus maintain the same time efficiency at inference, such as methods based on low-rank adaptation @xcite or sparse fine-tuning @xcite .