Learning to Adapt Large Language Models to One-Shot In-Context Intent Classification on Unseen Domains (2024)

Abstract: In this paper, we explore one-shot in-context intent classification using large language models (LLMs) with the goal of minimizing the effort required to adapt models to unseen domains. To enhance the one-shot in-context learning capabilities of LLMs, we employ in-context tuning, leveraging its cross-domain transferability to unseen domains.To this end, we introduce the IC-collection, a compilation of open-source intent classification datasets from diverse domains, which are meticulously divided into held-in and held-out datasets.Our experiments demonstrate the effectiveness of the proposed method, showing that our model, with only 7B parameters, not only outperforms GPT-4 on intent classification but also achieves state-of-the-art in unseen domains with only one-shot demonstrations.Both our benchmark and model will be made publicly available to advance research in the chatbot systems.

## introduction building accurate intent classifiers remains a significant challenge for chatbot systems in real-world scenarios. the labor-intensive process of labeling utterances for new and evolving intents complicates the development and maintenance of chatbots across diverse domains @xcite . in this study, we aim to minimize the effort required to adapt intent classification (ic) models to unseen domains and intents. to this end, in-context learning, which leverages large language models (llms) to achieve high performance on various tasks with only a few inputoutput pairs, presents a promising direction @xcite . recent research has demonstrated that prompting llms only with few-shot examples of text-label pairs can outperform fine-tuned models @xcite . despite these advancements, they primarily focus on * indicates equal contribution. this leads us to a research question: can we push the limits of in-context learning ability to perform one-shot in-context intent classification? to address this, we propose adopting in-context tuning @xcite for training on seen domains to enhance in-context learning ability on unseen domains. we leverage the crosstask transferability of in-context tuning to improve cross-domain transferability. to this end, we first construct ic-collection, a benchmark designed for training a model in intent classification across diverse domains and evaluating performance on unseen domains. the iccollection is a mixture of open-source intent classification datasets, encompassing 13 held-in and 3 held-out datasets foot_0 . after that, we present osic2-7b, where osic2 stands for one-shot in-context intent classification, by training a 7b language model on our benchmark. our results demonstrate that osic2-7b achieves state-of-the-art (sota) in unseen domains with only one-shot demonstration, even surpassing gpt-4 in the same setting. our contributions are as follows: ## data collection we compile the ic-collection from diverse opensource resources to cover various domains. during the dataset collection process, several considerations are made: (i) non-english utterances are excluded from multilingual datasets, (ii) only the initial turns from multi-turn interaction dataset are utilized, (iii) duplicate utterances within the dataset were removed, and (iv) multi-labeled utterances are excluded. in order to evaluate generalization capabilities on unseen domains, the datasets are divided into heldin and held-out sets foot_1 . held-out datasets were selected based on their minimal overlap with the heldin datasets, both in terms of domains and intents foot_2 . through preliminary experiments conducted to assess the overlap between the held-out and held-in datasets, we confirmed that the impact of the overlap is negligible (see details in ## our approach task definition given an utterance, one-shot incontext intent classification (osic2) identifies the correct intent from a list of provided intents, using only one example per intent as a reference. the following is the formulation of osic2: training our ultimate goal is to train a model that can effectively leverage one-shot examples at test time, enabling it to adapt proficiently to new domains. to this end, we employ the in-context tuning framework following @xcite ; @xcite . while the original concept of in-context tuning is developed for cross-task transfer learning, we apply this approach to cross-domain transfer learning within the intent classification task. prompt construction as illustrated in appendix a, we concatenate the task instruction, the in-context examples, and the target utterance into a single input sequence. also, we use a fixed prompt format for ic-collection to focus on the changes in the intent list along with the corresponding oneshot examples. specifically, we first construct the training pool by randomly selecting maximum of ten examples foot_6 for each intent from the held-in datasets in order to ensure a balance for each intent. from this training pool, we randomly draw a one-shot example for each instance, based on the following assumpof intents @xcite and labeling noise in training instances @xcite . when inferring on held-out datasets, we utilize a fixedrepresentative utterances sourced from the original training datasets as in-context examples. note that the intent list at the instance level is confined within each dataset. preprocessing for all datasets, minimal processing is applied to ensure data quality. the pattern of intent name is standardized with the steps: (i) converting all letters to lowercase, (ii) joining words with underscores, and (iii) writing out abbreviations. furthermore, intents categorized as outof-scope (e.g., no_nodes_detected) are excluded across all datasets due to their differing levels of granularity compared to other defined intents. ## conclusion this paper investigates one-shot intent classification as an extreme case of data scarcity in realworld scenarios. by enhancing the in-context learning capabilities of llms, our specialized 7b-scale model achieves state-of-the-art performances on held-out datasets, even in the context of unseen domains. to promote research on the creation of accurate intent classifiers that are easily adaptable to any domain, we release our data collection and model.