LARA : Linguistic-Adaptive Retrieval-Augmentation for Multi-Turn Intent Classification (2024)

Abstract: Multi-turn intent classification is notably challenging due to the complexity and evolving nature of conversational contexts. This paper introduces LARA, a Linguistic-Adaptive Retrieval-Augmentation framework to enhance accuracy in multi-turn classification tasks across six languages, accommodating numerous intents in chatbot interactions. LARA combines a fine-tuned smaller model with a retrieval-augmented mechanism, integrated within the architecture of LLMs. The integration allows LARA to dynamically utilize past dialogues and relevant intents, thereby improving the understanding of the context. Furthermore, our adaptive retrieval techniques bolster the cross-lingual capabilities of LLMs without extensive retraining and fine-tuning. Comprehensive experiments demonstrate that LARA achieves state-of-the-art performance on multi-turn intent classification tasks, enhancing the average accuracy by 3.67% from state-of-the-art single-turn intent classifiers.

## introduction a chatbot is an essential tool that automatically interacts or converses with customers. it plays a crucial role for international e-commerce platforms due to the rising consumer demand for instant and efficient customer service. chatbots represent a critical component of dialogue systems @xcite ) that can answer multiple queries simultaneously by classifying intent from the user's utterance to reduce waiting times and operational costs. naturally, the interaction with users could turn into a multi-turn conversation if they require more detailed information about the query. developing an intent classification model for a dialogue system is not trivial, even if it is a typical text classification task. as we must consider contextual factors such as historical utterances and intents, failing to understand session context while recognizing user intention often leads to visible errors. it would invoke a completely wrong application or provide an unrelated answer @xcite . as a result, it faces several challenges in dialogue understanding. the biggest challenge is that multi-turn datasets are hard to collect. while there are some studies on dialogue understanding in multi-turn intent classification @xcite @xcite , they are made under the assumption of the availability of multi-turn training data, which is usually not the case in the real world. figure 1 shows the annotation pipeline of multiturn intent classification. unlike emotion recognition in conversation (erc) with only less than 10 classes or topic classification within dialogue state tracking (dst) with tens of topics, there are hundreds of intents within the knowledge base of a chatbot to cover users' specific intents in each market, which increases the complexity of classification tasks and multi-turn data annotation. annotators can easily make mistakes and spend more time making decisions due to the numerous intents. combined, these make it a high-cost and timeconsuming annotation task, and it is unrealistic to annotate large-scale multi-turn datasets manually. however, the performance will most likely suffer without enough training sample size. this calls for a more efficient method in solving the challenge @xcite . to tackle the above challenge, we propose linguistic-adaptive retrieval-augmentation, or lara, which offers a pipeline of techniques to adopt only single-turn training data to optimize multi-turn dialogue classification. lara first leverages an xlm-based model trained on singleturn classification datasets for each market, thus simplifying data construction and maintenance. subsequently, lara advances the field by selectin summary, the contributions of this paper are as follows: in-context retrieval: in-context learning (icl) with llm like gpt-3 (brown et al., 2020) demonstrates the significant improvement on fewshot/zero-shot nlp tasks. icl has been successful in utterance-level tasks like intent classification @xcite . as for the retrieval part, most research on in-context learning (icl) usually deals with single sentences or document retrieval, but we are interested in finding and understanding dialogues. generally, there are two types of systems to find the relevant dialogues: the first is lm-score based retrieval. they @xcite check the probability of a language model, like gpt-3, to decode the right answer based on an example. the second type defines similarity metrics between task results and uses them as the training objective for the retriever. both k-highest and lowest examples are used as positive and negative samples to help the system learn. the most pertinent research on dialogue retrieval concentrates on areas such as knowledge identification @xcite and response selection @xcite . our objectives and settings differ from them. ## lara: linguistic-adaptive retrieval-augmentation the lara framework shown in figure 2 addresses the multi-turn intent recognition challenge through zero-shot in-context learning with singleturn demonstrations guided by a crafted instruction prompt. first, a single-turn classification model m c is trained on single-turn dataset and used to narrow down the intents to be included in the icl prompt, which are henceforth referred to as candidate intents. this step is necessary due to the limited llm context window, and it also helps to filter out extra noise from direct demonstration retrieval. then, for every candidate intent, in-context demonstrations are selected by retrieving single-turn examples that are semantically similar to the multiturn test sample. finally, an instruction prompt for multi-turn classification is formulated by combining the demonstrations and test user queries. ## results and discussions lara, on the other hand, with prompt p f ormatted , achieves the best results on most markets without any multi-turn training data. on average, it improved accuracy by 3.67% compared with selective concatenation. especially on the non-english markets, it also improved by 2.96%, 4.18% and 4.00% on th, vn and br separately. this highlights the linguistic-adaptivity of the method on broad languages. the only market that does not outperform the baselines is id, which most probably can be attributed to the language ability of open-sourced llms in handling the local slang and abbreviations in casual conversation. after all, the backbone model used in baselines is pre-trained directly on the in-domain chat log data, while the llm models are used out of the box. replacing the label names with non-related symbols in p symbolic significantly hurts the performance of in-context learning. on the other hand, minimal changes to label names in p prepend does ## ablation studies to validate our motivation and model design, we ablate single-turn model m c in candidate intent selection and retrievers in icl retrieval. the comparison is made on the original p prompt variant. ## limitation lara does not currently address the detection of out-of-domain utterances, a critical aspect for online dialogue systems. future research is necessary to explore methods for incorporating this capability and to assess their feasibility. furthermore, the resilience of the method to user intent shifts has not been examined. additionally, the multi-component architecture, which integrates text classification, retrieval, and icl, adds to the implementation complexity. in appendix d, we propose a lightweight solution that is more suitable for applications with limited deployment resources. ## conclusion this paper introduced lara, a framework that leverages linguistic-adaptive retrieval-augmentation to address multi-turn intent classification challenges through zero-shot settings across multiple languages. unlike other supervised fine-tuning (sft) models, which require a hard-tocollect multi-turn dialogue set, our method requires only a single-turn training set to train a conventional model, combining it with an innovative incontext retrieval augmentation for multi-turn intent classification. lara substantially improved user satisfaction by 1% over multi-turn sessions, reduced the transfer-to-agent rate by 0.5% and saved the cost of hundreds of agents, which is a key business metric in our industrial application. the empirical results underscore lara's capability to enhance intent classification accuracy by 3.67% over existing methods while reducing inference time, thus facilitating real-time application adaptability. its strategic approach to managing extensive intent varieties without exhaustive dataset requirements presents a scalable solution for complex, multi-lingual conversational systems. for future work, we intend to extend and apply the lara framework to recommendation tasks in other domains, such as understanding how user intent may shift during a poi or itinerary recommendation for tourism purposes @xcite . this will enable us to better capture evolving user preferences due to temporal shifts, changing contexts, and individual or group behavioural patterns, which is also applicable to general sequence recommendations. dian yu, luheng he, yuan zhang, xinya du, panupong pasupat, and qi li. 2021. few-shot intent classification and slot filling with retrieved examples. in north american chapter of the association for computational linguistics. chunyuan yuan, wen jie zhou, mingming li, shangwen lv, fuqing zhu, jizhong han, and songlin hu. 2019. multi-hop selector network for multi-turn response selection in retrieval-based chatbots. in conference on empirical methods in natural language processing. jinghan zhang, yuxiao ye, yue zhang, likun qiu, bin fu, yang li, zhenglu yang, and jian sun. 2020. multi-point semantic representation for intent classification. in proceedings of the aaai conference on artificial intelligence, volume 34, pages 9531-9538. he zhu, chong zhang, junjie huang, junran wu, and ke xu. 2023. hitin: hierarchy-aware tree isomorphism network for hierarchical text classification. in annual meeting of the association for computational linguistics.