Going beyond research datasets: Novel intent discovery in the industry setting (2023)

Abstract: Novel intent discovery automates the process of grouping similar messages (questions) to identify previously unknown intents. However, current research focuses on publicly available datasets which have only the question field and significantly differ from real-life datasets. This paper proposes methods to improve the intent discovery pipeline deployed in a large e-commerce platform. We show the benefit of pre-training language models on in-domain data: both self-supervised and with weak supervision. We also devise the best method to utilize the conversational structure (i.e., question and answer) of real-life datasets during fine-tuning for clustering tasks, which we call Conv. All our methods combined to fully utilize real-life datasets give up to 33pp performance boost over state-of-the-art Constrained Deep Adaptive Clustering (CDAC) model for question only. By comparison CDAC model for the question data only gives only up to 13pp performance boost over the naive baseline.

## introduction allegro is one of largest the e-commerce marketplace in central eastern europe region that connects buyers and merchants. it has millions of active users. therefore, the good functioning of the customer experience (cx) department is crucial as it provides the necessary support, resolves emerging issues, and answers user questions. task-oriented chatbots relieve humans by automatically resolving the most repetitive and trivial issues. they usually have a pre-defined set of user intents with matching template answers. then, when a user asks a question, the intent classifier detects the question intent and returns the matching response. creating a reliable and comprehensive chatbot requires massive work to discover, define, and maintain a set of intents with training examples. with the continuous development of marketplace platforms, new intents constantly appear as new features are introduced. therefore, the automated intent discovery system becomes a critical component. novel intent discovery is performed offline on historical data. in the context of personalized intelligence assistants existing approaches @xcite @xcite focus on learning transferable features with utterance encoders that guide the discovery on unlabeled data with a handful of labeled examples belonging to known intents. however, at allegro our main communication form is emails, and we have access to much richer conversational data that can improve discovery performance. a large body of historical conversational data (user questions and consultants' answers) can be leveraged in two ways. firstly, to better initialize message encoders and secondly by performing intent discovery on conversational data as an additional signal. additionally, a form of weak supervision is available: keywords (or tags) added by the consultants that help them understand past cases. the paper's main contribution is the demonstration that incorporating additional signals like conversational structure or weak labels into the existing intent discovery method results in better overall performance. we pre-trained for domain adaptation three encoders using conversational data and weak labels. we devised conv, a method for fine-tuning on conversational data (i.e., question and answer) for the clustering task using a three-headed encoder. to the best of our knowledge, this result was not reported in the public literature. ## evaluation we describe our experimental setup for novel intent discovery. we prove the efficiency of the proposed method on real-world communication datasets. to verify gains from different framework components, we present more results in the ablation section (sec. 5). ## ablation we attribute the improvement in performance to all three method components: domain adaptation during pre-training with conversational and weak label signal, state-of-the-art training scheme cdac, and leveraging of conversation structure with our conv method introduced in section 3.4. ## conclusions this paper describes an intent discovery pipeline deployed on a large e-commerce platform. the access to real-life datasets allows extending the established intent discovery models to better leverage vast amounts of unlabelled data, its conversational structure, and additional signals like weak labels. in particular, we learn the following lessons: