Supervised Neural Clustering via Latent Structured Output Learning: Application to Question Intents (2021)

Abstract: Previous pre-neural work on structured prediction has produced very effective supervised clustering algorithms using linear classifiers, e.g., structured SVM or perceptron. However, these cannot exploit the representation learning ability of neural networks, which would make supervised clustering even more powerful, i.e., general clustering patterns can be learned automatically. In this paper, we design neural networks based on latent structured prediction loss and Transformer models to approach supervised clustering. We tested our methods on the task of automatically recreating categories of intents from publicly available question intent corpora. The results show that our approach delivers 95.65% of F1, outperforming the state of the art by 17.24%.

## introduction recent years have witnessed a vast spread of virtual assistants, such as google home, siri and alexa, which are based on the research areas of conversational agents and question answering. when designing such systems, the creation of classes of expected questions, aka intents, is essential for building the main states of a dialog manager. in particular, when an assistant is designed for a specific domain, a knowledge engineer needs to analyze typical user's questions, answered by human operators. this work would be greatly sped up, if the engineers could have questions clustered according to the different topics they ask for. for example, the following questions/requests from the intent dataset by @xcite : however, for being effective, the clustering algorithm must demonstrate a sufficient accuracy, which is often not the case for completely unsupervised methods. thus, supervised clustering @xcite , which exploits some training data of the target domain, e.g., previously designed clusters, to discover new clusters, is a viable approach. a seminal work on structured prediction was latent structural support vector machines (lssvm) by @xcite . recently, @xcite have shown that lssvm as well as the latent structured perceptron (lsp) by @xcite , originally designed for coreference resolution, were also effective, when provided with the appropriate node similarity function, for clustering questions into intents. these approaches used traditional feature engineering (question similarity) and a linear classifier, i.e., svm, which can be highly improved by neural networks, and pre-trained transformers, e.g., @xcite . indeed, neural models enable representation learning, which can amplify the generalization ability of supervised clustering algorithms. in this paper, we design neural supervised clustering (nsc) models using the structured prediction algorithms, lssvm and lsp. these are based on a latent representation of clusters using graph structures, which are used to compute an augmented loss. the latter, in turn, is used together with the model score to globally select the maxviolating constraint at each learning step. this is the clustering that maximally penalizes the current model, which is used for a model update. we apply the same idea by computing the margin loss for our neural model and then back-propagating it, as any other differentiable loss. the augmented loss does not depend on the neural model, thus our approach can be applied to arbitrary learning settings. we applied nsc to two different question intent clustering tasks, defined by two datasets: ic&oos @xcite , which is an intent classifica-tion corpus, and quora intent corpus @xcite . another interesting contribution of our work is the creation of a training set for clustering from ic&oos, which enables an effective training of nsc. our corpus and software are available to the research community 1 . the comparative results of nsc using traditional cnn networks and transformer models against traditional methods, e.g., spectral clustering, show an impressive boost in f1 of our nsc-bert model, e.g., 95.65% vs. 78.38%, more than 17% of improvement over the best spectral clustering method. this accuracy enables the use of our approach for dialog applications and opens up further directions for other clustering tasks. ## related work this paper touches two main research areas: structured prediction, in particular with neural models, and intent clustering, which are described below. structured prediction has shown powerful machine learning algorithms for solving nlp tasks requiring complex output, e.g., syntactic parsing @xcite , coreference resolution (yu and @xcite . this work has mainly regarded traditional frameworks, e.g., svms, crf, perceptron. only little work has been devoted to the integration of the above theory in neural networks @xcite @xcite @xcite @xcite , and, to the best of our knowledge, none to supervised clustering. this is partially due to the fact that local solutions have usually produced optimal results. for example, in case of supervised clustering, it is difficult to design a loss function that captures the global information about the clusters. work in neural coreference resolution, e.g., @xcite , uses simple losses, which deliver state-of-the-art results but do not strictly take into account the cluster structure. secondly, this is also due to the complexity associated with adapting the methods from previous work to neural frameworks. for example, using ilp @xcite for clustering inference in spigot @xcite , which facilitates the backpropagation through argmax based on a projection onto the feasible set of structured outputs, would inevitably require reducing the com-1 https://github.com/ikernels/intent-qa putational overhead @xcite . on the line of research of question clustering, @xcite proposed to cluster queries with respect to a group of web pages frequently selected by users. @xcite describes a k-means like algorithm, mixkmeans, that can cluster threads in community question answering websites. these methods are unsupervised and, thus, are likely sensitive to setting the optimal number of clusters or to a heuristic adopted for the clustering criterion. also among the classification approaches, there are semi-supervised and mixed classification methods which advance on the use of vast amounts of unlabelled queries. @xcite classify unlabeled queries using their proximity to labeled queries in a click graph. @xcite classify queries from logs into topics using supervised or unsupervised methods. the following classification approaches address new emerging intents. @xcite enable a neural model to detect unknown intents as outliers using a novelty detection technique. this model, however, does not have the capability to distinguish between different unknown intents. @xcite devise a capsule neural network able to discriminate between different emerging intents. its zero-shot learning ability critically depends on the definition of a similarity between existing and new intents. our approach does not hold any explicit representation of intents. the recent work by @xcite proposes a deep intent clustering model which takes advantage of labeled data for discovering new user intents, but it requires the indication of the exact number of output clusters. finally, @xcite propose deep aligned clustering, a semisupervised method, to discover new intents using limited knowledge over intent data. we believe their approach is completely compatible with ours, i.e., our supervised clustering models can be integrated in their approach to improve intent discovering. ## structured output for clustering lssvms train a clustering function from a series of training examples {(x i , y i )} n @xmath0, where x i are input sets of elements, and y i are structured outputs, i.e., gold clusters. this function applied to unseen elements x predicts their clusters y. ## neural structured output clustering we propose a model for optimizing a structural clustering loss with neural networks. ## our baseline network we intentionally do not specify the architecture of net θ in sec. 4.2 as it could be of any form once encoding a pair of questions (x i , x j ). however, we further describe the architecture with which we experiment in this work. we use a simple feedforward neural network, which consists of (i) an input layer encoding a pair (x i , x j ), (ii) one fully connected hidden layer with relu activation functions, and (iii) an output layer, which is a linear operation over the outputs of the hidden layer. this way, for edge e, net θ (e) is a real number without any restriction on its range. the pairwise encoder is practically trained to score good edges higher than bad edges. however, doing it jointly for all the edges over a sample, in a structured way, has the goal of producing a more consistent decision in terms of clustering. ## question clustering data in this section, we describe two datasets: (i) ic&oos for intent classification; and (ii) quora for intent clustering. we illustrate our procedure to transform the former into a dataset for clustering. ## experiments we present the results of our empirical evaluation of the neural clustering models using ic&oos data, followed by that using quora intent corpus. we summarize afterwards the highlights of a deeper investigation, we conducted on the performance of our approach and of its errors. ## conclusions in this work, we firstly proposed supervised neural clustering based on traditional lssvm and lsp models, which hinge on optimizing the structural margin loss. this extends the structured prediction methods for supervised clustering to a neural setting. our experiments on ic&oos and quora intent corpora show an impressive improvement over the state of the art, 17.24% absolute over unsupervised models, and 8% points more than our proposed semi-supervised approaches. this suggests that our neural structured prediction can (i) effectively optimize a structural clustering objective function on structured examples, such as sets of questions for intent detection, and (ii) uncover clusters of questions of unseen classes, i.e., potential intents not seen in training.