Improving Out-of-Scope Detection in Intent Classification by Using Embeddings of the Word Graph Space of the Classes (2020)

Abstract: This paper explores how intent classification can be improved by representing the class labels not as a discrete set of symbols but as a space where the word graphs associated to each class are mapped using typical graph embedding techniques. The approach, inspired by a previous algorithm used for an inverse dictionary task, allows the classification algorithm to take in account inter-class similarities provided by the repeated occurrence of some words in the training examples of the different classes. The classification is carried out by mapping text embeddings to the word graph embeddings of the classes. Focusing solely on improving the representation of the class label set, we show in experiments conducted in both private and public intent classification datasets, that better detection of out-of-scope examples (OOS) is achieved and, as a consequence, that the overall accuracy of intent classification is also improved. In particular, using the recently-released Larson dataset , an error of about 9.9% has been achieved for OOS detection, beating the previous state-of-the-art result by more than 31 percentage points.

## introduction intent classification is usually applied for response selection in conversational systems, such as textbased chatbots. for the end-user to have the best possible experience with those systems, it is expected that an intent classifier is able not only to map an input utterance to the correct intent but also to detect when the utterance is not related to any of the intents, to which we refer to as out-ofscope (oos) foot_0 inputs or samples. in the light of this, this paper describes and evaluates a method which tries to capture the complexity of the set of intents by embedding them into a vector space created using word graphs, as described later. we show that, although the method in some cases is able to improve the accuracy of a text classifier in in-scope examples, it has often a tremendous impact on improving the ability of text classifier to reject oos text, without relying on oos examples in the training set. notice that the intent classifier is typically implemented using standard text classification algorithms @xcite @xcite . consequently, to perform oos sample detection, methods often rely on one-class classification or threshold rejectionbased techniques using the probability outputs for each class @xcite or reconstruction errors @xcite . there also exist approaches based on the assumption that oos data can be collected and included in the training set @xcite . however, in practice, collecting oos data can be a burden for intent classifier creation, which is generally carried out by domain experts and not by machine learning experts. thus, in the ideal world, one should rely solely on in-scope data for this task because it is very difficult to collect a set of data that appropriately represents the space of the very unpredictable oos inputs. the classes in a traditional text classifier are generally represented by a discrete set of symbols and the classifier is trained with the help of a finite set of examples, where the classes are assumed to be independent and the set of examples to be disjoint. but, in many cases, the classes are in fact associated with inter-connected higher-level concepts which could be formatted into more meaningful representations and better exploited in the classification process for an enhanced representation of the scope of the classifier. in particular we explore here the use of graphs which represent information by means of nodes connected to each other by arcs. recent research has demonstrated that nodes in a graph can be converted to an embedding, that is, projected into a vector space, which can then be mapped to sentences to cope with tasks such as the reverse dictionary problem @xcite . we propose here an adaptation of those ideas to an intent classifier so it uses such mappings to expand the representation of the class space, its scope, and class inter-dependencies, and thus possibly making the oos detection task easier. this paper presents an investigation of exploiting information from word graphs associated to the intent classes to improve oos sample detection in intent classification. by considering that each class is represented by a set of text examples and that different classes can be connected to each other by means of the repeated occurrence of words in their respective examples, we build a word graph where both class labels and words are represented by single nodes. the word nodes are connected to the class label nodes in accordance to their occurrence in the training samples and their respective class labels. then, a typical graph embedding technique is used to represent classes with the embedding of their corresponding class label node. instead of finding the classes with the highest probability, the intent classifier search for the class embedding which maps best to the sentence embedding of a given input sample. we have implemented and tested this idea with different types of base methods for sentence embedding, such as long-short term memory (lstm) neural networks and bidirectional encoder representations from transformers (bert), and performed oos detection by means of a simple threshold-based rejection. we conducted a thorough evaluation on both private and public intent classification datasets, such as the larson dataset for this specific task @xcite . our results show that the proposed word-graph based method improves considerably oos detection, compared against the corresponding traditional classification algorithms, based on combining the sentence embedding algorithm with softmax probabilities. in the case of the larson dataset, where comparison against varied oos detection methods is available, we show that our proposed approach reduces dramatically the previous stateof-the-art (sota) false acceptance rate in more than 30 percentage points, from 41.1% to 9.9%. ## the word graph method this section presents a formal description of the methodology employed in this work. ## experimental evaluation we performed a comparative evaluation of the performance of the classifiers described in the previous section using a public dataset described in @xcite called here the larson dataset foot_1 ; a real dataset from a finance chatbot; and a pool of 40 datasets in two different languages from chatbots built using the same platform, to check for the reproducibility of the results from the finance chatbot dataset. for those experiments, the methods were implemented as follows. for deepwalk, the embedding size was set to 150, and the walk sizes to 20, for undirected graphs. for lstm and lstm+, we considered word embeddings with 200 elements, output sentence embeddings of size 150, and both methods were trained for 50 epochs. for both mapping sentence to graph embeddings and the softmax classifiers, we trained two-layer neural networks with 800 hidden neurons for 1,000 epochs on larson dataset, and 300 hidden neurons for 20 epochs on the other datasets. those parameters were set after preliminary evaluations. ## related work classification methods, such as those used for intent classification, have been broadly applied to several areas, with the goal of predicting, for an input sample, which of the classes of the problem that sample is associated to. in the case of singlelabel classification, the training process consists of approximating a probability function, for instance a softmax function for neural networks, by using as reference an one-hot-encoding representation of class labels @xcite . oos sample detection is a problem which may be critical for intent recognition in chatbots, so that applying rejection mechanisms are important for detecting those cases @xcite @xcite . traditional classification can be implemented, for example, by training a specific oos class to set up a rejection threshold, or even by training a binary classifier @xcite . given that no specific domain information or structure are taken into account, those methods are roughly the same that have been previously applied for other classification problems @xcite . some recent effort has been put specifically for oos sample detection for intent recognition, either by considering oos data during the training process @xcite or solely by improving in-scope sample representation by means of auto encoders @xcite and generative adversarial neural networks (gans) @xcite , which is more desirable since there is no reliance on tedious data gathering processes to represent unpredictable oos inputs. the latter two methods are directly related to ours but, unfortunately, the lack of publicly-available source codes and datasets has made it a challenge to reproduce the methods for a fair direct comparison with ours. recently, methods which are able to take advantage of graph information in machine learning models have been proposed. some of them take advantage at the sample level, such as label propagation @xcite . others, though, take advantage of graphs at concept level, such as in @xcite @xcite . @xcite demonstrate the sentence embeddings could be mapped onto graph embeddings, in reverse dictionary-like problems. following, @xcite demonstrated that textual features can improve considerably such a mapping. those findings have opened an opportunity to enhance class modeling and hopefully better define the scope of a classifier, in special intent classifiers, since classes can be easily represented in a graph space by means of their relationship with individual words extracted from the training samples as we did in this paper. the previously-mentioned research have been put in practice mostly by advances in sentence embedding @xcite and graph embedding techniques @xcite . some of them are directly inspired by advances in word embeddings and convolutional neural networks, such as deepwalk @xcite and node2vec @xcite . ## final remarks in this paper we propose the use of information from word graphs to enhance intent classification, more specifically, for the detection of out-of-scope examples. instead of working on the representation of the input text, we enhance the representation of the outputs, i.e. how classes and their corresponding labels are represented. the results demonstrate the approach has a considerable positive impact for the detection of out-of-scope examples when an appropriate sentence embedding such as lstm and bert is used. in the publicly-available larson dataset, the proposed approach beats the previouslypublished results by a high margin, and particularly enhancing the false acceptance rate (far) from 41.1% to 9.9%. in our view, the improved results are due to a better representation of the higher-level concepts associated to the classes. by connecting the intents to lower-level entities, i.e. the words associated to the intents, and therefore establishing inter-connections between the classes, the word graph space enriches the traditional representation of classes by means of classifier parameters which are learned solely from input examples. we believe that the approach is general enough to be applied to others areas and presents ideas to develop more accurate classifiers in general, across multiple areas, particularly in contexts where outof-scope samples are common. in image classification problems, for instance, word graphs related to visual words could be computed. in addition, the proposed word graph method can be improved by exploiting combinations of the proposed expanded class representation with the traditional softmaxbased method, what may also provide better accuracy for in-scope samples in some situations.