Using Meta-Knowledge Mined from Identifiers to Improve Intent Recognition in Conversational Systems (2021)

Abstract: In this paper we explore the improvement of intent recognition in conversational systems by the use of meta-knowledge embedded in intent identifiers. Developers often include such knowledge, structure as taxonomies, in the documentation of chatbots. By using neuro-symbolic algorithms to incorporate those taxonomies into embeddings of the output space, we were able to improve accuracy in intent recognition. In datasets with intents and example utterances from 200 professional chatbots, we saw decreases in the equal error rate (EER) in more than 40% of the chatbots in comparison to the baseline of the same algorithm without the meta-knowledge. The meta-knowledge proved also to be effective in detecting out-of-scope utterances, improving the false acceptance rate (FAR) in two thirds of the chatbots, with decreases of 0.05 or more in FAR in almost 40% of the chatbots. When considering only the well-developed workspaces with a high level use of taxonomies, FAR decreased more than 0.05 in 77% of them, and more than 0.1 in 39% of the chatbots.

## introduction classification of sentences into a discrete set of classes is a key part of professional conversational systems. in fact, most of those systems require developers to define the different classes, or intents, by enumerating exemplars of each of them, since classification is often performed using machine learning (ml) methods. the process of classifying an input sentence into a specific intent or signaling it as out-of-scope (oos) of the system is often referred to as intent recognition. determining a class solely on a list of exemplars is a practical method to implement ml systems but it is hardly a natural way for human beings to define a class. in real life, people define classes often using a rich mix of symbolic definitions, sometimes taxonomic in nature, such as in "a credit card is a type of bank card", coupled with its sub-classes, for instance, "basic", "premium", and typical features such as "international". people also use exemplars, "card x of bank y is a credit card", as well as particular examples to describe a sub-class, such as in "card w is an international card". they also use counter examples, either categorically, "a debit card is not a credit card", or in examples, "card z is not a credit card". defining and specifying classes in the real world is, in fact, a cultural, contextual, and linguistic construct, and how people and societies perform this process is a traditional research subject in social sciences, notably in anthropology @xcite @xcite . this paper explores algorithms for intent recognition which use both the sets of exemplars and taxonomic-like symbolic descriptions of a class to define and train intents in conversational systems using ml methods. we aim not only to provide methods more aligned to everyday class definition practices of developers but also to improve the accuracy of the ml methods. inspired by a reverse dictionary algorithm @xcite and previous work on keyword-based classification @xcite , we propose three neuro-symbolic algorithms which combine taxonomic descriptions of classes with traditional exemplar-based supervised learning. we show that those novel algorithms are able to decrease error rates for a significant number of datasets, particularly in the difficult task of detecting oos cases in real, professional chatbots. the key idea behind our algorithms is to substitute the typical softmax used in the output layer of a ml text classifier with a space of embeddings of the taxonomic descriptions of the intents. the training process uses the exemplars in standard ways while the recognition process is performed using similarity distances in the embedding output space. this is similar to ideas used in zero-shot learning methods @xcite @xcite , in which classes defined by sub-concepts are also encoded with special embeddings to allow detection of new classes without exemplars. we tested our algorithms using real datasets by exploring a common practice among developers of conversational systems, who often embed symbolic knowledge as documentation in intent identifiers. in a previous work @xcite , we observed a pattern among developers of using taxonomic-like structures to name the intents in which strings of reoccurring concepts are used to identify and document the different classes. for example, an intent about utterances where users ask for the balance of a credit card may be named "checking credit card balance", while an intent related to finding out the date of payment of the balance could be identified as "asking credit card balance payment date". we call those structures intent proto-taxonomies, and real examples are shown in figures 1 and 2. in @xcite , we studied the use by developers of intent proto-taxonomies quantitatively and qualitatively, as well as proposed an algorithm to mine this meta-knowledge automatically, and concluded that their use is fairly common in at least one professional chatbot development platform. this paper focuses on the algorithms to use the meta-knowledge and on evaluating their impact on the accuracy of intent recognition. the paper starts by looking into the recent advances in neuro-symbolic systems and describing briefly the practice of developers of conversational systems of embedding meta-knowledge within the source code of their systems. we follow by describing the proposed three algorithms integrating such meta-knowledge into intent recognition ml algorithms and by evaluating them first with two typical intent recognition datasets, and then with hundreds of workspaces created in a professional tool called here chatworks 1 . the results show most of those workspaces can benefit from the techniques described in this paper, notably for oos detection tasks, often with accuracy improvements of 5% or more solely derived from the use of the additional symbolic description from the documentation. 1 we use an anonymous name for the tool due to publication restrictions from the platform company. ## related work the value and limits of symbolic categorization in ai have been of interest since the early days @xcite @xcite . but our work fits more in the context of a growing belief that symbolic knowledge needs to be included in ml systems, materialized in the so called neuro-symbolic approaches @xcite @xcite @xcite @xcite . neuro-symbolic methods "aim to transfer principles and mechanisms between (often nonclassical) logic-based computation and neural computation" @xcite . such kind of systems are viewed by some researchers as a way to embed high-level knowledge and even some form of "consciousness" into machine learning systems, making the language to develop them closer to "what passes in a man's own mind" @xcite . in recent years, ai has witnessed a myriad of novel neuro-symbolic techniques and their application to different problems, contexts, and scenarios @xcite @xcite @xcite . for instance, in @xcite , an approach for image understanding is suggested which takes the object-based scene representations and translates sentences into executable, symbolic programs. in @xcite , embeddings of knowledge graphs are used as attention layers for tasks such as autonomous driving (av) and question-answering. and in @xcite , random walks in a knowledge graph are mapped as sentence embeddings for use in an inverse dictionary problem. one important requirement for many neurosymbolic systems is to represent knowledge in a structured format such as knowledge graphs, ontologies, or taxonomies @xcite . in some cases, such as the scene ontology for autonomous vehicles in @xcite , a lot of effort was needed for manual annotation. nevertheless, as presented in @xcite , an unsupervised approach can sometimes be used to mine the meta-knowledge introduced by the experts, such as the classes in wikipedia pages. considering our context, intent identifiers are sometimes described using high-level representations of the class as we detail later. this is similar to what is used in some zero-shot learning techniques @xcite in which classes manually defined by sub-concepts are encoded with special embeddings so new classes can be detected without training @xcite @xcite @xcite . in @xcite , for example, intent identifiers can be formatted as natural language sentences to learn a model which maps training examples into those sentences, so that the meta-knowledge can be used in zero-shot learning. however, the dataset explored in that work is very limited. recent work has also demonstrated that intent recognition can be improved by enhancing class representations with "keywords" which are extracted from exemplar utterances considering their most common words @xcite . this work focuses on high level class representations based on taxonomies and aims to explore their usefulness as enhancers of ml intent recognition algorithms. it also explores different ways of embedding taxonomy-like meta-knowledge considering different methods of representation. ## the knowledge embedded in intent identifiers of conversational systems most real-world, deployed conversational systems in use today have been built based on the rule-based intent-action paradigm, using platforms such as luis.ai, watson assistant, or alexa skills. each intent corresponds to a desired information or answer from the user and is defined by a set of exemplar utterances by the chatbot developers. during runtime, each utterance from the user is recognized as one of the defined intents or as out-of-scope (oos), and then the associated action is outputted. in the context of the chatbots built using the chatworks platform explored in this paper, a previous work of the authors of this paper @xcite has shown that the curators and developers of chatbots often store symbolic knowledge in a taxonomic form about the intent classes in a documentation field called nameid. figure 1 shows some examples of those nameids, obtained from a professional finance chatbot, here translated from the original in portuguese and anonymized to preserve confidential information. this practice was studied in workshops with @xcite , which determined that the goal of the taxonomic description is to provide the intent classes with a summarized description of each intent. such taxonomic naming patterns are also common in the way people organize files and e-mails in computers @xcite and how software developers name functions @xcite . as described in (pinhanez et al., 2021), such knowledge-embedding practices are, in fact, fairly common among curators in the chatworks platform. using the algorithm reproduced in appendix a, taxonomic-like symbolic knowledge was automatically extracted from workspaces defining almost 7,000 professional chatbots, in two different languages. by considering the different words in the nameids as basic concepts and consecutive concepts as having connections between them, we can structure the set of nameids as a very basic knowledge graph @xcite , hereby referred as an intent proto-taxonomy. figure 2 depicts the intent proto-taxonomy associated to the nameids in fig. 1 . next, as proposed in @xcite , it is possible to compute the taxonomy rate of a workspace by calculating the ratio between the number of intents with taxonomies and the total number of intents. in 3,840 professsional workspaces in the english language, it was found that 76% of them had a taxonomy rate above 10%, almost 52% had a taxonomy rate above 50%, and 16% had a very high taxonomy rate, above 90%. moreover, the distribution followed a sort of "step" function where, as the threshold of 32 in the number of intents in a workspace was crossed, the majority of the workspaces had a taxonomy rate of more than 50%. it seems that, as the complexity of the workspace increases with the number of intents, more often developers resort to document them using an intent proto-taxonomy (see appendix b for details). the use in our work of the intent prototaxonomies as a symbolic description of classes is feasible because: (1) they are part of the documentation of the conversational system, so there is no need of acquiring knowledge from experts; (2) they are easily mined, as described in appendix a. ## using taxonomic intent descriptions to improve intent recognition we present now a formal description of the methodology employed in this work which takes advantage of the intent proto-taxonomies using a neurosymbolic approach. it expands some previous work which focused on the use of keywords as the source of symbolic information @xcite . . ## metrics, datasets, and experiments in this section we present the experiments to evaluate the three algorithms described in the previous section, using each of the input embeddings lstm and use. we explore the impact on intent recognition both in terms of classifying correctly utterances (is accuracy) and of finding which utterances are not covered by the intents (oos accuracy). ## discussion and future work we started this paper by proposing the combination of exemplars and symbolic characterizations of a class as a way to enhance ml-based intent recognition. we proposed 3 new neuro-symbolic algorithms and tested them using datasets built using data from intent identifiers of conversational systems. such identifiers often store taxonomic-like structures, due to a common practice among developers of professional conversational systems @xcite . the results of the experiments indicate that the intent proto-taxonomies embedded by those developers can indeed be used by many workspaces to improve accuracy in intent recognition, notably in oos detection. we see as one of the main contributions of this paper the creation of methods with which ml engineers can improve the accuracy of their systems by simply mining "documentation" from chatbots, without any further data and annotation. our results show that almost 40% of the 200 professional workspaces drawn from chatworks saw decreases of more than 0.05 in oos detection error rates. also, in 42% of them the overall error rate was improved, using the use+c algorithm. when considering the more well-structured and developed 18 workspaces, we saw much higher gains with the use+t algorithm. those accuracy improvements were achieved without any change in the training set but simply by incorporating the meta-knowledge into intent recognition. notice that the testing methodology used in this work is considerably harder than the practice of the majority of research papers, since it evaluates performance in 200 professional, non-edited workspaces from different domains. in reality, most ml algorithms do not perform well in all datasets, and ml practitioners often test different algorithms and parameters until accuracy is good enough. however, the improvement in oos detection (far) was not mirrored in classification error (iser). first, we must keep in mind that intent classification is often performed in two steps, first oos sentences detection and removal, followed by intent classification of the is sentences. given the improvements observed in oos detection, it would make sense to use our algorithms in the first step for many of the chatworks workspaces (about 60% of them), and selectively use it for is classification only when it works better than the baseline. but why were there so many workspaces where we did not see impact? it is important to take into account that the chatworks dataset has workspaces in different stages of development and deployment. by selecting better quality workspaces, we saw much higher gains. we explored briefly characterizations of the intent proto-taxonomy quality, such as taxonomy rate, depth of the taxonomy, and number of concepts, but we saw no clear correlation with decreases in error rates. we believe more complex metrics of knowledge structure need to be employed to characterize which intent prototaxonomies are likely to have the greatest impacts. we plan to do so in our future work. it is important to notice that, in the workspaces where we did see impact, the symbolic knowledge was mined from an absolutely "raw" format. in spite of that, by using the basic graph mining method described in appendix a, it was possible to obtain a "meaningful" taxonomic structure, similar to a knowledge graph which could be used by our neuro-symbolic algorithms. to improve the quality of the taxonomies, we are working on designing an interface which allows the developers to manipu-late directly the intent proto-taxonomy to make it more correct and complete, so to possibly decrease even more the intent recognition error rates. we have demonstrated in this work that combining exemplar and symbolic ways of defining classes can have a positive impact in the performance of the recognition system. this was done in the context of conversational systems where developers fortuitously embed such alternative descriptions of classes in their name identifiers. we believe it is possible to find in other machine learning development platforms similar patterns of knowledge embedding. for example, we know that it is common for people to use similar taxonomic structures when naming file and e-mail folders, giving names to functions and variables in programs and data, and writing comments into jupyter notebooks. also, ml platforms can further foster the use of metadata by developers by explicitly asking them to input, besides exemplars, categorical or textual descriptions of the classes. as we move along the path of creating such neuro-symbolic systems, not only we should expect that the job of developers becomes easier, as they follow their own cultural and linguistic practices, but also that machines became better in recognizing those classes accurately. using multiple forms of class definitions can be a winning proposition for both ml systems and their developers.