Intent Features for Rich Natural Language Understanding (2021)

Abstract: Complex natural language understanding modules in dialog systems have a richer understanding of user utterances, and thus are critical in providing a better user experience. However, these models are often created from scratch, for specific clients and use cases and require the annotation of large datasets. This encourages the sharing of annotated data across multiple clients. To facilitate this we introduce the idea of intent features : domain and topic agnostic properties of intents that can be learnt from the syntactic cues only, and hence can be shared. We introduce a new neural network architecture, the Global-Local model, that shows significant improvement over strong baselines for identifying these features in a deployed, multi-intent natural language understanding module, and more generally in a classification setting where a part of an utterance has to be classified utilizing the whole context.

## introduction while generic dialog systems, or chatbots, such as amazon alexa or google assistant, are increasingly popular, to date, most industrial dialog systems are built for specific clients and use cases. typically, these systems have the following: 1. a natural language understanding (nlu) module to analyze the user utterance, 2. a dialog manager module to reason over the analyzed utterance and decide on an action, and 3. a natural language generation module to generate an appropriate response based on the action. typically, an nlu module has two purposes: understanding the intent or goal of an utterance (classification) and identifying the entities in the utterance (slot filling). as dialog managers have evolved from simple flow-based systems to information state update systems @xcite , nlu modules have progressed past simple single intent detection and flat slot filling to multiple intents and nested entities @xcite . as these dialog systems need to be rebuilt for each client, the nlu module faces a significant data bottleneck; it is time-consuming and expensive to collect data, develop a domain-specific annotation scheme, and annotate data. therefore, it is imperative that the data is shared across clients as much as possible. in a production dialogue system, there are often similar situations that require drastically different responses. for example, "i want to cancel my subscription." and "i am thinking about canceling my subscription." are very similar. they are both about the canceling of a subscription. however, they differ in the users conviction. the latter user is much more likely to not cancel if offered a discount. making this distinction is critical for creating sophisticated and nuanced dialogue systems. a common approach to solve this problem would be to split the intent space so the dialogue manager can differentiate between these examples, creating a cancel and a think-cancel intent. using intents to recognize specific situations leads to data sparsity as each intent is broken into many subcategories like present vs. past tense, how certain a user is in their actions, and if the user has tried an action or not. there would be very few examples of each intent. additionally, the combinations of different sub-categories would cause a combinatorial explosion of intents. another short-coming of finegrained intents is the loss of compositionality. fundamentally the cancel and think-cancel intents are very similar, but because they are modeled as independent output classes, there is not a shared representation of these labels the model can lean on. in order to avoid these shortcomings, and allow for many examples per intent, we factor out these small differences in situations into what we call intent features. intent features are a set of domainindependent properties for intents that can primarily be understood from the syntax of the utterance. these intent features represent specifics of situation, such as tense, without having a massive intent space. by decoupling these small differences, we can keep the intent categories general, while still providing the dialogue manager with the information it needs for nuanced, human-like responses. in a multi-intent setting where each clause in the utterance has an intent, intent features reduce to the problem of classification of a span embedded within a larger utterance. we propose a new model, the global-local model, for this problem which shows significant improvement over strong baselines. ## intent features communicative functions: the communicative functions (cf) captures what kind of response (or action) the user is trying to elicit from the system. we define five such functions: often this warrants a yes/no answer. for example, one expects a yes or no from, "was my installation successful?" all of our running examples above share the intent of installing software; however, differences in phrasing warrants different responses. an inform does not typically require a targeted reply from the system, whereas for an issue, the system should start the response with "i am sorry you are having trouble." attribution: attribution is concerned with agency. there are two types of attribution. the first type is the of attribution of the communicative function (attr-cf) and it deals with who is the primary source of the content of the topic. the second type is the attribution of the event/action (attr-ev) of a topic and describes who is the agent of the event or action. this is perhaps best elucidated by an example. in negation: topics of many intents are represented in their negated versions, as well. for example, in the software domain, the compatibility intent models whether a piece of software is compatible with some device. a negation feature would denote incompatibility. the negation feature takes values positive and negative. tense: events and actions can occur in the past, present, or future, which is modeled by the tense feature using values of past, present, or future. the steps to solve a problem as it occurs are often quick-fixes, whereas the first step when fixing a problem that occurred in the past is often information gathering. the tense feature allows the dialogue manager to distinguish between these two possibilities. tense information is common in the annotation of event extraction, such as in ace 2005 dataset @xcite . modality: the real-world actions and events represented by an intent can also be viewed in terms of a modality of certainty, that is, whether or not the event or action actually occurred, and to what degree. we consider two types of modality. the first text topic/intent attr-cf attr-ev cf modality negation tense i am trying to install install self self inform modal-try positive present and ------i see a problem general self self issue other positive present utterance attribution cf attribution ev i have paid @xmath0 other self i was charged @xmath1 other other ## modeling there are four different model types we explored for intent features that we detail below. however, before we can annotate an intent with a feature, we need to have an intent span. first, we describe our intent span extraction model whose predictions are used as intent spans. ## dataset the data consists of customer utterances. they were collected from the first customer turn in webchat conversations between customers and agents there are four distinct phases of the model. first, the input is encoded into a sequence of vector representations. this can be as simple as word embeddings or it can use a more complex encoding like a bilstm. then, the local span is extracted from the sequence using the input mask. global and local pooling functions are applied to create two vectors, which are joined by concatenation. the local vector encodes the features of the span while the global vector encodes the features of the span as contextualized by the whole input. finally, this joint representation is used for classification. from a software company after filtering out low content first turns such as "hi", "hello", and "hey". our training, validation, and testing datasets have 36,725; 9,256; and 4,993 examples respectively. the data was annotated by a team of six (nonoverlapping) commercial annotators over a period of a month and then corrected by an expert annotator. a small subset of the data was annotated (before the error correction) by two expert annotators. the agreement was 53% between two expert annotators and 42% between one expert and the other non-expert annotators. ## experiments the f1 scores for these models are reported in we further ablate the global-local model to understand the reasons for the performance gain in the current implementation has a shared encoder step where the entire utterance in encoded into a sequence of vectors before the span is extracted and processed by the local pooling function separately. doing this efficiently in a batched computing environment, like tensorflow @xcite , is slightly tricky to implement. a much simpler model would feed the global utterance and the span separately, to be encoded and processed independently. our ablations in the "-shared embedding" row of all models were trained with mead-baseline @xcite , an open-source library for the development, training, and export for deep neural networks for nlp. ## deployment we have deployed a nlu component of a taskoriented, production dialogue system that produces intent features. the dialogue system deals with customer service in the retail software domain. the dialogue manager currently makes use of several intent features. the easier feature to use is negation and it is critical to understand user intent. it also uses the tense feature to understand if it needs to wait because a user is currently performing an action or if it can ask about the result because the action had already been performed. the next feature the dialogue manager plans to leverage is the modality features. understanding the user's convection in an action, like canceling, can help make decisions about whether an upsale or discount would be effective. in designing these intent features, we hoped they would be general enough to be transferable across domains without retraining a model on the new domain. recent work with a new client in the general retail domain gave the opportunity for a small scale test. we were given approximately 500 sample utterances that had been annotated with general labels like, "is this utterance equivalent to an faq?" this is very similar to our request-info intent feature. we ran our intent feature model on this new data and compared how many faq questions were labeled with request-info. we found that our model had high precision, 83.3% of request-info utterances were in fact faq questions, but had low recall, only 40.5%. this small scale experiment suggests that our intent features are general, but the low recall means our specific model is probably overfit to the lexical features in our original domain. ## previous work most popular intent taxonomies such as atis @xcite are domain-specific. dialog acts (da) @xcite are more formalized and generalized versions of intents. the international standard for da annotations @xcite @xcite defined the concept of commu-nicative functions in a dialog act. however, these functions are defined for a wide range of use cases. we note that a very restrictive and reworked subset of these suffices for our use cases. we believe the other features in the annotation scheme are novel or have an expanded range of possible values. the global-local model draws inspiration from the @xcite model for end-to-end neural coreference resolution. like us, they have regions on interest embedded in a larger context. however, our models differ in several key ways: their span representation is a hand-crafted combination of token features while ours is a learned pooling of token representations. also, their model is restricted to operating on contiguous spans (possibly due to unavailability of spans a priori, or that noncontiguous spans would lead to a combinatorial explosion), while our model has no such restriction. ## conclusion improvements in the complexity of conversations that a dialogue system can handle have put tremendous pressure on nlu systems to capture finegrained and domain-specific information. difficulty in the data generation process means the ability to share data across clients is critical. we define intent features, a core set of general annotations, on intents that provide context and clarity on the exact nature of the user requests, and allow for a more natural and intelligent response from the dialogue manager. a nlu system that produces these intent features has been deployed in a production system with a dialogue manager that makes use of them. to extract these intent features from an utterance, we propose a new neural network architecture, the global-local model, that fuses the representation of the content of a span of text and its global context through learned pooling functions. this model shows large improvements over several strong baselines.