D ialog A cts based Search and Retrieval for Response Generation in Conversation Systems (2021)

Abstract: Designing robust conversation systems with great customer experience requires a team of design experts to think of all probable ways a customer can interact with the system and then author responses for each use case individually. The responses are authored from scratch for each new client and application even though similar responses have been created in the past. This happens largely because the responses are encoded using domain specific set of intents and entities. In this paper, we present preliminary work to define a dialog act schema to merge and map responses from different domains and applications using a consistent domain-independent representation. These representations are stored and maintained using an Elasticsearch system to facilitate generation of responses through a search and retrieval process. We experimented generating different surface realizations for a response given a desired information state of the dialog.

## introduction a good conversation system is the one that enables its users to converse freely in natural language text. to handle conversations in a robust manner, the system should have set of responses covering many possible ways the end-customer can interact with the system. response generation is a challenging problem for spoken dialog systems, with the quality of the generator depending on factors such as adequacy, fluency, variation and readability @xcite . partly because of the need to adapt to requirements of the specific domain for which the system is designed, many deployed applications, including the ones that provide context for our work, follow a template-based approach, in which response templates with possible slot fillers are manually authored by application designers. such responses largely satisfy the quality measures of adequacy and fluency, where adequacy measures whether the response conveys the intended meaning completely, non-redundantly, and unambiguously, while fluency measures linguistic correctness and appropriateness of style. however, there can be challenges with respect to variation and readability. variation is intended to avoid repetitiveness so that the responses in multi-turn dialogs sound natural and human-like, while readability ensures that responses are interpretable in their dialog context. for example, asking users for basic personal information or task specific details is a common task across many business needs. however, repeatedly using the same small set of scripts such as what is your account number? or please share your account number has the undesirable effect of sounding predictable and unnatural. with a team of experts authoring prompts for diverse applications across different domains, collecting response variations for different response types for various stages of a conversation is effortless. the challenge however is that the collection is useful only if the variants are maintained with dialog state specific equivalence classes that are consistent across domains for authors to access and reuse. fig 1 presents some alternative realizations for frequently occurring use cases of asking customers for their name. while r1 is simply querying the end user to provide their name, r2 acknowledges the capture of information and requests for confirmation of correctness. responses r3 -r5 are used when the customer has not provided the desired information during the prior turn and needs to be prompted again; these are paraphrases providing reasons for why the information is necessary. the motivation behind this work is to cluster these variations using a systematic and consistent framework to promote response sharing across different domains and to suggest possible variations to choose from. in this paper, we describe our preliminary work on dialog act classification of a large database of responses authored by a design team for different applications using a consistent and common domain independent annotation scheme. these response variants can then be used by the designers to provide possible alternative realizations of the content they want to convey. furthermore, since selecting an appropriate variant is partly a function of the dialog context, our future goal is also to develop a context aware response suggestion model that can account for readability through the proper use of context-dependent elements such as referring expressions (e.g., please share your account number with me vs. please share that with me) and discourse markers (e.g., what is your account number? vs. and what is your account number?). our main contributions in this paper are: section 2 presents our approach to dialog act (da) classification, focusing on the taxonomy for the communication function (cf) component. section 3 explains the use of elasticsearch for maintaining a repository of utterances indexed along many different dimenions. section 4 presents our preliminary experiments on classification and static response generation. ## communicative function of dialog acts the meaning of an utterance in dialog has long been characterized as a dialog act, designed to capture the communicative behavior of a participant in terms of speech acts @xcite . many da schemes have been developed over the years, but most were designed for specific domains and applications @xcite @xcite . here, we adapt the iso standard for da annotation @xcite because it provides a domain independent representation that covers a broad range of intents for all aspects of a conversation state. based on the information-state update approach to meaning in dialogue @xcite , a da in the iso framework (iso-da) has two main components: a semantic content (sc), which describes the entities, events, actions, properties or relations that the da is about, and a communicative function (cf), which specifies how addressees should update their information state with the semantic content. for example, the utterance what is your account number? has some representation of the customer's account number as the semantic content, while the cf should represent the fact that the value of this entity is not known and that it is being requested of the customer. since the focus here is on the cf classification of responses, we have adapted the iso-da cfs to reflect the commonly occurring functions in our data of approximately 37k unique response utterances. the following provides the cfs, with definitions and examples. broadly, the functions can be classified as the iso-da categories of informationseeking functions, information-providing functions, commissives and directives. for some of the cfs, such as query info intro, further refinements are needed, however, our preliminary goal for this work is to explore the feasibility of the classification task with a coarse-grained taxonomy. additionally, of the nine dimensions in the iso-da taxonomy @xcite , we have focused here on classifying cfs in the task-related dimension. cfs that fall in other dimensions such as turn management, feedback, and social obligations are all treated as an other category but will be handled in future work. the cf taxonomy was developed first over a seed set of 200 utterances and validated over successive iterations as part of the active learning experiments described in section 5. for example, the utterance "i understand you want to speak to someone, but if you give me your credit card number, i can process your payment for you." has three functional segments with three cfs: deflect request, query info intro and offer. in this stage of our work, we ignore the ordering of the segments, and therefore, the classes as well. ## response generation for designing robust conversational systems, including ours, there exists a team of experts who list down alternative ways of how end customer might interact with the system and create responses for each such use case individually. the wording of the prompt has to be carefully chosen both to convey the desired message and to query for further information. this process is repeated from scratch for each new client and application, even though similar prompts may have been authored in the past for similar scenarios. for example, asking users for their personal information for authentication is a common task across many different applications. we observed many such situations where very similar system responses were present in different applications but were created from scratch because of no means available to access and re-use responses generated in the past. in this section we describe the mechanism we devised for maintaining a repository of responses that can be used either for designing new conversation flows or for reuse directly as templates. as mentioned above each system response is characterized by both the communicative function and semantic content. we thought of providing search interface where designers could mention one or more of these dimensions to specify these requirements and access different realizations of the response they want to generate. one dimension is to provide a text span describing the theme of the current response such as validating gift cards, informing about longer wait times, calling about a new product launch, querying for name or date of birth. communicative functions provides another dimension to search and filter responses by the desired intent. while we need an exact match to search for communicative functions, text based specifications should be able to retrieve responses that are semantically similar to the specified constraints. our requirements prompted us to use elasticsearch because it facilitates both exact search as required for cfs as well as similarity based search in case designer describes a text phrase to specify key aspects of the content they wish to communicate. elasticsearch (es) is an open source search and analytics platform widely used for non-structured text data, hence it perfectly matches our requirements. each indexed field helps to filter responses according to the desired specifications, for example that the entity must be billing address or account number. these filters help to obtain responses that are appropriate for a given context. the communicative function would be query inf o when asking for phone number or f irst name of a person, however, please say and spell your first name is more appropriate than please say and spell your phone number. since the current system implementation is context dependent, search fields such as entities help to provide some context specific information thereby retrieving responses that are more relevant to the current context. ## models in order to investigate response generation, we need to annotate a collection of system responses with the set of communicative functions defined above. the annotation task at hand essentially is a classification task, where given a system prompt we want to predict the relevant communicative function. for example, given system response " can you please tell me the phone number associated with this account", output should be query info and given input utterance as "i can help with automatic payments, but first for security purposes please share the phone number linked to this account " model should predict three labels as offer, query info intro. many machine learning classifiers are available in the literature for supervised multi-class classification problem such as svm, knn, and gradient boosting etc, but being supervised algorithms they @xcite which is very small to be used for training such classifiers and also more generic than ours. we collected a set of approximately 37k system responses from twenty different applications across eight different domains providing enough response variations for commonly occurring modules. these responses are a subset of around 2 billion system responses being used by our conversation assistants on a daily basis for various clients across different domains. the distribution of different domains is present in bert @xcite and t5 @xcite are two of the most widely adopted transfer learning approaches that are known to yield reasonably good performance even for a very small data set. for this study, we used huggingface implementation of bertforsequenceclassification, where the final hidden state of the sequence is input to a fully connected softmax layer with cross-entropy loss function. there are two standard approaches to train multi-label classifier. the first being to train individual binary classifiers for each class in onevs-rest approach and the second is to list down all possible combinations treating each one as an independent class. for example, if there are 3 unique classes, a, b and c, then we can have at most 7 distinct class labels (a), (b), (c), (a, b), (a, c), (b, c), (a, b, c) where each combination is treated as an atomic class. the latter approach though works well for smaller set of unique labels but becomes difficult as the number of classes increase and the distribution varies a lot. of the two approaches for solving multi-label classification problems, the preliminary set of experiments indicated that training ensemble of individual binary classifiers resulted in better performance than treating each combination as an atomic class. this implies that we trained and saved individual classifiers for each of the 13 communicative functions as binary classifiers. we experimented with different learning rates and found best results for 1e-5 with batch size 16 and max epochs 20. also, to account for varied distribution of labels, we computed class weights for each class label as size(label i)/max, where max is over all sizes. ## experiments we conducted experiments in two phases. the first phase is to train multi-label classifier for classifying system responses into space of communicative functions with a reasonable degree of accuracy. in the second phase, we analysed the quality of responses retrieved by the system for both simple queries given only cf or textual description and complex queries defined using a combination of other querying dimensions. ## literature review accurately predicting the speakers communicative intent is extremely important for a successful communication and thus intent detection has always been widely pursued research thread. as virtual assistants are becoming a part of daily life, it has been acknowledged that most a times speaker is communicating multiple aspects with in a single utterance. there is an increasing trend towards training joint models for intent detection as multi-label classification (mlc) and entity detection(also called slot filling @xcite ) @xcite . these systems compute relevance score for each label and utterance combination and then select the labels with maximum similarity score. some of these approaches are few shot learning approaches proposing techniques to perform mlc with fewer examples, but they all pretrain on domain specific data and then extend this to out-domain dataset. in contrast, our work aims to annotate with domain-independent dialog act labels and only focuses on predicting communicative functions, hence we adopted conventional machine learning approaches for classifying communicative functions. the concept of representing dialog acts using domain independent general purpose schemas has been studied multiple times as dialog act markup in several layers (damsl) by allen et. al @xcite and as iso standard by bunt et.al. @xcite . the iso taxonomy pro-cf response of f er due to heavy call volume at this time it could take over 90 minutes to talk to a representative, lets see if i can help you. inf orm issue we are sorry for your inconvenience, however, we are experiencing extremely high call volume due to the recall and this has caused extremely long wait times to connect with an agent inf orm wait a moment while this call is being transferred to our system.wait times are longer due to heavy call volumes. ## conclusion in this paper, we proposed a taxonomy of communicative functions that effectively captures the communicative intent of a dialog turn using domain independent labels providing means for flexible and generic dialog modelling. the taxonomy was used to annotate a subset of user responses from human-machine conversations used by our real-life applications on day-to-day basis. we experimented with this annotated dataset to generate different linguistic variations of the system responses for given communicative function and desired keywords indicating the essence of the current dialog turn. our experiments indicated that the proposed taxonomy can successfully learn representations that capture what the dialog is written to accomplish across different applications and verticals. we experimented with these annotations in a dialog generation settings and found that we are able to generate system responses given desired specifications from the existing data itself.