Content Selection for Explanation Requests in Customer-Care Domain (2020)

Abstract: This paper describes a content selection module for the generation of explanations in a dialogue system designed for customer care domain. First we describe the construction of a corpus of a dialogues containing explanation requests from customers to a virtual agent of a telco, and second we study and formalize the importance of a specific information content for the generated message. In particular, we adapt the notions of importance and relevance in the case of schematic knowledge bases.

## introduction customer care is one of the application domains where dialogue systems (dss) represent an emerging technology used by many big companies to satisfy customer requests @xcite . customer care dialogues can have a specific linguistic characterization @xcite , and often the customer preferences lean toward short dialogues @xcite . moreover, in the customer care domain the users' requests often regard some form of explanation about their past transactions with the company. to provide explanations, commercial dss often provide long lists of data entries extracted from databases containing companycustomer relationship data. therefore, there is the necessity to give some form of priority to data entries to present just -or to give more prominence to -the information that is most relevant for the user @xcite . most commercial dss follow the classical cascade architecture n lu nderstanding ↔ dialoguem anager ↔ n lgeneration @xcite . this architecture relies, as a working hypothesis, on the assumption that most of necessary information is provided by the user utterance. however, this assumption is sometimes false or only partially true. for instance, in the sentence "scusami ma vorrei sapere come mai mi vengono fatti certi addebiti?" ("excuse me, i'd like to know why i'm charged certain fees?"), even a very advanced nlu module can produce only vague information about the user's request to the dialogue manager. indeed, to provide an appropriate response, the dialogue manager might need to ask for additional clarification or, in alternative, to access some contextual information to obviate the lack of linguistic information. in the case of customer care, this contextual information can be found as schematic knowledge bases arising from databases. as a result, when linguistic information is scarce (or absent in the case of ungrammatical/incomprehensible input) retrieving and giving priority to contextual information in dss is essentially a problem of content selection @xcite . therefore, as a working hypothesis, in this paper we consider negligible the linguistic input given by the user. however, also when the linguistic input is comprehensible, a good balance between the information carried by the linguistic input and by the specific domain context is a key goal for the dialogue manager. the idea to use nlg techniques for explaining rationales inside data is a topic that is drawing growing attention @xcite . one of the few papers providing a quantitative evaluation of explanations was produced by @xcite . in this work the authors proposed a model for quantifying the relevance of a feature for a specific class of machine learning algorithms, i.e. linear classifiers. the authors introduced two notions, importance and effect, to evaluate the relevance of a feature in the general classification model and for a specific classification instance respectively. the basic idea was to determine the narrative role of a feature based on the combination of its importance and its effect; for example, a feature may have the narrative role of exceptional evidence in the case of low importance and high effect. in this way, the authors have been able to communicate the key data elements into core messages for an explanation (justification in their terminology). in this paper, we present some initial results of an ongoing study on the design of a generation module of a ds in the domain of telco customer care. we focus our study on customers' requests of explanations @xcite . the study presented here, in fact, is part of a wider project that aims to improve the answers provided by a virtual agent of an online customer service, by creating a nlg system that could also take into account various dimensions in the generation process, such as possible errors in the conversations (see e.g. @xcite , @xcite , higashinaka et al. (2015) ) and the presence of emotions (especially negative ones) in the user messages. at this stage of the project, we use the model presented in @xcite to give relevance to the content units in the knowledge about the customer. in particular, we adapt the definition of the narrative roles for importance and effect to the case of a knowledge base consisting of database entries. this paper provides two main specific contributions: (1) the analysis of a corpus consisting of real dialogues containing explanation requests (section 2), (2) the proposal of a content-selection procedure based on narrative roles in explanation when the ds contextual data is a schematic knowledge base arising from a database (section 3). in the final section of the paper we discuss these contributions in relation to our ongoing work. ## building a corpus of explanation requests this study builds upon the analysis of a corpus of dialogues between customers and a virtual agent for customer service developed by an italian telecommunications company @xcite . the dialogues, which take place by means of a textual chat, mainly deal with requests for commercial assistance, both on landline and mobile phones. for the purpose of this study, the corpus created was extracted by selecting, from a sample of dialogues held over 24 hours, a reduced subset that included requests for explanations from customers. the selection criteria were conceived so as to include all the dialogues where at least one message from the user contained a clearly stated potete verificare le ultime fatture non mi sembrano esatte grazie (c d ea e e if a i ice , he d ' ee c ec e, ha ) ecco il dettaglio delle voci di spesa presenti nella fattura in oicen mbe del mese di gennaio per la linea phonen mbe :in oicede ail . vuoi avere il dettaglio di una specifica voce di spesa presente in fattura? (he e i he de ai f he i e he i ice in oicen mber f ja a f he phonen mber : in oicede ail . d a he de ai f a ecific i e he i ice?) no vorrei capire perch non 29,00 come da promessa (n i j a h i ' 29 a i ed) s e ha f a ed vuoi cambiare argomento? (d a cha ge ic?) t ic cha ge grazie non arriviamo a niente (f ge i , hi i g i g he e) n -c e a i i -ve f a ed request for explanation. a simple string search method was thus carried out to filter such kind of dialogues, using the following strings: sapere/capire perché foot_0 ("know/understand why") and come mai ("why/how come"). the resulting corpus consists of 142 dialogues, with an average of 11 turns per dialogue, and an average length of 9 tokens in customer messages and 38 tokens in the bot messages. such difference in the message length is due to the way the assistant's responses are currently structured, in that they usually include detailed information e.g. on invoice items or options available, while, on the other hand, customer's messages are most often quite concise. also, the relatively high number of turns per dialogue might be explained with the high occurrence in the corpus of repeated or rephrased messages, both by the virtual agent and the user, due to recurring misunderstandings on both sides. the corpus underwent an annotation process that involved multiple, complementary, dimensions, such as errors in conversation and emotions (see figure 1 for an example). the explanation request and its sub-types have been included as one of such dimensions and we mainly focused our attention on these in this phase of the study. the types of requests for explanations in this collection reflect the different kinds of problems typically encountered with a telephone operator. based on a preliminary analysis of the corpus, we distinguished 5 main types of requests plus a generic category that includes a variety of cases that is more heterogeneous and not classifiable according to the main types. hence, we identified requests for explanations or clarifications regarding the following topics: (1) charges in the invoice or in the phone credit (about 52% of cases), (2) timing and methods of receipt of the invoice (10.5%), (3) unpaid invoice reminders erroneously received (10.5%), (4) currently active promotions (8%), (5) payment methods (5%). the remaining cases (14%) were included in the more generic "other" class. starting from this analysis we thus defined a reduced set of possible scenarios, i.e. prototypical situations that can be found in the dialogues and grouped together according to similar characteristics. for illustrative purposes, we describe in this paper the three main scenarios defined for the first request type, i.e the one regarding undue or unclear charges, being by far the most frequent case of request. in scenario 1 (31% of the occurrences) the customer asks for an explanation about a higher charge with respect to previous ones, also providing specific information on the amount charged; in scenario 2 (58% of the occurrences), a charge in the account is claimed, but no further information is provided in the user's message. in scenario 3 (11% of the occurrences) the customer asks for an explanation about a negative balance. ## importance and effect for content selection in the customer care domain we consider three scenarios arising form the corpus analysis (section 2). formally, each scenario consists of a set of sequences of transactions, where a transaction is a money transfer operation between a customer and the company (i.e., an amount paid for a certain service). as a result, each transaction sequence represents the different amounts paid along a time period for a specific service (transaction type). to determine the importance and the effect of a transaction sequence, we assume to know all the transactions on the user's account in the last seven months. it is worth pointing out that the two most important elements in this specific context are money and time. therefore, we want to formalize the intuition that the importance (in biran and mckeown's terminology) of a telco service can be associated with the amount of money that the user usually spends for such service, while its effect can be associated with the amount of money that the user spent for the service in the last month. we thus define the importance of a transaction sequence as the mean of the normalized values of the transactions in the past six months. moreover, we define the effect of a transaction sequence as the normalized value of the transactions in the current month. normalization is carried out by dividing the amount of the transactions by the maximum amount that the user has paid for that transaction. an important point in the biran and mckeown (2017) model is the procedure for transforming importance/effect numeric values in the discrete {low, high} values. in accordance to the original model, we determine the smallest subset h of transaction sequences such that the sum of their importance/effect values is greater than the 75% of the total importance/effect. when such a smallest subset is not unique, we consider the union of all the smallest subsets. note that the value of 75% has been set in order to adhere to the original model, that has been proposed in @xcite without a specific motivation. however, we consider this limit as a tunable value that should be empirically validated on the specific domain. in the following discussions we analyse the scenarios for three common requests of explanation. we separately analyse these scenarios but not that they are not mutually exclusive. it is worth noting that a complete nlg architecture could account their possible coexistence by using some form of syntactic or semantic aggregation. finally, note that in the discussion on these scenarios we are completely neglecting both the linguistic information arising from the dialogue (the user's question) as well as any kind of information on the customer. in other words, we are inferring the customer's explanation request as a content selection task only, without taking user utterances and user model into account. as a matter of fact, there are some cases where the user searches for a complete information about its transactions: for instance, the user wants to review all the transactions of the last months. in this case, the linguistic input should trigger the dialogue manager and the nlg system to provide information on transactions with normal evidence after the information on exceptional evidence. in contrast, there are situations such that the user wants to have only a short summary on its transactions and in this case the nlg system should only provide information on transactions with exceptional evidence. in future research, we plan to study how to merge the linguistic, the domain and the user model information. m1 m2 m3 m4 m5 m6 m7 s1 10 10 10 10 10 ## conclusions and future work this paper reports the first results of an ongoing study on the role of nlg for a ds in the customer care domain. we provided a corpus analysis that shed some light on the customer requests regarding explanations foot_2 . moreover, we adapted the model proposed in @xcite for narrative roles in explanation for this specific kind of input data. in this way, we designed a content selection procedure accounting for evidence of data. we are working on the inclusion of the content selection procedure described in this paper into a complete nlg architecture for ds. in this linguistically sound nlg architecture, we use a simple rule-based sentence planner @xcite in combination with the italian version of simplenlg @xcite for generating messages that give emphasis and priority to the content elements with high evidence. for instance, in this architecture we can decide to generate final messages that contain only (or mention primarily) contents with exceptional evidence. as a future work, we are designing a user-based comparative evaluation of the ds exploiting the complete nlg architecture following the schema adopted in @xcite . the idea is to