A Case Study of User Communication Styles with Customer Service Agents versus Intelligent Virtual Agents (2020)

Abstract: We investigate differences in user communication with live chat agents versus a commercial Intelligent Virtual Agent (IVA). This case study compares the two types of interactions in the same domain for the same company filling the same purposes. We compared 16,794 human-to-human conversations and 27,674 conversations with the IVA. Of those IVA conversations, 8,324 escalated to human live chat agents. We then investigated how human-to-human communication strategies change when users first communicate with an IVA in the same conversation thread. We measured quantity, quality, and diversity of language, and analyzed complexity using numerous features. We find that while the complexity of language did not significantly change between modes, the quantity and some quality metrics did vary significantly. This fair comparison provides unique insight into how humans interact with commercial IVAs and how IVA and chatbot designers might better curate training data when automating customer service tasks.

## introduction an intelligent virtual agent (iva) is a subset of chatbots designed for the commercial enterprise realm to mimic a human customer service agent. a popular use case for ivas is live chat deflection, where they are trained to handle the most common interactions while still allowing for escalation to a human agent when required or requested. as a company that has designed and built ivas for enterprise applications for many years, we had intuition that the language we saw in live chat interaction was different from the language we saw coming into the iva, but the difference had not yet been quantified. after using live chat data for training an iva, we were occasionally surprised at the gaps in understanding it presented once in production, even though the training data originated from the same company the iva was built for. in order to improve analysis and training, we sought a standard to create and gather data more consistent with actual iva usage and filter out some of the non-representative live chat data. we also wanted to investigate how the iva was affecting conversations with live chat operators. while there are differences, a user behaves consistently when he/she is chatting with a human, similarly they are consistent when chatting with an iva. in this paper we demonstrate that chatting with an iva has significant impact on language beyond what has been documented by human-tohuman computer mediated conversation such as instant messenger or live chat. the iva and live chat corpora used in this study originated from a financial services company where customers are interacting with the iva and live chat on their website for the same purposes making the comparisons extremely relevant. however, due to data use agreements with the financial services company, the identification of the origin and corpora cannot be made public. while tools to improve the training process of ivas from live chat or call center transcriptions exist such as @xcite , there has not been a focused linguistic study on the difference in communication styles between human-human and human-machine in service dialogs. such a study could inform such tools where specific samples may or may not make good training samples due to projected communication differences with ivas. to our knowledge this is the first study to compare real world language of users with ivas and live chat from the same origin. ## method the iva for this research was originally trained on live chat conversations from the financial sector and continuously refined while in production. it was designed to understand frequently asked questions and conversational work flows around the largest business use case: waiving fees (for example conversations see appendix b). besides business intents, the iva also responds to persona (e.g. asking if the iva is married), common courtesy, and profanity. escalation points were designed where human involvement was desired (e.g. credit limit changes, account closure). there was no dynamic response delay, no avatar, and users were informed at the beginning of the conversation that they were speaking with an iva (see appendix a). for our corpus, we selected 16,794 conversations with live chat agents from june through october 2017 and 27,674 conversations with the iva that occurred in january 2020. within the iva conversations, 19,350 conversations were completed with iva only while 8,324 escalated at some point to a live chat agent. for the purpose of this work we only looked at the user language and actions and not the iva or live agent responses. the iva was launched in 2017 on the company website along side a live chat option. after 2017, access to live chat without first talking to the iva was disabled due to the success of the iva at automating a continuously expanding set of use cases. we chose to sample iva data from 2020 to allow for adequate refinement time to present statistics representative of communicating with a mature iva implementation. ## analysis conversation level: we begin with the full conversation level metrics shown in live chat conversations take 1.7 times more turns with more than 2.5 times more words. where escalation is not required, a user can achieve a more efficient resolution with the iva. however, if the iva is in fact deflecting the easier to handle issues this could explain some of the differences. on the other hand, the user experience for escalation is substantially less efficient. first the user has an average length iva conversation and then escalates to the human agent for a more involved conversation with an average of 1.5 more turns and 14 more words than the live chat sessions alone. this indicates the user's tasks presented to the iva are not being properly reviewed by the live chat agents, requiring substantial additional effort turn level: human-to-human gratitude is significantly more frequent than with an iva. however, after continuing to a human after the iva, gratitude is almost halved. this reflects the more difficult conversations when live chat is tier 2 support. sentiment for human-to-human was significantly more positive. iva turns were neutral. iva-only turns averaged 0.08 where as live chat conversation turns averaged 0.2. one would expect the live chat conversations that were preceded by the iva to be more negative reflecting the decrease in user efficiency discussed in the previous section. however, sentiment for live chat after iva actually remained at 0.2, perhaps indicating that live chat was usually leading to a reasonable (if not always satisfactory) resolution or the additional effort seemed justified to the users as they were in a sense restarting the conversation with a new party. @xcite showed significant profanity in chatbot language and burton and gaskin (2019) showed a self-reported tendency to be less polite to digital assistants. in our data, only live chat sessions had any profanity to speak of. we speculate that the overall lack of profanity has to do with the professional setting of the customer service environment where previous studies were on open domain chatbots and personal assistants such as amazon's alexa. pronouns: live chat users were almost 2.9 times more likely to refer to the human as 'you' than they were with an iva ( l2sca returned results that could be explained by shorter turns and fewer words shown between live chat and iva ( l2sca did show that live chat language after iva was less complex across every measure. this may be part of the explanation for the reduction in gratitude in those conversations: they were less polite because they were more concise. it may be that as the conversation is less efficient, the language becomes more efficient to compensate, but more research is needed to prove this hypothesis. ## application and conclusion when designing an iva and when given live chat data for training, it's tempting to tag random inputs indiscriminately for training. however, indiscriminately adding longer inputs more when live chat data is not available, a synthetic strategy must take place. one such strategy outlined by @xcite is to give a human a sample input and ask them to synthesize new data. a better plan would be to give the user a task of retrieving information and then asking them what questions they would use to get that information. this would encourage the simple direct language that ivas are more likely to see in the wild instead of forcing the human to be creative, which may result in language unlikely to be seen by an iva. users are currently more likely to be concise with ivas than human live chat operators in the same domain. this is an advantage to the user as an iva can respond more quickly and get them the needed information with less language production. modern contact centers use various performance metrics to rate contact center agents, which has a direct impact on their compensation and recognition @xcite . a successful iva will significantly reduce the number of conversations coming into a contact center, and the conversations that do will largely consist of more difficult cases. as these types of conversations now make up a much larger part of the performance metrics, and the operators no longer get the positive feedback from the easy cases, they appear less effective than before the iva was implemented. this decrease in performance should be expected and these metrics adjusted, perhaps by weighting by the difficultly of the task, so that human contact center agents are not punished by the deployment of ivas alongside them. there exists substantial research regarding how to measure the performance of a dialog system, but the study of how people communicate through language with artificial intelligence in the wild is still in its infancy. the nature of the data originating from commercial ivas means that corpora are seldom shared, making the research more challenging. in spite of these sharing restrictions, in this paper we have presented an approach to analyze the nature of language use between humans and ivas compared to that of human chat operators in a way that still allows the research community to understand in what way humans currently communicate differently with ivas than other humans in the same domain. if others with commercially deployed ivas repeat such experiments we can observe how humans adapt to ivas over time in the wild, and change the way iva conversations are designed accordingly.