Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Support (2022)

Abstract: Building Agent Assistants that can help improve customer service support requires inputs from industry users and their customers, as well as knowledge about state-of-the-art Natural Language Processing (NLP) technology. We combine expertise from academia and industry to bridge the gap and build task/domain-specific Neural Agent Assistants (NAA) with three high-level components for: (1) Intent Identification, (2) Context Retrieval, and (3) Response Generation. In this paper, we outline the pipeline of the NAA’s core system and also present three case studies in which three industry partners successfully adapt the framework to find solutions to their unique challenges. Our findings suggest that a collaborative process is instrumental in spurring the development of emerging NLP models for Conversational AI tasks in industry. The full reference implementation code and results are available at https://github.com/VectorInstitute/NAA .

## introduction rising demand for ai-powered conversational agents @xcite , especially for customer support service, is estimated to grow at a compound annual growth rate (cagr) of 23.4%, earning a predicted revenue of around $29.9b usd by 2028 @xcite . as such, conversational ai research has increased substantially, especially to enhance customer service support @xcite . despite a proliferation of agent assistants from microsoft, ibm, oracle, and google, there remain many unanswered questions and challenges @xcite that need to be addressed before the widespread proliferation of ai in practice. we argue that bridging the gap between natural language processing (nlp) research in academia and industry is an overlooked issue in conversational ai. apart from the handful of aforementioned conglomerates, it is difficult for most other companies, which are not in the process of conducting cutting-edge nlp research, to benefit from the recent progress of nlp. conventional conversational ai architectures are delicate and complex, and require a large degree of specialised knowledge to bring a full system into fruition. however, with the introduction of the large-scale pretrained transformer-based language models such as bert @xcite and @xcite , it is possible for smaller teams to take advantage of this "monopoly" by being able to fine-tune these powerful models on their comparatively small amount of data and achieve high performance, therefore harnessing the architectural achievements of those powerful language models to devise their own conversational ai systems that exploit their more fine-grained expertise and knowledge of their customers' needs. the interdisciplinary nature of ai-enabled customer service support makes for an inherently difficulty task @xcite . to arrive at an answer to a user query, a system must bore through several layers of complexity: first, the intent behind the question must be categorized and quantized into a form which can be manipulated by the system; second, determining from thousands or millions of stored records the information relevant to that intent; and finally to manipulate said information into a form which the user may understand. to handle such complexity, techniques from across nlp must be employed, raising the barrierof-entry for companies with significant customer support needs. larger, well-established companies can offer solutions as a service, but said solutions are often closed-source, trained on generic data with few in-domain terms, and may not be easily integrated into a company's workflow. this paper represents the cumulative efforts of both researchers and industry practitioners over a yearlong project (whose details may be found in appendix a) to develop state-of-the-art customer service support systems. in its publication, we hope to help lower the barrier-of-entry for future industry practitioners by inspiring similar collaborations. our contributions are twofold and in line with the strengths of the contributors: in their presentation, we stress that a perfect, universal customer service system is unattainable as their real-world applications are not identical. as such, any dialogue about real-world naa must feature a discussion of both the technologies shared with, and the differences between tasks. ## neural agent assistant framework for customer service support as the basis of our collaborative project, we established a common neural agent assistant (naa) framework to serve as a guide for industry partners to adapt to their companies' workflows. those technologies were made available in a clear, easy-to-use engineering pipeline, alleviating one of the greater challenges for industry in obtaining a foothold in building competent neural agents. the framework features three high-level components shown in figure 1: (1) intent identification, (2) context retrieval, and (3) response generation. though designed as parts in a pipeline in generating a natural language response to customer questions (figure 1 (b) ), the output from any given component can nonetheless be transferred back to the human agent when needed (figure 1 (a) ). ## industry-specific proof of concept implementations in this section, we demonstrate how the naa framework outlined in section 2 was adapted to our industry partners' use-cases and settings, which is continuing to be refined for deployment in realworld scenarios. canadian imperial bank of commerce (cibc) is a financial institution with an advanced analytics team focusing on finding the correct curated response to banking queries. kpmg is a professional services firm which is looking to naa to service internal queries over large bodies of legal and regulatory documents. pricewaterhousecoopers (pwc) is also a professional services firm looking to better advise its banking clients on building naa technologies. in the following subsections we clarify the companies' motivations in participating in the project, how they adapted our naa framework, and their plans for deployment. ## summary and discussion we summarize our experiences in this section. domain/task-specific system works better: as stated earlier, the task/domain agnostic system is not well-suited for different use-cases as our industry partners found, and as confirmed by our experiences. real-world conversational datasets are noisy and contain domain-specific concepts, and as a result, domain agnostic systems that are trained on clean open-source datasets are likely to have poor performance when evaluated on realistic inputs. the best practice would be to have a system designed/adapted based on domain and task. finding a good performance evaluation method: due to the inherent difficulty in evaluating the quality of retrieved contexts and generated answers, finding a suitable model assessment strategy is imperative (khapra and sai, 2021), which includes the processes of model selection, bench-marking, and deciding between manual and automated assessment. automatic evaluation metrics in particular, although flawed, allow for the ability to benchmark many different model variations, leading to a strong model being picked if there is a general agreement between different metrics. the companies found that using a combination of automatic evaluation metrics like bleu score, f1-score, rouge-1, along with expert manual evaluation, helped better inform model selection.