Evaluation and Continual Improvement for an Enterprise AI Assistant (2024)

Abstract: The development of conversational AI assistants is an iterative process with many components involved. As such, the evaluation and continual improvement of these assistants is a complex and multifaceted problem. This paper introduces the challenges in evaluating and improving a generative AI assistant for enterprise that is under active development and how we address these challenges. We also share preliminary results and discuss lessons learned.

## introduction generative ai assistants for enterprises hold the great promise of significantly improved productivity, lowered barrier-to-entry, drastically increased product adoption, transformative amplification of creativity, and delivery of better customer and employee experiences @xcite . developing such an ai assistant is typically an iterative process, with its evaluation and continual improvement at the center. fig. 1 depicts the high-level architecture of adobe experience platform ai assistant 1 @xcite , a generative ai assistant built for an enterprise data platform. as can be seen, it is a complex pipeline with multiple underlying components consisting of one or more machine learning models based on large language models (llms) or small language models (slms). users interact with the system via a conversational interface to obtain answers based on heterogeneous data sources. the evaluation and continual improvement of such a system is a complex and multifaceted problem with the following key challenges. metrics. the success of assistant is ultimately measured by metrics such as user engagement, user satisfaction, and user retention. however, such metrics are lag measures obtainable only after building and deploying assistant in production. to 1 hereafter referred to as assistant guide continual improvement of assistant, we also need to define metrics that are lead measures for various aspects of assistant that are likely to impact the lag measures. data. to produce reliable evaluation metrics for assistant, we need data that are both representative and high-quality. we need a systematic approach to obtain such high-quality data at scale. dynamics. as shown in fig. 1 , a real-world ai assistant usually consists of a complex pipeline of components. each component evolves over time as both the underlying models and the assistant's functionalities change. further, in enterprise settings, the distribution of questions asked is ever-changing as the customer base shifts and grows and existing customers mature in their adoption of the assistant. we need to consider such customer dynamics. ## limitations of existing approaches common approaches for evaluating ai assistants include evaluation using explicit feedback, evaluation using implicit feedback, benchmarking @xcite , and human evaluation @xcite . explicit feedback is collected from users through feedback buttons, direct prompts, or questions on their preferences. in contrast, implicit feedback is gathered from user actions within a system, such as clicks, views, or navigation patterns, providing insights into user behavior and preferences without requiring direct input. evaluating with benchmark datasets is also a common way to evaluate ai assistants. these approaches, while important and effective to a certain degree, suffer from various limitations when it comes to evaluating an enterprise ai assistant such as assistant, which is under active development and improvement. ## our approach in this section, we introduce our framework to overcome the aforementioned challenges (section 1) and limitations of existing approaches (section 2) for evaluating an enterprise-grade ai assistant under active development. ## preliminary results: examples while assistant remains in active development, our evaluation and continual improvement framework already show promising impacts on both the prioritization and the design of improvements. in this section, we share the preliminary results obtained so far by examples. fig. 3 illustrates an example error dashboard produced by the annotation tool, showing componentwise and end-to-end errors with further breakdown by severity levels as well as how they change over time. this dashboard is monitored by all stakeholders and is used to track the impact of feature releases and improvements. while ideas for improving assistant may be endless, detailed error analysis allows the team to follow a powerful organizing principle: focusing on reducing error rates based on their actual impact on the users. for instance, the example report in however, the new classifier also led to a new, particularly frustrating source of errors: in-scope questions misclassified as out-of-scope would no longer be answered. without being able to quickly improve the classifier's precision, we used our other available lever of improvement and designed an override mechanism in the ui to allow users to receive an answer. as the sprint 2 report shows, this ui change converted a potential sev-1 error (refusal to answer) into a sev-2 error (the user could now recover), showcasing how human-centered ui design allows holistic improvement of assistant. explainability is important for improving user trust and comprehension. by helping users discover wrong answers with better explainability, we can reduce sev-0 errors and move them to sev-1/sev-2 error buckets. we took a data-driven approach to choose from many applicable explainability techniques @xcite . we first went through the sev-0 queries obtained during a certain window and examined which technique(s) can be used to alleviate the severity of each error based on the potential overall impact of each explainability technique, its implementation difficulty, and human cognitive load. we created a decision matrix ( ## discussion this framework has organically evolved during the development of assistant. while many of the design choices laid out may seem obvious in hindsight, they were not as clear at the beginning of this project, and so it is worth discussing the lessons we have learned along the way. first, we have found that metric design is of paramount importance. the severity framework came after many iterations in trying to connect enthusiastic early customer feedback with a seemingly large overall error rate. the insight that customers have varying tolerance depending on the class of errors has become a powerful organizing principle for our prioritization and resource allocation to improve assistant. next, we have seen firsthand the benefits of building a decomposed system as opposed to depending on a single, monolithic model. the choice to decompose into multiple, orchestrating models was led by constraints such as task specialization and the need to query real-time data. we have also reaped the secondary benefit of having many available "levers of improvement" (prompts, in-house models, specialized indexes, ux improvements, etc.), many more than what is possible in a single language model paradigm. finally, iterative and agile development are more important than designing everything upfront and building specialized tools. for instance, while it is tempting to build in-house tools, using spreadsheets as a simple alternative initially allows us to learn important lessons on designing the annotation tasks, from annotation guidelines to the actual ui. ## future work as we continue to develop assistant and onboard more customers, we plan to extend our evaluation and continual improvement framework with more human-in-the-loop/llm-in-the-loop automation to scale our evaluation and error analysis processes @xcite . in addition, the current framework heavily focuses on retrospective analysis based on past customer interactions. we plan to extend it with more proactive user studies and evaluation of in-development functionalities. moreover, personalization is also important for enterprise ai assistants since we have customers with different technical levels. to provide the best experience to various personas in potentially different languages, additional evaluation metrics and datasets proposed in @xcite may also be considered. as we have emphasized, human-centered design is essential for the success of assistant. we plan to further explore how the deeper interplay between ml and ux components in this new paradigm of hci can lead to more explainable and accurate assistants. finally, the impact of generative ai applications in the workplace is an important new area of study @xcite . as we enroll new customers, we intend to run a/b tests (hussey and hughes, 2007) that assess the causal impact of assistant on the engagement and productivity of customers.