Enhancing Pipeline-Based Conversational Agents with Large Language Models (2023)

Abstract: The latest advancements in AI and deep learning have led to a breakthrough in large language model (LLM)-based agents such as GPT-4. However, many commercial conversational agent development tools are pipeline-based and have limitations in holding a human-like conversation. This paper investigates the capabilities of LLMs to enhance pipeline-based conversational agents during two phases: 1) in the design and development phase and 2) during operations. In 1) LLMs can aid in generating training data, extracting entities and synonyms, localization, and persona design. In 2) LLMs can assist in contextualization, intent classification to prevent conversational breakdown and handle out-of-scope questions, auto-correcting utterances, rephrasing responses, formulating disambiguation questions, summarization, and enabling closed question-answering capabilities. We conducted informal experiments with GPT-4 in the private banking domain to demonstrate the scenarios above with a practical example. Companies may be hesitant to replace their pipeline-based agents with LLMs entirely due to privacy concerns and the need for deep integration within their existing ecosystems. A hybrid approach in which LLMs’ are integrated into the pipeline-based agents allows them to save time and costs of building and running agents by capitalizing on the capabilities of LLMs while retaining the integration and privacy safeguards of their existing systems.

## introduction the field of conversational artificial intelligence (cai) has experienced significant advances in recent years, with the emergence of both commercial and open-source cai development platforms such as google dialogflow, amazon's alexa skills kit, cognigy, and rasa, as well as the more recent large language model (llm)-based conversational agents (ca) like chatgpt. cas can be text-based agents (chatbots), voice-user interfaces (vui), or embodied-dialog agents (eda) @xcite and generally aim to replace or empower humans through natural language interaction. cas can be pipeline-based or end-to-end @xcite . in pipeline-based cas, the natural language understanding (nlu) component processes the user's message sequentially to identify their goal (intent recognition), and extract information pieces called entities. the dialog management component tracks the dialog state and decides on the next action based on the current state. finally, the natural language generation (nlg) component builds and returns the response. the cas "intelligence" relies on the agent's training data and internal logic used to create its nlu and dialog management models @xcite . the end-to-end cas rely on dialog models trained with large training datasets @xcite . these models learned hidden relations between input and output utterances, effectively avoiding that developers create interim representations @xcite . a downside is that the necessity of larger datasets makes end-to-end approaches less applicable in domains such as manufacturing, where developers cannot derive training data from existing human-human conversations. end-to-end cas also bear substantial safety issues, such as generating offensive language and responding inappropriately to offensive content or in safetycritical situations @xcite . combinations of pipeline-based and end-to-end approaches are also feasible. rasa open source, for instance, already supports both @xcite . this article uses the term llm to refer to language models trained with an end-to-end approach and an amount of high-quality training data that only a few organizations in the world can afford e.g., gpt-3 and gpt-4 by openai, bert, lamda, and palm by google, llama by meta ai. such models can, for instance, possess emergent abilities, be hard to steer, and humans will likely have difficulties interpreting how they work @xcite . in this article, we demonstrate how llms can expand the capabilities of pipeline-based cas without removing the pipeline altogether. the impact of llms helps the pipeline-based cas in generating training data for intent classification, the identification of domain-specific entities and synonyms, requirement characterization for the agent and its personalization and localization, among others. during deployment, llms can provide auto-correction to user input, handle context switching and outof-scope questions, introduce response variability, create conversation summarizations and perform closed question-answering (q&a). ## the state of conversational agents broadly, cas can be categorized into two main categories based on the design methodology employed: pipeline methods and end-to-end methods @xcite . agents that are developed using conversational ai platforms (task-oriented cas), such as rasa, google dialogflow, cognigy, and ibm watson, fall into the first category. llmbased cas such as chatgpt can be identified as cas belonging to the second category. while explicit architectural components can be identified in the pipeline-based cas, such clear distinctions cannot be identified in end-to-end cas. ## llms to overcome limitations of pipeline-based cas despite the various frameworks for building pipeline-based cas, it still requires substantial time and expertise to design and develop successful cas. related tasks concern the design of high-quality training utterances, the definition of intents and consistent and accurate named entities, the selection of domain-specific synonyms, and the localization of training data and responses. besides, designers must modulate, for instance, training data, dialog management rules, and pre-defined responses to represent desired assistant traits (e.g., client orientation) or personas. we assume that the strengths of llms in processing natural language from different countries and domains can substantially shorten at least the time and potentially also the expertise needed to build pipeline-based cas. their capability to generate responses matching the style of a generated persona or mimicking an actual person's style could provide new techniques to create attractive cas. a second area for improvement is the robustness of a ca at run-time, i.e., when it interacts with a user. often, pipeline-based cas produce repetitive responses (robust but less attractive) or experience conversational breakdown because users switched contexts (not robust). in addition, pipeline-based cas' narrow domain knowledge provokes out-ofscope answers due to smaller training data and limited responses. all of the situations above lower the user's satisfaction and could encourage them to give up on the agent. we assume that llms' extensive general and domain knowledge, coupled with their capability of generating attractive and diverse natural language texts, has the potential to achieve more robust and attractive cas. we conclude that llms have the capability to enhance pipeline-based cas during the design and development phase (delivery accelerator) and a dialogue with a user (real-time booster). in contrast to relying on llms only, this hybrid approach is helpful because the pipeline-based approach grants the ca designer more control and transparency over the agent's behavior. the former is critical to counter, for instance, hallucinations, while the latter helps trace and potentially explain unexpected or unwanted behavior. ## gpt-experiments to demonstrate the impact of llms on cas, we conducted a series of experiments with gpt-4. the example scenario is a chat agent serving as a client advisor for private banking. a supporting document contains the exact prompts and replies in the conducted experiments. parameters we used the azure open ai playground with the default parameters for our experiments: max response: 800, temperature: 0.7, top p: 0.95 frequency penalty: 0, presence penalty: 0, deployment: gpt-4, past messages included: 10, max tokens: 8192. the temperature value of 0.7 means that generated responses are not deterministic, i.e., the exact response may vary during reproduction. to keep this article short, we sometimes shorten the actual prompts and answers by inserting an ellipsis. ## outlook this paper proposes a hybrid approach that leverages llms, in particular gpt-4, to enhance pipeline-based cas. using this approach, maintainers of existing cas can adopt new domains and overcome the limitations in conversations with users while ensuring seamless integration with the existing ecosystem. this approach accelerates the ca delivery process through the assistance of llms in generating intents, entities, synonyms, respective training data, and agent personality traits. during deployment, llms can boost pipeline-based cas' performance by utilizing autocorrect, context-switching capabilities, answering out-of-scope questions, creating diverse and stylistically richer responses, and incorporating closed q&a and summarization. this paper presented experiments to showcase the scenarios mentioned above. in future work, we will extend the ad-hoc subjective assessment to a more rigorous evaluation among different llms and provide an integrated solution to demonstrate the proposed hybrid approach. given the existing risks regarding the reliability of llms @xcite @xcite , our future research will focus on examining the factors that prompt business owners to consider the integration of llms within their pipeline-based cas.