D eep P avlov Dream: Platform for Building Generative AI Assistants (2023)

Abstract: An open-source DeepPavlov Dream Platform is specifically tailored for development of complex dialog systems like Generative AI Assistants. The stack prioritizes efficiency, modularity, scalability, and extensibility with the goal to make it easier to develop complex dialog systems from scratch. It supports modular approach to implementation of conversational agents enabling their development through the choice of NLP components and conversational skills from a rich library organized into the distributions of ready-for-use multi-skill AI assistant systems. In DeepPavlov Dream, multi-skill Generative AI Assistant consists of NLP components that extract features from user utterances, conversational skills that generate or retrieve a response, skill and response selectors that facilitate choice of relevant skills and the best response, as well as a conversational orchestrator that enables creation of multi-skill Generative AI Assistants scalable up to industrial grade AI assistants. The platform allows to integrate large language models into dialog pipeline, customize with prompt engineering, handle multiple prompts during the same dialog session and create simple multimodal assistants.

## introduction complex ai assistants are becoming more and more widespread. as a result, interest in technology for building complex conversational interfaces has grown significantly over the last years. at the same time, most of the available systems enabling complex dialog systems development are proprietary or limited. this opens up new opportunities for open-source systems that facilitate the development of complex ai assistants. the deeppavlov dream platform foot_0 provides a stack of apache 2.0-licensed open-source technologies that enable development of complex dialog systems such as enterprise ai assistants. the platform features a conversational ai orchestrator called deeppavlov agent to coordinate an asynchronous scalable dialog pipeline; a framework called @xcite to facilitate development of the multi-step skills; support for wikidata and custom knowledge graphs; a library of modern nlp components and conversational ai skills (script-based, chit-chat, question answering (qa), and generative skills) organized into a set of distributions of multi-skill conversational ai systems; and a visual designer. these components make it possible for developers and researchers to implement complex dialog systems ranging from multi-domain task-oriented or chit-chat chatbots to voice and multimodal ai assistants suitable for academic and enterprise use cases. the deeppavlov dream platform supports integrating large language models (llms) into production-ready dialog systems with the help of general and custom knowledge graphs (kgs) for fact checking, pre-and post-filters for filtering out unsuitable responses, and prompt-based generation for indirect control of llms. multimodality in deeppavlov dream provides an opportunity to operate with images and to perform actions via apis based on the user's commands extracted during the conversation. in this paper, we present the deeppavlov dream platform for dialog systems development. in section 2, we compare the platform with existing competitors. sections 3, 4 and 5 introduce the pipeline's orchestrator, distributions and other development tools. the platform's components and proposed approaches for dialog system development are described in section 6. section 7 presents our methodology for building prompt-based generative ai assistants. approaches to managing multimodality are described in section 8. finally, section 9 provides an overview of a custom dialog system development process. ## comparison with competitors in ## pipeline the deeppavlov dream is built upon the deep-pavlov agent, an open-source framework for orchestrating complex systems. the full dialog system pipeline of the deeppavlov dream is presented in figure 1 . there are four component groups -annotators, skills, candidate annotators, and response annotators -and two dialog management components -skill selector and response selector. dialog state is a shared memory that contains all the information about the dialog as deeppavlov agent expects that services are stateless and can be run as multiple instances. there are two synchronization points -skill selector and response selector. the other services can be run in parallel, although the agent allows dependencies between services within the group to effectively chain them. deeppavlov agent's pipeline is asynchronous, i.e., one user's request does not block the agent and does not prevent the agent from receiving and processing requests from other users. each service is deployed in a separate docker container. in addition, separate containers are used for the agent itself and the database. for development, one can run the dialog system locally using docker-compose or kubernetes. ## distributions original dream socialbot @xcite included a large number of components for natural language understanding (nlu) and natural language generation (nlg). different components are run independently and accept dialog state in a required format. although some components may depend on the other ones' annotations, in general modular system's elements can be safely removed or replaced with their analogues. platform users can re-use existing components to design their own dialog systems or develop custom components and include them into existing or custom dialog systems. the deeppavlov dream platform utilizes a distribution-based approach for dialog systems development. a distribution is a set of yml-files specifying parameters of docker containers, and a configuration json-file determining a processing pipeline for deeppavlov agent. platform contains different distributions including script-based english distributions, generative-based english, russian and multi-lingual distributions, multimodal distribution, robot controller distribution, and lots of multi-skill distributions utilizing prompt-based generation with llms (details in section 6). ## development tools platform is supported by development tools -deeppavlov dreamtools and df designer @xcite , a visual aid in developing scenario-driven skills for the dream platform using dff foot_1 . deeppavlov dreamtools is a set of tools in python with a built-in command line tool which allows developers to operate with platform distributions in a programmatic way. python package the package exposes dream distribution api via configuration and component objects. it supports a strict set of configuration files: one pipeline json-file and 4 docker-compose yml configuration files (for production, development, proxy, and local deployments). for each file, the package implements configuration definitions domain-specific nlu no no yes 3rd-party limited 3rd-party yes yes yes yes external nlu yes yes no yes no partial no yes no unknown scalability yes no yes yes yes yes yes yes yes yes multimodality text,voice-3, image(wip), video(wip) text, voice-3 text, voice text, voice-3 text, voice text, voice text, voice text, voice-3 text, voice-3 text, voice additionally, its higher-level functionality allows for creating new distributions from scratch or using the existing ones as a template, editing distributions, verifying consistency between component definitions (e.g., correct port forwarding, component naming, etc.), and creating dff-based skills with all the necessary template files. we have also trnasferred distribution testing utilities previously bundled with dream: an automated dialog tester and a file validation script. cli deeppavlov dreamtools can be used as a command line utility that wraps most of high-level package functionality. one of the common uses is creating a local.yml configuration file, which contains instructions on whether to deploy a component locally or redirect to a proxied deeppavlov deployment. ## components the platform contains english, russian, and multilingual annotators and skills. skill and response selectors are language-agnostic in general. the skill selector's output may include skills not present in the running pipeline, making the skill selector's code reusable. there are several available algorithms for response selector -tag-based for script-based distributions, llm-based and rankingbased for generative distributions. modular container-based architecture of the dream platform allows for integrating components from different frameworks and with different requirements. for many conversational nlp models, deeppavlov @xcite library is extensively used in several annotators and skills. deeppavlov library, an open-source conversational nlp framework that is based on pytorch and supports huggingface transformers, which allows it to use various transformer-based models from huggingface hub @xcite . traditionally, deeppavlov contains several variants of models: the best-performing one with the highest score, the resource-efficient one with the fastest inference time @xcite , and the multilingual one with the support of several languages. deeppavlov library provides ready-for-use tools for training and inference of nlp pipelines along with appropriate docker images that simplify integrating and adapting new components. annotators solve a variety of nlp tasks: text re-writing (sentence segmentation, punctuation recovery, spelling preprocessing, coreference resolution ( le et al., 2019) ), text classification (sentiment, toxicity, emotions, factoidness, topics @xcite , dialog acts @xcite , intents and speech functions @xcite ), token classification (ner, entity detection), knowledge re-trieval (requests to external apis, entity linking to kbs (evseev), knowledge extraction from structured and textual kbs), text ranking (selecting most relevant prompts for conditional generation, response candidate ranking @xcite ). conversational input usually does not imply correct case sensitivity, so deeppavlov dream utilizes ner model @xcite adapted to both cased and uncased inputs. additionally, the model is based on the multilingual bert (mbert), which allows to support entity extraction in multiple languages due to the mbert cross-lingual transferability @xcite . resource consumption is also one of the most challenging parts of the dialog systems development. often, the production-level annotators quality can be reached only by nn-based models each of which may require gigabytes of gpu memory. to tackle this problem, deeppavlov equips dream with a multi-task learning (mtl) classifier @xcite trained to solve nine problems within a single model and decreasing gpu memory usage ninefold. ## prompt-based generative ai assistants since the emergence of llms, they have been used to tackle a variety of natural language tasks. the earlier and smaller models had to be fine-tuned for each specific task. however, very large models have shown a remarkable capacity of handling the same tasks few-shot and zero-shot using prompts, which are tokens appended or prepended to the model's as an instruction to "guide" its behavior. when using llms in dialog modeling, it is crucially important to avoid insensitive or potentially harmful content and tailor the responses for the specific user needs. that is where prompt engineering allows to steer the model in the right direction with no need for fine-tuning. deeppavlov dream provides an approach to building generative ai assistants with promptbased control and lets the user develop their own ones with the use of prompt engineering. for that, the platform features generative skills, which en-capsulate generative model services used to locally run the llms of the user's choice or utilize external generative api services. each generative skill is controlled by one user-specified prompt, utilizes selected generative service and is built using dialog flow framework to provide an opportunity for the developer to control the skill in a script-based manner. to build an assistant, which comes in a form of a custom generative ai assistant distribution, several generative skills can be combined with help of prompt selector picking the most relevant prompts among presented and response selector managing the dialog. prompt selection could be performed in different ways: simple ranking of prompt-context pairs, ranking of pairs of context and prompts goals extracted from prompts using llms, predicting with llms based on prompts descriptions. in upcoming releases, we plan to enhance our skills by incorporating structured, vector-based, and rmt-based @xcite representations of episodic and working memory based on the dialog state and conversation history stored within deeppavlov dream. given the importance of supporting queries over documents and knowledge bases, future versions of deeppavlov dream will support llm-driven interaction with external data sources like corporate databases. we also plan to add support for prompt-chaining to enable reasoning simulation to address more complex problems through planning and to facilitate development of the autonomous ai agents. ## multimodal generative ai assistants users' expectations from chatbots are rapidly increasing, so multimodality, which is operating with images, audio and video, is becoming an important direction in dialog systems' development. while the audio input can be converted to text almost without losing sense (except of intonations and emotions), received images may bring a key information to a dialog. deeppavlov dream utilizes stateless paradigm, which means that components do not store any information about the dialog. the dialog state contains all the information and is forwarded through the full pipeline (partially, according to the given formatters) by the agent component. sending images or video between containers can be timeconsuming, so we offer to use a special database storing images and videos and send file paths in this database through the pipeline. we implemented one of the most obvious ideas, which is to convert images to textual modality using image captioning models. textual input replacement with an original image caption does not work as expected, so we created a scripted approach which extracts topics and entities from the caption, detects the type of the objects, and returns scripted responses for several particular types of objects, such as people, animals and food. we are also working on the skills responding to the user with images to provide a more engaging and human-like experience. in addition to that, we are working on enabling socratic models-like @xcite approach to address user tasks by running conversations between skills understanding different modalities. ## developer experience deeppavlov dream platform enables developers to build their own multi-skill dialog systems. despite of low-code/no-code trends for development tools, our platform requires its users to work with python and docker that will give them enough flexibility and customization opportunities. developer's path starts with repository on github foot_2 . developer needs a pc with ubuntu or other unix-like os capable of running docker. to run heavy nlp components, locally dedicated gpus are necessary. any ide can be utilized, however, vs code is needed to use our df designer to create custom scenario-driven skills in a visual interface. to create a custom distribution, developer can either make it by hands or use a single command from deeppavlov dreamtools specifying a list of selected components. developer can talk to the system by running a chat in a developer's mode in a command line, via telegram or using web interface on "/chat" endpoint of the agent foot_3 . one can utilize some components via proxy. agent and database components are always run locally, while proxied components are light-weight containers. all components not present in the proxy yml-file for the current distribution are run locally, which allows the developer to run and debug any new or existing component locally. here are the main opportunities for customizing a dialog system in deeppavlov dream: one may combine a new dialog system with particular existing components, change the parameters of these components in configuration files, create new components from scratch or change the existing components. to create a custom scenario-driven dff skill and add it to the distribution, the developer can use a single command from deeppavlov dreamtools. to visually design a custom scenario-driven dff skill, the developer can use df designer (refer to the workshop video foot_4 for detailed instructions). prompt-based generative ai assistants can be created by hands using a template distribution. one can add any number of prompt-based skills and customize them by prompt engineering and selecting generative services of interest. multi-skill dialog management is handled automatically. debugging a complex distributed platform is always a challenge. in the deeppavlov dream platform, one can use different techniques to successfully debug their distribution. one can utilize deeppavlov agent's console to debug based on the entire dialog state, docker logging output to debug individual components, or post requests to deeppavlov agent via postman or similar tools. the developer can build a custom user experience around their dream-based multi-skill ai assistant or connect it to some of the existing channels, such as amazon alexa, to make it available for the end users. there is a workshop video foot_5 available with detailed instructions. we also provide the documentation site for deep-pavlov dream foot_6 . the site provides access to comprehensive resources for building intelligent conversational assistants tailored to users' specific needs. the site offers extensive documentation, release notes, and detailed examples to facilitate the development of advanced conversational ai applications. the site also provides opportunities to join the community of developers leveraging deep-pavlov dream to shape the future of conversational ai technology. ## conclusion as complex conversational systems are becoming more and more popular, it is important to make the development process of such systems easier for researchers and developers. deeppavlov dream is an open-source conversational platform designed to let the users develop their own complex dialog systems and access existing nlp components for feature extraction and classification of user utterances. the platform features a variety of skills developed with scenario-driven, retrieval, and generative approaches using modern nlp techniques including prompt-based generation. these skills are organized into ready-to-use distributions of the multi-skill ai assistants. deeppavlov dream allows to customize dialog systems at all levels. the platform was battle-tested during amazon alexa prize 3 and 4 and is now providing core infrastructure to facilitate the development of multi-skill generative ai assistants for industry and academia. we built a deeppavlov dream foot_7 platform's website foot_8 including documentation, tutorials and useful links, chat with the demo distribution. we have published a series of articles about deeppavlov dream on medium foot_9 . we also have a youtube channel foot_10 where we publish video workshops and seminars. to communicate with our team, one can use our forum foot_11 . short demo video is available on youtube foot_12 . deeppavlov dream covers open-domain hybrid conversational systems able to chat on any topic in addition to the commercial scenario-driven template-based chatbots. therefore, we propose to use generative models. generative-based distributions are able to respond to most of the dialog contexts with various replies while consume significant computational resources requiring at least a single gpu to run models like dialogpt. current distributions have a limited number of task-oriented skills, like factoid qa and weather skill.