Athena 2.0: Contextualized Dialogue Management for an A lexa P rize S ocial B ot (2021)

Abstract: Athena 2.0 is an Alexa Prize SocialBot that has been a finalist in the last two Alexa Prize Grand Challenges. One reason for Athena’s success is its novel dialogue management strategy, which allows it to dynamically construct dialogues and responses from component modules, leading to novel conversations with every interaction. Here we describe Athena’s system design and performance in the Alexa Prize during the 20/21 competition. A live demo of Athena as well as video recordings will provoke discussion on the state of the art in conversational AI.

## introduction there has been tremendous progress over the last 10 years on conversational ai, and a number of practical systems have been deployed. the alexa prize competition seeks to stimulate research and development on conversational ai for open-domain topic-oriented dialogue @xcite @xcite @xcite . however, the longstanding tension between hand-scripting the dialogue interaction, and producing systems that scale to new domains and types of interaction still remains @xcite neural end-to-end spoken dialogue systems are not yet at a point where they perform well in interactions with real users @xcite @xcite . athena's dialogue management architecture aims to be scalable and dynamic, by supporting many different interactions for every topic, and by constructing system utterances by concatenating multiple dialogue acts that achieve different purposes @xcite . a key aspect of athena is the existence of multiple response generators (rgs) for each topic, which can be flexibly interleaved during a particular interaction, as illustrated in figure 1 . 1 this approach contrasts with the commonly used approach of handcrafting conversation flowgraphs for each topic, a static directed graph where the nodes are the system utterances, and outgoing edges are represent possible user replies. this approach has not changed for over 20 years @xcite @xcite , and its strengths and limitations are well known. flow-graphs are hand constructed and thus do not scale well. however, each system response can assume a fixed prior context, which allows it to support fluent and coherent dialogues with sufficient handcrafting. in contrast, the ability of athena's dialogue manager (dm) to interleave rgs allows athena to dynamically construct conversations that never follow the same path. however, this more flexible approach requires rgs to pay the overhead cost of continuously adapting to the current context, as described in section 3. by eschewing a graph-based representation of dialogue state, athena's dm is flexible enough to use rg responses in contexts that were not planned out prior to the conversation starting, and that do not need to follow rigid guidelines. we believe this modular dialogue management approach promises to scale to deeper and richer conversations, while at the same time allow new conversational topics to be easily added to and integrated into the system. ## athena architecture and overview figure 2 details athena's architecture. athena is built using the cobot framework provided by amazon @xcite . it runs as an on-demand application that is initiated by an "alexa, let's chat" user request to any alexa-enabled device, such as an amazon echo or the alexa app installed on a phone. during the alexa prize, athena participates in about 9k conversations a week. the cobot framework provides support for automatically scaling to large volumes of user traffic. the inputs to athena are the asr hypotheses for a user's turn from amazon, and a conversation id that is used to retrieve the conversation history and state information from a back-end database. the asr hypothesis is fed into a natural language understanding (nlu) pipeline that produces a set of nlu features for the user utterance and conversation context. the nlu consists of cobot's module for topic classification, and athena modules for utterance segmentation, dialogue act tagging, named entity recognition and linking, and coreference resolution @xcite . the right-hand side of figure 2 indicates how athena's rgs use knowledge bases and fun facts databases organized by topic and named entity. athena uses the wikidata knowledge graph to aid in named entity resolution and for knowledge-graph based rgs. these are essential for creating an intelligent and versatile conversational agent @xcite . based on the nlu features and conversation context, the dialogue manager (dm) calls specific response generators (rgs) to populate a response pool. the dm then applies a trained neural response ranker to select from the response pool generated by the rgs. finally, athena's responses are spoken by amazon's text-to-speech service. ## dialogue management a dialogue manager (dm) for open-domain conversation faces a particularly challenging task due to the universe of possible valid responses at each point of a conversation. while goal-oriented dialogues have a clear task completion objective which the dm can optimize when making decisions @xcite @xcite , the dm for open-domain dialogues does not have an obvious way to measure the appropriateness of possible candidate responses. athena's dm architecture can be decomposed into a number of sub-components, corresponding to phases of dialogue management, oriented as a pipeline. the dm sub-modules in figure 3 are described in more detail in @xcite . the topic manager in figure 3 is responsible for classifying user utterances into topics, and the implementation of the dm's topic hierarchy. the topic hierarchy is a partially ordered list of topics in order of predicted "goodness" learned from past conversations, using a scoring function that combines user ratings and the number of turns per topic per conversation, as described in section 5. the topic hierarchy is a parameter for system-initiative topic initiations as well as suggesting topics for users to initiate. this makes it extremely easy to change which topics are promoted at any time, e.g., for collecting more data on a particular topic. it can also be personalized for each user. for example, if when asked about weekend activities, the user describes playing in a baseball league, we can prioritize talking about sports. this information persists across conversations. if the user is also an avid painter, but our system did not get a chance to discuss painting in the previous conversation, we will prioritize it when the user returns. the interface between the dm and the rgs in figure 3 , is a contract-based approach. the dm passes a set of response conditions to the rgs, which the rgs must meet for their response to be considered. this approach allows athena to have many rg types (see section 4). the response ranker is based on a bert-based ranker fine-tuned on hand-annotated alexa prize conversation data @xcite . the current tuning set size is ~10k utterances. annotation involves ranking candidate responses within a context of five turns. we have repeatedly annotated additional data and retrained our response ranker, which is useful when, for example, new rgs are added to athena. ## response generation athena uses four types of rgs: flow-rgs, knowledge-graph rgs, entity-based indexing rgs, and neural nlg rgs. ## evaluation and analysis the two criteria that are specified in the alexa prize grand challenge that systems aim to optimize are length of conversation and user ratings. the grand prize will go to a system that achieves conversations of at least 20 minutes with average ratings of 4.0 on a scale of 1 to 5. 3 over the 4 years our team has been in the competition, we have found that interactions with users are vulnerable to noise due to the competition setup @xcite . users often get into the alexa prize skill by accident leading to many conversations of only 1 or 2 turns @xcite . surprisingly, even for single turn conversations, some users still provide ratings. to improve our analysis of system performance, we remove these very short conversations from the data. obviously, user's interactions with different rgs and topics affect their conversations and therefore their ratings. while only about 20% of users actually provide ratings, over the course of this year, we collected about 38k conversations with ratings. the distribution of ratings by topic presence in conversations from january to june are in figure 6 . the purple and red bars indicate proportions of the topic that occur in conversations with ratings of 4 and 5 respectively. this suggests that the highest performing topics include animals, comic books, harry potter, hobbies, and video games, and that only a few topics are actually performing poorly, such as dinosaurs, news and sports. however, presence in a conversation is a rather imprecise indicator of topic quality. in order to better understand the contribution of each topic to athena's overall ratings, we developed a novel scoring function that aims to optimize topic selection over the prizes' user ratings and conversation duration criteria. thus, our scoring function gives credit based on the number of utterances in a conversation that are contributed by each topic in the conversation. the number of utterances is multiplied by the conversation rating and summed for each topic over all rated conversations. this sum is then normalized to produce its z-score. the plot shown in figure 7 indicates for each topic, how many standard deviations its performance is above or below the athena's mean performance for two weeks in june, 2021. this plot indicates that the topics that contribute most to long dialogues that are more highly rated include movies, animals, video games, music and hobbies, all of which are 1 or more standard deviations above the mean in performance. @xcite discusses the impact of topic selection and dialogue behaviors on user ratings and conversation length. we also investigated the impact of personalization on the dialogue manager with an a/b study run over 22 days. the a system has the fully personalized dialogue manager, while b utilizes our previously existing heuristic topic selection strat-egy. we selected conversations of 6 turns or more to allow personalization to have an effect and also filter out conversations that never perform topic promotion; thus, we only inspect conversations that differ due to the topic selection strategy. after filtering, personalization occurred in roughly 25% of conversations. our results, shown in a: but most people seem to find dinosaurs really interesting. there are dinosaur movies and whole floors of natural history museums dedicated to dinosaurs. personally, i find it fascinating that modern birds are descended from a particular family of dinosaurs. u: hella cool a: definitely. i was surprised to hear how much work went into creating the movie land before time. over 600 background paintings were made for the film. the artists had to create a believable environment in which there was almost no foliage. most paintings depicted beautiful but barren wastelands. u: surprising a: right? my favorite dinosaur is the tyrannosaurus rex. it's huge and ferocious, but the little arms make it somewhat adorable. what about you? what's your favorite dinosaur? u: i'm not sure i have to say a t. rex a: okay. there are lots of movies about dinosaurs. one of the most famous dinosaur movies is jurassic park, old but classic. are there any dinosaur movies that you love? u: i have to say jurassic park because that's the only one i've seen a: awesome! that one is well known! i seem to remember people saying the dinosaurs were very vivid u: i never knew that a: yeah, i love to collect these fun facts! but anyways, i'm interested in your opinion. would you rather ride on the back of a pterodactyl or brontosaurus as your primary means of transportation? u: i have to say brontosaurus and i hate heights a: choosing a brontosaurus you say, great pick! i would rather ride a pterodactyl so i could experience flight as well as move around more efficiently. riding a brontosaurus would be thrilling because of their size but less practical than a pterodactyl. so, okay. do you know why dinosaurs are extinct? u: yeah cause i'm eat your head it right a: ah nice, you must be super smart. let me tell you what i heard.the mass extinction of dinosaurs is a scientific mystery, but most hypotheses include an extraterrestrial impact or a massive bout of volcanism. what do you think happened? u: i'd say aliens a: i'd like to hear your ideas on this. let's say you are trapped on jurassic world alone with the dinosaurs, what would you do to survive? u: i'm not sure i just say find sheltering something and start there a: it's a hard question! i would try to become closer to the younger dinosaurs so that they would grow up with me and feel more comfortable with me. i would also try to help them by finding food and shelter for them. at any rate, dinosaurs had a lot of different ways to attack other dinosaurs and to defend themselves. to help fight meat eaters such as the allosaurus or spinosaurus, many plant eaters had natural weapons at their disposal.