Theano: A G reek-speaking conversational agent for COVID -19 (2021)

Abstract: Conversational Agents (CAs) can be a proxy for disseminating information and providing support to the public, especially in times of crisis. CAs can scale to reach larger numbers of end-users than human operators, while they can offer information interactively and engagingly. In this work, we present Theano, a Greek-speaking virtual assistant for COVID-19. Theano presents users with COVID-19 statistics and facts and informs users about the best health practices as well as the latest COVID-19 related guidelines. Additionally, Theano provides support to end-users by helping them self-assess their symptoms and redirecting them to first-line health workers. The relevant, localized information that Theano provides, makes it a valuable tool for combating COVID-19 in Greece. Theano has already conversed with different users in more than 170 different conversations through a web interface as a chatbot and over the phone as a voice bot.

## introduction the current covid-19 pandemic has presented a global challenge for citizens, health and government structures, and the global economy. one key characteristic of the current crisis is, that it cannot be combated with centralized handling alone, but requires a significant degree of cooperation from the public. specifically, citizens adapt their lifestyle by adhering to measures limiting their daily contacts through physical distancing guidelines, stay-at-home orders, and curfews. additionally, citizens are urged to avoid unnecessary visits to the hospitals. this transformation in everyday life has been a cause of stress for all areas of society. the best way to help people handle this crisis responsibly is to disseminate accurate information about the current state of affairs so that citizens are well-informed of expert recommendations and the status of the pandemic. intelligent conversational agents (cas) can be a powerful tool for information dissemination and user support. specifically, cas have been extensively used for health applications, e.g., to help users quit smoking 1 , for mental health support @xcite @xcite , and hiv counselling @xcite . cas have also been used for other social applications, e.g., fact checking @xcite and encouraging altruistic behavior @xcite . in the context of the covid-19 crisis, cas have been proposed for symptom self-checking, i.e., clara 2 , information dissemination, i.e., healthybuddy+ 3 , and combating misinformation, i.e., jennifer 4 @xcite . one issue that arises is that most cas are created for english-speaking demographics, but in many cases the policies and information about the pandemic are region-dependent. developing applications for under-resourced languages is challenging, due to the lack of data and the limited user base @xcite . nevertheless, we need to overcome language barriers and to provide contextualized information with respect to local policies, circumstances and statistics. theano is debunking myths and conspiracy theories that are prevalent in greece, such as the negative effects that masks have in children or the involvement of the greek church in covid-19 policies. we have not added information about fake news that have spread abroad and do not interest the greek public (e.g., drinking disinfectant to stop covid-19). therefore, it is essential to develop localized, native cas. theano aims to provide a first line of support for end users during the pandemic. our design choices are based on two key goals: a) keeping users wellinformed about the current status of the pandemic in order to enable them to make good health choices and b) reducing panic by helping users self-assess their symptoms and directing them to first-line health-care providers. this objective has also been suggested by @xcite and @xcite . on the one hand, theano provides general information about covid-19. there is access to the number of covid-19 new cases and deaths in greece, other countries, and worldwide. moreover, she knows the availability of intensive care units, as well as the number of people that have been vaccinated on a specific date and in total in greece. we have also included myth busters concerning the usage of masks and other conspiracy theories, e.g., that masks delay the development of children's lungs. there is a wide range of frequently asked questions about covid-19 supported, from how it started, ways of transmission and protection to how to wash hands or whether one should be wearing gloves in public. 6 . on the other hand, theano can list the common covid-19 symptoms and in case the user experiences symptoms, through a mixed initiative conversation, she starts a diagnostic survey. in all stages of the dialogue, she clearly states that she is not a doctor and the user should speak with the local authorities if they continue having symptoms. she can also find pharmacies that are open in the area that the user requests. in addition, theano can advise users who have been in contact with a covid-19 patient, i.e., she suggests that they get tested and informs them where they can obtain free tests provided by local authorities. to better facilitate the fulfilment of the two goals mentioned earlier, namely, being informative, and also keeping users calm, we have created a relevant persona for theano. the importance of analytical design of the persona is analyzed by @xcite . theano has the persona of a support agent that is honest, polite, and direct, but at the same time friendly. the core design principles we followed are: engagement. we want theano to be part of the daily life of users, so she has to be interesting and not only respond to questions. @xcite highlight the importance of engagement when dealing with precarious topics. theano tries to keep the users interested in the conversation at all dialogue stages, by suggesting new and relevant conversation topics. theano is also ready to chitchat, which helps keeping certain users better engaged. moreover, chitchatting includes general questions about theano, e.g. age, favourite color and music, holidays, hobbies, weather, languages, so that the users can get to know the persona in-depth and have an alternative from the covid-related topics. as shown in fig. 3 , and will be discussed in the following, chitchat has actually been found to be one of theano's most popular features. consistency. we want users to get the information they ask for, fast. theano has short responses and doesn't contradict herself by giving conflicting information. we want the user to trust theano. short responses are also preferable for allowing spoken communication with theano (on the phone). longer responses on the phone tend to quickly tire users. this objective has been proposed by @xcite . clarity. the topic that theano is handling is serious, that is the reason why we try to present pieces of information in a clear and concise manner, that is easily understood by everyone. by being explicit in every stage of the dialogue, we also ensure that the user acknowledges the capabilities of the ca and does not expect anything more than what is offered about a sensitive medical topic like covid-19 @xcite . empathy. theano is handling questions that are serious and have an impact on people's lives, in a period of profound crisis. she tries to mitigate user fear, comprehends the worries that the users express and offers emotional support with her responses, which is very valuable during the pandemic, as supported by @xcite , de gennaro et al. (2020) and rashkin et al. (2019). for example: user : i am positive. what can i do? theano : since you know you are positive, it is important to stay calm and not panic. you should stay in contact with your doctor and the local authorities. in case your symptoms worsen you should go to the hospital. i think that it's all going to be okay. try to stay calm. intents are organized in a hierarchy, where highlevel intents are recognized first (e.g., faq), and then fine-grained intents are extracted. this decision allows easily scaling up the dialogue system to a large number of intents with limited training data. therefore, intent recognition is performed using a hierarchy of dual intent entity transformer (diet) @xcite classifiers. diet is a transformer based architecture that operates on bag of words features. the sparse features are embedded using a feedforward layer and then encoded using a 2-layer transformer encoder with relative positional embeddings in the attention layers @xcite . the first diet classifier classifies coarse-grained intent classes, and then the input is routed to the second level of classifiers (i.e. response selectors). the output is the fine-grained recognized intent and the respective classifier confidence. an intent is considered to be recognized if the confidence level exceeds a specified threshold, otherwise a fallback strategy is adopted (e.g. "i did not understand your question. can you rephrase?"). for dialogue management, we use three policies with different levels of complexity and priority. first, we use a rule-based policy that explicitly maps a small, selected set of important intents to the desired actions or responses. then, we fall back to a memoization policy that retrieves identical user inputs from past conversations. if the rule-based policy and the memoization policy do not produce a response, we use ted policy @xcite , a transformer-based classifier for dialogue act classification. ted receives the user inputs, recognized intents, and extracted entities across a fixed history window, and outputs the appropriate response with respect to the user input. it is important that detailed statistics and logs of past conversations are kept, in order to continu-figure 2 : overview of the synthetic data generator ously improve theano's performance. to this end, we track all conversations, and anonymized versions of the user inputs, recognized intents, and extracted entities are saved into a database. thus, we can evaluate past system performance and improve the dialogue stories using expert knowledge from linguists. ## system evaluation we have constructed a training set that consists of 3719 real and synthetic conversations, which is constantly evolving based on real-world usage. our dataset includes 37 core intents at the first level of the intent hierarchy. four of these intents, i.e. chitchat, general faq, mask faq and outof-scope, extend to a second level in the hierarchy with 13, 31, 13 and 10 sub-intents respectively. fig. 3 shows the ten most frequent questions asked by users during a release in april 2021. there is a preference in the covid-19 statistics and the pharmacy finder features, with the out-ofscope response selector as a close second. the latter includes topics that theano does not cover, for example, lockdown updates or hospital operating hours; instead of resulting in a fallback action, in this case, theano redirects the user to the appropriate website. an interesting fact is the tendency of users to chit-chatting. we train diet classifier for the 37 core intents for 10 epochs with batch size 32. we use the default values provided by rasa for all other hyperparameters. similarly we train separate classifiers for the sub-intent categories of chitchat, mask faq, general faq and out of scope for 25, 60, 50 and 50 epochs respectively. this yields an overall intent recognition accuracy of 92.2% ± 0.002 for the core 37 intents and 92.0%±0.002 for the 67 sub-intents during 5-fold cross-validation. the respective f1 scores are 92.1% ± 0.008 and 91.8% ± 0.016. however, these proportions fall down to 86% when the system interacts with real users. during inference, if classifier confidence is small (i.e. less than 0.5), or the classifier is unsure (i.e. confidence for top two intents close -less than 0.1 difference) we choose the intent fallback action. for the dialogue manager, we configure memoization policy to memorize a window of 5 dialogue turns. ted classifier is configured with a history window of 10 turns and we set the output dimension to 64. we use default values for all other hyperparameters. ted is trained for 8 epochs with batch size 16, yielding 95.3% test accuracy. during inference we choose the fallback response if the output confidence of ted is less than 0.4. additionally to the test accuracy, we measure intent classification accuracy when the system is used by real users in the wild. fig. 4 shows the confusion matrices over two user evaluation periods. the first user evaluation was performed in december 2020 in tandem with the first release of theano, when we received 101 conversations. the second evaluation was completed in april 2021 with the second release of theano, when we received 73 conversations. all the incoming data have been annotated by linguists. one thing to notice is that theano was updated with more intents during the second release. included intents are selected based on user feedback and the current developments. while more intents are introduced, the overall nlu performance is improved (79.5% accuracy in december 2020 versus 86.7% recognition accuracy in april). finally we observe in both evaluation periods, most misclassified intents are successfully captured by the fallback strategy. ## conclusions and future work we present theano, a ca for covid-19 information dissemination and symptom self-checking in greek. we believe, theano is a valuable contribution to the list of digital tools for battling covid-19, as it provides a feature-rich and scalable support system tailored to the needs of our user-base. theano needs to be constantly updated with new intents and conversation topics without sacrificing nlu performance, in order to keep up with the latest developments. we show that this is possible; in the second iteration, we double the number of supported intents and even improve the intent recognition performance in the real-world user evaluation. we also propose a simple smart suggestion feature, in order to improve user engagement and interactively show theano's capabilities to users. during our a/b testing, we receive encouraging -though not conclusive-results, that this can improve user engagement, as the use of this feature leads to higher number of conversation turns. in the future, we want to keep improving theano's abilities by continuously adding intents based on our users needs. we also want to introduce a reinforcement learning based smart suggestion module, for open domain smart suggestions. finally, we want to develop a large scale deployment of theano and reach a wider user base.