Benchmark Data and Evaluation Framework for Intent Discovery Around COVID -19 Vaccine Hesitancy (2023)

Abstract: The COVID-19 pandemic has made a huge global impact and cost millions of lives. As COVID-19 vaccines were rolled out, they were quickly met with widespread hesitancy. To address the concerns of hesitant people, we launched VIRA, a public dialogue system aimed at addressing questions and concerns surrounding the COVID-19 vaccines. Here, we release VIRADialogs, a dataset of over 8k dialogues conducted by actual users with VIRA, providing a unique real-world conversational dataset. In light of rapid changes in users’ intents, due to updates in guidelines or in response to new information, we highlight the important task of intent discovery in this use-case. We introduce a novel automatic evaluation framework for intent discovery, leveraging the existing intent classifier of VIRA. We use this framework to report baseline intent discovery results over VIRADialogs, that highlight the difficulty of this task.

## introduction as covid-19 vaccines became available in late 2020, they were met with widespread vaccine hesitancy (goldstein et al., 2015; sallam, 2021), a phenomena recognized as a top global concern by the world health organization (who) in 2019. to address such hesitancy, one needs accurate, reliable, and up to date information, constantly available to the general public. in recent years, task-oriented dialogue systems (dss) have become an integral part of our daily lives, covering domains such as banking, tourism, and government agencies (androutsopoulou et al., 2019). correspondingly, we introduced vira, 1 the vaccine information resource assistant -an informational ds that aims to engage with covid-19 vaccination concerns and questions. vira is able to respond to 181 different concerns, accumulated over the course of the pandemic. vira responses were written and vetted by leading medical and public health experts, relying on up-to-date facts and guidelines. an example of a dialog conducted with vira is presented in we refer to this list of concerns and questions as intents. this is a slightly modified definition of intents, which are usually general tasks or goals which the user tries to accomplish (jurafsky and martin, 2009). in vira the intents are phrased as complete sentences, e.g., "is the vaccine safe?", as opposed to a synthetic class name like "vac-cine_safe", and the goal of the intent classifier is to predict the correct intent, to which vira responds to the user with a pre-defined message. advancing dss depends on the availability of conversational datasets on which models can be trained. in domains where fact-based information is a necessity, e.g., healthcare, curating such datasets is often challenging: users in a crowdsourcing setting may not share their authentic concerns, may not represent public opinion, or may even attempt trolling. in addition, creating responses by highly-trained individuals is a very demanding process (liu et al., 2021). furthermore, even if one has collected data from a real-world ds, there could be limitations for making such data public. the availability of vira enabled us to collect dialogs with real-world users, following word-ofmouth or social media advertising, presumably conveying genuine interest or concerns related to the vaccines. vira was launched in july 2021 and over the course of 10 months it accumulated over 8k conversations. we refer to this collection of conversations as viradialogs and release it as part of this work. @xcite after deploying a ds in a real-world setting, users may introduce new intents, which are not side text intent system hey! ask me your vaccine questions. ## the vira system users communicate with vira using either a web-based user interface (ui) @xcite or a whatsapp application. the general flow is that users enter free text expressing their questions and concerns about the vaccine, vira detects the intent within a pre-defined intent list, and in turn provides a suitable response, reviewed by medical experts. vira supports conversations in english. @xcite below we describe vira's main components. profanity classifier. we use a dictionary foot_4 to identify utterances with suspected offensive language, to which vira presents a generic response. dialog-act classifier. we classify each user input to one of the supported dialog acts. for certain dialog acts, e.g., 'hi', vira presents a generic response. full details can be found in appendix a. intent classifier. intents representing distinct vaccine concerns were carefully curated through various means: using a twitter analysis, reviewing audience questions in zoom-based public forums hosted by authors' affiliated academic centers, and synthesizing web pages with frequently asked questions. the intents were defined also by taking into consideration the scientific knowledge towards the vaccine at that point. over time, new concerns were identified by monitoring incoming queries to vira and eventually the list comprised of 181 intents, presented in appendix g. the requirement from vira was to provide specific answers to specific concerns, and general answers to general concerns -hence, "i am afraid the vaccine will change my dna" and "i distrust this vaccine" required different answers, and thus were represented as separate intents, although the latter can be entailed from the former. the intent classifier was trained on data collected from crowd annotators using the appen platform. @xcite annotators were presented with an intent and asked to express it in three different ways, as if conversing with a knowledgeable friend (see section 6.1 for more details). the classifier's top-ranked intent is selected for providing a response from the response database. if no intent passed a pre-defined threshold, a corresponding response is given. response database. this database contains vira's responses to intents. each entry specifies multiple responses to a specific intent, to increase output diversity. the responses contain varying information and tone from which vira selects one randomly. the database was created and is maintained by experts in the field based on up-todate facts and guidelines. all responses sought to minimize technical language and maintain brevity through a 280-character limit. feedback mechanism. vira incorporates a feedback mechanism that enables users to correct the course of conversation. this feedback allows vira's personnel to improve the system over time ## the viradialogs dataset viradialogs contains the interactions conducted with vira by actual users from july 2021 to may 2022. the full dialogues, as well as user feedback, predicted intents, dialog acts, and offensive language predictions are released to the research community. the data has been anonymized by masking locations, names, e-mails, phone numbers, and birth-dates, along with suspected offensive terms, using a range of regular expressions, the profanity classifier, and the spacy named entity recognizer. @xcite in addition, we have excluded dialogues between 29-30, july 2021, in which vira was confronted with multiple chats containing offensive language, presumably from individuals who attempted to break the system. stats of viradialogs are presented in ## retrospective intent discovery evaluation an important contribution of this work is to show how to leverage an existing ds intent classifierlike the one described in section 3, referred to as an oracle -to automatically evaluate intent discovery methods over a collection of dialogs. an overview of the proposed approach is depicted in figure 1 . the underlying components are described below, using the following terminology: oracle intents: the intents supported by the oracle. silver labels: subset of oracle intents, induced over a given data. predicted intents: intents predicted and phrased by an intent discovery method. predicted oracle in-tents: subset of predicted intents mapped by the oracle to oracle intents. 8 https://spacy.io/ ## experimental setup in this section we present a concrete implementation of the framework described in section 5 using vira and viradialogs to automatically evaluate various unsupervised intent discovery methods. ## ethics statement this paper describes work around vira, a realworld ds addressing covid-19 vaccine hesitancy. in an attempt to alleviate concerns that users would take action based on information given to them by vira which might harm them, the terms of use of the ds state that "this information ... is not intended as a substitute for medical advice". we were guided with the principle of providing accurate information, thus when building vira we incorporated a direct mapping between intents and responses. future endeavours based on this dataset, e.g., for building a generative bot for addressing vaccine hesitancy, should be aware of the ramifications of showing to users such content. in addition, the terms of use stated that queries are stored and may be used for research purposes. the chats collected might have originally contained offensive language, often as a result of the sensitivity of the domain to some users. we made a dedicated effort to flag these cases and mask problematic terms. however, we did so with automatic measures, so the dataset might still contain such language. finally, although the data was anonymized by masking various expressions, it is still possible that some sensitive medical concerns remain.