Domain- and Task-Adaptation for V accin C hat NL , a D utch COVID -19 FAQ Answering Corpus and Classification Model (2022)

Abstract: FAQs are important resources to find information. However, especially if a FAQ concerns many question-answer pairs, it can be a difficult and time-consuming job to find the answer you are looking for. A FAQ chatbot can ease this process by automatically retrieving the relevant answer to a user’s question. We present VaccinChatNL, a Dutch FAQ corpus on the topic of COVID-19 vaccination. Starting with 50 question-answer pairs we built VaccinChat, a FAQ chatbot, which we used to gather more user questions that were also annotated with the appropriate or new answer classes. This iterative process of gathering user questions, annotating them, and retraining the model with the increased data set led to a corpus that now contains 12,883 user questions divided over 181 answers. We provide the first publicly available Dutch FAQ answering data set of this size with large groups of semantically equivalent human-paraphrased questions. Furthermore, our study shows that before fine-tuning a classifier, continued pre-training of Dutch language models with task- and/or domain-specific data improves classification results. In addition, we show that large groups of semantically similar questions are important for obtaining well-performing intent classification models.

## introduction in quickly changing contexts, like the covid-19 pandemic, getting access to relevant and correct information in a fast and easy way is of crucial importance. although websites with frequently-askedquestions (faq) sections provide such information, finding the representative question that matches a user's request can be hard and time-consuming, especially when the list of question-answer (qa) pairs is long. a faq chatbot does this matching for the user, speeding up the process of finding an answer to the posed question: users speak or type their questions and the system retrieves the best matching answer available. intent: faq_ask_why waarom zou ik mij laten vaccineren? why would i get vaccinated? wat zijn de voordelen? what are the advantages? waarom moet je je laten inenten? why do you need to get vaccinated? ik snap niet waarom ik me moet laten vaccineren. i don't understand why i must get vaccinated. waarom is vaccineren belangrijk? why is vaccination important? hoezo moet ik mij laten inenten tegen covid? why should i get vaccinated against covid? waarom dringen ze zo aan op het vaccin? why are they so insistent on the vaccine? intent: faq_ask_certificate krijgen we een attest na vaccinatie? do we get a certificate after vaccination? moet ik later kunnen bevestigen dat ik een vaccinatie gehad heb? do i have to be able to confirm later that i have had a vaccination? kunnen instanties me vragen te bewijzen of ik gevaccineerd ben? can authorities ask me to prove that i have been vaccinated? standard faqs consist of a list of qa pairs, where each answer is coupled with one instance of a general question. however, to train an automatic classifier of answers, more than just one example of questions is needed. different paraphrases or versions of the same question will improve the classification performance. in this work we describe the data collection process for vaccinchat foot_0 , a dutch faq chatbot on the covid-19 vaccines. the result is vaccinchatnl, the first dutch faq answering data set, containing 12,883 questions that refer to 181 different answers. with dutch we refer to the dutch used in the flemish region of belgium as opposed to the dutch used in the netherlands. commercial organizations that use or develop faq chatbots, usually keep their faq answering data sets private. however, vaccinchat was built for research purposes and the released corpus was checked and anonymized with respect to privacysensitive information. the number of faq answers (181) is a lot higher than on the average faq page on the web holding only six answers @xcite . in addition, the corpus does not just hold qa pairs, but groups of questions per answer, providing a unique set of many-to-one mappings of questions to answers. unlike faqs on the web with 'clean', grammatically correct questions, the questions in our corpus come from actual users and may contain typing errors and/or other omis-sions/mistakes. questions referring to the same answer are not always paraphrases (e.g., "should we get a vaccine on a yearly basis?" and "how long will the vaccine protect me?": both refer to the same answer), but in a lot of cases they are (see the examples in our faq chatbot uses an intent classifier, meaning that user questions are classified into an answer class (intent), after which the answer of the classified intent is returned to the user. to train such a classifier we need the user questions and their intent labels. since we also have the text of the answers, the vaccinchatnl corpus could be used to train a model that considers the similarity between (the representations of) a question and its answer. convert @xcite is an example of such a model for english, and a dutch version was developed by de bruyn et al. (2021a). on top of the above-mentioned qualities of the corpus, the topic of the corpus -covid-19 vaccines -makes vaccinchatnl relevant. although the data provides a blueprint of questions and answers for the flemish situation in the first half of 2021, the topic is relevant and similar for the whole world. translated data could be used for training models in other languages. since dutch (faq) data is scarce, the corpus is an excellent addition to multilingual faq approaches and data sets. despite the multipurpose nature of the corpus, we focus on using the data for training an intent classifier as one of the possible applications for the covid-19 faq domain. rather than finding the best possible model for this purpose, we focus on inspecting the effect of pre-training with task-and/or domain-specific data. in this work we describe those experiments as well as the data collection process. the vaccinchatnl corpus is publicly available foot_1 on the huggingface dataset hub @xcite . ## related work this section contains related work concerning domain-and task-adaptation, and considering the overlap of vaccinchatnl with faq and paraphrase corpora, we review the existing literature on these types of data resources. ## dutch vaccinchatnl corpus in this section, we present our new dutch vaccin-chatnl corpus, a faq answering data set with many-to-one mappings of user questions to faq answers. ## experiments in this section we describe a number of different classification models, built with vaccinchatnl data. ## discussion when taking a more in-depth look into the type of errors made by the bertje model and its domainand task-adapted versions, a main observation is that all models make errors for user questions with infrequent words, like place names (e.g., "kessel-lo", "rotselaar") or typos (see with respect to the fallback and chitchat intents, the results show a better performance for the con-tact models than for the bertje models (see we also see a decrease in faq errors with domain-specific pre-training, but only for the case where no task-specific pre-training was done (con-tact vs. bertje). for contact+, the model that was pre-trained on both domain-and taskspecific data, we only see faq improvements compared to contact, but not compared to bertje+. unlike the faq intents, fallback and chitchat intents are better recognized with contact+ than with bertje+, but there is no difference with con-tact. in general, contact+ shows the best performance. for task-specific pre-training a clear improvement was observed for user questions containing domain terminology, i.e., words like ku leuven, facebook, sars-cov-2, kovid 19, jnj, qvax, etc. this was to be expected, since these words were actually in the task-specific data. in summary, we can conclude that task-specific pre-training improves faq intent classification, whereas domain-adaptation mainly benefits the fallback and chitchat intent classification, i.e., the outof-domain user questions. ## conclusions and future work we have presented vaccinchatnl, the first dutch faq answering corpus with over 12k of user queries about covid-19 vaccines, and on average 71 example questions per answer. as the corpus holds lots of paraphrased versions of questions, it can serve as a paraphrase corpus as well. existing dutch paraphrase corpora, or dutch parts of multilingual corpora are sometimes bigger, but do not contain sentential paraphrases (less long phrases or single words), or have less paraphrases per semantically similar group (typically paraphrase pairs). the many-to-one mapping characteristic of this corpus is unique in faq and paraphrasing corpora. as an example application of vaccinchatnl, we described training an intent classifier and we have shown that for a classifier with many classes (i.e., different answers), performance improves with more examples of user questions per class. this work also showed the importance of using domain-or task-adapted pre-trained language models for the fine-tuning task of sentence classification. more specifically, domain-adaptation (covid-related tweets) led to improvements for the non-faq questions, whereas task-adaptation improved performance for faq questions with domain-specific terminology.