An Open-Domain QA System for e-Governance (2022)

Abstract: The paper presents an open-domain Question Answering system for Romanian, answering COVID-19 related questions. The QA system pipeline involves automatic question processing, automatic query generation, web searching for the top 10 most relevant documents and answer extraction using a fine-tuned BERT model for Extractive QA, trained on a COVID-19 data set that we have manually created. The paper will present the QA system and its integration with the Romanian language technologies portal RELATE, the COVID-19 data set and different evaluations of the QA performance.

## introduction according to @xcite 1), open-domain question answering (qa) has the ability "to answer a given question without any specified context", by searching for the relevant documents on the web and extracting the relevant answer from one (or more) of the retrieved documents. in contrast, machine reading comprehension "aims to enable machines to read and comprehend specified context passage(s) for answering a given question" which entails that, given a question and one (or more) passage(s) of text that (can) contain the answer, the qa system is able to identify it in the given text piece. the "open-domain" designation of a qa system also pertains to the ability of the system to answer factoid questions (factual questions) from any domain, according to @xcite . the qa system that is presented in this paper is "open-domain" from both points of view: it only takes the input question and automatically searches for the relevant documents on the web but, for the answer selection, it employs a fine-tuned bert model for extractive qa that, using the input question together with the snippet that the web search engine produces for each relevant document, highlights the answer to the input question. although we present an instance of this system for the covid-19 domain, given other targeted data sets, the exact same pipeline can be applied to answer questions from those domains. the qa system was developed in the european project enrich4all foot_0 , a project aiming at a digital single market strategy, which is linked with lowering language barriers for online services and public administration procedures. the architecture of the qa system enables it to answer administrative questions about a certain topic (e.g. covid-19, construction permits, etc.) that citizens may have for public authorities, by automatically searching for relevant documents on the public authority web site. being available 24/7, it has the potential to reduce the administrative burden for public authorities. in what follows, we present related approaches to open-domain qa in section 2, followed by a description of the covid-19 data set in section 3. section 4 details the fine-tuning of different romanian bert models to covid-19 extractive qa, while section 5 describes the architecture and the underpinnings of the open-domain qa system. we end the paper with section 6 devoted to the evaluation of the qa system and section 7 presenting concluding remarks and future work plans. ## related work open-domain qa (odqa) aims at answering questions from large open-domain corpora (e.g., wikipedia). recent success in this field mainly comes from fine-tuning and improving the pretrained lms, like @xcite and bert @xcite . @xcite proposed a multi-passage bert model to globally normalize answer scores across all passages of the same question, enabling the qa model to find more precise answers utilizing more text passages. splitting articles into passages with the length of 100 words improved performance by 4% and trained on the opensquad data set, the model gained 21.4% em and 21.5% f1 over all non-bert models, and 5.8% em (exact match) and 6.5% f1 over bert-based models. @xcite integrated bert with the open-source anserini information retrieval toolkit. they showed that combining a bert-based reader with passage retrieval using the anserini ir toolkit yields towards an improvement in question answering directly from a wikipedia corpus. during training, passages corresponding to the same question are taken as independent training instances. the authors report that fine-tuning pre-trained bert with squad is sufficient to achieve high accuracy in identifying answer spans. @xcite showed that it is sub-optimal to incorporate a standalone ir system in an openqa system, therefore they developed and they develop an openretrieval question answering system (orqa) system that treats the document retrieval from the information source as a latent variable and trains the whole system only from question-answer string pairs based on bert. the system was evaluated on open versions of five qa data sets and outperformed bm25 model by up to 19 points in exact match. @xcite used bert pre-trained model @xcite and a dual-encoder architecture @xcite in order to develop a training scheme that uses a relatively small number of question and passage pairs. the authors demonstrated that by fine-tuning the question and passage encoders on existing question-passage pairs the system outperformed models, such as tf-idf or bm25 and also that applying a reader model to the retrieved passages leads to comparable or better results on multiple qa data sets in the openretrieval setting. furthermore, the study showed that in the context of open-domain question answering, a higher retrieval precision translates to a higher end-to-end qa accuracy. @xcite used contextualized word representations to predict a span as answer. the authors showed the effectiveness of retrieval-augmented language model pre-training (realm) by finetuning on the task of odqa. the system outperformed previous methods by a significant margin (4-16% absolute accuracy),and also provided qualitative benefits such as interpretability and modularity. @xcite introduced binary passage retriever (bpr), a memory-efficient neural retrieval model that integrates a learning-to-hash technique into a dense passage retriever (dpr) @xcite . bpr has two main objectives: to generate efficient candidates based on binary codes and re-ranking based on continuous vectors. when compared with dpr, bpr reduced the memory cost from 65gb to 2gb without a loss of accuracy. ## the covid-19 data set the covid-19 data sets we designed are a small corpus and a question-answer data set. the targeted sources were official websites of romanian institutions involved in managing the covid-19 pandemic, like the ministry of health, bucharest public health directorate, the national information platform on vaccination against covid-19, the ministry of foreign affairs, as well as of the european union. we also harvested the website of a non-profit organization initiative, in partnership with the romanian government through the romanian digitization authority, that developed an ample platform with different sections dedicated to covid-19 official news and recommendations. news websites were avoided due to the volatile character of the continuously changing pandemic situation, but a reliable source of information was the website of a major private medical clinic, that provided detailed medical articles on important subjects of immediate interest for the readers and patients, like immunity, the emergent treating protocols, or the new variants of the virus. both the corpus and the question-answer data set were manually collected and revised. data was checked for grammatical correctness and missing diacritics were introduced. the corpus is structured in 55 utf-8 documents and contains 147,297 words. the question-answer data set comprises 185 entries made up of a label (see the list of labels below), a question and its answer. the questions have been multiplied manually: rephrasing techniques have been used (such as active-passive constructions, proceedings of clib 2022 personal-impersonal ones, constructions with or without (epistemic and/or deontic) modality, synonyms, antonyms, hypernyms, etc.), always making sure the question's meaning is not altered, so that the existing answer could be appropriate for each of the resulted questions. all these diverse ways of expressing the same question are meant to serve as a train base for the bert model so that it can recognize alternative ways of inquiring about a certain topic. each entry in the question-answer data set was associated with one of the labels: covid-spread, covid-symptoms, covid-treatment, covid-vaccination, covid-logistics, covid-passport, covid-testing and covid-others. to use the bert model as a qa system (see section 4), we had to manually mark the relevant answer to the question in the provided answer paragraph. an example of such an entry in the questionanswer data set is given below: three different formulations of the same question, all marked with "q:", synonyms for a word or phrase enumerated within the question, using square brackets and slashes, and finally, the enlarged answer comes last, marked with "a:", in which the more to starting from the example above we can generate 3 • 2 • 4 = 24 different, but semantically equivalent formulations of the question "does the warm weather protect us from the coronavirus?" to which the answer is highlighted in the answer para-graph: "existing data shows that [the infection can be acquired in all climates, including the warm ones]." having question-answer data points annotated in this way, we were able to automatically generate a squad 2.0 data set @xcite ) on which we fine-tuned a romanian bert model for extractive qa, as described in the next section. the covid-19 squad 2.0 data set foot_1 contains 1,388 question-answer data points, after automatically expanding all possible question formulations as described above. since each question entry has multiple formulations, in order to be fair to the extractive learning model, we randomly set aside 10% of a question alternative formulations for the test set (if there were less than 10 alternative formulations for a question, we kept a single formulation for the test set). this selection procedure gave us 180 question-answer data points in the development set and 1,208 question-answer data points in the training set, which represents a 13%-87% split of the data. ## fine-tuning bert for covid-19 to create the qa model, we employed the standard bert fine-tuning procedure described in (kenton and toutanova, 2019) that consists of putting two feed-forward layers on top of the contextualized embeddings to predict the start and the end of an answer. on a more granular level, this operation is equivalent to taking the dot product between either a start vector s or an end vector e and each of the contextualized embedding t i produced by the bert model, and then applying the softmax function over the results: we trained each model for 5 epochs, using a batch size of 8, a learning rate of 3 • 10 -5 , a weight decay of 10 - foot_2 and a maximum span ξ = 30 of tokens. we used the train-test split of the data set that was described in section 3. the results are depicted in ## the open-domain qa pipeline the trainable, open-domain qa system 3 executes the following operations, in sequence, for an input question: ## evaluation to fairly evaluate our qa system, we have developed a new test set, containing 65 covid-19 related questions, that were created to be different from the ones in the data set presented in section 3. thus, each of the authors of this paper independently recorded the most 10 interesting questions, from a personal point of view, mimicking real usage of the system. the intent behind this decision was to evaluate our qa system in real-world scenario where users may ask all sorts of questions, using romanian diacritics or not, about a very rapidly evolving subject such as covid-19. we have incrementally developed our covid-19 data set in a time frame of about 6 months, in which time some understudied or evolving aspects of covid-19 (e.g. the duration of the vaccine-induced immunity, the different vaccines efficiencies or the number of days the infected persons are quarantined) have figure 2: question answers changed, as results of ongoing studies were published that shed new light on these aspects. we have found out that, for most of the questions, bing retrieves more than one document containing relevant information for the answer, even if not spot on. this is because many questions in our newly developed test set are open-ended and opinions about the correct answer vary. there were questions with contradictory answers that were being marked as correct since it is not the duty of (this) qa system to infer the correct answer to a input question, but merely to present likely options as possible answers. one such example is the question "ajutȃ vitamina d la prevenirea covid?" ("does vitamin d help preventing covid infections?") for which we find that either "lipsa vitaminei d cres , te riscul de infect , ie, inclusiv cu sars-cov-2." ("the lack of vitamin d causes a rise in the risk of a sars-cov-2 infection.") or "... dovezile existente nu sust , in eficacitatea vitaminei d pentru tratarea virusului covid-19." ("... existing evidence does not support the efficiency of vitamin d in treating covid-19"). consequently, our 65-question test set contains, for each question, a list of the urls of the documents containing the relevant and recent (if necessary) answer. for each url, we retain the bingextracted text snippet in which we manually highlight one or more likely answers. for 10 questions out of 65, our automatic query generation algorithm did not find any suitable documents. we can thus evaluate the effectiveness of the content word selection query generation algorithm at 1 -10 65 ≈ 85%. ## conclusions we have presented an open-domain qa system that uses a fine-tuned bert model to highlight probable answers to the input question in the bingreturned text snippets. with a mrr bigger than 0.5, we have the guarantee that the user sees relevant content in the first two results returned by the qa system, with more relevant content following, as bing finds useful information in more than one document. furthermore, a character overlap f1 of almost 77% between the correct answer and the bert supplied answer will steer user's attention effectively towards the correct answer. comparing the data sets sizes, we see that our covid-19 data set is two orders of magnitude smaller than the data sets presented in @xcite . the exact match and f1 scores on the squad 1.1 data set are 78.6% and 85.8% respectively, suggesting that we have room to improve in these areas, provided that we can grow our data set significantly. but even with the current performance, the qa system is useful as it is. the open-domain qa pipeline is trainable in the sense that, given a data set similar to the one presented in section 3 but in a different domain, one can fine-tune the chosen bert model to answer questions in the domain of the new data set. relying on a web search engine such as bing, indexing billions of documents on a regular basis, the part of answer retrieving is assured, irrespective of the chosen domain. in a different use case, the qa system can be adapted to work on e.g. public institution web sites, by having the public institution web sites indexed using either a local search engine or local bing/google indexing and using these specialized search results instead of the web-wide search results. coupled with a data set of questions on the specific topic of interest (e.g. "tax payments", "public transportation", "resident parking", etc.), the qa pipeline can work to answer targeted questions from citizens. the next steps in the development of this qa system are: