Evaluating Recognizing Question Entailment Methods for a P ortuguese Community Question-Answering System about Diabetes Mellitus (2021)

Abstract: This study describes the development of a Portuguese Community-Question Answering benchmark in the domain of Diabetes Mellitus using a Recognizing Question Entailment (RQE) approach. Given a premise question, RQE aims to retrieve semantically similar, already answered, archived questions. We build a new Portuguese benchmark corpus with 785 pairs between premise questions and archived answered questions marked with relevance judgments by medical experts. Based on the benchmark corpus, we leveraged and evaluated several RQE approaches ranging from traditional information retrieval methods to novel large pre-trained language models and ensemble techniques using learn-to-rank approaches. Our experimental results show that a supervised transformer-based method trained with multiple languages and for multiple tasks (MUSE) outperforms the alternatives. Our results also show that ensembles of methods (stacking) as well as a traditional (light) information retrieval method (BM25) can produce competitive results. Finally, among the tested strategies, those that exploit only the question (not the answer), provide the best effectiveness-efficiency trade-off. Code is publicly available.

## introduction question answering (qa) aims to automatically retrieve precise, rather than merely relevant, answers to a given question. the field has faced exponential progress along the years with new corpora @xcite as well as computational models which approach the task from different perspectives. one of these is known as recognizing question entailment (rqe). given a premise question (aka query), a rqe approach aims to retrieve semantically similar archived questions which have been already answered @xcite . the task became relevant thanks to popular community qa forums, such as yahoo answers, quora and stack overflow, where an rqe approach is used to automatically search a large body of material looking up for archived question-answer pairs entailing a user posed question. besides popular forums, a domain in which rqe approaches are highly beneficial and have been extensively studied is the medical one. @xcite showed that, when using traditional information retrieval search engines to query medical information, healthcare professionals spend on average 60 minutes to formulate a search strategy, 3 minutes to analyse the relevance of a retrieved document, and 4 hours of total search time. @xcite suggest that healthcare consumers may also benefit from qa systems through which they can ask for desired information in natural language instead of having to perform complex search strategies. in fact, a study reveals that, in 2013, 59% of u.s. adults searched for health information online and 35% used healthcare search engines to figure out what medical condition they or someone else had foot_1 . to advance the state-of-the-art in rqe approaches for medical-specific applications, some benchmarks have been proposed. targeting frequently asked question (faq) by healthcare consumers, @xcite introduced a collection of 8,890 pairs of questions labelled as having or not the same meaning. in the trec 2017 liveqa track, a medical question answering task was proposed addressing the automatic answering of consumer health questions received by the u.s. national library of medicine @xcite . under the shared-task, a total of 738 question-answer pairs were publicly released. @xcite released medquad, a dataset with 47,457 medical question-answer pairs of 37 question types extracted from 12 websites of the the national institutes of health of united states. although the rqe task has been extensively studied, as is the case with other nlp subfields, studies and approaches largely focus on the english language. in this study, we goes in a different direction, facing the challenge of developing an nlp approach for a low resource language. specifically, our study focuses on the portuguese language and investigates the development of a community question answering system in the particular domain of diabetes mellitus using rqe. this allows investigating new challenges in terms of multi-lingual nlp problems. from an application domain (health) perspective, world-widely, the number of people with diabetes increased from 108 million in 1980 to 422 million in 2014 foot_2 . it is expected that the disease will reach 629 million in 2045. these numbers are partially explained by the rapidly increase of the disease in low-and middle-income countries, such as brazil, where around 12 million people have this health condition. aiming to cope with this disease and to have a better quality of life, a significant number of portuguese speakers with this condition engages in dedicated public forums. in order to improve the access to information about this health condition for these speakers, this study aims to leverage rqe approaches to build beteqa, a portuguese community question-answering system to provide prompt and accurate answers to questions about diabetes mellitus posed by social forum users. to meet our goal, similar to abacha and demner-fushman (2016); abacha et al. @xcite and ben abacha and demner-fushman (2019), we first built a portuguese benchmark with 785 pairs between premise questions and archived answered questions annotated as perfect match, relevant and irrelevant, following @xcite . based on our benchmark corpus, we leveraged and evaluated several rqe approaches ranging from traditional information retrieval (ir) methods (bm25 and tf-idf cosine similarity) to novel large pre-trained language models such as bert @xcite . for the latter methods, we evaluated them according to both zero-shot and fine-tuned learning setups. we also used ensemble models (stacking) based on a learn-to-rank model to combine the outputs of the previous methods with other traditional linguistic features. experimental results show that a supervised transformer-based method trained in multiple languages and for multiple tasks (muse) outperforms the alternatives in a zero-shot setting. moreover, results show that the ensemble method (stacking) as well as a traditional (light) ir method (bm25) have the potential to provide competitive results. finally, among the tested strategies, those that exploit only the question (not the answer), provide the best effectiveness-efficiency trade-off. our failure case analysis also reveals that most failures occur for longer sentences containing a smaller number of relevant candidates and harder separability. the remainder of this paper is organized as follows: section 2 describes how the benchmark was built. section 3 explains the rqe models leveraged to rank similar questions about diabetes. section 4 describes the experimental methods while section 5 discusses the experiment results and a failure analysis. section 6 presents related work and section 7 concludes our study. ## data collection to build a portuguese rqe benchmark in the domain of diabetes mellitus, we first manually extracted portuguese questions about this health condition from specialized websites and social media forums. in particular, most questions were extracted from the faq section in the website of the brazilian association about diabetes foot_3 as well as in forums about this health condition, such as diabetes -diab éticos foot_4 , a facebook community about diabetes with around 85 thousand portuguese speaking users. preprocessing and anonymization to keep users' privacy, the extracted questions from forums were manually de-identified by first removing emojis, fixing orthographic mistakes and paraphrasing non-fluent syntactic structures which could point to idiosyncratic wordings by users. then any identifier, such as name, phone or address, was removed from the questions. moreover, quasi-identifiers, such as age and relative mentions, were modified. users' age were modified by randomly choosing a number in the interval of [age-5; age+5], whereas mentions to relatives were randomly changed by the reference to a relative with similar age, such as parent ↔ uncle, parent in law; sibling ↔ cousin, partner; soon ↔ nephew. we collected a total of 1474 questions. answers 4 medical students were recruited to review the question-answer pairs extracted from faq sections of websites as well as to formulate answers to the de-identified questions from public online forums. during the evaluation process, each question was answered by a single medical student. in case of doubts, the respective question would be discussed among the students together with a medical specialist. to organize the process, the students kept a standardized database of answers to the collected questions. each answer in this database was classified according to 10 topics: 1) general information about diabetes; 2) diagnosis; 3) chronic complications; 4) acute complications; 5) treatment; 6) treatment control; 7) comorbidities; 8) signs and symptoms; 9) motivation; and 10) highly frequent, though unrelated to diabetes. answers were elaborated pursuant to article 37 of chapter v of the brazilian code of medical ethics foot_5 , which prohibits treatment prescription without actual patient examination. hence, the answers were constructed with the aim of informing the user about diabetes and related issues, without offering any diagnosis or treatment. in cases where the user requested some type of intervention, answers were prepared in order to guide them to seek a public healthcare unit, both to obtain an accurate diagnosis and to have adequate therapeutic plans designed by healthcare professionals. ## models drawing on our collected portuguese benchmark about diabetes, we evaluated several approaches to rank question-answer pairs to their premise questions. such approaches range from traditional bagof-words information retrieval techniques to novel methods based on continuous vector representations and learn-to-rank ensembles. ## evaluation we evaluated the proposed approaches as a ranking problem. given a premise question from the described benchmark, our goal is to investigate which model can better rank "perfect match" and "relevant" candidate question-answer pairs ahead of "irrelevant" ones. following @xcite , we treated the problem as a binary one, not distinguishing "perfect match" and "relevant" candidate questions. metrics we evaluated the approaches using two popular ranking measures: the mean averaged-precision (map) as the main metric and the mean reciprocal rank (mrr) as the secondary one. since each premise question of our benchmark is attached to 5 candidates, we used this length to compute the metrics (e.g., map@5 and mrr@5). ## related work recognizing question entailment (rqe) has been extensively investigated in the field so that it was included as a shared task in semeval-2016/2017 (task 3 -subtask b) @xcite . under the domain of the qatar living corpus, the task consisted of reranking 10 candidate questionanswer pairs retrieved by google for a premise question about qatar. among the promising participant approaches, we highlight simbow @xcite , the winner of the shared-task in semeval 2017. the approach works by computing the semantic similarity over the vector representations of a premise question and a corresponding candidate question-answer pair using the softcosine metric. another promising participant was kelp @xcite , an approach based on tree kernels and svms which provided top results in the task. after the shared-tasks, @xcite conducted a study with these approaches in order to understand the effects of particular design choices, such as the adopted preprocessing methods and word-similarity metrics. in the medical domain, @xcite proposed an answer recommendation algorithm for medical community question answering. given a user query, the system starts by looking for similar archived questions using a paragraph vector based language model (pvlm) as a similarity metric. this metric measures the distance between a premise question and a candidate one by multiplying the cosine distances among the word embedding of each word of the premise question with the paragraph vector of the archived question. in the same year, abacha and demner-fushman (2016) proposed a supervised machine learning approach which classifies whether or not a candidate question can be inferred from a premise question. the questions were represented based on lexical and semantic features. more recently, ben abacha and demner-fushman (2019) proposed a siamese neural network to predict whether a candidate question is a perfect match, a relevant one or an irrelevant one to a premise question. ## conclusion this study investigates recognizing question entailment approaches to build a community question answering about diabetes mellitus. unlike previous studies in the field, ours focuses on a language other than english. specifically, we focused on the portuguese language. due to the lack of resources for the language in this domain, we built a benchmark corpus, which was used to test several rqe models ranging from traditional information retrieval methods to novel large pre-trained language models and ensemble techniques using learnto-rank techniques. results show the power of multilingual and multi-task large neural networks such a muse. this sentence encoder obtained the best results of our evaluation in a zero-shot learning setup, i.e. this means it was not optimized to the target task. results of the evaluation also show that bm25, a traditional and light information retrieval method, can obtain competitive results in the task. different from what was expected, state-of-art fine-tuned methods such as our bertimbau classifier did not perform better than our muse zeroshot approach. we believe this may be caused by lack of training data, since our benchmark is relatively small. in future work, we plan to overcome this problem by collecting more data and expanding the corpus to other conditions related to diabetes, such as hypertension. like @xcite , we also plan to augment our portuguese training corpus by translating english questions from corpora such as medquad into portuguese.