V i M ed AQA : A V ietnamese Medical Abstractive Question-Answering Dataset and Findings of Large Language Model (2024)

Abstract: Question answering involves creating answers to questions. With the growth of large language models, the ability of question-answering systems has dramatically improved. However, there is a lack of Vietnamese abstractive question-answering datasets, especially in the medical domain. Therefore, this research aims to mitigate this gap by introducing ViMedAQA. This **Vi**etnamese **Med**ical **A**bstractive **Q**uestion-**A**nswering dataset covers four topics in the Vietnamese medical domain, including body parts, disease, drugs and medicine. Additionally, the empirical results on the proposed dataset examine the capability of the large language models in the Vietnamese medical domain, including reasoning, memorizing and awareness of essential information.

## introduction question-answering (qa) is a natural language processing (nlp) task that aims to generate an appropriate response to a given question. qa systems are categorized based on their answer format. while extractive qa systems return the sub-strings from the provided context as the answer, abstractive qa systems identify keywords within the context and then rewrite this information to answer the question. several qa datasets are squad @xcite and hotpotqa @xcite for extractive qa, and aquamuse @xcite and ms marco @xcite are notable abstractive qa datasets. in the field of vietnamese nlp, various extractive qa datasets exist, including @xcite and vimqa @xcite , both of which serve for general knowledge (opendomain qa). within the specific context of the vietnamese medical domain, datasets such as uit-vinewsqa ( @xcite and @xcite are available for extractive qa. however, there is a shortage of a vietnamese abstractive question-answering corpus, especially in the medical domain. to address the identified problem, we have developed and introduced vimedaqa, a vietnamese medical abstractive qa dataset. the corpus undergoes question-answer generation and human annotation stages to ensure quality while minimizing construction time. this proposed dataset is also leveraged to investigate the reasoning, denoising, and memorizing capabilities of large language models (llms) within the vietnamese medical and healthcare domain. the contributions of this research work are listed as follows: ## related work extractive and abstractive qa: extractive qa systems answer the question by extracting parts of the context @xcite . common extractive qa datasets include squad @xcite , natural questions by @xcite , triviaqa by @xcite and searchqa by @xcite . conversely, the abstractive qa task generates responses using the model's knowledge. when provided with context, this task becomes open-book abstractive qa, and in the absence of context, it is closed-book qa @xcite . common datasets for abstractive qa are eli5 by @xcite , aquamuse by @xcite , ms marco by @xcite , polqa by @xcite and natural questions @xcite . open-domain and close-domain qa: opendomain qa systems assist with general knowledge. some common open-domain qa datasets include triviaqa @xcite , searchqa @xcite , and ms marco @xcite . conversely, close-domain qa systems answer questions in specific domains such as healthcare, law, and finance. close-domain biomedical and healthcare qa datasets include medquad (ben abacha and demner-fushman, 2019), healthqa @xcite , medmcqa @xcite and biqa @xcite . vietnamese qa datasets: multiple qa datasets have been widely published in vietnamese. uit-viquad by @xcite , which follows the squad format, is constructed from wikipedia text. vimqa @xcite ) is a multihop extractive qa dataset based on wikipedia. uit-vinewsqa, introduced by @xcite , is built on top of vietnamese healthcare news articles. uit-vicoqa, developed by @xcite , is a medical extractive qa dataset for machine reading comprehension evaluation. ## methodology this study assesses the capabilities of generative language models for learning, memorizing, and understanding medical information in vietnamese. experiments were conducted using the proposed vimedaqa dataset. using an open-book qa format, the first experiment examined the model's reasoning ability in the vietnamese medical and healthcare domain. each input to the model consisted of a pair, {p i , q i }, where p i is a paragraph and q i is a corresponding question. to assess the model's ability to extract essential information to answer q i , additional unrelated paragraphs, {p j } where (j ̸ = i and |{p j }| ∈ {0, 1, 2, 4, 8}), are included in the prompt. it is hypothesized that adding more noise paragraphs would degrade the model's performance. a second experiment measured the amount of vietnamese medical knowledge within the language models under a closed-book qa setting. the model is prompted with only the question q i and answers q i using its internal knowledge acquired during pretraining and finetuning. we use greedy search decoding during the evaluation process to prompt the model, resulting in highly reproducible experiments. the bleu @xcite , meteor @xcite , rouge-l @xcite , and bertscore @xcite metrics are utilized to compare model's outputs and labels. ## experimental results and analysis this section reports the empirical results for the experiments following setups in section 4. all of the experiments use medium to small llms, including llama2-7b and llama3-7b @xcite , gemma-2b and gemma-7b by @xcite , phogpt-4b by @xcite , vinallama-7b and vinallama-2.7b by @xcite , and vigpt by @xcite . all the models are explored with the chat or instruction tuning versions. ## conclusion vimedaqa, a vietnamese medical abstractive question-answering dataset, is published to mitigate the lack of an abstractive qa corpus for the vietnamese medical domain. by leveraging the available llms, raw question-answer pairs are automatically generated before being verified by expert annotators to ensure the dataset's quality. additionally, experiments to study the capability of llms are examined, including reasoning, memorizing, and awareness of critical information. the empirical results show that vinallama-7b is a large language model with powerful reasoning skills in the vietnamese medical domain, and the gemma model family is robust in realizing essential information across multiple noise contexts. there are several potential directions for future work, including (1) expanding the scope of topics covered by vimedaqa in the vietnamese medical domain, (2) investigating the performance of llms in this domain under different fine-tuning methodologies, and (3) delving into the extraction of critical data by increasing the number of incorrect paragraphs (m), and exploring solutions to mitigate performance degradation as m increases.