B angla CHQ -Summ: An Abstractive Summarization Dataset for Medical Queries in B angla Conversational Speech (2023)

Abstract: Online health consultation is steadily gaining popularity as a platform for patients to discuss their medical health inquiries, known as Consumer Health Questions (CHQs). The emergence of the COVID-19 pandemic has also led to a surge in the use of such platforms, creating a significant burden for the limited number of healthcare professionals attempting to respond to the influx of questions. Abstractive text summarization is a promising solution to this challenge, since shortening CHQs to only the information essential to answering them reduces the amount of time spent parsing unnecessary information. The summarization process can also serve as an intermediate step towards the eventual development of an automated medical question-answering system. This paper presents ‘BanglaCHQ-Summ’, the first CHQ summarization dataset for the Bangla language, consisting of 2,350 question-summary pairs. It is benchmarked on state-of-the-art Bangla and multilingual text generation models, with the best-performing model, BanglaT5, achieving a ROUGE-L score of 48.35%. In addition, we address the limitations of existing automatic metrics for summarization by conducting a human evaluation. The dataset and all relevant code used in this work have been made publicly available.

## introduction the answers to general health inquiries can often be obtained by utilizing internet search engines, but addressing queries by individual users in a manner that caters to their specific circumstances remains a manual process. such queries, known as consumer health questions (chqs), are frequently found on online health forums, and answering them is becoming increasingly timeconsuming and labour-intensive for medical professionals @xcite . the task is made even more difficult by the fact that patients are often overly descriptive when asking questions, providing unnecessary details @xcite . the ability to identify and discard these unnecessary details would save a lot of time for the response providers and would also be an important step towards the eventual development of an automated question-answering system @xcite . abstractive text summarization is the task of generating a shortened and human-readable version of the original text that retains the important information @xcite . despite the recent improvement in this domain due to the development of transformer-based architectures as well as the greater availability of data, progress has been somewhat limited in chq summarization @xcite @xcite . this shortcoming is particularly notable for low-resource languages like bangla @xcite , for which there is no existing work on this task. developing a dataset dedicated to this language presents a substantial challenge to existing architectures for several reasons. firstly, bangla is an exceedingly complicated language in comparison to english, allowing for more flexible sentence structuring @xcite and a significantly greater number of inflections (220 as opposed to just 9 in english @xcite ), resulting in noisier text. furthermore, the diversified dialects exacerbate the issue, with the language used in one region frequently being entirely unintelligible in another @xcite . navigating this complexity and successfully identifying the relevant medical information is a significantly complicated task. unfortunately, the bangla text summarization architectures currently available do not account for the complications of informal speech in medical contexts since they were mostly trained on news articles @xcite , making them unsuitable for summarizing medical text. this paper addresses the lack of medically relevant data by introducing the first human-annotated bangla chq summarization dataset, 'banglachq-summ', consisting of 2350 question-summary pairs. the data was collected from a public online health forum used by hundreds of native bangla speakers, allowing it to present an accurate representation of the health questions that are generally present on online forums. in addition to the dataset, we also discuss the shortcomings of established evaluation metrics of text summarization tasks and explore a methodology for human evaluation that addresses the shortcomings. ## related work although a large amount of work has been dedicated to text summarization in general @xcite @xcite , very limited literature is devoted to chq summarization. to the best of our knowledge, there are only two datasets available for the task, 'meqsum' (abacha and demner-fushman, 2019b) and 'chq-summ' @xcite , and both consist exclusively of english text. the lack of work addressing chq summarization is a major limitation for the domain since domain-specific models are known to outperform general ones @xcite . meqsum was the first dataset for consumer health question summarization, consisting of 1,000 samples collected from the u.s. national library of medicine. the dataset has a relatively small size but was also the only medical question summarization dataset available at that time. @xcite attempted to address the lack of available datasets by introducing the 'chq-summ' dataset. this dataset consists of 1,507 samples collected from the yahoo community question-answering forum. the informal source of the data enhances its diversity and presents a more realistic depiction of the questions that medical professionals are likely to encounter. a notable shortcoming of the existing literature is the lack of diversity in language. the advantages of chq summarization should prove extremely beneficial if its application can be extended to support overpopulated regions such as bangladesh, where healthcare workers are fre-quently overwhelmed by the volume of patients @xcite . introducing a bangla dataset contributes towards solving this issue, and the knowledge gained is also transferable to other indo-aryan languages of the indian subcontinent. ## the banglachq-summ dataset in this section, we demonstrate how we curated the proposed banglachq-summ dataset. ## experiments to benchmark model performance on our proposed dataset, we conduct two types of evaluation: (i) automatic and (ii) human. we split the dataset into training, validation, and test sets following an 80:10:10 ratio. below, we present our findings. ## conclusion in this paper, we propose the first chq summarization dataset for the bangla language. the source of the data used in the creation of the dataset also presents an advancement towards a more accurate representation of the diversity of the language. in addition, we explore a methodology for human evaluation that addresses the limitations of existing text summarization evaluation metrics. given the sensitive nature of the public health domain, improvements in the performance of the summarization models, alongside evaluating how large language models @xcite perform in this dataset could be a good direction for future research.