NER vous About My Health: Constructing a B engali Medical Named Entity Recognition Dataset (2023)

Abstract: The ability to identify important entities in a text, known as Named Entity Recognition (NER), is useful in a large variety of downstream tasks in the biomedical domain. This is a considerably difficult task when working with Consumer Health Questions (CHQs), which consist of informal language used in day-to-day life by patients. These difficulties are amplified in the case of Bengali, which allows for a huge amount of flexibility in sentence structures and has significant variances in regional dialects. Unfortunately, the complexity of the language is not accurately reflected in the limited amount of available data, which makes it difficult to build a reliable decision-making system. To address the scarcity of data, this paper presents ‘Bangla-HealthNER’, a comprehensive dataset designed to identify named entities in health-related texts in the Bengali language. It consists of 31,783 samples sourced from a popular online public health platform, which allows it to capture the diverse range of linguistic styles and dialects used by native speakers from various regions in their day-to-day lives. The insight into this diversity in language will prove useful to any medical decision-making systems that are developed for use in real-world applications. To highlight the difficulty of the dataset, it has been benchmarked on state-of-the-art token classification models, where BanglishBERT achieved the highest performance with an F1-score of 56.13 \pm 0.75 %. The dataset and all relevant code used in this work have been made publicly available.

## introduction a fundamental component of nearly all automated question answering (qa) systems is a named entity recognition (ner) model @xcite . to accu-1 https://github.com/alvi-khan/ bangla-healthner rately answer a question, the system must be capable of identifying important entities in the text. this is even more critical when addressing healthrelated queries from the general public, known as consumer health questions (chqs). chqs are often difficult to answer due to the informal language used @xcite , and most patients only provide a vague description of their issues. this presents a unique challenge to current systems, most of which were developed for use with medical documents that use formal language and concrete scientific terms to describe health issues @xcite . being able to extract the correct medical knowledge required to answer the question from the vague descriptions containing unstandardized medical terms used by the general public is a challenging task. the difficulties described are further exacerbated in the bengali language, which poses greater challenges compared to english for several reasons. bengali allows for a significantly larger number of inflections, with 220 inflections existing in total compared to just 9 for english @xcite . this forces the models to learn a large number of words that are nearly identical but have slightly different meanings. for example, in the sentence 'i have fallen ill', the verb 'fallen' remains unchanged even if the subject is changed to 'you' or 'she'. in bengali, however, the same sentence ('আিম অসু হেয় পেড়িছ।') will change the verb depending on the subject: 'পেড়িছ' for the first person, 'পেড়েছা' for the second-person, and 'পেড়েছ' for the third-person. there is also greater flexibility regarding sentence structuring. whereas the order of words is relatively rigid in english, and breaking the order makes sentences grammatically incorrect, the same is not true for bengali. for example, 'i have a headache' is correct in english, but 'headache have i' sounds odd and is unlikely to be used. however, the bengali versions of these sentences ('আমার মাথা বয্থা' and 'মাথা বয্থা আমার' re-spectively) are both correct and commonly used. furthermore, there is a large amount of diversity in the language as there are numerous dialects that are so significantly different from each other that people from one region are often entirely unable to comprehend the language of another @xcite . these attributes make bengali an incredibly complex language with a huge amount of variation and intricacies. unfortunately, the amount of data available for bengali, especially in a medical context, is severely limited and fails to reflect the complex nature of the language. this makes it challenging to create a system that accurately understands the language in the context of chqs. to address this issue, we present 'bangla-healthner', the largest human-annotated bengali medical ner dataset comprising 31,783 samples. the data has been collected from a public online health platform in bangladesh, ensuring that it accurately captures the variety and complexity of the language used by native bengali speakers in their dayto-day lives. the samples also consist of a large amount of text that combines bengali and english, using words from both languages or transliterating words from one language using alphabets from the other, a common occurrence in informal conversations known as 'code-switching'. the large variety in language present in the dataset should allow models to gain a thorough understanding of medical terminology in informal contexts. ## literature review although there is a significant amount of data available for general bengali ner @xcite @xcite , the availability of data specifically tailored for bengali medical ner remains limited. currently, there are two datasets publicly available: one introduced by @xcite , which we will be referring to in this paper as the 'telemedicine dataset', and the 'banglabiomed' dataset @xcite . both of these datasets have relatively small sizes, with the telemedicine dataset consisting of 190 samples and nine entity labels and banglabiomed consisting of 1100 sentences and four entity labels. this makes the datasets unsuitable for modern transformer-based architectures, which rely on large amounts of data for effective learning. additionally, the data for both datasets were collected from sources that use metric count sample count 31,783 sentences 144,136 percentage of entity tokens 23.47% avg. words per sample 175.81 the small dataset sizes combined with their formal nature means that systems trained on the existing datasets will be unable to perform well in real-world use cases. the introduction of bangla-healthner addresses both of these limitations since it is larger in size and also accommodates much more variety in language. ## benchmarking to establish a benchmark, the dataset was fine-tuned on three token classification models-banglabert @xcite , ban-glishbert @xcite , and mbert @xcite . aside from this, to analyze the performance of large language models on bengali medical ner, we also included the inference results of zero-shot chatgpt (gpt-3.5turbo, accessed june 2023) @xcite . for the benchmarking process, the dataset was divided into training, validation, and test splits using the ratio 80 : 10 : 10. further details about the experimental setup are provided in appendix a.1, and both banglabert and banglishbert outperformed the mbert model, which reinforces the understanding that models pre-trained on a specific language achieve better performance on downstream tasks than multilingual ones. ban-glishbert achieved the highest scores, likely due to the presence of text that combines bangla and english in the dataset. the gpt 3.5 model was not fine-tuned on our dataset but was given prompts to assign labels to each word in the test samples. past literature studying the performance of chatgpt on downstream tasks related to bio-medicine without fine-tuning found that domain-specific models achieve significantly better results, even if expert prompting is used @xcite . our results agree with this conclusion, as all the fine-tuned models outperform zero-shot chatgpt by a large margin. a thorough error analysis of the predictions of the fine-tuned models reveals that the expected ## conclusion in this paper, we introduced a manually annotated bengali named entity recognition dataset of health-related texts. the paper highlights several key features of the dataset, including its ability to capture the linguistic diversity and complexity of the bengali language, as well as its emphasis on the informal nature of speech used by the average patient in medical contexts. we evaluated the dataset on bengali and multilingual token classification models to establish a benchmark and further explored the performance of the large language model gpt 3.5 on the dataset. given that the dataset introduces significant variety in language to the domain compared to existing data, architectures trained on past data struggled to perform well. this opens up a new direction for future work, which can concentrate on exploring methods to improve this performance. the research and data presented in this paper have implications that extend beyond the domain of ner in chqs. the challenges addressed, such as dealing with informal language, complex sentence structures, and regional dialects, are observed in various downstream tasks, meaning our findings can be generalized to broader contexts and applications. the dataset can also be leveraged to achieve cross-domain adaptability in fields such as social media, customer reviews, and online forums, where users communicate using colloquial and unstructured language. the proposed insights also have cross-linguistic transferability to other indo-aryan languages with similar linguistic characteristics. this could prove beneficial in developing ner models for languages with limited labelled data. the adaptability of ner as a feature can facilitate automated clinical decisionmaking systems as well.