Harnessing the Power of BERT in the T urkish Clinical Domain: Pretraining Approaches for Limited Data Scenarios (2023)

Abstract: Recent advancements in natural language processing (NLP) have been driven by large language models (LLMs), thereby revolutionizing the field. Our study investigates the impact of diverse pre-training strategies on the performance of Turkish clinical language models in a multi-label classification task involving radiology reports, with a focus on overcoming language resource limitations. Additionally, for the first time, we evaluated the simultaneous pre-training approach by utilizing limited clinical task data. We developed four models: TurkRadBERT-task v1, TurkRadBERT-task v2, TurkRadBERT-sim v1, and TurkRadBERT-sim v2. Our results revealed superior performance from BERTurk and TurkRadBERT-task v1, both of which leverage a broad general-domain corpus. Although task-adaptive pre-training is capable of identifying domain-specific patterns, it may be prone to overfitting because of the constraints of the task-specific corpus. Our findings highlight the importance of domain-specific vocabulary during pre-training to improve performance. They also affirmed that a combination of general domain knowledge and task-specific fine-tuning is crucial for optimal performance across various categories. This study offers key insights for future research on pre-training techniques in the clinical domain, particularly for low-resource languages.

## introduction language models have undergone a significant transformation in the field of natural language processing, demonstrating exceptional capabilities in executing tasks with minimal guidance. this shift can be attributed to pivotal milestones, such as word2vec @xcite , which replaced feature engineering methods with deep learningbased representation learning. furthermore, the emergence of contextualized word embeddings with elmo has led to the development of @xcite pre-trained transformer-based models such as bert @xcite , gpt @xcite , t5 @xcite , and bart @xcite . recent advancements in large language models (llms) have led to the development of models with parameter sizes exceeding hundred billion, including the gpt (generative pre-trained transformer) series @xcite @xcite , such as chatgpt and @xcite , which are pre-trained on massive datasets. however, research focusing on llms architectures within specialized domains characterized by limited resources is scarce. a range of approaches for developing language models exists to address the issue of limited language resources, including simultaneous pretraining with in-domain data @xcite and domain-adaptive pretraining by fine-tuning an existing generic language model with in-domain data @xcite . the choice of pre-training technique depends on the specific task data and available resources, but determining the optimal utilization of limited clinical task data in pretraining and selecting the most suitable data for pretraining methods remain open questions. this study aimed to assess and contrast different techniques using a limited task corpus for pretraining bert models in the turkish clinical domain, a low-resource setting. we introduce two pretrained language model families, turkradbertsim and turkradbert-task, each comprising two models for the clinical domain in the turkish language. these models explore the effects of different corpus selections that combine small taskrelated corpora and pretraining strategies in the turkish clinical domain. the turkradbert-sim pre-trained model family, developed via simultaneous pre-training @xcite , involves a balanced combination of two distinct corpora: one general and one limited task-specific. both corpora were upsampled to create pretraining instances, resulting in robust neural language models. the turkradbert-task pretrained model family, developed via task-adaptive pre-training, involves an additional pretraining stage where the model is adaptively pretrained on a smaller, task-specific dataset following the initial pretraining. we also created a labeled dataset for multi-label document classification using head ct radiology reports to evaluate the models. the main contributions can be listed as follows: ## related work to optimize natural language processing models for specialized domains, various studies have explored different approaches to adapt general bert models for the biomedical domain. biobert @xcite , an early attempt to adapt general bert models to the biomedical domain, employed continual pretraining to enhance performance. initialized from the general bert model, biobert was further trained on pubmed abstracts and full-text articles, yielding an improved performance for tasks such as named entity recognition, relation extraction, and question answering. similarly, clinical-bert @xcite , a domain-specific language model, was created using continual pretraining with mimic data, demonstrating its effectiveness in improving clinical task performance. other studies have explored continual pretraining for biomedical language models, such as scib-ert @xcite and bluebert @xcite , which were pretrained on a mix of biomedical and general domain corpora. an alternative approach, pretraining from scratch, focuses exclusively on in-domain data, without relying on a generic language model. this method has been effective in creating models, such as pubmedbert @xcite , which is pretrained solely on pubmed abstracts. comparisons between the two pretraining methods reveal that continual pretraining often leads to more successful transfers from general to specialized domains. for example, one study proposed four bert models @xcite , two pretrained on german radiology freetext reports (fs-bert and rad-bert), and two based on open-source models (multi-bert and ger-bert). the fs-bert model, which used the pretraining from scratch approach, performed poorly compared to the other models, suggesting that domain-specific corpora alone might be insufficient for learning proper embeddings. another study developed radbert @xcite , a set of six transformer-based language models pretrained on radiology reports with various language models for initialization, to explore their performance in radiology nlp applications. although pretraining bert models can improve performance across various biomedical nlp tasks, they require significant domain-specific data. biomedical text data are often limited and scattered across various sources, and few publicly available medical databases are written in languages other than english. this creates a high demand for effective techniques that can work well even with limited resources. one solution to this problem is the simultaneous pre-training technique proposed in @xcite , which up-samples a limited domain-specific corpus and uses it for pre-training in a balanced manner with a larger corpus. using small japanese medical article abstracts and japanese wikipedia texts, the authors created a simultaneous pretrained bert model, oubiobert. the study confirmed that their japanese medical bert model performed better than conventional baselines and other bert models in a medical japanese document classification task. however, they did not focus on applying the simultaneous pre-training approach to limited clinical task radiology data. building upon this work, our study shifts the focus towards applying the simultaneous pretraining approach to limited clinical task data for the first time. to overcome the limitations of the limited resources problem, many researchers have explored the benefits of continued pretraining on a smaller corpus drawn from the task distribution as task-adaptive pre-training @xcite . in addition, @xcite previously demonstrated that their biomedical bert models, the bioberturk family, which were continuously pre-trained on a limited turkish radiology thesis corpus, exhibited improved performance in clinical tasks. however, the authors also highlighted the potential ineffectiveness of domain incompatibility when evaluating turkish language models, emphasizing the need for a closer alignment between domain-specific data and evaluation tasks. ## materials and methods in this section, we provide a concise overview of the pre-training methods employed for the development of turkish clinical language models and the characteristics of the corpora used in this process. we developed four turkish clinical language models, leveraging the bert-base architecture and constrained language resources by employing two pre-training strategies: simultaneous pre-training and continual pre-training, referred to as task-adaptive pretraining. two models, referred to as the turkradbert-sim family, were developed by employing simultaneous pre-training techniques that combined general, biomedical, and clinical task corpora, while utilizing distinct vocabularies. in contrast, two models, the turkradberttask family, were developed by employing taskadaptive pretraining using the task corpus. to construct these clinical models, we employed four distinct corpora: the turkish biomedical corpus compiled from open-source medical articles @xcite , turkish electronic radiology theses corpus @xcite , turkish web corpus @xcite , and newly created turkish radiology report corpus, which is a limited task corpus. while all corpora were utilized in simultaneous pre-training, only turkish radiology reports were used in task-adaptive pre-training. subsequently, the clinical language models were fine-tuned on a downstream nlp task within the turkish clinical domain. finally, the clinical language models were compared to the general turkish domain bert model, berturk @xcite , and the bioberturk variant @xcite , which was continually pretrained on turkish radiology theses. ## experimental results in this study, we evaluated the performance of five different models, including berturk, turkradbert-task v1, turkradbert-task v2, turkradbert-sim v1, and turkradbert-sim v2, for turkish clinical multi-label classification. we compared their performance over ten runs in terms of average precision, recall, and f1 score. additionally, we analyzed the performance of wining two model (berturk, turkradbert-task v1) on individual categories using their respective f1 scores. the results are presented in tables 2 and 3 . ## discussion by assessing the experiments as a whole, we derived the following conclusions. when comparing simultaneous pre-training and task-adaptive pre-training, it is observed that, owing to the size difference between the task data and the general data, the limited domain-specific data may be overshadowed by the large general-domain data. this causes the model to focus more on learning general rather than task-specific features. this phenomenon highlights the importance of carefully balancing general and domain-specific data during the pretraining process to ensure that the model effectively captures the nuances of the specialized domain. the performances of the berturk and turkradbert-task v1 models are quite similar because both models leverage the knowledge gained from the large general-domain corpus during pre-training. berturk is directly pre-trained on this large corpus, while turkradbert-task v1 is initialized with berturk's weights and then fine-tuned using task-adaptive pre-training on a smaller clinical corpus. this fine-tuning enables turkradbert-task v1 to capture domain-specific patterns, structures, and terminologies absent in the general-domain corpus. however, the small task-specific corpus used in task-adaptive pretraining may limit the model's learning of domain-specific knowledge. consequently, despite the benefits of task-adaptive pre-training, turkradbert-task v1, which utilized this approach, had a slightly lower performance than berturk. in limited data scenarios, the task-adaptive pre-training approach may be prone to overfitting, especially when pre-trained on a small task-specific corpus. the model may become overly specialized in training data and fail to generalize well to unseen examples @xcite . in terms of performance, turkradbert-task v1 has a slightly higher f1 score (0.9556) than bioberturk (0.9535) and turkradbert-task v2 (0.9470). this suggests that despite the more specialized biomedical knowledge in bioberturk, the general-domain berturk model provides a more robust foundation for task-adaptive pre-training in this specific clinical task. another conclusion reached in this study is that comparison between turkradbert-sim v1 and v2 offers insights into the impact of domain-specific vocabulary on model performance. turkradbert-sim v1, which used an amplified vocabulary built from the generated corpus, outperformed turkradbert-sim v2 that employed the general domain vocabulary. this finding indicates that using a domain-specific vocabulary during pre-training can enhance the ability of the model to capture and understand domain-specific language patterns, ultimately leading to improved performance on clinical nlp tasks. examining the f1 scores for each label in first, the optimal performance on specific labels, such as meningioma and chronic ischemic changes, might be attributed to the use of precise, standard reporting terminology to define these pathologies, a factor that likely provides high results, regardless of the classifier employed. berturk outperforms turkradbert-task v1 in certain labels, such as intraventricular, gliosis, epidural, leukoaraiosis, mega cisterna magna, with no findings. the higher performance of berturk on certain labels could be attributed to the general domain knowledge acquired during direct pre-training (different from other pre-training methods), which may provide better coverage for specific categories, particularly those with a lower frequency in the task-specific corpus. berturk's broader pre-training data exposure could potentially give it an advantage over models like turkradbert-task v1 when dealing with specific labels that have lower representation in the task-specific corpus, even though turkradbert-task v1 is initialized with berturk. this suggests that a combination of general domain knowledge and task-specific fine-tuning may be critical for optimal performance across diverse categories. on the other hand, turkradbert-task v1 exhibits superior performance for labels like hydrocephalus, encephalomalacia, subarachnoid bleeding, and subdural. this suggests that task-adaptive pre-training can offer a performance boost in some instances by fine-tuning the model based on domain-specific information. however, it is worth noting that the overall performance differences between the two models are relatively small, highlighting the importance of leveraging both general-domain and task-specific knowledge in these models. ## conclusion this study provides a comprehensive comparison of the performance of various models, including berturk, turkradbert-task v1, turkradbert-task v2, turkradbert-sim v1, and turkradbert-sim v2, on a radiology report classification task. our findings demonstrate that the berturk model achieved the best overall performance, closely followed by the turkradbert-task v1 model. this highlights the importance of leveraging both general domain knowledge acquired during pre-training and task-specific knowledge through fine-tuning to achieve optimal performance on complex tasks. we also observed that the performance of these models varies across different labels, with berturk performing better on certain categories, particularly those with lower representation in the task-specific corpus. this finding suggests that a combination of general domain knowledge and task-specific fine-tuning may be critical for achieving optimal performance across diverse categories. additionally, it is essential to consider label frequencies when interpreting results because performance on rare labels may be more susceptible to noise and overfitting. the simultaneous pre-training models turkradbert-sim v1 and v2 exhibit lower performance compared to their task-adaptive counterparts, indicating that task-adaptive pre-training is more effective in capturing domain-specific knowledge. nevertheless, further investigation of alternative pre-training and fine-tuning strategies could help enhance the performance of these models. future research could focus on expanding the task-specific corpus to improve domain-specific knowledge and performance on rare labels as well as explore alternative pre-training and fine-tuning strategies to further enhance model performance. moreover, investigating the factors contributing to the performance differences between the models for each label could provide valuable insights for developing more effective models in the field of medical natural language processing.