TALP - UPC at P rob S um 2023: Fine-tuning and Data Augmentation Strategies for NER (2023)

Abstract: This paper describes the submission of the TALP-UPC team to the Problem List Summarization task from the BioNLP 2023 workshop. This task consists of automatically extracting a list of health issues from the e-health medical record of a given patient. Our submission combines additional steps of data annotationwith finetuning of BERT pre-trained language models. Our experiments focus on the impact of finetuning on different datasets as well as the addition of data augmentation techniques to delay overfitting.

## introduction healthcare is a vital sector in our society, playing a crucial role in both the prevention and treatment of diseases for the general population. in addition, healthcare workers often face large workloads due to the number of patients and emergencies that may occur during their shifts. therefore, finding ways to automate their most repetitive tasks is worth exploring with ai system. with the addition of these systems, healthcare workers could reduce their cognitive load while at the same time providing better care and diagnosis to their patients. with these objectives in mind, this shared task aims to develop systems able to summarize medical health records. healthcare workers collect daily notes on the treatment of patients, which are stored in electronic ehealth records. the summarizer must be able to extract the most relevant health issues from these records. one of the main difficulties when creating these systems is the lack of available annotated data. medical records are sensible material requiring a thorough revision in order to ensure the privacy of the patients, which implies some degree of human supervision. as a result of these steps, these datasets are usually expensive to create and include a limited number of examples. a popular approach for these tasks is resorting to recent nlp techniques such as transformer @xcite that allow working with large text context as well as pre-trained language models such as bert @xcite that allow models to benefit from transfer learning from large corpora for several languages. in this paper, we present our submission to the shared task of problem list summarization of the bionlp 2023 workshop. the main contributions of our system are: all code use for the this work is available at the following link: https://github.com/neiltorrero/bionlp . ## related work in this section, we introduce the general concepts of named-entity-recognition (ner) as well as previous methods based on pre-trained language models and data augmentation to fine-tune them on limited amounts of data. ## methodology inspecting the mimic dataset, we can observe that most of the example's ground truths follow the same pattern. we have a list of diagnoses split by ; without connectors or additional information for each example. in order to generate a summary that follows these ground truths, we propose a system based on bio tagging using two additional corpora and a fine-tuning step based on bert. then we process the inputs of the patient's treatment notes to extract the keywords and transform them into the expected test format. ## available here we noticed that medical notes usually included the same problem several times, while the label did not include repetitions. another problem was the differences in using acronyms between notes and labels. as a result, some diseases correctly identified in the text were considered wrong by the metric as the strings did not match. we applied several post-processing steps to mitigate these issues. we removed duplicated terms for all outputs, considering acronyms and partial forms of multi-word entities. finally, we concatenated and separated all entities using the ; delimiter. ## results this section will discuss the main results from the different experiments performed during this shared task. we performed experiments on fine-tuning strategies. results show that the two-step fine-tuning approach (tsf) outperforms fine-tuning on all available data by more than 1% on private and public test sets. this performance increment is even more significant when the post-processing step is applied. we observe a 4% f1 score improvement on the private test set, much larger than the 0,05% observed on the public test set. an explanation for this difference may be that the public test set is cleaner than the provided training data. these steps are applied to all our experiments. another critical factor in our experiments was data scarcity. to mitigate it, we evaluated different levels of data augmentation by randomly masking different percentages of the input tokens. we observe slight improvements when adding more data augmentation up to 40% of tokens. these results are consistent for both private and public test sets, being this the best model on the public test set and our final submission to the competition. finally, once we had the final hyperparameters for all models, we decided to run a final experiment, including the development data for the training to increase the amount of data available. although this configuration performed best on our private test set, it yielded slightly worse results on the public test set compared to the original model that used less data. overall, these results showcase the importance of fine-tuning domain-specific data and using data augmentation to maximize the performance of our systems. ## conclusions in this paper, we have presented the upc submission to the bionlp 2023 problem list summarization. our results demonstrate that modeling this task as a ner problem and combining pre-train model fine-tuning and data augmentation is an effective approach to solving this task, even when limited training data is available. experimental results show how these two techniques can outperform fine-tuning on additional out-of-domain data or adding small amounts of domain data during the fine-tuning process. in future work, we plan to explore leveraging additional non-annotated data from the biomedical domain.