Using Bottleneck Adapters to Identify Cancer in Clinical Notes under Low-Resource Constraints (2023)

Abstract: Processing information locked within clinical health records is a challenging task that remains an active area of research in biomedical NLP. In this work, we evaluate a broad set of machine learning techniques ranging from simple RNNs to specialised transformers such as BioBERT on a dataset containing clinical notes along with a set of annotations indicating whether a sample is cancer-related or not. Furthermore, we specifically employ efficient fine-tuning methods from NLP, namely, bottleneck adapters and prompt tuning, to adapt the models to our specialised task. Our evaluations suggest that fine-tuning a frozen BERT model pre-trained on natural language and with bottleneck adapters outperforms all other strategies, including full fine-tuning of the specialised BioBERT model. Based on our findings, we suggest that using bottleneck adapters in low-resource situations with limited access to labelled data or processing capacity could be a viable strategy in biomedical text mining.

## introduction clinical notes involve important information about patients and their current state and medical history. automatic processing of these notes and the terms that appear in them would help researchers classify them into standard conditions that can also be † please refer to appendix a.4 for the full list of collaborators. in this work, we looked at a real clinical notes database and designed a pilot experiment in which a set of different ml models were used to predict whether a clinical note is cancer-related or not. the incentive behind this experiment is to help clinicians and data curators to automatically search for and identify notes that signal a particular medical condition, instead of solely relying on laborious human annotation and keyword-based search. the promise of ml is in automating this task reasonably close to human-level performance and ultimately expanding this work to include other conditions in a multi-class scenario. ideally a model would be able to identify cancer types that are not seen during training and would be able to have some understanding of context and grammar to be sensitive to negation. ## pre-trained transformers and fine-tuning in recent years, the transformers architecture @xcite and large language models (lms) have become the staple baseline for many nlp tasks. the conventional paradigm is to first pre-train an lm on a large corpus of general text (e.g. wikipedia) with a pre-training objective such as masked or causal language modeling and then fine-tune the lm on downstream tasks. in our task, we focus on transformers pre-trained with the masked language modeling (mlm) objective. in mlm, a portion of the text is masked out and the objective of the model is to learn to reconstruct the masked portion based on the available context. the most commonly used model pretrained with mlm is named bert @xcite . despite bert's promising results on many downstream nlp tasks, it has been shown that large lms pre-trained on generic text do not always perform well on specialised domains like biomedical tasks @xcite . the standard approach, therefore, is to pre-train models on corpora that are related to the target domain. @xcite is an example of an lm trained on specialized data. it is trained on a large corpus of general and biomedical texts making it a strong model for biomedical text mining. ## challenges of identifying cancer-related records clinical notes usually involve abbreviated and nonstandard language. a single concept like cancer is mentioned in different ways depending on cancer subtype. the same subtype might have a scientific and a commonly known variant and both can appear in the text. grammar is sometimes broken and language can appear cryptic. another issue is the prevalence of misspellings which further complicates this task. there are also words that co-occur with a condition and can easily confound the model. for instance, words like 'breast' and 'lung' which are not specific to cancer appear a lot in cancer-related samples and the model can mistake them for a cancer signal. another important issue is negation. if a condition is ruled out, ideally a model should not return positive. however since most rows that are classed as positive in the dataset include the token 'cancer', an example like 'not cancer' could be mistaken as positive. encoding awareness of negation into the model is a challenge since it is known that pre-trained lms lack an innate ability to handle negation @xcite . ## dataset and annotation the dataset in this pilot experiment was provided by isaric, a global initiative that, among other things, provides tools and resources to facilitate clinical research 1 . the larger dataset contains 1 the isaric covid-19 data platform is a global partnership of more than 1, 700 institutions across more than 60 countries. accreditation of the individuals, institutions and funders that contributed to this effort can be found in the supplementary material. these partners have combined data and expertise to accelerate the pandemic response and improve patient outcomes. for more information on isaric, see https: //isaric.org . data are available for access via application to the data access committee at www.iddo.org/covid-19 . 125381 rows corresponding to clinical notes related to different conditions and patients. for the purposes of this experiment a portion of this data was annotated for presence of cancer. the annotated subset contains 2563 rows that include cancer labels, out of which 343 are repeated notes where the doctors have written the same cancer-related note for a different patient. the human experts who tagged the data for cancer, had access to a set of cancer-related terms to guide them in the annotation. the negative cohort of 3k rows was generated by filtering out the larger data by any row that contained keywords that could potentially signal cancer definitively or with a very high possibility. the details of the lists and more information on the annotation scheme are included in the appendix (a.1). ## experiments the experiments in this work are divided into two categories, namely, attention-based and rnnbased methods. we conducted all our experiments on an internal cancer detection dataset with ~6k labeled samples with roughly equal instances in each classes and evaluated them on a gold standard consisting of 1k samples, 31 of which were positive and the rest negative. note the distributional shift between training and test sets which reflect the real clinical setting under which the models are expected to perform. ## results reported results in tuning with bas was done with 10 epochs and a learning rate of 1e-3. pt was used with 10 epochs and a learning rate of 1e -4. all approaches used a batch size of 64, adamw optimizer, weight decay of 0.01, and a cosine scheduler. as can be seen, the best performing model is the bert trained with adapters (including variants which are equipped with some notion of negation as explained in 5.2.3). analysing the outputs of individual models, we found that the majority of positive labels in the test set are correctly identified by most models. the bottleneck, however, is the false positives that happen due to the presence of certain words (e.g. 'diagnosed with', 'lung', 'breast' etc) that co-occur with cancer and can cause models to incorrectly label an instance as positive. the best model had only 4 false positives and no false negatives. the values for the confusion matrices of all the models are provided in a.3. to alleviate the false positive issue, using the method explained in 5.2.3, we trained our best model (bert with adapter-tuning) with additional 250 and 500 generated negative samples. the model was subsequently able to predict cases such as 'neither cancer nor covid', 'lung infection but no cancer', and 'diagnosed with covid but not cancer' correctly with only minor performance drops. a point of strength in all the models was their ability to correctly identify cancer, given rare cancer types that had not occurred in the training set. this generalisation to unseen cancer types indicates that the models can effectively use information from the pre-trained resources they rely upon. ## conclusion in this work, we trained and tested a number of classification approaches as part of a preliminary experiment on a dataset of clinical notes annotated for presence of cancer. we compared a number of rnn models utilising pre-trained biomedical embeddings with two different pretrained transformer-based models that were finetuned in separate ways. we also addressed the issue of negation by integrating negation patterns into the negative training samples. our find-