B i CAL : Bi-directional Contrastive Active Learning for Clinical Report Generation (2024)

Abstract: State-of-the-art performance by large pre-trained models in computer vision (CV) and natural language processing (NLP) suggests their potential for domain-specific tasks. However, training these models requires vast amounts of labelled data, a challenge in many domains due to the cost and expertise required for data labelling. Active Learning (AL) can mitigate this by selecting minimal yet informative data for model training. While AL has been mainly applied to single-modal tasks in the fields of NLP and CV, its application in multi-modal tasks remains underexplored. In this work, we proposed a novel AL strategy, Bidirectional Contrastive Active Learning strategy (BiCAL), that used both image and text latent spaces to identify contrastive samples to select batches to query for labels. BiCAL was robust to class imbalance data problems by its design, which is a problem that is commonly seen in training domain-specific models. We assessed BiCAL’s performance in domain-specific learning on the clinical report generation tasks from chest X-ray images. Our experiments showed that BiCAL outperforms State-of-the-art methods in clinical efficacy metrics, improving recall by 2.4% and F1 score by 9.5%, showcasing its effectiveness in actively training domain-specific multi-modal models.

## introduction active learning (al) is a branch of machine learning that aims to select a small set of the most informative data to annotate for model training @xcite . this technique allows the model to achieve optimal performance while lowering the cost of annotation. moreover, by actively selecting data to train on, a model trained under active learning can sometimes surpass the performance that is trained on the full dataset. al has shown its great potential in the field of natural language processing (nlp) @xcite ; shen @xcite and computer vision (cv) @xcite . however, relatively few have explored the application of active learning in a multi-modal setting. in addition, as the capabilities of general largepretrained models arise @xcite , a rising interest has been seen in fine-tuning them to become domain-specific models. however, when training models in specific domains, obtaining quality labelled data is challenging due to the domain expertise required for accurate annotation, which is costly in both time and money. this motivates us to explore active learning's application in the domain-specific setting. we identify that in domain-specific active learning, there exists one key challenge -class imbalance in datasets is often seen in domain-specific settings, existing al methods struggle to actively select samples that have less population but may be more importantin medicine, common (healthy) samples often out populate rare (unhealthy) samples. models trained under such active learning strategies converge on the commonly seen samples and perform poorly in identifying rare sickness cases. in this study, we introduce a novel al strategy bidirectional contrastive active learning strategy (bical) that is tailored to address the challenge in domain-specific active learning. we assess bical and other established al methods on clinical report generation from chest x-ray images. our key contributions are: ## related work this section provides the background of our proposed al strategy bical. we first formalize the active learning problem under the image-to-text generation task and set up the notation for the rest of the paper. given a model m, unlabelled image data pool x pool . we denote an unlabelled input image as @xmath0, and the labelled text report as @xmath1, where @xmath2, ..., y n ), n is the number of tokens in the generated report. we define the labelled data pool x label to contain image-report pairs, where x label ∩ x @xmath3. the whole data pool is x all := x label ∪ x pool . the model is parameterized by vector w, as follows: an acquisition function representing the query heuristic in the al setting is denoted as a(x, m). at each active learning iteration, we acquire the label of a batch q of b number of unlabelled instances from x pool and add to the labelled data pool x label using a(x, m). the updated labelled data pool x label is used to train the underlying model every iteration. this process iterates until a predefined budget b is depleted. sampling from the pool is determined by the acquisition function as follows : ## bidirectional contrastive active learning we identify the following limitations that existing al methods have when training models in a clinical setting. in clinical settings, data for healthy or common sicknesses is often seen, while unhealthy or rare sicknesses are rare in the population, leading to an imbalanced dataset. this leads to models trained on such datasets that can converge easily on the common cases, and have poor performance on rare but important cases. previous al methods have not yet addressed this problem, as they are not able to explicitly identify important cases within the dataset automatically. the original cal would identify two data points are neighbours if two data points have the same sickness, and if the model predicts differently for the two data points, they are considered as 'contrastive' and queried. such a heuristic cannot locate the positive (unhealthy) cases efficiently, because negative (healthy) neighbour pairs would outweigh the positive (unhealthy) neighbour pair in the population, leading to the sampling process suffering from class imbalance and queries too many negative instances. therefore, models trained using cal achieve a bad performance in clinical efficacy and recalling positive cases, as revealed by our experiments in ## experiment settings we assess bical and other established al methods' performance in training general multi-modal models to specify on the task of clinical report generation from chest x-ray images. in every active learning loop, the underlying model denoted as m, was fine-tuned twice on the labelled pool x label . subsequently, we evaluated the model on the test dataset using various nlg metrics. each experiment was run in 3 folds with different random seeds, each fold containing 10 active learning iterations, where 100 data points were queried per iteration, i.e. 1000 data points were queried in total. this choice of 1000 data points reflects realworld scenarios where active learning is applied when labelled data is not available. our goal was to examine the efficiency and performance of active learning methods under constrained labelling budgets in a medical setting. in real-world al sce-narios, labelling a large size of unlabelled data is often impractical due to the significant expertise labelling effort required. therefore, 1000 data points were deemed sufficient to assess the performance of the al methods in our focus while mimicking a real-world al situation. future work could explore varying the number of training examples (e.g., @xcite to understand further the impact of labelled data quantity on active learning strategies in training medical models. ## results and analysis we used two types of evaluation metrics: natural language generation (nlg) metrics and domainspecific (clinical efficacy) metrics. this provided a comprehensive evaluation of the generated reports in terms of general and domain-specific performance. for nlg metrics, we reported bleu @xcite and rouge @xcite scores at each active learning iteration. for clinical efficacy metrics, we used the chexpert @xcite model to label the generated and reference reports. we reported precision, recall, and f1 scores for the labeled categories of the generated and reference reports. this evaluation approach is widely used in chest x-ray clinical report gen-eration tasks @xcite @xcite . ## conclusion in this work, we present a study on the effectiveness of current active learning methods for domainspecific multi-modal learning, specifically on the task of clinical report generation from chest xray images. we identified the challenge of class imbalance in domain-specific active learning and addressed it by introducing bical, a new active learning technique. bical excelled in both nlg and domain-specific (clinical efficacy) metrics, notably outperforming baselines in clinical recall and f1-score. we found that existing al strategies demonstrate similar performance in nlg metrics for the task of clinical report generation from chest x-ray images. this may be due to the complexity of our task, which requires training the model to acquire clinical expertise to generate accurate and clinically sound reports. interestingly, our tests revealed that an al strategy's high performance in nlg metrics does not ensure equal success in domain-specific (clinical) performance, possibly due to the hallucination properties of language models. we hope this work provides valuable insights and can act as a starting point for researchers in the future on the task of active learning in multi-modal clinical tasks.