Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
Abstract
Learning medical visual representations directly from paired radiology reports hasbecome an emerging topic in representation learning.
However, existing medicalimage-text joint learning methods are limited by instance or local supervision analysis, ignoring disease-level semantic correspondences.
In this paper, we present anovel M ulti-
G ranularity C ross-modal A lignment (MGCA) framework for generalized medical visual representation learning by harnessing the naturally exhibitedsemantic correspondences between medical image and radiology reports at threedifferent levels, i.e. , pathological region-level, instance-level, and disease-level.
Specifically, we first incorporate the instance-wise alignment module by maximizing the agreement between image-report pairs.
Further, for token-wise alignment,we introduce a bidirectional cross-attention strategy to explicitly learn the matching between fine-grained visual tokens and text tokens, followed by contrastivelearning to align them.
More important, to leverage the high-level inter-subject relationship semantic ( e.g. , disease) correspondences, we design a novel cross-modaldisease-level alignment paradigm to enforce the cross-modal cluster assignmentconsistency.
Extensive experimental results on seven downstream medical imagedatasets covering image classification, object detection, and semantic segmentationtasks demonstrate the stable and superior performance of our framework.
Introduction
In recent decades, deep learning techniques have significantly advanced medical image understandingwhen large-scale labeled datasets are available [42, 18, 12, 8, 40].
However, assembling suchlarge annotated data is expensive and time-consuming.
As an alternative, learning directly fromradiology reports accompanied by medical images becomes mainstream without any extra manualannotation which aims to learn general medical vision representations from physicians’ detailedmedical records and then transfer the learned representations to downstream tasks.
In the previousliterature, image-text contrastive learning has achieved huge success for a wide range of medicaldownstream tasks [55, 24, 56] by predicting which radiology report goes with which medical image.
Considering one limitation that pathologies usually occupy a small part of the whole image, Huang etal.
[24] proposed attention-based contrastive learning strategy to learn local representations, whichcan capture fine-grained semantics in medical images to facilitate localized downstream tasks, e.g. ,medical object detection and medical semantic segmentation.
As we can see in Figure 1, medical images and radiology reports naturally exhibit multi-granularitysemantic correspondence at different levels, e.g. , disease-level, instance-level, and pathological regionlevel.
However, existing image-text joint learning methods are limited to explore correspondencesupervision from only a part of levels [4, 21, 55, 24, 38], leading to inefficient usage of the valuablemedical image/report information and insufficient representation capability of models.
Thus, how to
[Figure]
i s
[Figure]
a s
[Figure]
- l
[Figure]
v e l
[Figure]
effectively leverage this intrinsic multi-granularity cross-modal correspondences between medicalimages and radiology reports from all three levels to enhance medical visual representation remainsan open question.
In this paper, we present a novel M ulti-
G ranularity C ross-modal A lignment (MGCA) frameworkto seamlessly leverage the multi-granular semantic correspondences between medical images andradiology reports for generalized medical visual representation learning.
Specifically, we incorporate well-known instance-wise alignment, fine-grained token-wise alignment, and disease-levelalignment via contrastive learning to enhance the generalizability of learned visual representations.
Particularly, for token-wise alignment, we introduce an effective bidirectional cross-attentionstrategy to explicitly learn the soft matching between local image-text representations, and thenadopt a cross-modal contrastive learning scheme to improve the sensitivity of local features.
Fordisease-level alignment, we leverage the inter-subject relationship correspondence between medical images and radiology reports by enforcing the cross-modal cluster assignment consistency.
By exploiting multi-granular cross-modal correspondences from three aspects, our MGCA framework has the ability to boost downstream tasks at both image and pixel levels, where only limited annotated data are required.
We pre-train our MGCA framework with a large-scale medicalimage and report dataset, i.e. , MIMIC-CXR, and then validate our learned medical visual representations with seven downstream datasets, belonging to three medical tasks: image classification, object detection, and semantic segmentation.
Experimental results demonstrate that ourmodel achieves stable superior transfer performance even training with 1% of training data, whencompared with existing state-of-the-art medical image-text pre-training methods.
Our code is inhttps://anonymous.4open.science/r/Multimodal-contrastive-learning-4A7A .
Related Work
Learning Medical Visual Representations from Reports There are two mainstream paradigmsto learn medical visual representations from report supervision.
The first is to extract disease labelsfrom radiology reports via human-designed rules [26, 28, 50] and then pre-train an image modelfor downstream tasks.
However, defining such rules requires a lot of human labor and expertknowledge.
Also, the trained models may be suboptimal, as the extracted labels are usually noisy [50].
The second focuses on leveraging vision-language contrastive learning to pre-train image and textencoders in an unsupervised manner [4, 21, 55, 24, 38].
Supervised by naturally occurring ofmedical images and radiology reports, these methods demonstrate remarkable performance in variousmedical image downstream tasks ( e.g. , image classification [55, 24], semantic segmentation [24],image-image retrieval [55], image-text retrieval [55, 24]).
However, these methods only utilizepartial correspondence supervision of cross-modal semantics, leading to suboptimal performance fordownstream tasks.
Contrastive Learning Contrastive learning [20, 4, 21] aims to learn an embedding space wherepositive instances stay close to each other, while negative pairs are far apart.
One of the key technique
[Figure]
is to find effective positive and negative pairs.
To improve the efficiency of contrastive learning,some recent works proposed to predict one view’s representations from another view [4, 17].
Werefer the readers to the survey [29, 27] for more details.
Recently, some prototypical contrastlearning approaches were proposed to exploit the prototype-level semantics in the dataset.
Forexample, [32, 49, 19] proposed contrasting instance features with its paired prototype features, while[34, 2] proposed clustering-based methods to contrast prototype-prototype pairs in embedding space.
Different from these methods working on a single modality, we propose a cross-modal prototypealignment module to exploit the cross-modal prototype-level semantic consistency.
Unsupervised Learning for Dense Prediction Learning fine-grained semantic correspondence isessential for dense image prediction tasks, e.g. , object detection [13, 35], semantic segmentation [5],etc. .
Recent studies have proposed various paradigms to tackle this problem [30, 25, 33, 10].
However,most of these methods require a pre-trained object detector to generate proposals of interested objects.
Dense contrastive learning is thus proposed to learn fine-grained visual representations without apre-trained object detector [53, 52, 51, 3].
By optimizing a contrastive loss at the pixel-level (orregion-level) between two views of input images, these methods effectively learn the correspondencebetween local features, which significantly benefits dense prediction downstream tasks.
Method
3.1 Overview
We aim to learn a generalized medical image representations from radiology reports to benefit variousdownstream medical image recognition tasks where annotated data is limited.
Given a trainingset of N image-report pairs D = { ( x v, 1 , x t, 1 ) , ( x v, 2 , x t, 2 ) , ..., ( x v,N , x t,N ) } , we utilize an imageencoder f v ( e.g. , ViT [11]) and a text encoder f t ( e.g. , BERT [9]) to map D into the latent space
E = { ( v 1 , t 1 ) , ( v 2 , t 2 ) , ..., ( v N , t N ) } , where v i = f v ( x v,i ) , t i =
f t ( x t,i ) .
In detail, for the i -thimage-report pair ( x v,i , x t,i ) , the image encoder generates a sequence of encoded visual tokens
R i = { r 1 i , r 2 i , ..., r Si } and a global image representation v i .
Similarly, the text encoder generates asequence of encoded text tokens Z i = { z 1 i , z 2 i , ..., z Li } and a global report representation t i .
Here, Sand L denote the total number of visual tokens and text tokens, respectively.
As illustrated in Figure 2, we design a novel multi-granularity cross-modal alignment framework forrepresentation learning by exploiting the naturally exhibited multi-granularity cross-modal correspondences, i.e. , disease (prototype)-level, instance-level, and pathology region (token)-level, betweenimages and reports.
Specifically, we incorporate an instance-wise image-text alignment (ITA) moduleto retain the cross-modal smoothness by maximizing the agreement between true image-report pairs( v i , t i ) versus random pairs.
Meanwhile, to harness the benefits of naturally existing fine-grained cor- respondence between visual and text tokens, we also introduce a bidirectional cross-attention-basedtoken-wise alignment (CTA) module to learn the soft matching between the visual token sequence
R i and text token sequence Z i , as well as align them via the contrastive learning.
More important,we design a novel cross-modal prototype alignment (CPA) module to benefit the high-level semantic( e.g. , disease) understanding by capturing the inter-subject relationship correspondence between twomodalities.
3.2 Multi-granularity Cross-modal Alignment
Instance-wise Cross-modal Alignment As the core workhorses in our framework, we incorporatean instance-wise Image-Text Alignment (ITA) module to encourage the framework to map correctimage-report pairs nearby in the latent space, while mapping random pairs far apart.
Specifically,following common practice [4], we first use two non-linear projection layers ( g v and g t ) to transformv i and t i into normalized lower-dimensional embeddings ˜ v i ∈ R d and ˜ t i ∈ R d , respectively.
Then,the cosine similarity of i -th image-report pair is denoted as:
[Equation]
˜
[Equation]
[Equation]
For i -th image-report pair ( x v,i , x t,i ) in a minibatch, we regard two modality data as queries and keysalternatively to learn the correct image-report pairings.
This results in two symmetric temperaturenormalized InfoNCE [39] losses (image-to-text contrastive loss and text-to-image contrastive loss) tomaximally preserve the mutual information between the true pairs in latent space:
[Equation]
where B is the batch size and τ 1 is the instance-level temperature hyperparameter.
The overallobjective of our instance-wise cross-modal alignment is the average of the two losses:
[Equation]
where N is the total number of image-report pairs.
Token-wise Cross-modal Alignment Fine-grained information is more significant in medicalfield: pathologies only occupy a small portion of the whole image and only a few disease tags in areport depict the crucial medical condition.
Considering that these important subtle clues are likelyto be ignored when optimizing the global instance-wise representations, we introduce an effectivebidirectional Cross-attention-based Token-wise Alignment (CTA) module to explicitly match andalign the cross-modal local presentations between medical images and radiology reports.
Specifically, for the i -th image-report pair ( x v,i , x t,i ) , the generated visual and text token embeddingswill first be projected into normalized lower-dimensional embeddings, which results in ˜ R i =
{ ˜ r 1 i , ˜ r 2 i , ..., ˜ r Si } and ˜
Z i = {
˜ z 1 i , ˜ z
2 i , ..., ˜ z Li } , where ˜ r i ∈ R d , ˜ z i ∈ R d .
In order to conduct the tokenwise alignment, we need to find the matching between visual and text tokens.
Instead of directlycomputing the cosine similarity of different tokens [24], we propose to calculate the soft matchingbetween generated visual and text tokens with the cross-attention mechanism [47].
Formally, for thej -th visual token embedding ˜ r ji in i -th image-report pair, we let ˜ r ji attend to all text token embeddingsin
˜ Z i and then calculate its corresponded cross-modal text embedding o ji
[Equation]
[Equation]
where Q ∈ R d × d , K ∈ R d × d , V ∈ R
d × d are learnable matrices.
After that, we adopt a Local
Image-to-text Alignment (LIA) loss L LIA to pull ˜ r ji close to its cross-modal text embedding o jibut push ˜
r ji away from other cross-modal text embeddings, which maximizes the lower bound onlocal cross-modal mutual information within each image-report pair [39].
Considering that differentvisual tokens have various importance ( e.g. , visual tokens containing pathologies are obviously more important than those with irrelevant information), we further assign a weight w ji to the j -th visualtoken when calculating the LIA loss.
So the LIA loss can be formulated as:
[Equation]
where τ 2 is the token-level temperature hyperparameter.
Similar with the instance-level alignment,we also employ two symmetric InfoNCE losses by taking the visual token embedding ˜ r ji and itscross-modal text embedding o ji as queries, respectively.
Note that we set w ji as the last-layer attentionweight from j -th visual token to [CLS] token averaged across multiple heads.
Similarly, for j -th texttoken embedding ˜ z ji in i -th image-report pair, we also calculate a cross-modal image embedding ˆ o jiwith the same manner and construct a Local Text-to-image alignment ( LTA ) loss L LTA by contrasting˜ z ij with ˆ o ji .
The final objective of our CTA module is the combination of LIA and LTA losses:
[Equation]
Discussion: Note that our token-wise alignment is different from the local contrastive loss in [24].
Ourmodule explicitly contrasts similarities between local tokens to maximize their mutual informationand calculates per-token InfoNCE, whereas the local contrastive loss in [24] contrasts aggregatedinstance-level similarities and still calculates per-instance InfoNCE.
Disease-level Cross-modal Alignment Both ITA and CTA treat two samples as a negative pair aslong as they are from different instances.
Many pairs sharing the similar high-level semantics ( e.g. ,disease) are undesirably pushed apart in the embedding space.
Therefore, we design a novel Crossmodal Prototype Alignment (CPA) module to harness the cross-modal inter-subject correspondencesbetween medical images and reports.
For each image-report embedding pair (˜ v i , ˜ t i ) in (1), we employ the iterative Sinkhorn-Knoppclustering algorithm [7] to acquire two soft cluster assignment codes q v,
i ∈
R K and q t,i ∈ R K , byindividually assigning ˜ v i and ˜ t i into K clusters.
Meanwhile, we also pre-define K trainable crossmodal prototypes as C = { c 1 , ..., c K } , where c k ∈ R d .
After that, we calculate the visual softmaxprobability p v,i ∈
R K of the cosine similarities between ˜ v i and all cross-modal prototypes in C ,and the text softmax probability p t,i ∈
R K of the cosine similarities between ˜ t i and all cross-modalprototypes in C ,
[Equation]
where τ 3 is the prototype-level temperature parameter and ( k ) indicates the k -th element of the vector.
The cross-modal disease-level ( i.e. , prototype) alignment is achieved by conducting cross-modalprediction and optimizing the following two cross-entropy losses:
[Equation]
Here, the cross-modal prediction is implemented by taking the soft text cluster assignment code q t,ias "pseudo-label" to train the image representation ˜ v i and taking the soft image cluster assignmentcode q v,i as "pseudo-label" to train the report representation ˜ t i .
Finally, the overall CPA loss is theaverage of two prediction losses over all the image-report pairs:
[Equation]
3.3 Overall Objective
We train our MGCA framework with jointly optimizing the three cross-modal alignment modules,encouraging the network to learn discriminative and generalizable medical image representation.
Theoverall training objective can be represented as:
[Equation]
[Equation]
where λ 1 , λ 2 , and λ 3 are hyperparameters to balance three-level cross-modal alignments.
Experiments
We pre-train our MGCA framework on a large-scale medical image-report dataset and then evaluatethe effectiveness of learned medical image representations on seven datasets from three importantdownstream tasks in medical imaging.
In the following subsections, we first introduce the experimental setup of pre-training in Section 4.1 and three downstream tasks in Section 4.2.
Then, wecompare our proposed framework with state-of-the-art medical image-text pre-training methods andshow the comparison results in Section 4.3-4.5.
Finally, we analyze our framework in Section 4.6.
More analysis results can be found in the Appendix.
4.1 Pre-Training Setup
Dataset We pre
-train our MGCA framework on the JPG version of MIMIC-CXR 2.0.0 dataset [28].
We follow [55] to preprocess the dataset.
We remove all lateral views from the dataset, as thedownstream datasets only contain frontal-view chest images.
Also, we extract the impression andfinding sections from free-text reports to obtain detailed descriptions of medical diseases and removereports which are empty or have less than 3 tokens, resulting in roughly 217 k image-text pairs.
Implementation details Following [24], we use BioClinicalBERT [1] as the text encoder.
Wechoose ViT-B/16 [11] as the image encoder backbone by default for unified modal architecture design.
It is worth noting that our framework is model-agnostic to the image encoder backbone and we alsoreport the results with ResNet50 [22] as image encoder backbone for fair comparison with othermethods.
We train our framework 50 epochs on 2 pieces of RTX 3090 GPUs with batch size of144 .
The optimizer is AdamW [37] with learning rate of 2 e − 5 and weight decay of 0 . 05 .
We usea linear warmup with cosine annealing scheduler [36].
We initialize learning rate as 1 e − 8 andwarmup epoch as 20 .
Following the practice in contrastive learning [4, 21], the dimension d = 128and the temperature hyperparameters are τ 1 = 0 .
1 , τ 2 = 0 .
07 , τ 3 = 0 .
The number of prototypes is
K = 500 .
We set λ 1 = 1 , λ 2 = 1 , λ 3 = 1 .
More pre-training details can be found in the Appendix.
4.2 Downstream Tasks and Experimental Setup
Medical Image Classification We conduct medical image classification on three representativedatasets: (1) CheXpert [26], which contains 191 , 229 frontal chest radiographs.
The task is toclassify each image into 5 individual binary labels: atelectasis , cardiomegaly , consolidation , edema ,and pleural effusion .
Following [55, 24], we hold out the expert-labeled validation set as test dataand randomly select 5 , 000 radiographs from training data for validation.
(2) RSNA Pneumonia [43].
We use the stage 2 version, which contains around 29 , 700 frontal view chest radiographs.
Thetask is a binary classification, i.e. , classifying each chest image into normal or pneumothoraxpositive .
Following [24], we manually split the dataset into training, validation, and test set with70% / 15% / 15% ratio.
(3) COVIDx [48], which contains over 30 k CXR images from a multinationalcohort of over 16 , 600 patients.
This dataset contains 16 , 490 positive COVID-19 images from over2 , 800 patients.
We use the latest version 6 of this dataset.
The task is a three-class classification, i.e. ,classifying each radiograph into COVID-19 , non-COVID pneumonia or normal .
We use the originalvalidation dataset as test data and manually split 10% of original training set for validation.
Following the previous work [24], we use the Linear Classification setting to evaluate the transferability of our pre-trained image encoder, i.e. , freezing the pre-trained ViT/ResNet-50 image encoder andonly training a randomly initialized linear classification head for the downstream classification task.
Also, we evaluate our model with 1% , 10% , and 100% training data on each classification dataset tofurther verify the data efficiency of our method.
We report area under the ROC curve (AUROC) on
CheXpert and RSNA and acuracy (ACC) on COVIDx-v6 as the evaluation metric following [55].
Medical Object Detection We evaluate the localized performance of pre-trained image encoderon two object detection tasks: (1) RNSA Pneumonia [43] contains 29700 frontal view radiograph.
The task is to predict bounding boxes indicating evidence of pneumonia.
We randomly split theoriginal training set into 16 , 010 / 5 , 337 / 5 , 337 for training/validation/testing.
(2) Object CXR [23]contains 9 , 000 frontal-view chest X-rays with detection targets for foreign objects.
We use theoriginal development set as test set ( 1 , 000 ) and randomly split the original training set into training( 6 , 400 ) and validation ( 1 , 600 ) sets. [Table] pre
[Table]
on MIMIC-CXR
Caption-Transformer [6]
[Table]
Caption
[Table]
LSTM [
[Table]
[Table]
Contrastive-Binary [45][44]
[Table]
[Table]
GLoRIA-MIMIC [24]
[Table]
[Table]
We evaluate the detection performance by YOLOv3 [41] frozen setting, i
[Table]
e. , using the pre-trained
ResNet-50 image encoder as a frozen backbone of a YOLOv3 model and only fine-tuning the nonbackbone layers.
Similarly, we fine-tune the model by 1% , 10% and 100% training data to evaluatethe data efficiency.
Mean Average Precisions (mAP) are reported as evaluation metric with IOUthresholds 0 .
Medical Semantic Segmentation We also evaluate the performance of our framework for medicalsemantic segmentation on SIIM and RNSA datasets: (1) SIIM Pneumothorax [15] dataset contains12047 chest radiographs with manually annotated segmentation mask of pneumothorax.
Following [24], train/validation/test split respectively constitutes 70% / 30% / 30% of original dataset.
RNSA Pneumonia [43] is with the same split protocol as object detection task.
We convert objectdetection ground truths into masks for semantic segmentation.
Following [24], we evaluate the segmentation performance by U-Net [42] fine-tuning protocol.
Weuse the pre-trained ResNet-50 image encoder as a frozen encoder backbone of U-Net and train thedecoder portion using 1% , 10% and 100% training data.
Dice scores are reported to evaluate thesegmentation performance.
The other downstream experimental setup can be found in the Appendix.
4.3 Results
Results on Medical Image Classification Table 1 reports the results on three classification tasks.
The results of other methods on CheXpert and RSNA are from original papers 1 .
According to the opacities pneumothorax cardiomegaly atelectasis
Atelectasis Cardiomegaly Edema
Fracture Pleural Effusion Pneumonia
Pneumothorax No Finding
[Table]
pre-training dataset, we group the existing pre-training methods into two categories : pre-trainedon CheXpert and pre-trained on MIMIC-CXR .
As GLoRIA only reports results with pre-trainingon CheXpert dataset, we also reimplement their method with pre-training on MIMIC-CXR dataset(GLoRIA-MIMIC) for a fair comparison.
It is observed that our MGCA with ViT-B/16 backboneshows the best performance in all nine settings, outperforming state-of-the-art GLoRIA [24] and
ConVIRT
With the same ResNet-50 image encoder backbone, our framework also achievessecond-best performance on four settings and competitive performance on the remaining five settings,showing the effectiveness of our framework.
When fine-tuning with 1% proportion of data, our
MGCA with ViT-B/16 backbone outperforms GLoRIA-MIMIC with 1 . 7% AUROC on CheXpert,2 .
1% AUROC on RSNA dataset and 8 .
3% ACC on COVIDx dataset, showing larger improvementthan other methods and also indicating the data efficiency of our method.
Results on Medical Object Detection Table 2 reports the object detection performance on RSNA and
Object CXR datasets.
All methods adopt the same ResNet-50-YOLOv3 architecture.
It is observedthat under each setting, our MGCA outperforms ConVIRT, GLoRIA, and GLoRIA-MIMIC by alarge margin.
Importantly, our model shows superior detection performance when fine-tuning on 1%training data, indicating that multi-granularity semantic alignment benefits the image encoder to learnmore discriminative localized representations.
Results on Medical Semantic Segmentation Table 3 shows the semantic segmentation performanceon SIIM and RSNA datasets with the same ResNet50-U-Net architecture.
Compared with GLoRIA,
GLoRIA-MIMIC, and ConVIRT
, our MGCA shows higher dice scores on five over six settings.
When training with 1% portion of data, our MGCA achieves 12 .
3% and 2 .
7% Dice improvementthan GLoRIA-MIMIC on SIIM and RSNA segmentation tasks, respectively.
This comparison furthervalidates the data efficiency of our method when transferring into dense prediction tasks.
4.4 Analysis of Our Framework
Visualization To better understand the behaviour of our MGCA framework, we visualize thelearned local correspondence of radiographs and medical reports in Figure 3.
Our MGCA learnsmeaningful local correspondence between visual tokens and text tokens, which is helpful for the localdiscriminative feature learning.
Moreover, we select 1600 medical images, each with one excluded
[Table] [Table]
abnormality, and present t-SNE plots [46] in Figure 4 to visualize image embeddings.
The colorsrepresent the ground truth and cluster assignment in left and right sub-figures.
It is observed that ourmulti-modal prototypes can learn reasonable disease-level semantic information.
Ablation Study of Component Design Table 4 shows the ablation study results of medical imageclassification on CheXpert and RSNA datasets with ViT-B/16 as the image encoder backbone.
It isobserved that CTA and CPA modules can both improve the classification performance, indicatingthat token-level alignment and prototype-level alignment facilitate the image encoder to learn moregeneralizable representations for downstream tasks.
When combining CTA and CPA, we can obtainfurther improvement on all datasets, indicating that the benefits of CTA and CPA are complementary.
Results of Natural Vision-Language Pre-trained Model Table 5 shows the results of fine-tuning thestate-of-the-art natural Vision-Language Pre-trained (VLP) model BLIP [31], which is pre-trained on14M image-text pairs.
Due to the large domain discrepancy between natural image-text and medicalimage-text, directly transferring the pre-trained BLIP model to the downstream medical image tasksleads to inferior performance.
This comparison indicates that pre-training on medical image-textdatasets is necessary for capturing useful medical prior knowledge.
Discussion and Conclusion
This work presents MGCA, a multi-granularity cross-modal alignment framework for learning generalized medical visual representations from free-text radiology reports.
By harnessing the naturallyexhibited multi-granularity semantic correspondences across medical images and reports, our framework can learn generalized and discriminative medical image presentations for versatile downstreamtasks to reduce the annotation burden.
Extensive experimental results on seven downstream datasetsdemonstrate that our framework achieves substantial performance with limited annotated data.
Limitations and Future Work
As our work mainly focuses on medical visual representation learning,we did not conduct experiments on image-image or image-text retrieval downstream tasks, whichcan be regarded as a limitation of our work.
Our current framework learns the multi-granularitycross-modal alignment in parallel.
In future work, we would like to explore how to leverage themulti-granularity correspondence in a holistic manner.
Moreover, this paper mainly investigatesthe discrimination-based image-text pre-training.
We also plan to extend our framework as theintegration of discrimination-based and generation-based pre-training methods for medical imageand text learning.
Social Impacts Our MGCA provides a promising solution to automatically diagnose abnormality ofchest X-rays with limited annotated data, which can assist in reducing the workload of radiologistsand promote the health in poor area.
On the other hand, medical data ( e.g. , chest X-rays, radiologyreports etc. ) may contain unintended private information or harmful texts, and we highly recommendusers conduct a careful analysis of data before employing our model into practical applications.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See the second paragraph in Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See thelast paragraph in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
Our work
mainly involves empirical contributions.
(b) Did you include complete proofs of all theoretical results?
[N/A]
Our work mainly
involves empirical contributions.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See the Section4.1 for pre-training dataset and model details.
More experimental details is includedin the supplementary material.
Our code is in the supplementary material and will bereleased later.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.2 for data splits.
Most of dataset details arefollowing previous work.(c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] See the second paragraph of
Section 4.1.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all usedpublic datasets in Section 4.1 and 4.2.
(b) Did you mention the license of the assets?
[Yes] All dataset are publicly available.
[Table]
They are under a non-commercial license.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] [Table] Our code and pre-trained model will be released later for the assets.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We are using publicly available datasets for all experiments.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A] We are using publicly available datasets for
[Table]
all experiments.
No personally identifiable information involved.
[Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
No human subjects involved in our experiments.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review [Table] Board (IRB) approvals, if applicable?
[N/A]
No human subjects involved in our
[Table]
experiments.
(c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
No human subjects involved in our experi- [Table] ments. [Table]