When training deep neural networks for medical image classification, obtaining a sufficient number of manually annotated images is often a significant challenge. We propose to use textual findings , which are routinely written by clinicians during manual image analysis, to help overcome this problem. The key idea is to use a contrastive loss to train image and text feature extractors to recognize if a given image-finding pair is a true match. The learned image feature extractor is then fine-tuned, in a transfer learning setting , for a supervised classification task . This approach makes it possible to train using large datasets because pairs of images and textual findings are widely availablein medical records. We evaluate our method on three datasets and find consistent performance improvements . The biggest gains are realized when fewer manually labeled examples are available. In some cases, our method achieves the same performance as the baseline even when using 70%DIFDELCMD < \\%%% %DIF < --98\\% fewer labeled examples.abstract 
A key challenge in training neural networks for a given medical imaging task is often the difficulty of obtaining a sufficient number of manually labeled examples. In contrast, textual imaging reports , which are often readily available in medical records, contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it in a transfer learning setting for a supervised task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. This approach can be applied to any task for which text-image pairs are readily available . We evaluate our method on three classification tasks and find consistent performance improvements %DIFDELCMD < \\%%% %DIF < --98\\% fewer labeled examples.abstract , reducing the need for labeled data by 67\%-98\%.
R
When training deep
A key challenge in training
R
medical image classification,
a given medical imaging task is often the difficulty of
R
annotated images is often a significant challenge. We propose to use textual findings
labeled examples. In contrast, textual imaging reports
R
routinely written by clinicians during manual image analysis, to help overcome this problem. The key idea is to use a contrastive loss to train image and text feature extractors to recognize if a given image-finding pair is a true match. The learned image feature extractor is then fine-tuned,
often readily available in medical records, contain rich but unstructured interpretations written by experts as part of standard clinical practice. We propose using these textual reports as a form of weak supervision to improve the image interpretation performance of a neural network without requiring additional manually labeled examples. We use an image-text matching task to train a feature extractor and then fine-tune it
D
,
R
classification task
task using a small labeled dataset. The end result is a neural network that automatically interprets imagery without requiring textual reports during inference. This approach can be applied to any task for which text-image pairs are readily available
D
This approach makes it possible to train using large datasets because pairs of images and textual findings are widely availablein medical records.
R
datasets
classification tasks
D
. The biggest gains are realized when fewer manually labeled examples are available. In some cases, our method achieves the same performance as the baseline even when using 70
A
, reducing the need for labeled data by 67\%-98\%.