  Pre-trained models, e.g., from ImageNet, have proven to be effective in
boosting the performance of many downstream applications. It is too demanding
to acquire large-scale annotations to build such models for medical imaging.
Meanwhile, there are numerous clinical data (in the form of images and text
reports) stored in the hospital information systems. The paired image-text data
from the same patient study could be utilized for the pre-training task in a
weakly supervised manner. However, the integrity, accessibility, and amount of
such raw data vary across different institutes, e.g., paired vs. unpaired
(image-only or text-only). In this work, we introduce an image-text
pre-training framework that can learn from these raw data with mixed data
inputs, i.e., paired image-text data, a mixture of paired and unpaired data.
The unpaired data can be sourced from one or multiple institutes (e.g., images
from one institute coupled with texts from another). Specifically, we propose a
transformer-based training framework for jointly learning the representation of
both the image and text data. In addition to the existing masked language
modeling, multi-scale masked vision modeling is introduced as a self-supervised
training task for image patch regeneration. We not only demonstrate the
feasibility of pre-training across mixed data inputs but also illustrate the
benefits of adopting such pre-trained models in 3 chest X-ray applications,
i.e., classification, retrieval, and image regeneration. Superior results are
reported in comparison to prior art using MIMIC-CXR, NIH14-CXR, and OpenI-CXR
datasets.
