  Transfer learning has gained attention in medical image analysis due to
limited annotated 3D medical datasets for training data-driven deep learning
models in the real world. Existing 3D-based methods have transferred the
pre-trained models to downstream tasks, which achieved promising results with
only a small number of training samples. However, they demand a massive amount
of parameters to train the model for 3D medical imaging. In this work, we
propose a novel transfer learning framework, called Medical Transformer, that
effectively models 3D volumetric images in the form of a sequence of 2D image
slices. To make a high-level representation in 3D-form empowering spatial
relations better, we take a multi-view approach that leverages plenty of
information from the three planes of 3D volume, while providing
parameter-efficient training. For building a source model generally applicable
to various tasks, we pre-train the model in a self-supervised learning manner
for masked encoding vector prediction as a proxy task, using a large-scale
normal, healthy brain magnetic resonance imaging (MRI) dataset. Our pre-trained
model is evaluated on three downstream tasks: (i) brain disease diagnosis, (ii)
brain age prediction, and (iii) brain tumor segmentation, which are actively
studied in brain MRI research. The experimental results show that our Medical
Transformer outperforms the state-of-the-art transfer learning methods,
efficiently reducing the number of parameters up to about 92% for
classification and
