  Multi-modal MR images are widely used in neuroimaging applications to provide
complementary information about the brain structures. Recent works have
suggested that multi-modal deep learning analysis can benefit from explicitly
disentangling anatomical (shape) and modality (appearance) representations from
the images. In this work, we challenge existing strategies by showing that they
do not naturally lead to representation disentanglement both in theory and in
practice. To address this issue, we propose a margin loss that regularizes the
similarity relationships of the representations across subjects and modalities.
To enable a robust training, we further introduce a modified conditional
convolution to design a single model for encoding images of all modalities.
Lastly, we propose a fusion function to combine the disentangled anatomical
representations as a set of modality-invariant features for downstream tasks.
We evaluate the proposed method on three multi-modal neuroimaging datasets.
Experiments show that our proposed method can achieve superior disentangled
representations compared to existing disentanglement strategies. Results also
indicate that the fused anatomical representation has great potential in the
downstream task of zero-dose PET reconstruction and brain tumor segmentation.
