  Recent advances in neuroscience have highlighted the effectiveness of
multi-modal medical data for investigating certain pathologies and
understanding human cognition. However, obtaining full sets of different
modalities is limited by various factors, such as long acquisition times, high
examination costs and artifact suppression. In addition, the complexity, high
dimensionality and heterogeneity of neuroimaging data remains another key
challenge in leveraging existing randomized scans effectively, as data of the
same modality is often measured differently by different machines. There is a
clear need to go beyond the traditional imaging-dependent process and
synthesize anatomically specific target-modality data from a source input. In
this paper, we propose to learn dedicated features that cross both intre- and
intra-modal variations using a novel CSC$\ell_4$Net. Through an initial
unification of intra-modal data in the feature maps and multivariate canonical
adaptation, CSC$\ell_4$Net facilitates feature-level mutual transformation. The
positive definite Riemannian manifold-penalized data fidelity term further
enables CSC$\ell_4$Net to reconstruct missing measurements according to
transformed features. Finally, the maximization $\ell_4$-norm boils down to a
computationally efficient optimization problem. Extensive experiments validate
the ability and robustness of our CSC$\ell_4$Net compared to the
state-of-the-art methods on multiple datasets.
