  Deep learning (DL) is the state-of-the-art methodology in various medical
image segmentation tasks. However, it requires relatively large amounts of
manually labeled training data, which may be infeasible to generate in some
applications. In addition, DL methods have relatively poor generalizability to
out-of-sample data. Multi-atlas segmentation (MAS), on the other hand, has
promising performance using limited amounts of training data and good
generalizability. A hybrid method that integrates the high accuracy of DL and
good generalizability of MAS is highly desired and could play an important role
in segmentation problems where manually labeled data is hard to generate. Most
of the prior work focuses on improving single components of MAS using DL rather
than directly optimizing the final segmentation accuracy via an end-to-end
pipeline. Only one study explored this idea in binary segmentation of 2D
images, but it remains unknown whether it generalizes well to multi-class 3D
segmentation problems. In this study, we propose a 3D end-to-end hybrid
pipeline, named deep label fusion (DLF), that takes advantage of the strengths
of MAS and DL. Experimental results demonstrate that DLF yields significant
improvements over conventional label fusion methods and U-Net, a direct DL
approach, in the context of segmenting medial temporal lobe subregions using 3T
T1-weighted and T2-weighted MRI. Further, when applied to an unseen similar
dataset acquired in 7T, DLF maintains its superior performance, which
demonstrates its good generalizability.
