Zero-Shot Physics-Guided Deep Learning for Subject-Speciﬁc MRI Reconstruction
Abstract —Physics-guided deep learning (PG-DL) has emerged as a powerful tool for accelerated MRI reconstruction, while often necessitating a database of fully-sampled measure- ments for training.
Recent self-supervised and unsupervised learning approaches enable training without fully-sampled data.
However, a database of undersampled measurements may not be available in many scenarios, especially for scans involving contrast or recently developed sequences, necessitating new methodology for subject-speciﬁc PG-DL reconstructions.
A main challenge for developing subject-speciﬁc PG-DL methods is the large number of parameters, making it prone to over- ﬁtting.
Moreover, database-trained models may not generalize well to unseen measurements that differ in terms of SNR, image contrast, sampling pattern, and anatomy.
In this work, we propose a zero-shot self-supervised learning approach to perform subject-speciﬁc PG-DL reconstruction to tackle these issues.
The proposed approach splits available measurements for each scan into three disjoint sets.
Two of these sets are used to enforce data consistency and deﬁne loss during training, while the last set is used to establish an early stopping criterion.
In the presence of models pre-trained on a database, we show that the proposed approach can be adapted as subjectspeciﬁc ﬁne-tuning via transfer learning to further improve reconstruction quality.
I. INTRODUCTION
Lengthy acquisition times in MRI remain a challenge, necessitating the use of accelerated imaging techniques.
Conventional acceleration methods, such as parallel imaging [1] or compressed sensing [2] are commonly used, but the acceleration rates they achieve may be limited due to noise ampliﬁcation and/or residual aliasing artifacts in the reconstruction.
Recently, deep learning (DL) based methods have emerged for accelerating MRI further [3], [4].
Among
DL methods, physics-guided DL (PG-DL) approaches have received attention due to their robustness and improved reconstruction quality [3]–[6], while explicitly incorporating the physics of the encoding matrix to the neural network via a procedure known as algorithm unrolling [7].
Despite its effectiveness as an accelerated MRI reconstruction strategy, a number of challenges remain for DL methods.
Most training paradigms require large databases of fullysampled data, which may not be available.
Transfer learning has been proposed to enable training with smaller fullysampled datasets [8], while unsupervised training approaches [9], [10] have been proposed to tackle challenges associated with not having fully-sampled data in some applications.
In particular, self-supervised learning via data undersampling (SSDU) has shown that similar reconstructions to supervised
PG-DL can be achieved while training only on a database of undersampled measurements [9].
Although the aforementioned unsupervised learning strategies enable training from undersampled data, they still require a database for training in order to learn the large number of parameters for the neural network.
However, in some MRI applications involving time-varying physiological processes, dynamic information such as time courses of signal changes, contrast-related uptake or breathing patterns may differ substantially between subjects, making it difﬁcult to generate high-quality databases of sufﬁcient size for the aforementioned strategies.
Furthermore, database training in general brings along concerns about generalization [11].
Particularly, for MRI reconstruction, this may translate to training and test
dataset mismatches on image contrast, sampling pattern, SNR, vendor, and anatomy.
For instance, the fastMRI transfer track challenge shows that the performance of pretrained models degrades due to distribution shift or changes in acquisition parameters at inference time [12].
Moreover, bias due to datasets lacking examples of rare and/or subtle pathologies increases the risk of generalization failure [11], [13].
A recent work [14] proposed to ﬁne-tune a pre-trained neural network for MRI reconstruction in a subject-speciﬁc manner using transfer learning.
However, it required a pre-trained model and lacked an early stopping criterion to avoid overﬁtting [14], [15].
In this work, we develop a zero-shot self-supervised learning (ZS-SSL) strategy to enable subject-speciﬁc training of PG-DL methods and tackle the challenges associated with database training.
We propose a holdout self-supervision method, where the acquired data is split into at least three disjoint sets, which are respectively used only in the PGDL neural network, to deﬁne the training loss, and to establish an early stopping strategy to avoid overﬁtting.
In cases where a database-trained network may be available, we also propose a subject-speciﬁc ﬁne-tuning approach, which combines the concept of transfer learning and our zero-shot self-supervised learning to achieve improved reconstruction quality and further reduce computational complexity.
II. MATERIALS AND METHODS A. Problem Formulation and PG-DL Reconstruction
Let y Ω be the undersampled noisy data from a multi-coil
MRI system, where Ω denotes the under-sampling pattern, and E Ω : C M × N → C P be the forward encoding operator that includes a partial Fourier matrix sampling the locations in Ω and coil sensitivites [1].
The inverse problem for accelerated MRI reconstruction is given as
[Equation]
where R ( · ) is a regularizer.
Using standard optimization methods [7], Eq.
(1) can be decoupled into two subproblems, one involving a regularization step and the other forming a data consistency (DC) step.
In PG-DL methods, such iterative optimization algorithms are unrolled for a ﬁxed number of iterations [7].
Neural networks are used to implicitly solve the regularization subproblem, while the DC sub-problem is solved by conventional linear approaches [4].
B. Supervised Learning for PG-DL Reconstruction
In supervised PG-DL, training is performed using fullysampled data.
Let y i ref be fully-sampled k-space for subject i and f ( y i Ω , E i Ω ; θ ) be the output of the unrolled network for sub-sampled k-space y i Ω , where the network is parameterized by θ .
End-to-end training minimizes [4], [9]
[Equation]
where N is the number of samples in the training database,
E i full is the fully-sampled encoding operator that transform network output to k-space and L ( · , · ) is a loss function.
C. Self-Supervised Learning for PG-DL Reconstruction
Unlike supervised learning, SSDU performs training without fully-sampled data by only utilizing acquired measurements [9].
In SSDU, the acquired undersampled data indices, Ω are split into two disjoint sets Θ and Λ as Ω = Θ ∪ Λ .
Here, Θ is the set of k-space locations that is used within the
DC units of the PG-DL network during training, while Λ is a set of k-space locations used in the loss function.
End-to-end training is performed using a loss function of the form
[Equation]
Recently, this method was also extended to a multi-mask setting, which was shown to substantially improve the performance at the cost of increased training time [16].
D. Proposed Zero-Shot Learning for PG-DL Reconstruction
SSDU uses 2-way partitioning of acquired measurements for training and deﬁning the loss.
The use of this 2-way partitioning for subject-speciﬁc learning was explored in our previous work [14].
However, it was observed that training had to be stopped early to avoid overﬁtting.
This is similar to other single-image learning strategies, such as deep image prior (DIP) [15].
In both cases, it is difﬁcult to develop a stopping strategy with a 2-way partition.
Thus, in this work, we propose a zero-shot self-supervised learning (ZS-SSL) approach for subject-speciﬁc reconstruction with a welldeﬁned early stopping criterion.
1) ZS-SSL Formulation and Training : ZS-SSL extends the partitioning in SSDU to a 3-way split, which is reminiscent of the machine learning framework of using a validation set in addition to testing and training sets for hyperparameter tuning or for regularization by early stopping.
We deﬁne the following partition for Ω :
[Equation]
where (cid:116) denotes a disjoint union, i.e. Θ , Λ and Γ are pairwise disjoint (Figure 1).
Similar to Section II-C, Θ is used in the DC units of the unrolled network, and Λ is used to deﬁne the loss in k-space.
The third partition Γ is a set of acquired k-space indices that are set aside for deﬁning a kspace validation loss.
The proposed zero-shot learning for MRI reconstruction uses two types of losses at each epoch during training.
The training loss for ZS-SSL can be written as:
[Equation]
This is now supplemented by a new k-space validation loss, which tests the generalization performance of the trained network on the k-space validation partition Γ .
For the l th epoch, where the learned network weights are speciﬁed by θ ( l ) , this loss is given by:
[Equation]
Note that in Eq.
(6), the network output is calculated by applying the DC units on Ω \ Γ =
Θ ∪ Λ , i.e. all the acquired points outside of Γ to get a more accurate representation of its generalizability performance.
The key idea is that while the training loss will decrease over epochs, the kspace validation loss will start increasing once overﬁtting starts to be observed.
Thus, we monitor the loss in Eq.
(6) during training to deﬁne an early stopping criterion to avoid overﬁtting.
Let L represent the epoch in which the training needs to be stopped.
Then at inference time, the network output is calculated as f ( y Ω , E Ω ; θ ( L ) ) , i.e. all acquired points are used to calculate the ﬁnal network output [9].
2) Multi-Mask Augmentation for ZS-SSL Training : ZSSSL can be extended to multi-mask setting for improved performance [16], by ﬁxing a k-space validation partition Γ ⊂ Ω , and performing the multi-masking on Ω \ Γ .
Formally, Ω \ Γ is partitioned K times such that
[Equation]
where Λ k , Θ k and Γ are pairwise disjoint.
This leads to the following training loss
[Equation]
with the k-space validation loss
[Equation]
Note that this training strategy has a K -fold longer training time compared to that of Eq.
3) ZS-SSL Scan Speciﬁc Fine-Tuning via Transfer Learning: DL approaches are typically trained on large databases in a supervised manner.
Similarly, for smaller datasets, a network pre-trained on a large database can be re-trained with supervision via transfer learning [8].
In both cases, the learned models are ﬁxed during inference and used to reconstruct undersampled measurements that may have different acquisition parameters than the pre-trained model.
Hence, the models may not generalize well during inference, leading to sub-optimal reconstructions [13].
In the presence of pre-trained models, proposed ZS-SSL can be used to ﬁnetune this model using subject-speciﬁc transfer learning.
This approach, referred to as ZS-SSL-TL uses the network pretrained on a different database as the starting point and optimizes the network parameters for the scan of interest using the objective function in Eq. 8.
Furthermore, using pre-trained model weights as starting point leads to faster convergence and reduced computational time.
E. Knee MRI Datasets
Fully-sampled coronal proton density (Coronal PD) knee datasets [17] were obtained from the fastMRI database.
Relevant imaging parameters: matrix size = 320 × 368 , in-plane resolution = 0 .
49 × 0 .
44 mm 2 .
Fully-sampled knee datasets were retrospectively subsampled with an acceleration rate of 4 by keeping 24 lines of autocalibrated signal (ACS) from center of k-space using a uniform undersampling pattern.
F. Implementation Details
All PG-DL approaches were trained end-to-end usingunrolled iterations.
Conjugate gradient method and a ResNet structure were employed in DC and regularizer units of the unrolled network, respectively [9].
Coil sensitivity maps were generated from central 24 × 24 ACS using ESPIRiT. End-toend training was performed with a normalized (cid:96) 1 - (cid:96) 2 loss (Adam optimizer, learning rate= 5 · 10 − 4 , batch size=1)
The stopping criterion for the proposed ZS-SSL was investigated on slices from the knee dataset.
Γ was selected from the acquired measurements Ω using a uniformly random selection with | Γ | / | Ω | = 0 .
The remaining acquired measurements Ω \ Γ were retrospectively split into disjoint 2tuples multiple times based on uniformly random selection with the ratio ρ = | Λ k | / | Ω \ Γ | = 0 .
4 for ∀ k ∈ { 1 , . . .
, K } .
Network parameters for ZS-SSL were initialized randomly with a normal distribution.
The supervised PG-DL network was used for comparison purposes, as well as the starting point for ZS-SSL-TL.
CG-SENSE was also implemented for further comparisons
III. RESULTS
Figure 2a shows representative subject-speciﬁc training and validation loss curves at R=4 of ZS-SSL for K ∈ { 1 , 10 , 25 , 50 } .
As expected, training loss decreases with increasing epochs for all K .
Validation loss for K =decreases without showing a clear breaking point for stopping.
For K > 1 , validation loss forms an L-curve, and the breaking point of the L-curve is used as the stopping criterion.
Figure 2b shows reconstructions corresponding to the K values using the proposed stopping criterion.
For
K > 1 , good reconstruction quality is observed with no visible residual aliasing artifacts.
For K = 1 , without a clear breaking point for stopping criterion, reconstructions show lower visual and quantitative quality.
K
= 10 is used for the remainder of the study.
Figure 3a and b show loss curves and reconstruction results on a representative slice with and without transfer learning.
As expected, ZS-SSL-TL converges faster compared to ZS-SSL, substantially reducing the total training time.
Both ZS-SSL and ZS-SSL-TL remove residual artifacts, while the latter shows visually and quantitatively improved reconstruction performance.
Figure 4 shows reconstruction results for another test slice.
CG-SENSE suffers from major residual artifacts.
While supervised PG-DL further improves reconstruction performance, it still suffers from residual artifacts in this slice, marked by red arrows.
Both ZS-SSL and ZS-SSLTL achieves artifact-free reconstruction despite being trained on a single slice, while the latter expectedly achieves better quantitative metrics due to transfer learning.
IV. C ONCLUSION
In this study, we proposed a zero-shot self-supervised PGDL approach for subject-speciﬁc MRI reconstruction with a well-deﬁned stopping criterion that avoids overﬁtting.
ZSSSL was also used to ﬁne-tune a pretrained model via transfer learning for improved reconstruction quality and reduced risk of generalization at a lower computational cost.
Results on knee datasets showed that the proposed ZS-SSL achieves comparable or better image quality than supervised
PG-DL despite being trained on a single dataset