Image reconstruction is an inverse problem that solves for a computational image based on sampled sensor measurement.
Sparsely sampled image reconstruction poses addition challenges due to limited measurements.
In this work, we propose an implicit
Neural Representation learning methodology with Prior embedding (NeRP) to reconstruct a computational image from sparsely sampled measurements.
The method differs fundamentally from previous deep learning-based image reconstruction approaches in that NeRP exploits the internal information in an image prior, and the physics of the sparsely sampled measurements to produce a representation of the unknown subject.
No large-scale data is required to train the
NeRP except for a prior image and sparsely sampled measurements.
In addition, we demonstrate that NeRP is a general methodology that generalizes to different imaging modalities such as CT and MRI.
We also show that NeRP can robustly capture the subtle yet significant image changes required for assessing tumor progression.
I. I NTRODUCTION
MAGE reconstruction is conventionally formulated as an inverse problem, with the goal of obtaining the computational image of an unknown subject from measured sensor data.
For example, projection data are measured for computed tomography imaging (CT) while frequency domain (k-space) data are sampled for magnetic resonance imaging (MRI).
To reconstruct artifact-free images, dense sampling in measurement space is required to satisfy the Shannon-Nyquist theorem.
However, in many practical applications it would be desirable to reconstruct images from sparsely sampled data.
One important application is reducing radiation dose in CT imaging.
Another application is accelerating MRI.
The illposed nature of the sparse sampling image reconstruction problem poses a major challenge for algorithm development.
Many approaches have been studied to solve this problem.
One widely used approach is to exploit prior knowledge of the sparsity of the image in a transform domain, such as in compressed sensing, where total-variation, low-rank, and dictionary learning have been applied [1]-[7].
Unprecedented advances in deep learning driven by learning from large-scale data have achieved impressive progress in many fields, including computational image reconstruction.
Many researchers have introduced deep learning models for medical imaging modalities such as CT and MRI [8], [9].
The key to these deep learning approaches is training convolutional neural networks (CNNs) to learn the mapping from raw measurement data to the reconstructed image by exploiting the large-scale training data.
The network exploits the hidden transformation information embedded in the data through the data-driven training procedure.
More advanced methods have improved the conventional deep learning reconstruction by leveraging prior knowledge from other aspects, such as generative adversarial model [10], [11] and geometryintegrated deep learning frameworks [12]-[15].
Previous works have demonstrated the effectiveness of explicitly incorporating physics and geometry priors of imaging system with deep learning [12], [15].
Although these works show the advantage of deep learning for medical image reconstruction, they have also exposed some limitations.
For example, the acquisition of large-scale training data sets can be a bottleneck, the reconstructions may not be robust when deployed to unseen subjects, the reconstructions can be unstable with subtle yet significant structural changes such as tumor growth, and there can also be difficulties generalizing to different image modalities or anatomical sites [16].
To address these limitations, we propose a new insight for deep learning methodology for image reconstruction.
We propose to learn the implicit Neural Representation of an image with Prior embedding (NeRP), instead of learning the reconstruction mapping.
This is an essentially different perspective from previous deep learning-based reconstruction methods.
: Implicit Neural Representation Learning with Prior Embedding for Sparsely Sampled Image Reconstruction
I
[Figure]
(x, y, z
[Figure]
(x, y, z [Figure] [Figure]
y
[Figure]
z
[Figure]
Conventionally, a neural network is trained to learn the mapping from the sampled measurement data to reconstruct images based on a large-scale training database.
The proposed
NeRP model learns the network, i.e. multi-layer perceptron (MLP), to map the image spatial coordinates to the corresponding intensity values.
The neural network learns the continuous implicit neural representation of the entire image by encoding the full image spatial field into the weights of MLP model [17], [18].
The image reconstruction problem is transformed into a network optimization problem.
For sparse sampling, the measurements may not provide sufficient information to precisely reconstruct images of the unknown subject due to the ill-posed nature of the inverse problem.
The proposed NeRP framework exploits prior knowledge from a previous image for the same subject.
This is particularly applicable to clinical protocols where patients are scanned serially over time, such as monitoring tumor response to therapy.
The implicit neural representation first embeds the internal information of the prior image into the weights of MLP.
Note that NeRP requires no training data form external subjects except for the sparsely sampled measurements and a prior image of the subject.
The main contributions of this work are:
1) We present a novel deep learning methodology for sparsely sampled medical image reconstruction by learning the implicit neural representation of image with prior embedding (NeRP).
Our method requires no training data from external subjects and can be easily generalized across different imaging modalities and contrasts, and different anatomical sites.
We propose a prior embedding method in implicit neural representation learning by encoding internal information of the prior image into network parameters as the initialization of network optimization, which enables sparsely sampled image reconstruction.
3) We present extensive experiments for both 2D and 3D image reconstruction with various imaging modalities, including CT and MRI, and demonstrate the effectiveness and generalizability of the proposed NeRP method.
In particular, we show that our method is robust for capturing subtle yet significant structural changes such as those due to tumor progression.
II. M ETHOD
Fig.
1 illustrates the basic concept of the proposed implicit neural representation learning with prior embedding (NeRP) for image reconstruction.
NeRP contains three modules to obtain the final reconstruction images.
First, a prior image from earlier scan of the same subject is embedded as the implicit neural representation by encoding the entire spatial image field into the network’s parameters.
Specifically, the network is optimized to seek the continuous function that could precisely map the spatial coordinates to corresponding intensity values in the prior image.
Next, using the prior-embedded network as the initialization, we aim to learn the neural representation of the target reconstruction image from the subsampled measurements of an unknown subject without any ground truth, as shown in
Fig. 1(b).
The differentiable forward model corresponding to the imaging system (e.g. Radon transform for CT imaging or
Fourier transform for MRI imaging) is integrated to bridge between image space and sensor space.
In this way, the network is optimized in the continuous function space of the network’s parameters, with the constraints of the subsampled measurements from the unknown subject.
Finally, the reconstructed image can be obtained by inferring the trained network across all the spatial coordinates in the image field.
A. Problem Formulation
To formulate the inverse problem for computational image reconstruction, the forward process of imaging system can be modeled as: 𝑦 =
𝐴𝑥 + 𝑒 , where 𝑥 is the image of the unknown subject while 𝑦 is the sampled sensor measurements.
Matrix 𝐴 represents the forward model of the imaging system, and 𝑒 is the acquisition noise.
Image reconstruction aims to recover the computational image 𝑥 of the unknown subject, given the measurements 𝑦 from sensors.
In the sparsely sampled image reconstruction problem, the measurements 𝑦 are undersampled in sensor space for either accelerated acquisition, as in MRI, or reduction of radiation, as in CT.
The inverse problem for sparse sampling is ill-posed, and is typically formulated as an optimization problem with regularization: [Equation]
where ℰ(𝐴𝑥, 𝑦) is the data term, which measures the errors between 𝐴𝑥 and 𝑦 , so as to guarantee the data consistency with the sensor measurements.
Function ℰ can be different distance metrics such as L1 or L2 norm.
𝜌(𝑥) is the regularizer term characterizing the generic image prior.
The regularizer 𝜌(𝑥) can be determined in many different ways to capture various image characteristics.
For example, total variation of the image enforces smoothness, while sparsity in a transform domain is used in compressed sensing.
B. Neural Representation for Image
In implicit neural representation learning, the image is represented by a neural network as a continuous function.
The network ℳ 𝜃 with parameters 𝜃 can be defined as: ℳ 𝜃 : 𝑐 → 𝑣 𝑤𝑖𝑡ℎ 𝑐 ∈ [0,1) 𝑛 , 𝑣 ∈ ℝ , where the input 𝑐 is the normalized coordinate index in the image spatial field, and the output 𝑣 is the corresponding intensity value in the image.
The network function ℳ 𝜃 maps coordinates to the image intensities, which actually encodes the internal information of entire image into the network parameters.
Thus, network structure ℳ 𝜃 with the parameters 𝜃 is also regarded as the neural representation for the image.
Note that, theoretically, a random image in any modality or in any dimension 𝑥 ∈ ℝ 𝑛 can be parameterized by the network using this method.
Below we introduce the specific network structure used in our method.
1) Fourier Feature Embedding
Since Fourier features are shown to be effective for networks to learn high-frequency functions [20], we use a Fourier feature mapping 𝛾 to encode the input coordinates 𝑐 before applying them to the coordinate-based network.
Thus, the encoded coordinates are:
[Equation]
where matrix 𝐵 represents the coefficients for Fourier feature transformation.
Following [20], entries of matrix 𝐵 are sampled from Gaussian distribution 𝒩(0, 𝜎 2 ) , where 𝜎 is a hyperparameter characterizing the standard deviation of the prior distribution.
After the Fourier feature embedding, the input to the network ℳ 𝜃 is the encoded coordinates 𝛾(𝑐) .
Multi-Layer Perceptron Network
The network ℳ 𝜃 is implemented by a deep fully-connected network or multi-layer perceptron (MLP).
The coordinatebased MLP parameterizes the continuous function to represent the entire image.
This function is defined by the network structure as well as the network parameters.
In the next section, we will describe in detail how to obtain the network parameters through optimization.
For the network structure, the model depth and width of MLP are hyper-parameters, characterizing the representative capability of the MLP model.
Moreover, we use the periodic activation functions in our MLP model after each fully-connected layer, which are demonstrated to effectively represent fine details in signals [21].
C. NeRP for Sparsely Sampled Image Reconstruction
Next, we introduce how the proposed implicit neural representation learning with prior embedding (NeRP) is used to solve image reconstruction problem.
The goal is to recover the image 𝑥 of the target subject, given corresponding sparsely sampled measurements 𝑦 and a prior image 𝑥 𝑝𝑟 .
Note that 𝑥 𝑝𝑟 and 𝑥 are different scans for the same subject, but at different time points.
These capture the changing state of the subject such as tumor progression for monitoring therapy response.
Prior Embedding
In the first step, we embed the prior image 𝑥 𝑝𝑟 into the network.
We use the coordinate-based MLP ℳ 𝜙 introduced in
Sec.
III.B to map the spatial coordinates to corresponding intensity values in prior image 𝑥 𝑝𝑟 .
That is, ℳ 𝜙 : 𝑐 𝑖 → 𝑥 𝑖𝑝𝑟 , where 𝑖 denotes the coordinate index in image spatial field.
Given all the coordinate-intensity pairs in prior image {𝑐 𝑖 ,𝑥 𝑖𝑝𝑟 } 𝑖=1 𝑁 with a total of 𝑁 pixels in the image, the randomlyinitialized MLP is optimized based on the objective:
[Equation]
After optimization, the internal information of prior image 𝑥 𝑝𝑟 is encoded into the MLP network ℳ 𝜙 ∗ with the corresponding network parameters 𝜙 ∗ .
For clarity, we use ℳ 𝑝𝑟 to denote the prior-embedded MLP network, i.e. 𝑥 𝑝𝑟 =
ℳ 𝜙 ∗ =
ℳ 𝑝𝑟 .
2) Network Training
Given the prior-embedded MLP ℳ 𝑝𝑟 and measurements 𝑦 , we train the network to learn the neural representation of the target image.
Based on the formulation in Eq.
(2), the unknown target image 𝑥 is parametrized by a coordinate-based MLP ℳ 𝜃 with parameters 𝜃 .
Thus, the data term is defined as min 𝑥 ℰ(𝐴𝑥, 𝑦) =
min 𝜃 ℰ(𝐴ℳ 𝜃 ,𝑦) , where the optimization in image space is transformed to the optimization in the space of
MLP’s parameters.
Furthermore, the regularizer 𝜌(𝑥) is replaced by the implicit image priors from network parametrization, including the internal information from prior image embedded in ℳ 𝑝𝑟 as well as the low-level image statistics prior captured by network structure itself ℳ 𝜃 [24].
Thus, the optimization objective in Eq.
(2) can be formulated as follows:
[Equation]
The network ℳ 𝜃 is trained by minimizing the L2-norm loss, which is initialized by the prior-embedded network ℳ 𝑝𝑟 .
Note that forward model 𝐴 is adapted to the corresponding imaging system, such as Radon transform for CT imaging and Fourier transform for MRI imaging.
The operation 𝐴 is differentiable, which enables training the network ℳ 𝜃 in an end-to-end fashion.
3) Image Inference
Finally, after the network is well trained, the reconstruction image can be generated by inferring the trained network across all the spatial coordinates in the image field.
That is: 𝑥 ∗ : {𝑐 𝑖 , ℳ 𝜃 ∗ (𝑐 𝑖 )} 𝑖=1𝑁 , where 𝑖 denotes the coordinate index in image spatial field.
This is denoted in short as 𝑥 ∗ =
ℳ 𝜃 ∗ in
Eqs.
(6) and (7).
Filling the intensity values at all the coordinates in image grid constitutes the final reconstruction image 𝑥 ∗ .
[Figure]
w / o p r
i
o r
P ha s e
P ha s e
P ha s e
P ha s e
P ha s e
P ha s e
P ha s e
P ha s e
[Figure]
N o P r i
o r
I
[Figure]
N o P r i
o r
I m
[Figure]
III. E XPERIMENTS AND R ESULTS
To evaluate the proposed NeRP method, we conducted experiments for 2D/3D CT and MRI image reconstruction with sparsely sampling.
For CT image reconstruction we assumeprojections equally distributed across a semi-circle.
We compute parallel-beam projections for 2D CT and cone-beam projections for 3D CT.
For MRI image reconstruction, 40 radial spokes are sampled in k-space with golden angle as the angular interval.
Beyond sparsely-sampled measurements data, a prior image from an earlier scan is also given.
Since the prior image and reconstruction image are of the same patient at different time points, the prior image can provide useful information about the patient’s anatomic structure while still allowing crucial structural and functional differences such as tumor or lesion changes.
We will show the experimental results applying
NeRP for 2D/3D CT and MRI image reconstruction with various image contrasts and at various anatomical sites.
The experiments datasets contain: (1) a pancreas 4D CT withphases (phase 1 as the prior image); (2) two clinical patient cases including a head and neck CT case and a lung CT case, where each case has two longitudinal 3D CT images scanned for the same patient during radiation therapy (earlier CT as the prior image); (3) brain tumor regression MRI dataset: for each patient with primary newly diagnosed glioblastoma, there are two MRI exams within 90 days following chemo-radiation therapy completion and at tumor progression (set the first MRI exam as the prior image).
A. Experiments on CT Image Reconstruction
In Fig.
2, we show the 2D CT reconstruction results for phase 2 to phase 10 using the proposed NeRP algorithm by using phase 1 as prior image.
The reconstructed images can precisely capture the continuous changes with fine detail over different phases, although the same prior image is used and only sparse projections are sampled for reconstructing the target image in each phase.
For comparison, we conducted experiments to show the results of other reconstruction methods including filtered back projection (FBP) and Gaussian
random Fourier feature (GRFF) [20].
Going beyond 4D CT data, we also evaluated the clinical radiation therapy patient data with both head and neck CT and lung CT.
The quantitative results for 3D
CT reconstruction evaluated by PSNR and SSIM metrics are reported in Table I on different anatomic sites with all comparison methods.
B. Experiments on MRI Image Reconstruction
We conducted experiments to evaluate the proposed method for MRI image reconstruction.
Appendix Fig.
S1 demonstrates 2D MRI reconstruction results for multi-contrast MR images.
We can see the reconstructed images from sparsely subsampled k-space data can accurately capture the fine detailed structures especially in the tumor region, which differs from that in the prior image.
Quantitative results of 3D MRI image reconstruction evaluated by PSNR and SSIM metrics are reported in Table II for different image contrasts including T1,
T1c, T2 and FLAIR.
The reconstruction results indicate that our method is able to reconstruct the precise changes in brain tumor region even with sparsely sampled k-space data, which is crucial for clinical diagnosis and cancer treatment.
IV. C ONCLUSION
In this work, we propose a new deep learning-based medical image reconstruction methodology by learning implicit neural representations with prior embedding (NeRP), which efficiently incorporates the prior knowledge and learns to reconstruct the target image through implicit neural representations.
Through the experiments for 2D/3D MRI and
CT image reconstruction, we show that the proposed NeRP algorithm is able to provide high-quality reconstruction images even with sparsely sampled measurements data.
The NeRP approach possesses a number of unique advantages: (1) requires no training data from external subjects for developing networks; (2) accurate reconstruction of small and detailed changes such as anatomic structure or tumor progression; (3) broad applicability to different body sites, different imaging modalities and different patients. [Table] [Table] GRFF [20]
[Table]
[Table] [Table] [Table] [Table]
NeRP w/o prior
[Table]
NeRP
[Table]
ours)
[Table]
FBP
[Table]
GRFF [20]
[Table]
NeRP w/
[Table]
prior
[Table]
NeRP
[Table]
ours)
[Table]
[Table]
[Table]
ours)
[Table]
Evaluation
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
A CKNOWLEDGMENT
The authors acknowledge the funding supports from the
Stanford Bio-X Bowes Graduate Student Fellowship of
Stanford
University,
NIH/NCI 1R01CA and 1R01CA256890.