Paper: Gradient Descent Provably Solves Nonlinear Tomographic Reconstruction (ICLR-2024)

Reviewer comment (Theory):
The assumptions made by Theorems 1 and 2, which form the main contribution of the paper, do not match well the CT measurement setup. Specifically, in CT, the beams are well-structured, and if we consider the image of a beam, it does not resemble at all a sample from a 2D Gaussian field. Therefore, it is unclear how much the results on provable convergence carry over.

Author response:
This is a great point that we have now addressed in the revised paper (and in this comment) in two ways. 

First, we have added an explanation of the logical steps we followed to reach our Gaussian theoretical model, and why each of these steps is well-supported by prior literature. In particular, our Gaussian model follows from two approximations: (1) A projection in real-space is equivalent to a slice through the origin in Fourier space, according to the Fourier slice theorem. Our first approximation involves sampling random Fourier coefficients rather than radial slices in Fourier space. This approximation is somewhat standard in the literature, for example in the seminal compressive sensing paper by Candes, Romberg, and Tao (https://arxiv.org/abs/math/0503066). (2) Our second approximation is to replace random Fourier coefficients with random Gaussian measurements. This approximation is also well supported by prior work, namely https://arxiv.org/abs/1506.03521 which shows that “structured random matrices behave similarly to random Gaussian matrices” in compressive sensing type settings.

Second, and more importantly, we have independently verified that the key pseudoconvexity property we use in our proof—namely, the positivity of the correlation quantity <gradient of loss at z, z-x>/||z-x||^2, where x is the true signal and z is the current iterate—holds in practice even when we evaluate it using the true structure-preserving loss function instead of our Gaussian model. https://imgur.com/5rU3qgH (and the new Figure 1 in the revised manuscript) is a plot of the correlation value <gradient of loss at z, z-x>/||z-x||^2 where x is the synthetic phantom with ||x||= 7.4; the x-axis distance is normalized by this value so that the origin has normalized distance 1 relative to the global optimum. We sample 100 values of z at various distances from x, and show that this correlation is empirically positive for a wide range of distances including the distance between the origin and the true signal x. This experiment uses the true forward model that respects the ray structure, verifying that the same pseudoconvexity property we show in our Gaussian model also holds in the full forward model.

Reviewer comment (Theory):
If the authors can show or argue that the local pseudo-convexity property (in some reasonable neighborhood of the solution) carries over to the deterministic concentrated-beam setting, the connection with CT would be stronger.

Author response:
This is a great point that we have now addressed in the revised paper (and in this comment) in two ways. 

First, we have added an explanation of the logical steps we followed to reach our Gaussian theoretical model, and why each of these steps is well-supported by prior literature. In particular, our Gaussian model follows from two approximations: (1) A projection in real-space is equivalent to a slice through the origin in Fourier space, according to the Fourier slice theorem. Our first approximation involves sampling random Fourier coefficients rather than radial slices in Fourier space. This approximation is somewhat standard in the literature, for example in the seminal compressive sensing paper by Candes, Romberg, and Tao (https://arxiv.org/abs/math/0503066). (2) Our second approximation is to replace random Fourier coefficients with random Gaussian measurements. This approximation is also well supported by prior work, namely https://arxiv.org/abs/1506.03521 which shows that “structured random matrices behave similarly to random Gaussian matrices” in compressive sensing type settings.

Second, and more importantly, we have independently verified that the key pseudoconvexity property we use in our proof—namely, the positivity of the correlation quantity <gradient of loss at z, z-x>/||z-x||^2, where x is the true signal and z is the current iterate—holds in practice even when we evaluate it using the true structure-preserving loss function instead of our Gaussian model. https://imgur.com/5rU3qgH (and the new Figure 1 in the revised manuscript) is a plot of the correlation value <gradient of loss at z, z-x>/||z-x||^2 where x is the synthetic phantom with ||x||= 7.4; the x-axis distance is normalized by this value so that the origin has normalized distance 1 relative to the global optimum. We sample 100 values of z at various distances from x, and show that this correlation is empirically positive for a wide range of distances including the distance between the origin and the true signal x. This experiment uses the true forward model that respects the ray structure, verifying that the same pseudoconvexity property we show in our Gaussian model also holds in the full forward model.

Reviewer comment (Theory):
Since the paper is not concerned with proposing a non-obvious algorithm but proving the usefulness of a rather well-known algorithm, the mismatch between the theoretical assumptions and the CT setup is a serious shortcoming.

Author response:
This is a great point that we have now addressed in the revised paper (and in this comment) in two ways. 

First, we have added an explanation of the logical steps we followed to reach our Gaussian theoretical model, and why each of these steps is well-supported by prior literature. In particular, our Gaussian model follows from two approximations: (1) A projection in real-space is equivalent to a slice through the origin in Fourier space, according to the Fourier slice theorem. Our first approximation involves sampling random Fourier coefficients rather than radial slices in Fourier space. This approximation is somewhat standard in the literature, for example in the seminal compressive sensing paper by Candes, Romberg, and Tao (https://arxiv.org/abs/math/0503066). (2) Our second approximation is to replace random Fourier coefficients with random Gaussian measurements. This approximation is also well supported by prior work, namely https://arxiv.org/abs/1506.03521 which shows that “structured random matrices behave similarly to random Gaussian matrices” in compressive sensing type settings.

Second, and more importantly, we have independently verified that the key pseudoconvexity property we use in our proof—namely, the positivity of the correlation quantity <gradient of loss at z, z-x>/||z-x||^2, where x is the true signal and z is the current iterate—holds in practice even when we evaluate it using the true structure-preserving loss function instead of our Gaussian model. https://imgur.com/5rU3qgH (and the new Figure 1 in the revised manuscript) is a plot of the correlation value <gradient of loss at z, z-x>/||z-x||^2 where x is the synthetic phantom with ||x||= 7.4; the x-axis distance is normalized by this value so that the origin has normalized distance 1 relative to the global optimum. We sample 100 values of z at various distances from x, and show that this correlation is empirically positive for a wide range of distances including the distance between the origin and the true signal x. This experiment uses the true forward model that respects the ray structure, verifying that the same pseudoconvexity property we show in our Gaussian model also holds in the full forward model.

Reviewer comment (Experiments):
The experiments are not entirely convincing; reconstructions with the linear model could have been improved by including regularizers.

Author response:
To clarify, the baseline linearized method in the synthetic experiment is including exactly the same regularization (by total variation) as the nonlinear method; we also note that this synthetic experiment is not in the underdetermined regime, as more measurements are provided than there are voxels in the reconstruction. The baseline linearized method in the real experiment is not using gradient descent, but rather using a state-of-the-art proprietary algorithm provided by our industrial collaborators. This real-data linearized baseline is the closest we have to a ground truth volume. In the revised paper, we have added an additional comparison to a more standard (and open-source) linearized baseline algorithm, FDK, which also exhibits worse metal artifacts than our nonlinear reconstruction. The updated figure is in the revised manuscript, and at https://imgur.com/a/PmtZz2O for convenience.

Reviewer comment (Writing):
Page 2, "We propose a Gaussian model of eqn 1.2": Please clarify what you mean by a Gaussian model. It seems you are referring to modelling \( a_i \) to be samples of a Gaussian field, but this is not clear at this stage.

Author response:
Yes, this is what we mean. We have clarified this in the revised paper.

Reviewer comment (Theory):
Page 2, beginning of Section 2, "(1) we model \( a_i \) as a standard Gaussian...": This strongly contradicts the opening sentence of Section 2, where you state that "\( a_i \) are sparse, nonnegative, highly structured, dependent on \( i \)". What is the motivation behind this modelling? Is it mainly to obtain provable results? While some deviation from an accurate model is understandable, this seems extreme.

Author response:
Yes, this is to make the analysis more tractable. Please refer to our response above regarding the connection between the theoretical model and the true/practical model.

Reviewer comment (Presentation):
Page 5, footnote 1: This would be worth bringing into the main text and clarifying what \( m \) should exceed.

Author response:
Thank you for the suggestion; it has been implemented in the revised paper.

Reviewer comment (Theory):
In general, it would be helpful to provide concrete typical values for the constants \( c_i \) that appear in theorems, for specific cases or resolutions.

Author response:
We expect the values to be of the order of 10s. However, we note that exact calculation of these constants is cumbersome and this type of proof is more about demonstration of the result holding with numerical constants rather than identifying the exact values, for which asymptotic techniques are often used instead.

Reviewer comment (Experiments):
Section 5.1: Can you actually expect to see inside the metal? It seems that the regularizer plays a more crucial role here. What happens if you were to use a regularized formulation but with a linear measurement model (e.g., using one of the iterative solvers like FISTA)?

Author response:
The goal is less to see inside the metal (which at some point becomes unobservable, as the reviewer suggests), but to at least not have metal artifacts obscure tissue outside the metal. It is true that regularization can help reduce the severity of artifacts, but some noise amplified by the linearization step is unrecoverable even with regularization. For example, our synthetic experiments use the same regularization for both linearized and nonlinear reconstruction, and our real experiment uses a state-of-the-art linearized baseline that surely includes regularization (though the algorithm itself is proprietary). We have clarified the use of regularization in the linearized methods in the revised paper.

Reviewer comment (Novelty):
Regularization techniques have been widely used in solving inverse problems to address the ill-posedness. It is not clear about the contributions and novelty of the proposed approaches.

Author response:
The main problem we are addressing is separate from the ill-posedness problem from undersampling. Rather, we are addressing the issue of sensitivity to absorptive media like metal, which arises because of the poorly conditioned logarithmic preprocessing used to linearize the measurements in the standard reconstruction pipeline. This problem would persist with the linearized forward model even in the case of unlimited measurements. For example, our synthetic experiments are not in the ill-posed regime; instead they are overdetermined and have no added measurement noise, yet linearized reconstruction fails because the logarithmic preprocessing is sensitive even to the noise of machine precision.

Reviewer comment (Experiments):
In the experiments, there are no other related works for comparison. So, it is hard to see whether the proposed performance is standing out.

Author response:
We have added a standard linearized baseline method, FDK, in our real-data experiment, and we find that it also exhibits severe artifacts. The updated figure is in the revised manuscript, and at https://imgur.com/a/PmtZz2O for convenience. Please also note that the primary contribution of our work is the theoretical analysis, which is the first of its kind for this nonlinear forward model. The experiments are provided as a preliminary validation to support the theoretical results; they are not intended as a comprehensive comparison.

Reviewer comment (Theory):
Why was the regularization term in the objective function in (2.3) recast as an inequality constraint in (4.1)? Will they give two different solutions? In addition, how was $x$ in (4.1) selected/constructed?

Author response:
These two formulations (2.3 and 4.1) are equivalent in the sense that for any value of R(x) in the inequality constraint (4.1) there is a value of lambda in the regularization term (2.3) that will yield exactly the same solution. We also note that our theoretical results are robust to the exact choice of this regularization weight or inequality constraint threshold (see e.g. section 6.4 in https://arxiv.org/pdf/1507.04793.pdf), but we have stated the results in this form (with a precise choice of regularization or constraint) for simplicity of exposition.  

In (4.1) the x in the constraint refers to the unknown ground truth signal. Even if the signal is unknown, this formulation assumes that some structure of x is known via the value of the function R(x). In practice this value R(x) can be viewed as a hyperparameter.

Reviewer comment (Evaluation):
Experiments are unconvincing, particularly Fig.2, where except around the high-density object (the crown), the quality of the linearised reconstruction is visibly better.

Author response:
Just to clarify, when we describe the dental imaging target as a phantom, it is because it is not a living patient. This phantom is a real human skull, not a synthetic function, so we do not have a ground truth volume. We have removed the term “phantom” in the revised paper, when describing this skull dataset, to avoid any confusion for future readers. For this real dataset the linearized reconstruction is the closest we have to a ground truth; it is the state-of-the-art reconstruction provided by an industrial collaborator (using a proprietary method). We have also augmented our real-data experiment with a comparison to a more standard and open-source linearized baseline method, FDK, which likewise suffers metal artifacts as well as generally lower quality compared to either our nonlinear reconstruction or the commercial reference linearized reconstruction.

Reviewer comment (Experiments):
The experiments on the Shepp-Logan phantom are too simple to be convincing.

Author response:
Just to clarify, when we describe the dental imaging target as a phantom, it is because it is not a living patient. This phantom is a real human skull, not a synthetic function, so we do not have a ground truth volume. We have removed the term “phantom” in the revised paper, when describing this skull dataset, to avoid any confusion for future readers. For this real dataset the linearized reconstruction is the closest we have to a ground truth; it is the state-of-the-art reconstruction provided by an industrial collaborator (using a proprietary method). We have also augmented our real-data experiment with a comparison to a more standard and open-source linearized baseline method, FDK, which likewise suffers metal artifacts as well as generally lower quality compared to either our nonlinear reconstruction or the commercial reference linearized reconstruction.

Reviewer comment (Experiments):
Maybe the unconvincing experimental results are due to a poor choice of hyper-parameter or a sub-optimal regulariser.

Author response:
Indeed, by slightly reducing the total variation regularization we are able to reduce the blurriness in our nonlinear reconstruction. Thanks for the suggestion! The updated figure with our tuned results and the comparison to FDK is in the revised manuscript, and at https://imgur.com/a/PmtZz2O for convenience.

Reviewer comment (Reproducibility):
Reference code does not seem to be provided, limiting reproducibility.

Author response:
Our anonymized code has been uploaded as a supplemental file. We will also release the code publicly upon publication.

Reviewer comment (Presentation):
How could you improve the fuzziness of Fig.2? This is essential because the current result does not sell your method well.

Author response:
Please refer to our response above.

Reviewer comment (Experiments):
Could you compare with other non-linear tomography approaches?

Author response:
We agree it’s always preferable to compare to strong baselines. In this case, we are not aware of existing algorithms that use the nonlinear tomography forward model rather than the linearized version–though if the reviewer has suggestions we would be happy to look into them. To be clear, when we say non-linear we are referring to the measurement model, not the reconstruction algorithm. For example, there are some nonlinear methods (e.g. Neural Adaptive Tomography https://dl.acm.org/doi/abs/10.1145/3528223.3530121) that use a nonlinear model as the volume representation (e.g. neural net), but to our knowledge these still use the same linearized forward model and thus face the same sensitivity to absorptive media like metal.