Original: L OOP U NROLLED S HALLOW E QUILIBRIUM R EGULARIZER
Revised: L OOP U NROLLED S HALLOW E QUILIBRIUM R EGULARIZER

Original: (LUSER) -
Revised: (LUSER) -

Original: A
Revised: A

Original: M EMORY -E
Revised: M EMORY -E

Original: FFICIENT
Revised: FFICIENT

Original: I NVERSE
Revised: I NVERSE

Original: P ROBLEM S OLVER
Revised: P ROBLEM S OLVER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In inverse problems we aim to reconstruct some underlying signal of interest from potentially corrupted and often ill-posed measurements.
Revised: In inverse problems we aim to reconstruct some underlying signal of interest from potentially corrupted and often ill-posed measurements.

Original: Classical optimizationbased techniques proceed by optimizing a data consistency metric together with a regularizer.
Revised: Classical optimizationbased techniques proceed by optimizing a data consistency metric together with a regularizer.

Original: Current state-of-the-art machine learning approaches draw inspiration from such techniques by unrolling the iterative updates for an optimizationbased solver and then learning a regularizer from data.
Revised: Current state-of-the-art machine learning approaches draw inspiration from such techniques by unrolling the iterative updates for an optimizationbased solver and then learning a regularizer from data.

Original: This loop unrolling (LU) method has shown tremendous success, but often requires a deep model for the best performance leading to high memory costs during training.
Revised: This loop unrolling (LU) method has shown tremendous success, but often requires a deep model for the best performance leading to high memory costs during training.

Original: Thus, to address the balance between computation cost and network expressiveness, we propose an LU algorithm with shallow equilibrium regularizers (LUSER).
Revised: Thus, to address the balance between computation cost and network expressiveness, we propose an LU algorithm with shallow equilibrium regularizers (LUSER).

Original: These implicit models are as expressive as deeper convolutional networks, but far more memory efficient during training.
Revised: These implicit models are as expressive as deeper convolutional networks, but far more memory efficient during training.

Original: The proposed method is evaluated on image deblurring, computed tomography (CT), as well as single-coil Magnetic Resonance Imaging (MRI) tasks and shows similar, or even better, performance while requiring up to 8 × less computational resources during training when compared against a more typical LU architecture with feedforward convolutional regularizers.
Revised: The proposed method is evaluated on image deblurring, computed tomography (CT), as well as single-coil Magnetic Resonance Imaging (MRI) tasks and shows similar, or even better, performance while requiring up to 8 × less computational resources during training when compared against a more typical LU architecture with feedforward convolutional regularizers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In an inverse problems we face the task of reconstructing some data or parameters of an unknown signal from indirect observations.
Revised: In an inverse problems we face the task of reconstructing some data or parameters of an unknown signal from indirect observations.

Original: The forward process, or the mapping from the data to observations, is typically well known, but ill-posed or non-invertible.
Revised: The forward process, or the mapping from the data to observations, is typically well known, but ill-posed or non-invertible.

Original: More formally, we consider the task of recovering some underlying signal x from measurements y taken via some forward operator A according to
Revised: More formally, we consider the task of recovering some underlying signal x from measurements y taken via some forward operator A according to

Original: [Equation]
Revised: [Equation]

Original: where η represents noise.
Revised: where η represents noise.

Original: The forward operator can be nonlinear, but to simplify the notation, we illustrate the idea in linear form throughout this paper.
Revised: The forward operator can be nonlinear, but to simplify the notation, we illustrate the idea in linear form throughout this paper.

Original: A common approach to recover the signal is via an iterative method based on the least squares loss:
Revised: A common approach to recover the signal is via an iterative method based on the least squares loss:

Original: [Equation]
Revised: [Equation]

Original: For many problems of interest, A is ill-posed and does not have full column rank.
Revised: For many problems of interest, A is ill-posed and does not have full column rank.

Original: Thus, attempting to solve (2) does not yield a unique solution.
Revised: Thus, attempting to solve (2) does not yield a unique solution.

Original: To address this, we can extend (2) by including a regularizing term to bias the inversion towards solutions with favorable properties.
Revised: To address this, we can extend (2) by including a regularizing term to bias the inversion towards solutions with favorable properties.

Original: Common examples of regularization include ℓ 2 , ℓ 1 , and Total Variation (TV).
Revised: Common examples of regularization include ℓ 2 , ℓ 1 , and Total Variation (TV).

Original: Each regularizer encourages certain properties on the estimated signal (cid:98) x
Revised: Each regularizer encourages certain properties on the estimated signal (cid:98) x

Original: (e.g., smoothness, sparsity, piece-wise constant, etc.)
Revised: (e.g., smoothness, sparsity, piece-wise constant, etc.)

Original: and is often chosen based on task-specific prior knowledge.
Revised: and is often chosen based on task-specific prior knowledge.

Original: Recent works (Ongie et al., 2020) attempt to tackle inverse problems using more data-driven methods.
Revised: Recent works (Ongie et al., 2020) attempt to tackle inverse problems using more data-driven methods.

Original: Unlike typical supervised learning tasks that attempt to learn a mapping purely from examples, deep learning for inverse problems have access to the forward operator and thus should be able to guide the learning process for more accurate reconstructions.
Revised: Unlike typical supervised learning tasks that attempt to learn a mapping purely from examples, deep learning for inverse problems have access to the forward operator and thus should be able to guide the learning process for more accurate reconstructions.

Original: One popular approach to incorporating knowledge of the forward operator is termed loop unrolling (LU).
Revised: One popular approach to incorporating knowledge of the forward operator is termed loop unrolling (LU).

Original: These methods are heavily inspired by standard iterative inverse problem solvers, but rather than use a hand tuned regularizer, they instead learn the update with some parameterized model.
Revised: These methods are heavily inspired by standard iterative inverse problem solvers, but rather than use a hand tuned regularizer, they instead learn the update with some parameterized model.

Original: They tend to have a fixed number of iterations (typically around 5-10) due to computational constraints.
Revised: They tend to have a fixed number of iterations (typically around 5-10) due to computational constraints.

Original: Gilton et al.
Revised: Gilton et al.

Original: (2021) proposes an interesting alternative that takes advantage of deep equilibrium (DEQ) models (Bai et al., 2019; 2020;
Revised: (2021) proposes an interesting alternative that takes advantage of deep equilibrium (DEQ) models (Bai et al., 2019; 2020;

Original: Fung et al., 2021; El Ghaoui et al., 2021) that we refer to as DEQ4IP.
Revised: Fung et al., 2021; El Ghaoui et al., 2021) that we refer to as DEQ4IP.

Original: Equilibrium models are designed to recursively iterate on their input until a “fixed point” is found (i.e., the input no longer changes after passing through the model).
Revised: Equilibrium models are designed to recursively iterate on their input until a “fixed point” is found (i.e., the input no longer changes after passing through the model).

Original: They extend this principle to the LU method, choosing to iterate until convergence rather than for a “fixed budget”.
Revised: They extend this principle to the LU method, choosing to iterate until convergence rather than for a “fixed budget”.

Original: Our Contributions.
Revised: Our Contributions.

Original: We propose an alternative novel architecture for solving inverse problems called Loop Unrolled Shallow Equilibrium Regularizer (LUSER).
Revised: We propose an alternative novel architecture for solving inverse problems called Loop Unrolled Shallow Equilibrium Regularizer (LUSER).

Original: It incorporates knowledge of the forward model by adopting the principles of LU architectures while reducing its memory consumption by using a shallow (relative to existing feed-forward models) DEQ as the learned regularizer update.
Revised: It incorporates knowledge of the forward model by adopting the principles of LU architectures while reducing its memory consumption by using a shallow (relative to existing feed-forward models) DEQ as the learned regularizer update.

Original: Unlike DEQ4IP that converts the entire LU architecture into a DEQ model, we only convert the learned regularizer at each stage.
Revised: Unlike DEQ4IP that converts the entire LU architecture into a DEQ model, we only convert the learned regularizer at each stage.

Original: This has the advantage of simplifying the learning task for DEQ models, which can be unstable to train in practice.
Revised: This has the advantage of simplifying the learning task for DEQ models, which can be unstable to train in practice.

Original: To our knowledge, this is the first use of multiple sequential DEQ models within a single architecture for solving inverse problems.
Revised: To our knowledge, this is the first use of multiple sequential DEQ models within a single architecture for solving inverse problems.

Original: Our proposed architecture ( i ) reduces the number of forward/adjoint operations compared to the work proposed by Gilton et al.
Revised: Our proposed architecture ( i ) reduces the number of forward/adjoint operations compared to the work proposed by Gilton et al.

Original: (2021),  ( ii ) reduces the memory footprint during training without loss of expressiveness as demonstrated by our experiments, and ( iii ) is more stable to train in practice than
Revised: (2021), and ( ii ) reduces the memory footprint during training without loss of expressiveness as demonstrated by our experiments.

Original: DEQ4IP.

Original: We empirically demonstrate better reconstruction across multiple tasks thanstate-of-theart LU alternatives with a similar number of parameters, with the ability to reduce computational memory costs during training by a factor of up to 8 × .
Revised: We empirically demonstrate better reconstruction across multiple tasks than LU alternatives with comparable number of parameters, with the ability to reduce computational memory costs during training by a factor of up to 8 × .

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: Section 2 reviews related works in solving inverse problems.
Revised: Section 2 reviews related works in solving inverse problems.

Original: Section 3 introduces the proposed LUSER, which we compare with other baseline methods in image deblurring, CT, and MRI tasks in Section 4.
Revised: Section 3 introduces the proposed LUSER, which we compare with other baseline methods in image deblurring, CT, and MRI tasks in Section 4.

Original: We conclude in Section 5 with a brief discussion.
Revised: We conclude in Section 5 with a brief discussion.

Original:  R ELATED W ORK 2. L OOP U NROLLING
Revised:  R ELATED W ORK 2. L OOP U NROLLING

Original: As noted above, a common approach to tackling an inverse problem is to cast it as an optimization problem consisting of the sum of a data consistency term and a regularization term
Revised: As noted above, a common approach to tackling an inverse problem is to cast it as an optimization problem consisting of the sum of a data consistency term and a regularization term

Original: [Equation]
Revised: [Equation]

Original: where r is a regularization function mapping from the domain of the parameters of interest to a real number and γ ≥ 0 is a well-tuned coefficient.
Revised: where r is a regularization function mapping from the domain of the parameters of interest to a real number and γ ≥ 0 is a well-tuned coefficient.

Original: The regularization function is chosen for specific classes of signals to exploit any potential structure, e.g., ∥ x ∥ 2 for smooth signals and
Revised: The regularization function is chosen for specific classes of signals to exploit any potential structure, e.g., ∥ x ∥ 2 for smooth signals and

Original: ∥ x ∥ 0 or ∥ x ∥for sparse signals.
Revised: ∥ x ∥ 0 or ∥ x ∥for sparse signals.

Original: The total-variation (TV) norm is another popular example of a regularizer that promotes smoothness while preserving edges and is often used for imaging tasks.

Original: When r is differentiable, the solution of (3) can be obtained in an iterative fashion via gradient descent.
Revised: When r is differentiable, the solution of (3) can be obtained in an iterative fashion via gradient descent.

Original: For some step size λ at iteration k = 1 , 2 , . . .
Revised: For some step size λ at iteration k = 1 , 2 , . . .

Original: , K , we apply the update:
Revised: , K , we apply the update:

Original: [Equation]
Revised: [Equation]

Original: For non-differentiable r , the more generalized proximal gradient algorithm can be applied with the following update, where τ is a well-tuned hyperparameter related to the proximal operator:
Revised: For non-differentiable r , the more generalized proximal gradient algorithm can be applied with the following update, where τ is a well-tuned hyperparameter related to the proximal operator:

Original: [Equation]
Revised: [Equation]

Original: The loop unrolling (LU) method considers running the update in (4) or (5), but replaces λγ ∇ r or the proximal operator with a learned neural network instead.
Revised: The loop unrolling (LU) method performs the update in (4) or (5), but replaces λγ ∇ r or the proximal operator with a learned neural network instead.

Original: The overall architecture repeats the neural network based update for a pre-determined number of iterations, fixing the overall computational budget.
Revised: The overall architecture repeats the neural network based update for a pre-determined number of iterations, fixing the overall computational budget.

Original: Note that the network is only implicitly learning the regularizer.
Revised: Note that the network is only implicitly learning the regularizer.

Original: In practice, it is actually learning an update step, which can be thought of as de-noising or a projection onto the manifold of the data.
Revised: In practice, it is actually learning an update step, which can be thought of as de-noising or a projection onto the manifold of the data.

Original: LU is typically trained end-to-end, i.e., when fed some initialization x 0 , the network will output the final estimate x K , compute a loss with respect to the ground truth, and then back-propagate across the entire computational graph to update network parameters.
Revised: LU is typically trained end-to-end.

Original: While endto-end training is easier to perform and encourages faster convergence, it requires all intermediate activations to be stored in memory.
Revised: While end-to-end training is easier to perform and encourages faster convergence, it requires all intermediate activations to be stored in memory.

Original: Thus, the maximum number of iterations is always kept small compared to classical iterative inverse problem solvers.
Revised: Thus, the maximum number of iterations is always kept small compared to classical iterative inverse problem solvers.

Original: Due to the limitation in memory, there is a trade-off between the depth of a LU and the richness of each regularization network.
Revised: Due to the limitation in memory, there is a trade-off between the depth of a LU and the richness of each regularization network.

Original: Intuitively, one can raise the network performance by increasing the number of loop unrolled iterations.
Revised: Intuitively, one can raise the network performance by increasing the number of loop unrolled iterations.

Original: For example, Gilton et al.
Revised: For example, Gilton et al.

Original: (2021) extends the LU model to potentially infinite number of iterations using an implicit network, and (Putzky & Welling, 2019) allows deeper unrolling iterations using invertible networks, while requiring recalculation of the intermediate results from the output in training phase.
Revised: (2021) extends the LU model to potentially infinite number of iterations using an implicit network, and (Putzky & Welling, 2019) allows deeper unrolling iterations using invertible networks, while requiring recalculation of the intermediate results from the output in training phase.

Original: This approach can be computationally intensive for large-scale inverse problems or when the forward operator is nonlinear and computationally expensive to apply.
Revised: This approach can be computationally intensive for large-scale inverse problems or when the forward operator is nonlinear and computationally expensive to apply.

Original: For example, the forward operator may involve solving differential equations such as the wave equation for seismic wave propagation (Chapman, 2004) and the Lorenz equations for atmospheric modeling(Oestreicher, 2022).
Revised: For example, the forward operator may involve solving differential equations such as the wave equation for seismic wave propagation (Chapman, 2004) and the Lorenz equations for atmospheric modeling(Oestreicher, 2022).

Original: Alternatively, one can design a richer regularization network.
Revised: Alternatively, one can design a richer regularization network.

Original: For example Fabian & Soltanolkotabi (2022) uses a transformer as the regularization network and achieves extremely competitive results in the fastMRI challenge (Zbontar et al., 2018), but requires multiple 24GB GPU for training with batch size of 1, which is often impractical, especially for large systems.
Revised: For example Fabian & Soltanolkotabi (2022) uses a transformer as the regularization network and achieves extremely competitive results in the fastMRI challenge (Zbontar et al., 2018), but requires multiple 24GB GPU for training, which is often impractical, especially for large systems.

Original: Our design strikes a balance in the expressiveness in regularization networks and memory efficiency during training.
Revised: Our design strikes a balance in the expressiveness in regularization networks and memory efficiency during training.

Original: Our proposed work is an alternative method to achieve a rich regularization network without the additional computational memory costs during training.
Revised: Our proposed work is an alternative method to achieve a rich regularization network without the additional computational memory costs during training.

Original: 2. D EEP E QUILIBRIUM M ODELS FOR I NVERSE P ROBLEMS (DEQ4IP)
Revised: 2. D EEP E QUILIBRIUM M ODELS FOR I NVERSE P ROBLEMS (DEQ4IP)

Original: Deep equilibrium (DEQ) models introduce an alternative to traditional feed-forward networks (Bai et al., 2019; 2020;
Revised: Deep equilibrium (DEQ) models introduce an alternative to traditional feed-forward networks (Bai et al., 2019; 2020;

Original: Fung et al., 2021; El Ghaoui et al., 2021).
Revised: Fung et al., 2021; El Ghaoui et al., 2021).

Original: Rather than feed an input through a fixed (relatively small) number of layers, DEQ models solve for the “fixed-point” given some input.
Revised: Rather than feed an input through a fixed (relatively small) number of layers, DEQ models solve for the “fixed-point” given some input.

Original: More formally, given a network f θ and some input x (0) and y , we recursively apply the network via
Revised: More formally, given a network f θ and some input x (0) and y , we recursively apply the network via

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  until convergence.
Revised: 6) until convergence.

Original: In this instance, y acts as an input injection that determines the final output.
Revised: In this instance, y acts as an input injection that determines the final output.

Original: This is termed the forward process.
Revised: This is termed the forward process.

Original: The weights θ of the model can be trained via implicit differentiation, removing the need to store all intermediate activations from recursively applying the network.
Revised: The weights θ of the model can be trained via implicit differentiation, removing the need to store all intermediate activations from recursively applying the network (Bai et al., 2019; Fung et al., 2021).

Revised: In particular, we adopt the “Jacobian-free” backpropagation strategy outlined in Fung et al.

Original: This allows for deeper, more expressive models without the associated memory footprint to train such models.
Revised: This allows for deeper, more expressive models without the associated memory footprint to train such models.

Original: Gilton et al.
Revised: Gilton et al.

Original: (2021) demonstrates an application of one such model, applying similar principles to a single iteration of an LU architecture.
Revised: (2021) demonstrates an application of one such model, applying similar principles to a single iteration of an LU architecture.

Original: Such an idea is a natural extension as it allows the model to “iterate until convergence” rather than rely on a “fixed budget”.
Revised: Such an idea is a natural extension as it allows the model to “iterate until convergence” rather than rely on a “fixed budget”.

Original: In another word, looping over (4) and (5) many times (in practice, usually around 50 iterations) until x k converges.
Revised: More specifically, the model repeats (4) and (5) many times (in practice, usually around 50 iterations) until x k converges.

Original: However, such a model can be unstable to train and often performs best with pre-training of the learned portion of the model (typically acting as a learned regularizer/de-noiser).
Revised: However, such a model can be unstable to train and often performs best with pre-training of the learned portion of the model (typically acting as a learned regularizer/de-noiser).

Original: It is also important to note is that such a model would have to apply the forward operator (and potentially the adjoint)
Revised: It is also important to note is that such a model would have to apply the forward operator (and its adjoint)

Original: many times during the forward process.
Revised: many times during the forward process.

Original: Although this can be accelerated to reduce the number of applications, it is still often  more than the number of applications for an LU equivalent. This can be an issue if the forward operator is computationally expensive to apply, an issue LU methods somewhat mitigate by fixing the total number of iterations.
Revised: Although this can be accelerated to reduce the number of applications, it is still often signficiantly more than the number of applications for an LU equivalent which can be an issue if the forward operator is computationally expensive to apply.

Original: 2. A LTERNATIVE A PPROACHES TO T ACKLE M EMORY I SSUES
Revised: 2. A LTERNATIVE A PPROACHES TO T ACKLE M EMORY I SSUES

Original: I-RIM (Putzky & Welling, 2019) is a deep invertible network that address the memory issue by recalculating the intermediate results from the output.
Revised: I-RIM (Putzky & Welling, 2019) is a deep invertible network that address the memory issue by recalculating the intermediate results from the output.

Original: However it is not ideal when forward model is computationally expensive.
Revised: However it is not ideal when forward model is computationally expensive.

Original: Gradient checkpointing (Sohoni et al., 2019) is another practical technique to reduce memory costs for deep neural networks.
Revised: Gradient checkpointing (Sohoni et al., 2019) is another practical technique to reduce memory costs for deep neural networks.

Original: It saves intermediate activations of some checkpoint nodes, and recomputes the forward pass between two checkpoints for backpropagation.
Revised: It saves intermediate activations of some checkpoint nodes, and recomputes the forward pass between two checkpoints for backpropagation.

Original: However, it is not an easy and efficient technique to implement for a weight-tied neural network.
Revised: However, it is not an easy and efficient technique to implement for a weight-tied neural network.

Original: , to aid a we superscript

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: LU methods have been the state-of-the-art approach for solving inverse problems due to their stability when training, inclusion of the forward model, and their near instantaneous inference times.
Revised: LU methods currently dominate the state-of-the-art approaches for solving inverse problems due to their stability when training, inclusion of the forward model, and their near instantaneous inference times.

Original: However, there is a noticeable trade-off in terms of the memory requirements  when training these models and the accuracy foreven medium scale problems,  making them nearly impossible to use for much larger scale inverse problems.
Revised: However, there is a noticeable trade-off in terms of the memory requirements vs accuracy when training these models. Even for medium scale problems, some of the proposed architectures require multiple GPUs just to train, making this approach infeasible to use for much larger scale inverse problems.

Original: DEQ4IP offers an interesting alternative, drastically reducing the memory required during training and allowing the flexibility to adjust accuracy during reconstruction when performing inference.
Revised: DEQ4IP offers an interesting alternative, drastically reducing the memory required during training and allowing the flexibility to adjust accuracy during reconstruction when performing inference.

Original: However, these models can potentially suffer when the forward/adjoint operators are computationally expensive, particularly when it takes more iterations to converge than a standard fixed LU method.
Revised: However, these models can potentially suffer when the forward/adjoint operators are computationally expensive, particularly since it takes more iterations to converge than a standard fixed LU method.

Original: To address these concerns, we propose a novel architecture for solving inverse problems called Loop
Revised: To address these concerns, we propose a novel architecture for solving inverse problems called Loop

Original: Unrolled Shallow Equilibrium Regularizer (LUSER) as an approach that limits the number of forward/adjoint calls for particularly complex inverse problems while  drastically reducing the memory requirements for training, allowing us to scale up to much larger inverse problems (or require less GPU memory for existing problems).
Revised: Unrolled Shallow Equilibrium Regularizer (LUSER). We adopt a LU approach to limit the number of forward/adjoint calls for particularly complex inverse problems while also drastically reducing the memory requirements for training, allowing us to scale up to much larger inverse problems (or require less GPU memory for existing problems).

Original: LUSER achieves this by adopting DEQ models as the learned regularizer update in a standard LU architecture.
Revised: LUSER achieves this by adopting DEQ models as the trainable regularizer update in a standard LU architecture.

Original: The implicit DEQ models are smaller in size but just as expressive  as typical convolutional models used as the learned regularizer update allowing for an accurate reconstruction with less computational memory costs.
Revised: The implicit DEQ models are smaller in size but just as expressive (Bai et al., 2019) as typical convolutional models used as the learned regularizer update allowing for an accurate reconstruction with less computational memory costs.

Original: Furthermore, learning a proximal update is a far simpler task compared to solving the inverse problem as a whole, reducing the instability during training commonly faced by DEQ models.
Revised: Furthermore, learning a proximal update is a far simpler task compared to solving the inverse problem as a whole.

Original: We adopt a “proximal gradient descent” styled LU architecture.
Revised: We adopt a “proximal gradient descent” styled LU architecture.

Original: The network takes in measurements y and some initial estimate x 0 .
Revised: The network takes in measurements y and some initial estimate x 0 .

Original: The architecture consists of K stages, alternating between a gradient descent step d k = x k + λ
Revised: The architecture consists of K stages, alternating between a gradient descent step d k = x k + λ

Original: A ⊤ ( y − Ax k ) for k = 1 , 2 , . . .
Revised: A ⊤ ( y − Ax k ) for k = 1 , 2 , . . .

Original: , K , followed by a feed-forward pass of the shallow equilibrium model acting as a proximal update block.
Revised: , K , followed by a feed-forward pass of the shallow equilibrium model acting as a proximal update block.

Original:  Figure 1 illustrates a single block in LUSER as a learned proximal operator.
Revised: [Figure] Figure 1 illustrates a single block in LUSER as a learned proximal operator.

Original: The input d k from the previous stage is processed only once by a set of input injection layers to avoid redundant computation.
Revised: The input d k from the previous stage is processed only once by a set of input injection layers to avoid redundant computation.

Original: The input injection will determine the final fixed point output (Fung et al., 2021).
Revised: The input injection will determine the final fixed point output.

Original: The recursive portion of the proximal block consists of two sets of layers: the data layer and the mixing layer.
Revised: The recursive portion of the proximal block consists of two sets of layers: the data layer and the mixing layer.

Original: The current estimate of the fixed point is first passed through the data layer before being concatenated with the input injection and processed by the mixing layer.
Revised: The current estimate of the fixed point is first passed through the data layer before being concatenated with the input injection and processed by the mixing layer.

Original: This process repeats until a fixed point is found.
Revised: This process repeats until a fixed point is found.

Original: Although Figure 1 shows the simplest way of finding the fixed point, in practice acceleration or other fixed point solvers are applied to solve for the fixed point. We apply Anderson acceleration (Walker & Ni, 2011) for all of our models.
Revised: In practice, convergence to a fixed point is achieved when the difference between two time steps | r ( k +1) − r ( k ) | is within a small ϵ or when the maximum number of iterations is reached.

Original: [Figure]
Revised: A more thorough summary can be found in Fung et al.

Original: At each stage, the shallow equilibrium regularizer attempts to output its best estimate of the ground truth x ∗ .
Revised: At each stage, the shallow equilibrium regularizer attempts to output its best estimate of the ground truth x ∗ .

Original: We introduce a skip connection between the input d k and the final output so that the regularizer is learning the residual r between the input d k and the ground truth x ∗ instead.
Revised: We introduce a skip connection between the input d k and the final output so that the regularizer is learning the residual r between the input d k and the ground truth x ∗ instead, similar to the approach adopted by DnCNNs Ryu et al.

Original: When x k converges to x ∗ , d k will also converge to x ∗ and we expect the residual to be closer and closer to the zero vector.
Revised: When x k converges to x ∗ , d k will also converge to x ∗ and we expect the residual to be closer and closer to the zero vector.

Original: Therefore, we initialize the input of data layer with a zero vector with the same dimensions as the input d k in the hopes that fewer iterations will be needed at later stages as the current estimate x k converges towards the ground truth.
Revised: Therefore, we initialize the input of data layer with a zero vector with the same dimensions as the input d k in the hopes that fewer iterations will be needed at later stages as the current estimate x k converges towards the ground truth.

Original: Let r ( i ) denote the i th update of the residual, where r (0) = 0 and r ( ∞ ) be the fixed point solution of the residual.
Revised: Let r ( i ) denote the i th update of the residual, where r (0) = 0 and r ( ∞ ) be the fixed point solution of the residual.

Original: Let ⊕ denote the concatenation operator.
Revised: Let ⊕ denote the concatenation operator.

Original: The process in one loop unrolled block can be formulated as the following:
Revised: The process in one loop unrolled block can be formulated as the following:

Original: [Equation]
Revised: [Equation]

Original: Because of  the fixed  point solver within each regularization block, LUSERuses fewer layers to achieve a similar level of performance, thus we call LUSER a shallow equilibrium regularizer.
Revised: Although Figure 1 shows the simplest way of finding the fixed point, in practice acceleration or other fixed point solvers are applied to solve for the fixed point. Similar to Gilton et al. (2021), we also apply Anderson acceleration (Walker & Ni, 2011) for all of our models when searching for the fixed point. Using a fixed point solver based architecture for the regularization block allows LUSER to achieve a similar level of performance with a much shallower number of layers than traditional feedforward models.

Revised: For the scope of this article, we restrict ourselves to a total of 5 layers to demonstrate memory savings, but initial experiments have shown that increasing the number of layers can yield higher performance.

Original: We also explore two variants of LUSER, dubbed LUSER-SW and LUSER-DW.
Revised: We also explore two variants of LUSER, dubbed LUSER-SW and LUSER-DW.

Original: LUSER-SW refers to a shared-weight version of the proposed algorithm, where the proximal operator in all loop unrolled iterations are weight-tied and thus identical.
Revised: LUSER-SW refers to a shared-weight version of the proposed algorithm, where the proximal operator in all loop unrolled iterations are weight-tied and thus identical.

Original: Theoretically, the regularizer should be able to handle any input regardless of the iteration step.
Revised: Theoretically, the regularizer should be able to handle any input regardless of the iteration step.

Original: However, in practice, the distribution of intermediate reconstructions can be vastly different.
Revised: However, in practice, the distribution of intermediate reconstructions can be vastly different.

Original: Thus, training a single model to handle all these instances can be a daunting task, leading to poor generalization across the different stages.
Revised: Thus, training a single model to handle all these instances can be a daunting task, leading to poor generalization across the different stages.

Original: Since the total computational budget is fixed, one approach can be to use different weights (DW) for the learned proximal operator at each stage to handle the potentially different distributions.
Revised: Since the total computational budget is fixed, one approach can be to use different weights (DW) for the learned proximal operator at each stage to handle the potentially different distributions.

Original: This will increase the number of parameters that need to be trained and stored, but since LUSER already has so few parameters to begin with, expanding to the different weight variant is still feasible.
Revised: This will increase the number of parameters that need to be trained and stored, but since LUSER already has so few parameters to begin with, expanding to the different weight variant is still feasible.

Original: This paper will compare two variants of LUSER with other architectures in different tasks.
Revised: This paper will compare two variants of LUSER with other architectures in different tasks.

Original: DEQ4IP relies on the learned proximal operator to have the same weights, thus we cannot include a different weight variant for comparison.
Revised: DEQ4IP relies on the learned proximal operator to have the same weights, thus we cannot include a different weight variant for comparison.

Original: On the other hand, the same principles can be applied to the
Revised: On the other hand, the same principles can be applied to the

Original: LU variant of DEQ4IP.
Revised: LU variant of DEQ4IP.

Original: However, we aim to only compare models of similar number of parameters (or less in the case of LUSER-SW), and thus restrict our attentions to the shared weight variant of
Revised: However, we aim to only compare models of similar number of parameters (or less in the case of LUSER-SW), and thus restrict our attentions to the shared weight variant of

Original: LU only.
Revised: LU only.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: In this section, we compare our proposed networks to LU with DnCNN as proximal operator (LUDnCNN) and DEQ4IP on three different tasks: image deblurring, computed tomography (CT), and single-coil accelerated Magnetic Resonance Imaging (MRI).
Revised: In this section, we compare our proposed networks to LU with DnCNN as proximal operator (LUDnCNN) and DEQ4IP on three different tasks: image deblurring, computed tomography (CT), and single-coil accelerated Magnetic Resonance Imaging (MRI).

Original: The datasets we use are RGB CelebA
Revised: The datasets we use are RGB CelebA

Original: (Liu et al., 2015), LoDoPaB-CT (Leuschner et al., 2021), and single-coil knee data from fastMRI (Zbontar et al., 2018) respectively.
Revised: (Liu et al., 2015), LoDoPaB-CT (Leuschner et al., 2021), and single-coil knee data from fastMRI (Zbontar et al., 2018) respectively.

Original: We also experiment with incorporating an auxiliary loss of including an MSE loss on intermediate reconstructions with the ground-truth instead of just the final output.
Revised: We also experiment with incorporating an auxiliary loss of including an MSE loss on intermediate reconstructions with the ground-truth instead of just the final output.

Original: This is done for the LU models (LU-DnCNN, LUSER).
Revised: This is done for the LU models (LU-DnCNN, LUSER).

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: In the image deblurring task, the blurry image is obtained by applying a (9 × 9) Gaussian kernel with variance of 5 to an image with additive white Gaussian noise with a standard deviation of 0.01.
Revised: In the image deblurring task, the blurry image is obtained by applying a (9 × 9) Gaussian kernel with variance of 5 to an image with additive white Gaussian noise with a standard deviation of 0.01.

Original: If the image is RGB, the same kernel is applied to all channels.
Revised: If the image is RGB, the same kernel is applied to all channels.

Original: For accelerated MRI tasks, measurements in k-space (or frequency domain) are often subsampled due to the cost in measurement.
Revised: For accelerated MRI tasks, measurements in k-space (or frequency domain) are often subsampled due to the cost in measurement.

Original: The goal of MRI reconstruction is to recover the underlying physical structure from subsampled noisy measurements.
Revised: The goal of MRI reconstruction is to recover the underlying physical structure from subsampled noisy measurements.

Original: We simulate the forward operator A with a 2-dimensional Fourier transform with randomly selected rows.
Revised: We simulate the forward operator A with a 2-dimensional Fourier transform with randomly selected rows.

Original: We consider two common subsampling scenarios: 4x and 8x acceleration, or subsampling the columns in full measurement by a factor of 4 and 8 respectively.
Revised: We consider two common subsampling scenarios: 4 × and 8 × acceleration, or subsampling the columns in full measurement by a factor of 4 and 8 respectively.

Original: For the CT task, the forward operator is a Radon transform, and we uniformly select 200 out ofangles in measurements.
Revised: For the CT task, the forward operator is a Radon transform, and we uniformly select 200 out ofangles in measurements.

Original: The adjoint of measurement A ⊤ y is used as initialization for MRI and CT tasks, which brings the measurement back to the signal domain.
Revised: The adjoint of measurement A ⊤ y is used as initialization for MRI and CT tasks, which brings the measurement back to the signal domain.

Original: However, in the deblurring task, since the measurement lies in a same domain as the underlying clean image, y is used as the initial guess.
Revised: However, in the deblurring task, since the measurement lies in a same domain as the underlying clean image, y is used as the initial guess.

Original: Notice that some works use the filtered backprojection as the initialization for CT, such as (Khorashadizadeh et al., 2022), but we use the adjoint for the purpose of consistency.[Table] [Table] [Table]
Revised: Notice that some works use the filtered backprojection as the initialization for CT, such as (Khorashadizadeh et al., 2022), but we use the adjoint for the purpose of consistency.

Original: :1, C out :32, ks:3, pad:1))

Original: GN + LeakyReLU

Original: Mix Layer

Original: SN(conv( C in :64, C out :64, ks:3, pad:1))

Original: GN + LeakyReLU

Original: SN(conv( C in :64, C out :64, ks:3, pad:1))

Original: GN + LeakyReLU

Original: SN(conv( C in :64, C out :1, ks:3, pad:1))

Original: We fix the budget of LU-DnCNN and LUSER to be a total of 8 iterations, while as we allow DEQ4IP to iterate until it reaches a fixed point.
Revised: We fix the budget of LU-DnCNN and LUSER to be a total of 8 iterations, while as we allow DEQ4IP to iterate until it reaches a fixed point.

Original: We use a DnCNN adopted from Ryu et al.
Revised: We use a DnCNN adopted from Ryu et al.

Original: with 17 convolutional layers with 64 channels, followed by BatchNorm and ReLU activations for the regularizer for LU-DnCNN and DEQ4IP.
Revised: with 17 convolutional layers with 64 channels, followed by BatchNorm and ReLU activations for the regularizer for LU-DnCNN and DEQ4IP.

Original: For the learned regularizer update in LUSER, we useconvolutional layers each for the input injection layer and data layer.
Revised: For the learned regularizer update in LUSER, we useconvolutional layers each for the input injection layer and data layer.

Original: The mixing layer containsconvolutional layers.
Revised: The mixing layer containsconvolutional layers.

Original:  In order to stablize training for DEQ inspired models, we wrap all convolutional layers in DEQ4IP and LUSER with Spectral Norm (SN) (Miyato et al., 2018).
Revised: [Table] In order to stablize training for DEQ inspired models, we wrap all convolutional layers in DEQ4IP and LUSER with Spectral Norm (SN) (Miyato et al., 2018).

Original: We list more details for the learned proximal network for LUSER in Table 1 for the case when the input has a single channel.
Revised: We list more details for the learned proximal network for LUSER in Table 1 for the case when the input has a single channel.

Original: C in and
Revised: C in and

Original: C out denote the input and output channels, ks refers to the kernel size of a convolutional layer, pad denotes the padding in 2-dimension, and GN stands for GroupNorm.
Revised: C out denote the input and output channels, ks refers to the kernel size of a convolutional layer, pad denotes the padding in 2-dimension, and GN stands for GroupNorm.

Original: We use two metrics to evaluate the quality of reconstruction: Peak Signal-to-Noise Ratio (PSNR) in dB and the Structural Similarity Index (SSIM): [Equation] where MSE ( x , (cid:98) x ) = E [ ∥ x − (cid:98) x
Revised: We use two metrics to evaluate the quality of reconstruction: Peak Signal-to-Noise Ratio (PSNR) in dB and the Structural Similarity Index (SSIM).

Original: ∥ 2 ] is the mean squared error.

Original: In SSIM, µ x and µ (cid:98) x are pixel means of x and (cid:98) x , σ 2 x and σ 2 (cid:98) x are variances of x and (cid:98) x , and σ x (cid:98) x is their covariance.

Original: C 1 and C 2 are small positive numbers to stabilize SSIM.

Original: Note that although we use the same models as Gilton et al.
Revised: Note that although we use the same models as Gilton et al.

Original: (2021), we train our models from scratch and report lower values on the MRI task.
Revised: (2021), we train our models from scratch and report lower values on the MRI task.

Original: We suspect that this is due to evaluating with a single channel only.
Revised: We suspect that this is due to evaluating with a single channel only.

Original: When we include the imaginary channel (for a total of 2 channels), the metrics we recorded are more aligned with those reported in Gilton et al.
Revised: When we include the imaginary channel (for a total of 2 channels), the metrics we recorded are more aligned with those reported in Gilton et al.

Original: All models are trained with a single RTX6000 24GB GPU.
Revised: All models are trained with a single RTX6000 24GB GPU.

Original: 4.M AIN R ESULTS
Revised: 4. R ECONSTRUCTION Q UALITY

Original: Table 2 compares the average  PSNR and SSIM.
Revised: Table 2 compares the average testing PSNR and SSIM.

Original: The different weight version of LUSER outperforms LU-DnCNN with a similar number of network parameters.
Revised: The different weight version of LUSER outperforms LU-DnCNN with a similar number of network parameters as specified in Table 3.

Original: LUSER-SW achieves similar level of performance in most tasks with only 5 layers, versus 17 layers in LU-DnCNN.
Revised: LUSERSW achieves similar level of performance in most tasks with only 5 layers, versus 17 layers in LUDnCNN.

Original: DEQ4IP attains the best performance in image delurring task, but it requires repetitive computation of forward/adjoint operators.
Revised: DEQ4IP attains the best performance in image delurring task and CT, but LUSER achieves comparable quality.

Original: Training shared-weight architectures with auxillary losses improves the reconstruction quality in most tasks.
Revised: Training shared-weight architectures with auxiliary losses improves the reconstruction quality in most tasks.

Original: [Table]
Revised: Note that we restricted ourselves to only 5 layers for LUSER-SW and LUSER-DW for all reported experiments to highlight the memory savings without degradation in performance. In an initial exploration, we noticed that increasing the number of layers can lead to boosts in performance.

Original: 8x MRI 27.
Revised: For example, in the CT task, LUSER-SW with 6 layers increases the PSNR and SSIM by 1.04 dB and 0.022 respectively.

Original: Figure 2 shows representative reconstruction results.
Revised: Figure 2 shows representative reconstruction results.

Original: The overall performances are similar, but LUSER-DW  attains better qualities in detailed structures, especially compared to LU-DnCNN.
Revised: LUSER-DW achieves higher PSNR and SSIM in some cases, and attains better qualities in detailed structures, especially compared to LU-DnCNN.

Original: The areas with improvements are emphasized with red boxes in the ground truth images. 
Revised: The areas with improvements are emphasized with red boxes in the ground truth images. [Table] Deblurring [Table]

Original: Table 3 compares the network sizes (number of parameters) as well as the maximum training batch sizes for three commonly seen GPU RAM capacities: 8 GB, 10 GB and 24 GB.
Revised: [Table] MRI ( [Table] [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Original: Batch sizes are recorded with maximum even numbers, except when it is 1 for stochastic gradient descent.
Revised: [Table] [Table] GB10 GB24 GB MRI

Original: Notice
Revised: [Table] 8 GB10 GB    24 GB 4. M EMORY IN T RAINING

Original: [Table]
Revised: GPU capacity is a major bottleneck for training large-scale loop unrolled networks as discussed earlier. Table 3 compares the network sizes (number of parameters) as well as the maximum training batch sizes for three commonly seen GPU RAM capacities: 8 GB, 10 GB and 24 GB. Batch sizes are recorded with maximum even numbers, except when it is 1 for stochastic gradient descent.

Original:  that in MRI, because the Fourier transformation is implemented in tensor form, the minimum batch size it can take is 2.
Revised: Notice that in MRI, because the Fourier transformation is implemented in tensor form, the minimum batch size it can take is 2.

Original: We use the batch size as a proxy for the memory requirements during training.
Revised: We use the batch size as a proxy for the memory requirements during training.

Original: Since DEQ4IP is an extension of LU-DnCNN, their networks are of the same size, but implicit DEQ models support larger batch sizes making DEQ4IP far more memory-efficient during training.
Revised: Since DEQ4IP is an extension of LU-DnCNN, their networks are of the same size, but implicit DEQ models support larger batch sizes making DEQ4IP far more memory-efficient during training.

Original: The advantages of using DEQ models for LUSER and DEQ4IP are particularly highlighted in the case of limited memory (smaller GPUs).
Revised: The advantages of using DEQ models for LUSER and DEQ4IP are particularly highlighted in the case of limited memory (smaller GPUs).

Original: LU-DnCNN is unable to even train for the CT and MRI task with limited memory constraints, while LUSER and DEQ4IP can.
Revised: LU-DnCNN is unable to even train for the CT and MRI task with limited memory constraints, while LUSER and DEQ4IP can.

Original: This pattern is expected to repeat for more large scale tasks where standard LU architectures will be unable to train at all due to memory requirements.
Revised: This pattern is expected to repeat for more large scale tasks where standard LU architectures will be unable to train at all due to memory requirements.

Original: It is important to note that memory requirements depend more on the depth of the network than the number of parameters.
Revised: It is important to note that memory requirements depend more on the depth of the network than the number of parameters.

Original: For example, even though LUSER-DW and LUSER-SW have different numbers of parameters, they share the same architecture/depth and thus use roughly the same amount of memory during training. 
Revised: For example, even though LUSER-DW and LUSER-SW have different numbers of parameters, they share the same architecture/depth and thus use roughly the same amount of memory during training. 4. I NFERENCE T IME

Original: We observe that LUSER takes longer to train in each epoch than LU-DnCNN, but the training converges in fewer iterations than LU-DnCNN.
Revised: The evaluation time of these models depend on the complexity of the forward model. For the experiments performed, taking a gradient step on average took 5.3e-5, 6.9e-4, and 6.6e-3 seconds for deblurring, MRI and CT respectively. In this regime, the time to evaluate the DEQ models dominates the inference time. However, as the forward model time increases, it has the potential to dominate.

Original: The total training time for all methods are roughly the same for the image deblurring and MRI tasks, but DEQ4IP takes longer totrain on the CTproblem as it is finding a fixed-point with a more complicated forward operator.
Revised: In order to test this, we introduce artificial delays of 5e-2 and 5e-1 seconds to the CT forward model and evaluate the models on the test set. Because the performance does not significantly vary (as the trained models didn’t change), we only report the run times in Figure 3. In these regimes, the choice of adopting a Loop Unrolled structure presents an advantage over DEQ4IP.

Original: During evaluation,
Revised: In our experiments,

Original: LU-DnCNN is the fastest, and the DEQ4IP and LUSER are in similar speed. When the forward operator is computationally intensive, LUSER evaluates faster than DEQ4IP.
Revised: DEQ4IP took approximately 30-50 iterations to reach a reasonable level of performance, leading to 30-50 calls of the forward model.

Original: Table 4 compares the various properties among the three architectures. [Table]
Revised: The inference time is shown in the shaded region in green of

Original: The experiments above show the results of LUSER-SW with only 5 layers as extreme case to highlight the effectiveness of even a vastly smaller network.
Revised: Figure 3a, while as LUSER is shown in red for a max iteration of 10-30. It is interesting to note that as the forward model run time increases, so does the overlap between LUSER and DEQ4IP, with

Original: The number of layers typically correlate with the expressiveness of a deep feedforward neural network. We also explore the effect of number of layers in the reconstruction quality for the learned proximal operators of LUSER.
Revised: LUSER evaluating faster on average somewhere between 1e-2 and 1e-1 seconds.

Original: Table 5 shows the average PSNR and SSIM with one extra layer in  the Mix Layer.
Revised: The reason becomes more evident in Figure 3b which show cases the breakdown in timing for the solid line example from Figure 3a.

Original: It raises the PSNR by 1 dB and SSIM by 0.2, but sacrifices some memory in training. This suggests that one can design the network in LUSER according to their need in reconstruction quality and memory constraints.
Revised: The blue bar represents the approximate portion of time spent evaluating the forward model while as the green and red represent the learned model portion (as well as any overhead) associated with DEQ4IP and LUSER respectively. As the forward modeling time

Original: [Table]
Revised: [Figure]

Revised: becomes non-negligible, the evaluation time suffers accordingly with LUSER evaluating faster even for the 5e-2 second delay experiment.

Original: 4. E FFECT OF WEIGHT SHARING
Revised: 4. E FFECT OF WEIGHT SHARING

Original: As shown in Table 2, LUSER-DW has better performance than LUSER-SW with equivalent block structure, but has a larger number of parameters.
Revised: As shown in Table 2, LUSER-DW has better performance than LUSER-SW with equivalent block structure, but has a larger number of parameters.

Original: We also explore the possibility of reusing multiple
Revised: We also explore the possibility of reusing multiple

Original: LUSER proximal blocks across loop unrolled iterations.
Revised: LUSER proximal blocks across loop unrolled iterations.

Original: For example, in MRI tasks, we repeat 4 different proximal blocks over 8 iterations.
Revised: For example, in MRI tasks, we repeat 4 different proximal blocks over 8 iterations.

Original: Let g i denote the i th proximal block where i = { 1 , 2 , 3 , 4 } , and we form the LUSER network with the sequence of proximal blocks in order of g 1 , g 1 , g 2 , g 2 , g 3 , g 3 , g 4 , g 4 .
Revised: Let g i denote the i th proximal block where i = { 1 , 2 , 3 , 4 } , and we form the LUSER network with the sequence of proximal blocks in order of g 1 , g 1 , g 2 , g 2 , g 3 , g 3 , g 4 , g 4 .

Original: Now, the number of parameters is only half of that in LUSER-DW, but maintains a similar level of performance.
Revised: Now, the number of parameters is only half of that in LUSER-DW, but maintains a similar level of performance.

Original: We denote this variant LUSER-PSW, which stands for partially shared-weight.
Revised: We denote this variant LUSER-PSW, which stands for partially shared-weight.

Original: Table 6 compares the average PSNR and SSIM of LUSER-SW, LUSERPSW and LUSER-DW with the same block structure, which are trained using final loss only. 
Revised: Table 4 compares the average PSNR and SSIM of LUSER-SW, LUSERPSW and LUSER-DW with the same block structure, which are trained using final loss only. [Table]

Revised: Table 5 summarizes various prorperties among the three architectures.

Revised: In particular, we refer to the expressiveness of the network loosely as the performance relative to the depth.

Revised: LUSER is able to achieve comparable performance to both DEQ4IP and LU with a much shallower network.

Revised: We denote the rows related to timing with the asterisk to include the simulated regimes as well, though for negligible forward models, DEQ4IP can be faster than LUSER.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Loop unrolling architectures with deep convolutional layers as the learned regularizer update are a popular approach for solving inverse problems.
Revised: Loop unrolling architectures with deep convolutional layers as the learned regularizer update are a popular approach for solving inverse problems.

Original: Although its variants achieve state-of-the-art results across a variety of tasks, LU algorithms incur a huge memory cost when training due to the requirement of saving all intermediate activations, sometimes even requiring multiple GPUs to train on complex tasks.
Revised: Although its variants achieve state-of-the-art results across a variety of tasks, LU algorithms incur a huge memory cost when training due to the requirement of saving all intermediate activations, sometimes even requiring multiple GPUs to train on complex tasks.

Original: DEQ4IP offers an interesting alternative via extending LU to infinitely many layers by finding a fixed point solution, but can be impractical when the forward/adjoint operators are nonlinear or larger in scale.
Revised: DEQ4IP offers an interesting alternative via extending LU to infinitely many layers by finding a fixed point solution, but can be impractical when the forward/adjoint operators are nonlinear or larger in scale.

Original: To address the memory issue, we proposed two variants of loop unrolling architectures with deep equilibrium models as the learned regularizer updates, LUSER-SW and LUSER-DW.
Revised: To address the memory issue, we proposed two variants of loop unrolling architectures with deep equilibrium models as the learned regularizer updates, LUSER-SW and LUSER-DW.

Original: We verify the memory savings (by comparing batch sizes) relative to loop unrolling algorithms with a DnCNN model.
Revised: We verify the memory savings (by comparing batch sizes) relative to loop unrolling algorithms with a DnCNN model.

Original: Across all tasks, LUSER-DW outperforms LU-DnCNN with a similar number of network parameters, while reducing the memory requirements by a factor of 5 or more.
Revised: Across all tasks, LUSER-DW outperforms LU-DnCNN with a similar number of network parameters, while reducing the memory requirements by a factor of 5 or more.

Original: LUSER offers a path forward for large-scale, complex, non-linear inverse problems that are currently infeasible to train with existing methods.