Original: HUMUS-Net: Hybrid Unrolled Multi-Scale Network Architecture for Accelerated MRI Reconstruction
Revised: HUMUS-Net: Hybrid Unrolled Multi-Scale Network Architecture for Accelerated MRI Reconstruction

Original: Abstract
Revised: Abstract

Original: In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements.
Revised: In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements.

Original: Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions.
Revised: Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions.

Original: However, current architectures heavily rely on convolutions, that are content-independent and have difﬁculties modeling long-range dependencies in images.
Revised: However, current architectures heavily rely on convolutions, that are content-independent and have difﬁculties modeling long-range dependencies in images.

Original: Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks.
Revised: Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks.

Original: These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures.
Revised: These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures.

Original: However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve ﬁne detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect.
Revised: However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve ﬁne detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect.

Original: To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneﬁcial implicit bias and efﬁciency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network.
Revised: To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneﬁcial implicit bias and efﬁciency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network.

Original: HUMUS-Net extracts high-resolution features via convolutional blocks and reﬁnes low-resolution features via a novel
Revised: HUMUS-Net extracts high-resolution features via convolutional blocks and reﬁnes low-resolution features via a novel

Original: Transformer-based multi-scale feature extractor.
Revised: Transformer-based multi-scale feature extractor.

Original: Features from both levels are then synthesized into a high-resolution output reconstruction.
Revised: Features from both levels are then synthesized into a high-resolution output reconstruction.

Original: Our network establishes new state of the art on the largest publicly available MRI dataset, the fastMRI dataset.
Revised: Our network establishes new state of the art on the largest publicly available MRI dataset, the fastMRI dataset.

Original: We further demonstrate the performance of HUMUS-Net on
Revised: We further demonstrate the performance of HUMUS-Net on

Original: two other popular MRI datasets and perform ﬁne-grained ablation studies to validate our design.
Revised: two other popular MRI datasets and perform ﬁne-grained ablation studies to validate our design.

Original:  Introduction
Revised:  Introduction

Original: Magnetic resonance imaging (MRI) is a medical imaging technique that uses strong magnetic ﬁelds to picture the anatomy and physiological processes of the patient.
Revised: Magnetic resonance imaging (MRI) is a medical imaging technique that uses strong magnetic ﬁelds to picture the anatomy and physiological processes of the patient.

Original: MRI is one of the most popular imaging modalities as it is non-invasive and doesn’t expose the patient to harmful ionizing radiation.
Revised: MRI is one of the most popular imaging modalities as it is non-invasive and doesn’t expose the patient to harmful ionizing radiation.

Original: The MRI scanner obtains measurements of the body in the spatial frequency domain also called k-space .
Revised: The MRI scanner obtains measurements of the body in the spatial frequency domain also called k-space .

Original: The data acquisition process is MR is often time-consuming.
Revised: The data acquisition process is MR is often time-consuming.

Original: Accelerated MRI
Revised: Accelerated MRI

Original: [Lustig et al., 2008] addresses this challenge by undersampling in the k-space domain, thus reducing the time patients need to spend in the scanner.
Revised: [Lustig et al., 2008] addresses this challenge by undersampling in the k-space domain, thus reducing the time patients need to spend in the scanner.

Original: However, recovering the underlying anatomy from undersampled measurements is an ill-posed problem (less measurements than unknowns) and thus incorporating some form of prior knowledge is crucial in obtaining high quality reconstructions.
Revised: However, recovering the underlying anatomy from undersampled measurements is an ill-posed problem (less measurements than unknowns) and thus incorporating some form of prior knowledge is crucial in obtaining high quality reconstructions.

Original: Classical MRI reconstruction algorithms rely on the assumption that the underlying signal is sparse in some transform domain and attempt to recover a signal that best satisﬁes this assumption in a technique known as compressed sensing (CS)
Revised: Classical MRI reconstruction algorithms rely on the assumption that the underlying signal is sparse in some transform domain and attempt to recover a signal that best satisﬁes this assumption in a technique known as compressed sensing (CS)

Original: [Candes et al., 2006, Donoho, 2006].
Revised: [Candes et al., 2006, Donoho, 2006].

Original: These classical
Revised: These classical

Original: CS techniques have slow reconstruction speed and typically enforce limited forms of image priors.
Revised: CS techniques have slow reconstruction speed and typically enforce limited forms of image priors.

Original: With the emergence of deep learning (DL), data-driven reconstruction algorithms have far surpassed
Revised: With the emergence of deep learning (DL), data-driven reconstruction algorithms have far surpassed

Original: CS techniques (see Ongie et al.
Revised: CS techniques (see Ongie et al.

Original: [2020] for an overview).
Revised: [2020] for an overview).

Original: DL models utilize large training datasets to extract ﬂexible, nuanced priors directly from data resulting in excellent reconstruction quality.
Revised: DL models utilize large training datasets to extract ﬂexible, nuanced priors directly from data resulting in excellent reconstruction quality.

Original: In recent years, there has been a ﬂurry of activity aimed at designing DL architectures tailored to the MRI reconstruction problem.
Revised: In recent years, there has been a ﬂurry of activity aimed at designing DL architectures tailored to the MRI reconstruction problem.

Original: The most popular models are convolutional neural networks (CNNs) that typically incorporate the physics of the MRI reconstruction problem, and utilize tools from mainstream deep learning (residual learning, data augmentation, self-supervised learning).
Revised: The most popular models are convolutional neural networks (CNNs) that typically incorporate the physics of the MRI reconstruction problem, and utilize tools from mainstream deep learning (residual learning, data augmentation, self-supervised learning).

Original: Comparing the performance of such models has been difﬁcult mainly due to two reasons.
Revised: Comparing the performance of such models has been difﬁcult mainly due to two reasons.

Original: First, there has been a large variation in evaluation datasets spanning different scanners, anatomies, acquisition models and undersampling patterns rendering direct comparison challenging.
Revised: First, there has been a large variation in evaluation datasets spanning different scanners, anatomies, acquisition models and undersampling patterns rendering direct comparison challenging.

Original: Second, medical imaging datasets are often proprietary due to privacy concerns, hindering reproducibility.
Revised: Second, medical imaging datasets are often proprietary due to privacy concerns, hindering reproducibility.

Original: More recently, the fastMRI dataset [Zbontar et al., 2019], the largest publicly available MRI dataset, has been gaining ground as a standard benchmark to evaluate MRI reconstruction methods.
Revised: More recently, the fastMRI dataset [Zbontar et al., 2019], the largest publicly available MRI dataset, has been gaining ground as a standard benchmark to evaluate MRI reconstruction methods.

Original: An annual competition, the fastMRI Challenge [Muckley et al., 2021], attracts signiﬁcant attention from the machine learning community and acts a driver of innovation in MRI reconstruction.
Revised: An annual competition, the fastMRI Challenge [Muckley et al., 2021], attracts signiﬁcant attention from the machine learning community and acts a driver of innovation in MRI reconstruction.

Original: However, over the past years the public leaderboard has been dominated by a single architecture, the End-to-End VarNet [Sriram et al., 2020] 1 , with most models concentrating very closely around the same performance metrics, hinting at the saturation of current architectural choices.
Revised: However, over the past years the public leaderboard has been dominated by a single architecture, the End-to-End VarNet [Sriram et al., 2020] 1 , with most models concentrating very closely around the same performance metrics, hinting at the saturation of current architectural choices.

Original: In this work, we propose HUMUS-Net: a Hybrid, Unrolled, MUlti-Scale network architecture for accelerated MRI reconstruction that combines the advantages of well-established architectures in the ﬁeld with the power of contemporary Transformer-based models.
Revised: In this work, we propose HUMUS-Net: a Hybrid, Unrolled, MUlti-Scale network architecture for accelerated MRI reconstruction that combines the advantages of well-established architectures in the ﬁeld with the power of contemporary Transformer-based models.

Original: We utilize the strong implicit bias of convolutions, but also address their weaknesses, such as content-independence and inability to model long-range dependencies, by incorporating a novel multi-scale feature extractor that operates over embedded image patches via self-attention.
Revised: We utilize the strong implicit bias of convolutions, but also address their weaknesses, such as content-independence and inability to model long-range dependencies, by incorporating a novel multi-scale feature extractor that operates over embedded image patches via self-attention.

Original: Moreover, we tackle the challenge of high input resolution typical in MRI by performing the computationally most expensive operations on extracted low-resolution features.
Revised: Moreover, we tackle the challenge of high input resolution typical in MRI by performing the computationally most expensive operations on extracted low-resolution features.

Original: HUMUS-Net establishes new state of the art in accelerated
Revised: HUMUS-Net establishes new state of the art in accelerated

Original: MRI reconstruction on the largest available MRI knee dataset.
Revised: MRI reconstruction on the largest available MRI knee dataset.

Original: At the time of writing this paper,
Revised: At the time of writing this paper,

Original: HUMUS-Net is the only Transformer-based architecture on the highly competitive fastMRI Public
Revised: HUMUS-Net is the only Transformer-based architecture on the highly competitive fastMRI Public

Original: Leaderboard.
Revised: Leaderboard.

Original: Our results are fully reproducible and the source code is available online
Revised: Our results are fully reproducible and the source code is available online

Original:  Background 2.1 Inverse Problem Formulation of Accelerated MRI Reconstruction
Revised:  Background 2.1 Inverse Problem Formulation of Accelerated MRI Reconstruction

Original: An MR scanner obtains measurements of the patient anatomy in the frequency domain, also referred to as k-space .
Revised: An MR scanner obtains measurements of the patient anatomy in the frequency domain, also referred to as k-space .

Original: Data acquisition is performed via various receiver coils positioned around the anatomy being imaged, each with different spatial sensitivity.
Revised: Data acquisition is performed via various receiver coils positioned around the anatomy being imaged, each with different spatial sensitivity.

Original: Given a total number of N receiver coils, the measurements obtained by the i th coil can be written as
Revised: Given a total number of N receiver coils, the measurements obtained by the i th coil can be written as

Original: [Equation]
Revised: [Equation]

Original: where x ∗ ∈ C n is the underlying patient anatomy of interest, S i is a diagonal matrix that represents the sensitivity map of the i th coil, F is a multi-dimensional Fourier-transform, and z i denotes the measurement noise corrupting the observations obtained from coil i .
Revised: where x ∗ ∈ C n is the underlying patient anatomy of interest, S i is a diagonal matrix that represents the sensitivity map of the i th coil, F is a multi-dimensional Fourier-transform, and z i denotes the measurement noise corrupting the observations obtained from coil i .

Original: We use k = ( k 1 , ..., k N ) as a shorthand for the concatenation of individual coil measurements and x = ( x 1 , ..., x N ) as the corresponding image domain representation after inverse Fourier transformation.
Revised: We use k = ( k 1 , ..., k N ) as a shorthand for the concatenation of individual coil measurements and x = ( x 1 , ..., x N ) as the corresponding image domain representation after inverse Fourier transformation.

Original: Since MR data acquisition time is proportional to the portion of k-space being scanned, obtaining fully-sampled data is time-consuming.
Revised: Since MR data acquisition time is proportional to the portion of k-space being scanned, obtaining fully-sampled data is time-consuming.

Original: Therefore, in accelerated MRI scan times are reduced by undersampling in k-space domain.
Revised: Therefore, in accelerated MRI scan times are reduced by undersampling in k-space domain.

Original: The undersampled k-space measurements from coil i take the form
Revised: The undersampled k-space measurements from coil i take the form

Original: [Equation]
Revised: [Equation]

Original: where M is a diagonal matrix representing the binary undersampling mask, that has 0 values for all missing frequency components that have not been sampled during accelerated acquisition.
Revised: where M is a diagonal matrix representing the binary undersampling mask, that has 0 values for all missing frequency components that have not been sampled during accelerated acquisition.

Original: The forward model that maps the underlying anatomy to coil measurements can be written concisely as ˜ k
Revised: The forward model that maps the underlying anatomy to coil measurements can be written concisely as ˜ k

Original: = A ( x ∗ ) , where A ( · ) is the linear forward mapping and ˜ k is the stacked vector of all undersampled coil measurements.
Revised: = A ( x ∗ ) , where A ( · ) is the linear forward mapping and ˜ k is the stacked vector of all undersampled coil measurements.

Original: Our target is to reconstruct the ground truth object x ∗ from the noisy, undersampled measurements ˜ k .
Revised: Our target is to reconstruct the ground truth object x ∗ from the noisy, undersampled measurements ˜ k .

Original: Since we have fewer observations than variables to recover, perfect reconstruction in general is not possible.
Revised: Since we have fewer observations than variables to recover, perfect reconstruction in general is not possible.

Original: In order to make the problem solvable, prior knowledge on the underlying object is typically incorporated in the form of sparsity in some transform domain.
Revised: In order to make the problem solvable, prior knowledge on the underlying object is typically incorporated in the form of sparsity in some transform domain.

Original: This formulation, known as compressed sensing [Candes et al., 2006, Donoho, 2006], provides a classical framework for accelerated MRI reconstruction [Lustig et al., 2008].
Revised: This formulation, known as compressed sensing [Candes et al., 2006, Donoho, 2006], provides a classical framework for accelerated MRI reconstruction [Lustig et al., 2008].

Original: In particular, the above recovery problem can be formulated as a regularized inverse problem
Revised: In particular, the above recovery problem can be formulated as a regularized inverse problem

Original: [Equation]
Revised: [Equation]

Original: where R ( · ) is a regularizer that encapsulates prior knowledge on the object, such as sparsity in some wavelet domain.
Revised: where R ( · ) is a regularizer that encapsulates prior knowledge on the object, such as sparsity in some wavelet domain.

Original: 2.2 Deep Learning-based Accelerated MRI Reconstruction
Revised: 2.2 Deep Learning-based Accelerated MRI Reconstruction

Original: More recently, data-driven deep learning-based algorithms tailored to the accelerated MRI reconstruction problem have surpassed the classical compressed sensing baselines.
Revised: More recently, data-driven deep learning-based algorithms tailored to the accelerated MRI reconstruction problem have surpassed the classical compressed sensing baselines.

Original: Convolutional neural networks trained on large datasets have established new state of the art in many medical imaging tasks.
Revised: Convolutional neural networks trained on large datasets have established new state of the art in many medical imaging tasks.

Original: The highly popular U-Net [Ronneberger et al., 2015] and other similar encoder-decoder architectures have proven to be successful in a range of medical image reconstruction [Hyun et al., 2018, Han and
Revised: The highly popular U-Net [Ronneberger et al., 2015] and other similar encoder-decoder architectures have proven to be successful in a range of medical image reconstruction [Hyun et al., 2018, Han and

Original: Ye, 2018] and segmentation [Çiçek et al., 2016, Zhou et al., 2018] problems.
Revised: Ye, 2018] and segmentation [Çiçek et al., 2016, Zhou et al., 2018] problems.

Original: In the encoder path, the network learns to extract a set of deep, low-dimensional features from images via a series of convolutional and downsampling operations.
Revised: In the encoder path, the network learns to extract a set of deep, low-dimensional features from images via a series of convolutional and downsampling operations.

Original: These concise feature representations are then gradually upsampled and ﬁltered in the decoder to the original image dimensions.
Revised: These concise feature representations are then gradually upsampled and ﬁltered in the decoder to the original image dimensions.

Original: Thus the network learns a hierarchical representation over the input image distribution.
Revised: Thus the network learns a hierarchical representation over the input image distribution.

Original: Unrolled networks constitute another line of work that has been inspired by popular optimization algorithms used to solve compressed sensing reconstruction problems.
Revised: Unrolled networks constitute another line of work that has been inspired by popular optimization algorithms used to solve compressed sensing reconstruction problems.

Original: These deep learning models consist of a series of sub-networks, also known as cascades, where each sub-network corresponds to an unrolled iteration of popular algorithms such as gradient descent [Zhang and Ghanem, 2018] or
Revised: These deep learning models consist of a series of sub-networks, also known as cascades, where each sub-network corresponds to an unrolled iteration of popular algorithms such as gradient descent [Zhang and Ghanem, 2018] or

Original: ADMM [Sun et al., 2016].
Revised: ADMM [Sun et al., 2016].

Original: In the context of MRI reconstruction, one can view network unrolling as solving a sequence of smaller denoising problems, instead of the complete recovery problem in one step.
Revised: In the context of MRI reconstruction, one can view network unrolling as solving a sequence of smaller denoising problems, instead of the complete recovery problem in one step.

Original: Various convolutional neural networks have been employed in the unrolling framework achieving excellent performance in accelerated MRI reconstruction [Putzky et al., 2019, Hammernik et al., 2018, 2019].
Revised: Various convolutional neural networks have been employed in the unrolling framework achieving excellent performance in accelerated MRI reconstruction [Putzky et al., 2019, Hammernik et al., 2018, 2019].

Original: E2E-VarNet [Sriram et al., 2020] is the current state-of-the-art convolutional model on the fastMRI dataset.
Revised: E2E-VarNet [Sriram et al., 2020] is the current state-of-the-art convolutional model on the fastMRI dataset.

Original: E2E-VarNet transforms the optimization problem in (2.1) to the k-space domain and unrolls the gradient descent iterations into T cascades, where the t th cascade represents the computation
Revised: E2E-VarNet transforms the optimization problem in (2.1) to the k-space domain and unrolls the gradient descent iterations into T cascades, where the t th cascade represents the computation

Original: [Equation]
Revised: [Equation]

Original: where ˆ k t is the estimated reconstruction in the k-space domain at cascade t , µ t is a learnable step size parameter and G ( · ) is a learned mapping representing the gradient of the regularization term in (2.1).
Revised: where ˆ k t is the estimated reconstruction in the k-space domain at cascade t , µ t is a learnable step size parameter and G ( · ) is a learned mapping representing the gradient of the regularization term in (2.1).

Original: The ﬁrst term is also known as data consistency (DC it consistency estimate with the available measurements.
Revised: The ﬁrst term is also known as data consistency (DC it consistency estimate with the available measurements.

Original:  Related Work
Revised:  Related Work

Original: Transformers in Vision – Vision Transformer (ViT) [Dosovitskiy et al., 2020], a fully nonconvolutional vision architecture, has demonstrated state-of-the-art performance on image classiﬁcation problems when pre-trained on large-scale image datasets.
Revised: Transformers in Vision – Vision Transformer (ViT) [Dosovitskiy et al., 2020], a fully nonconvolutional vision architecture, has demonstrated state-of-the-art performance on image classiﬁcation problems when pre-trained on large-scale image datasets.

Original: The key idea of ViT is to split the input image into non-overlapping patches, embed each patch via a learned linear mapping and process the resulting tokens via stacked self-attention and multi-layer perceptron (MLP) blocks.
Revised: The key idea of ViT is to split the input image into non-overlapping patches, embed each patch via a learned linear mapping and process the resulting tokens via stacked self-attention and multi-layer perceptron (MLP) blocks.

Original: For more details we refer the reader to Appendix G and [Dosovitskiy et al., 2020].
Revised: For more details we refer the reader to the supplementary and [Dosovitskiy et al., 2020].

Original: The beneﬁt of Transformers over convolutional architectures in vision lies in their ability to capture long-range dependencies in images via the self-attention mechanism.
Revised: The beneﬁt of Transformers over convolutional architectures in vision lies in their ability to capture long-range dependencies in images via the self-attention mechanism.

Original: Since the introduction of ViT, similar attention-based architectures have been proposed for many other vision tasks such as object detection [Carion et al., 2020], image segmentation [Wang et al., 2021b] and restoration [Cao et al., 2021b, Chen et al., 2021, Liang et al., 2021, Zamir et al., 2021, Wang et al., 2021c].
Revised: Since the introduction of ViT, similar attention-based architectures have been proposed for many other vision tasks such as object detection [Carion et al., 2020], image segmentation [Wang et al., 2021b] and restoration [Cao et al., 2021b, Chen et al., 2021, Liang et al., 2021, Zamir et al., 2021, Wang et al., 2021c].

Original: A key challenge for Transformers in low-level vision problems is the quadratic compute complexity of the global self-attention with respect to the input dimension.
Revised: A key challenge for Transformers in low-level vision problems is the quadratic compute complexity of the global self-attention with respect to the input dimension.

Original: In some works, this issue has been mitigated by splitting the input image into ﬁxed size patches and processing the patches independently [Chen et al., 2021].
Revised: In some works, this issue has been mitigated by splitting the input image into ﬁxed size patches and processing the patches independently [Chen et al., 2021].

Original: Others focus on designing hierarchical Transformer architectures [Heo et al., 2021, Wang et al., 2021b] similar to popular ResNets
Revised: Others focus on designing hierarchical Transformer architectures [Heo et al., 2021, Wang et al., 2021b] similar to popular ResNets

Original: [He et al., 2015].
Revised: [He et al., 2015].

Original: Authors in Zamir et al.
Revised: Authors in Zamir et al.

Original: [2021] propose applying self-attention channel-wise rather than across the spatial dimension thus reducing the compute overhead to linear complexity.
Revised: [2021] propose applying self-attention channel-wise rather than across the spatial dimension thus reducing the compute overhead to linear complexity.

Original: Another successful architecture, the Swin
Revised: Another successful architecture, the Swin

Original: Transformer [Liu et al., 2021], tackles the quadratic scaling issue by computing self-attention on smaller local windows.
Revised: Transformer [Liu et al., 2021], tackles the quadratic scaling issue by computing self-attention on smaller local windows.

Original: To encourage cross-window interaction, windows in subsequent layers are shifted relative to each other.
Revised: To encourage cross-window interaction, windows in subsequent layers are shifted relative to each other.

Original: Transformers in Medical Imaging – Transformer architectures have been proposed recently to tackle various medical imaging problems.
Revised: Transformers in Medical Imaging – Transformer architectures have been proposed recently to tackle various medical imaging problems.

Original: Authors in Cao et al.
Revised: Authors in Cao et al.

Original: [2021a] design a U-Net-like architecture for medical image segmentation where the traditional convolutional layers are replaced by Swin
Revised: [2021a] design a U-Net-like architecture for medical image segmentation where the traditional convolutional layers are replaced by Swin

Original: Transformer blocks.
Revised: Transformer blocks.

Original: They report strong results on multi-organ and cardiac image segmentation.
Revised: They report strong results on multi-organ and cardiac image segmentation.

Original: In
Revised: In

Original: Zhou et al.
Revised: Zhou et al.

Original: [2021] a hybrid convolutional and Transformer-based U-Net architecture is proposed tailored to volumetric medical image segmentation with excellent results on benchmark datasets.
Revised: [2021] a hybrid convolutional and Transformer-based U-Net architecture is proposed tailored to volumetric medical image segmentation with excellent results on benchmark datasets.

Original: Similar encoder-decoder architectures for various medical segmentation tasks have been investigated in other works [Huang et al., 2021, Wu et al., 2022].
Revised: Similar encoder-decoder architectures for various medical segmentation tasks have been investigated in other works [Huang et al., 2021, Wu et al., 2022].

Original: However, these networks are tailored for image segmentation, a task less sensitive to ﬁne details in the input, and thus larger patch sizes are often used (for instance 4 in Cao et al.
Revised: However, these networks are tailored for image segmentation, a task less sensitive to ﬁne details in the input, and thus larger patch sizes are often used (for instance 4 in Cao et al.

Original: [2021a] ).
Revised: [2021a] ).

Original: This allows the network to process larger input images, as the number of token embedding is greatly reduced, but as we demonstrate in Section 5.2 embedding individual pixels as 1 × 1 patches is crucial for MRI reconstruction.
Revised: This allows the network to process larger input images, as the number of token embedding is greatly reduced, but as we demonstrate in Section 5.2 embedding individual pixels as 1 × 1 patches is crucial for MRI reconstruction.

Original: Thus, compute and memory barriers stemming from large input resolutions are more severe in the MRI reconstruction task and therefore novel approaches are needed.
Revised: Thus, compute and memory barriers stemming from large input resolutions are more severe in the MRI reconstruction task and therefore novel approaches are needed.

Original: Promising results have been reported employing Transformers in medical image denoising problems, such as low-dose CT denoising [Wang et al., 2021a, Luthra et al., 2021] and low-count PET/MRI denoising [Zhang et al., 2021].
Revised: Promising results have been reported employing Transformers in medical image denoising problems, such as low-dose CT denoising [Wang et al., 2021a, Luthra et al., 2021] and low-count PET/MRI denoising [Zhang et al., 2021].

Original: However, these studies fail to address the challenge of poor scaling to large input resolutions, and only work on small images via either downsampling the original dataset [Luthra et al., 2021] or by slicing the large input images into smaller patches [Wang et al., 2021a].
Revised: However, these studies fail to address the challenge of poor scaling to large input resolutions, and only work on small images via either downsampling the original dataset [Luthra et al., 2021] or by slicing the large input images into smaller patches [Wang et al., 2021a].

Original: In contrast, our proposed architecture works directly on the large resolution images that often arise in MRI reconstruction.
Revised: In contrast, our proposed architecture works directly on the large resolution images that often arise in MRI reconstruction.

Original: Even though some work exists on Transformer-based architectures for supervised accelerated MRI reconstruction [Huang et al., 2022, Lin and Heckel, 2022, Feng et al., 2021], and for unsupervised pre-trained reconstruction [Korkmaz et al., 2022], to the best of our knowledge ours is the ﬁrst work to demonstrate state-of-the-art results on large-scale MRI datasets such as the fastMRI dataset.
Revised: Even though some work exists on Transformer-based architectures for supervised accelerated MRI reconstruction [Huang et al., 2022, Lin and Heckel, 2022, Feng et al., 2021], and for unsupervised pre-trained reconstruction [Korkmaz et al., 2022], to the best of our knowledge ours is the ﬁrst work to demonstrate state-of-the-art results on large-scale MRI datasets such as the fastMRI dataset.

Original:  Method
Revised:  Method

Original: HUMUS-Net combines the efﬁciency and beneﬁcial implicit bias of convolutional networks with the powerful general representations of Transformers and their capability to capture long-range pixel dependencies.
Revised: HUMUS-Net combines the efﬁciency and beneﬁcial implicit bias of convolutional networks with the powerful general representations of Transformers and their capability to capture long-range pixel dependencies.

Original: The resulting hybrid network processes information both in image representation (via convolutions) and in patch-embedded token representation (via Transformer blocks).
Revised: The resulting hybrid network processes information both in image representation (via convolutions) and in patch-embedded token representation (via Transformer blocks).

Original: Our proposed architecture consists of a sequence of sub-networks, also called cascades.
Revised: Our proposed architecture consists of a sequence of sub-networks, also called cascades.

Original: Each cascade represents an unrolled iteration of an underlying optimization algorithm in k-space (see (2.2)), with an imagedomain denoiser, the HUMUS-Block.
Revised: Each cascade represents an unrolled iteration of an underlying optimization algorithm in k-space (see (2.2)), with an imagedomain denoiser, the HUMUS-Block.

Original: First, we describe the architecture of the HUMUS-Block, the core component in the sub-network.
Revised: First, we describe the architecture of the HUMUS-Block, the core component in the sub-network.

Original: Then, we specify the high-level, k-space unrolling architecture of HUMUS-Net in Section 4.3.
Revised: Then, we specify the high-level, k-space unrolling architecture of HUMUS-Net in Section 4.3.

Original: 4.1 HUMUS-Block Architecture
Revised: 4.1 HUMUS-Block Architecture

Original: The HUMUS-Block acts as an image-space denoiser that receives an intermediate reconstruction from the previous cascade and performs a single step of denoising to produce an improved reconstruction for the next cascade.
Revised: The HUMUS-Block acts as an image-space denoiser that receives an intermediate reconstruction from the previous cascade and performs a single step of denoising to produce an improved reconstruction for the next cascade.

Original: It extracts high-resolution, shallow features and low-resolution, deep features through a novel multi-scale transformer-based block, and synthesizes high-resolution features from those.
Revised: It extracts high-resolution, shallow features and low-resolution, deep features through a novel multi-scale transformer-based block, and synthesizes high-resolution features from those.

Original: The high-level overview of the HUMUS-Block is depicted in Fig.
Revised: The high-level overview of the HUMUS-Block is depicted in Fig.

Original: High-resolution Feature Extraction– The input to our network is a noisy complex-valued image x in ∈ R h × w × c in derived from undersampled k-space data, where the real and imaginary parts of the image are concatenated along the channel dimension.
Revised: High-resolution Feature Extraction– The input to our network is a noisy complex-valued image x in ∈ R h × w × c in derived from undersampled k-space data, where the real and imaginary parts of the image are concatenated along the channel dimension.

Original: First, we extract high-resolution features F H ∈ R h × w × d H from the input noisy image through a convolution layer f H written as
Revised: First, we extract high-resolution features F H ∈ R h × w × d H from the input noisy image through a convolution layer f H written as

Original: F H = f H ( x in ) .
Revised: F H = f H ( x in ) .

Original: This initial 3 × 3 convolution layer provides early visual processing at a relatively low cost and maps the input to a higher, d H dimensional feature space.
Revised: This initial 3 × 3 convolution layer provides early visual processing at a relatively low cost and maps the input to a higher, d H dimensional feature space.

Original: Important to note that the resolution of the extracted features is the same as the spatial resolution of the input image.
Revised: Important to note that the resolution of the extracted features is the same as the spatial resolution of the input image.

Original: Low-resolution Feature Extraction– In case of MRI reconstruction, input resolution is typically signiﬁcantly higher than commonly used image datasets ( 32 × 32 - 256 × 256 ), posing a signiﬁcant challenge to contemporary Transformer-based models.
Revised: Low-resolution Feature Extraction– In case of MRI reconstruction, input resolution is typically signiﬁcantly higher than commonly used image datasets ( 32 × 32 - 256 × 256 ), posing a signiﬁcant challenge to contemporary Transformer-based models.

Original: Thus we apply a convolutional feature extractor f L to obtain low-resolution features F L = f L ( F H ) , with F L ∈ R
Revised: Thus we apply a convolutional feature extractor f L to obtain low-resolution features F L = f L ( F H ) , with F L ∈ R

Original: h L × w L × d L where f L consists of a sequence of convolutional blocks and spatial downsampling operations.
Revised: h L × w L × d L where f L consists of a sequence of convolutional blocks and spatial downsampling operations.

Original: The speciﬁc architecture is depicted in Figure 4.
Revised: The speciﬁc architecture is depicted in Figure 4.

Original: The purpose of this module is to perform deeper visual processing and to provide manageable input size to the subsequent computation and memory heavy hybrid processing module.
Revised: The purpose of this module is to perform deeper visual processing and to provide manageable input size to the subsequent computation and memory heavy hybrid processing module.

Original: In this work, we choose h L = h 2 and w L = w 2 which strikes a balance between preserving spatial information and resource demands.
Revised: In this work, we choose h L = h 2 and w L = w 2 which strikes a balance between preserving spatial information and resource demands.

Original: Furthermore, in order to compensate for the reduced resolution we increase the feature dimension to d L = 2 · d H :=
Revised: Furthermore, in order to compensate for the reduced resolution we increase the feature dimension to d L = 2 · d H :=

Original: d .
Revised: d .

Original: Deep Feature Extraction– The most important part of our model is MUST, a MU lti-scale residual
Revised: Deep Feature Extraction– The most important part of our model is MUST, a MU lti-scale residual

Original: S win T ransformer network.
Revised: S win T ransformer network.

Original: MUST is a multi-scale hybrid feature extractor that takes the low- resolution image representations F L and performs hierarchical Transformer-convolutional hybrid processing in an encoder-decoder fashion, producing deep features F D = f D ( F L ) , where the speciﬁc architecture behind f D is detailed in Subsection 4.2.
Revised: MUST is a multi-scale hybrid feature extractor that takes the low- resolution image representations F L and performs hierarchical Transformer-convolutional hybrid processing in an encoder-decoder fashion, producing deep features F D = f D ( F L ) , where the speciﬁc architecture behind f D is detailed in Subsection 4.2.

Original: High-resolution Image Reconstruction– Finally, we combine information from shallow, highresolution features F H and deep, low-resolution features F D to reconstruct the high-resolution residual image via a convolutional reconstruction module f R .
Revised: High-resolution Image Reconstruction– Finally, we combine information from shallow, highresolution features F H and deep, low-resolution features F D to reconstruct the high-resolution residual image via a convolutional reconstruction module f R .

Original: The residual learning paradigm allows us to learn the difference between noisy and clean images and helps information ﬂow within the network [He et al., 2015].
Revised: The residual learning paradigm allows us to learn the difference between noisy and clean images and helps information ﬂow within the network [He et al., 2015].

Original: Thus the ﬁnal denoised image x out ∈ R h × w × c in is obtained as x out = x in + f R ( F H , F D ) .
Revised: Thus the ﬁnal denoised image x out ∈ R h × w × c in is obtained as x out = x in + f R ( F H , F D ) .

Original: The speciﬁc architecture of the reconstruction network is depicted in
Revised: The speciﬁc architecture of the reconstruction network is depicted in

Original: Figure 4.
Revised: Figure 4.

Original: 4.2 Multi-scale Hybrid Feature Extraction via MUST
Revised: 4.2 Multi-scale Hybrid Feature Extraction via MUST

Original: The key component to our architecture is MUST, a multi-scale hybrid encoder-decoder architecture that performs deep feature extraction in both image and token representation (Figure 1, bottom).
Revised: The key component to our architecture is MUST, a multi-scale hybrid encoder-decoder architecture that performs deep feature extraction in both image and token representation (Figure 1, bottom).

Original: First, individual pixels of the input representation of shape h 2 × w 2 ×
Revised: First, individual pixels of the input representation of shape h 2 × w 2 ×

Original: d are ﬂattened and passed through a learned linear mapping to yield h 2 · w 2 tokens of dimension d .
Revised: d are ﬂattened and passed through a learned linear mapping to yield h 2 · w 2 tokens of dimension d .

Original: Tokens corresponding to different image patches are subsequently merged in the encoder path, resulting in a concise latent representation.
Revised: Tokens corresponding to different image patches are subsequently merged in the encoder path, resulting in a concise latent representation.

Original: This highly descriptive representation is passed through a bottleneck block and progressively expanded by combining tokens from the encoder path via skip connections.
Revised: This highly descriptive representation is passed through a bottleneck block and progressively expanded by combining tokens from the encoder path via skip connections.

Original: The ﬁnal output is rearranged to match the exact shape of the input low-resolution features F L , yielding a deep feature representation F D .
Revised: The ﬁnal output is rearranged to match the exact shape of the input low-resolution features F L , yielding a deep feature representation F D .

Original: Our design is inspired by the success of Residual Swin Transformer Blocks (RSTB) in image denoising and super-resolution [Liang et al., 2021].
Revised: Our design is inspired by the success of Residual Swin Transformer Blocks (RSTB) in image denoising and super-resolution [Liang et al., 2021].

Original: RSTB features a stack of Swin Transformer layers (STL) that operate on tokens via a windowed self-attention mechanism [Liu et al., 2021], followed by convolution in image representation.
Revised: RSTB features a stack of Swin Transformer layers (STL) that operate on tokens via a windowed self-attention mechanism [Liu et al., 2021], followed by convolution in image representation.

Original: However, RSTB blocks operate on a single scale, therefore they cannot be readily applied in a hierarchical encoder-decoder architecture.
Revised: However, RSTB blocks operate on a single scale, therefore they cannot be readily applied in a hierarchical encoder-decoder architecture.

Original: Therefore, we design three variations of RSTB to facilitate multi-scale processing as depicted in Figure 2.
Revised: Therefore, we design three variations of RSTB to facilitate multi-scale processing as depicted in Figure 2.

Original: RSTB-B is the bottleneck block responsible for processing the encoded latent representation while maintaining feature dimensions.
Revised: RSTB-B is the bottleneck block responsible for processing the encoded latent representation while maintaining feature dimensions.

Original: Thus, we keep the default RSTB architecture for our bottleneck block, which already operates on a single scale.
Revised: Thus, we keep the default RSTB architecture for our bottleneck block, which already operates on a single scale.

Original: RSTB-D has a similar function to convolutional downsampling blocks in U-Nets, but it operates on embedded tokens.
Revised: RSTB-D has a similar function to convolutional downsampling blocks in U-Nets, but it operates on embedded tokens.

Original: Given an input with size h i · w i × d , we pass it through an RSTB-B block and apply
Revised: Given an input with size h i · w i × d , we pass it through an RSTB-B block and apply

Original: PatchMerge operation.
Revised: PatchMerge operation.

Original: PatchMerge linearly combines tokens corresponding to 2 × 2 non-overlapping image patches, while simultaneously increasing the embedding dimension (see Figure 3, top and Appendix I for more details) resulting in an output of size h i 2 · w i 2 × 2 · d .
Revised: PatchMerge linearly combines tokens corresponding to 2 × 2 non-overlapping image patches, while simultaneously increasing the embedding dimension (see Figure 3, top and the supplementary for more details) resulting in an output of size h i 2 · w i 2 × 2 · d .

Original: Furthermore, RSTB-D outputs the higher dimensional representation before patch merging to be subsequently used in the decoder path via skip connection.
Revised: Furthermore, RSTB-D outputs the higher dimensional representation before patch merging to be subsequently used in the decoder path via skip connection.

Original: RSTB-U used in the decoder path is analogous to convolutional upsampling blocks.
Revised: RSTB-U used in the decoder path is analogous to convolutional upsampling blocks.

Original: An input with size h i · w i × d is ﬁrst expanded into a larger number of lower dimensional tokens through a linear mapping via PatchExpand (see Figure 3, bottom and Appendix I for more details).
Revised: An input with size h i · w i × d is ﬁrst expanded into a larger number of lower dimensional tokens through a linear mapping via PatchExpand (see Figure 3, bottom and the supplementary for more details).

Original: PatchExpand reverses the effect of PatchMerge on feature size, thus resulting in 2 h i · 2 w i tokens of dimension d 2 .
Revised: PatchExpand reverses the effect of PatchMerge on feature size, thus resulting in 2 h i · 2 w i tokens of dimension d 2 .

Original: Next, we mix information from the obtained expanded tokens with skip embeddings from higher scales via TokenMix .
Revised: Next, we mix information from the obtained expanded tokens with skip embeddings from higher scales via TokenMix .

Original: This operation linearly combines tokens from both paths and normalizes the resulting vectors.
Revised: This operation linearly combines tokens from both paths and normalizes the resulting vectors.

Original: Finally, the mixed tokens are processed by an RSTB-B block.
Revised: Finally, the mixed tokens are processed by an RSTB-B block.

Original: 4.3 Iterative Unrolling
Revised: 4.3 Iterative Unrolling

Original: Architectures derived from unrolling the iterations of various optimization algorithms have proven to be successful in tackling various inverse problems including MRI reconstruction.
Revised: Architectures derived from unrolling the iterations of various optimization algorithms have proven to be successful in tackling various inverse problems including MRI reconstruction.

Original: These architecture can be interpreted as a cascade of simpler denoisers, each of which progressively reﬁnes the estimate from the preceding unrolled iteration (see more details in Appendix F).
Revised: These architecture can be interpreted as a cascade of simpler denoisers, each of which progressively reﬁnes the estimate from the preceding unrolled iteration (see more details in the supplementary).

Original: Following Sriram et al.
Revised: Following Sriram et al.

Original: [2020], we unroll the gradient descent iterations of the inverse problem in (2.1) in k-space domain, yielding the iterative update scheme in (2.2).
Revised: [2020], we unroll the gradient descent iterations of the inverse problem in (2.1) in k-space domain, yielding the iterative update scheme in (2.2).

Original: We apply regularization in image domain via our proposed HUMUS-Block, that is we have G ( k )
Revised: We apply regularization in image domain via our proposed HUMUS-Block, that is we have G ( k )

Original: =
Revised: =

Original: F (cid:0) E (cid:0) D (cid:0) R (cid:0)
Revised: F (cid:0) E (cid:0) D (cid:0) R (cid:0)

Original: F − 1 ( k ) (cid:1)(cid:1)(cid:1)(cid:1) , where
Revised: F − 1 ( k ) (cid:1)(cid:1)(cid:1)(cid:1) , where

Original: D denotes the HUMUS-Block, R ( x 1 , ..., x N ) =
Revised: D denotes the HUMUS-Block, R ( x 1 , ..., x N ) =

Original: (cid:80) Ni
Revised: (cid:80) Ni

Original: =1 S ∗
Revised: =1 S ∗

Original: i x
Revised: i x

Original: i is the reduce operator that combines coil images via the corresponding sensitivity maps and E ( x ) =
Revised: i is the reduce operator that combines coil images via the corresponding sensitivity maps and E ( x ) =

Original: ( S 1 x , ..., S N x ) is the expand operator that maps the combined image back to individual coil images.
Revised: ( S 1 x , ..., S N x ) is the expand operator that maps the combined image back to individual coil images.

Original: The sensitivity maps can be estimated a priori using methods such as ESPIRiT [Uecker et al., 2014] or learned in an end-to-end fashion via a Sensitivity Map Estimator (SME) network proposed in Sriram et al.
Revised: The sensitivity maps can be estimated a priori using methods such as ESPIRiT [Uecker et al., 2014] or learned in an end-to-end fashion via a Sensitivity Map Estimator (SME) network proposed in Sriram et al.

Original: In this work we aspire to design an end-to-end approach and thus we use the latter method and estimate the sensitivity maps the low-frequency (ACS) region of the undersampled input measurements a standard U- network.
Revised: In this work we aspire to design an end-to-end approach and thus we use the latter method and estimate the sensitivity maps the low-frequency (ACS) region of the undersampled input measurements a standard U- network.

Original: 4.4 Adjacent Slice Reconstruction (ASR)
Revised: 4.4 Adjacent Slice Reconstruction (ASR)

Original: We observe improvements in reconstruction quality when instead of processing the undersampled data slice-by-slice, we jointly reconstruct a set of adjacent slices via HUMUS-Net
Revised: We observe improvements in reconstruction quality when instead of processing the undersampled data slice-by-slice, we jointly reconstruct a set of adjacent slices via HUMUS-Net

Original: (Figure 5).
Revised: (Figure 5).

Original: That is, if we have a volume of undersampled data ˜ k vol =
Revised: That is, if we have a volume of undersampled data ˜ k vol =

Original: (˜ k 1 , ..., ˜ k K ) with K slices, when reconstructing slice c , we instead reconstruct the volume (˜ k c − a , ..., ˜ k
Revised: (˜ k 1 , ..., ˜ k K ) with K slices, when reconstructing slice c , we instead reconstruct the volume (˜ k c − a , ..., ˜ k

Original: c − 1 , ˜ k c , ˜ k
Revised: c − 1 , ˜ k c , ˜ k

Original: c +1 , ..., ˜ k
Revised: c +1 , ..., ˜ k

Original: c + a ) by concatenating them along the coil channel dimension, where a denotes the number of adjacent slices added on each side.
Revised: c + a ) by concatenating them along the coil channel dimension, where a denotes the number of adjacent slices added on each side.

Original: However, we only calculate and backpropagate the loss on the center slice c of the reconstructed volume.
Revised: However, we only calculate and backpropagate the loss on the center slice c of the reconstructed volume.

Original: The beneﬁt of ASR is that the network can remove artifacts corrupting individual slices as it sees a larger context of the slice by observing its neighbors.
Revised: The beneﬁt of ASR is that the network can remove artifacts corrupting individual slices as it sees a larger context of the slice by observing its neighbors.

Original: Even though ASR increases compute cost, it is important to note that it does not impact the number of token embeddings (spatial resolution is unchanged) and thus can be combined favorably with Transformer-based methods.
Revised: Even though ASR increases compute cost, it is important to note that it does not impact the number of token embeddings (spatial resolution is unchanged) and thus can be combined favorably with Transformer-based methods.

Original:  Experiments
Revised:  Experiments

Original: In this section we provide experimental results on our proposed architecture, HUMUS-Net.
Revised: In this section we provide experimental results on our proposed architecture, HUMUS-Net.

Original: First, we demonstrate the reconstruction performance of our model on various datasets, including the large-scale fastMRI dataset.
Revised: First, we demonstrate the reconstruction performance of our model on various datasets, including the large-scale fastMRI dataset.

Original: Then, we justify our design choices through a set of ablation studies.
Revised: Then, we justify our design choices through a set of ablation studies.

Original: 5.1 Benchmark Experiments
Revised: 5.1 Benchmark Experiments

Original: We investigate the performance of HUMUS-Net on three different datasets.
Revised: We investigate the performance of HUMUS-Net on three different datasets.

Original: We use the structural similarity index measure (SSIM)
Revised: We use the structural similarity index measure (SSIM)

Original: [Wang et al., 2003] as a basis of our evaluation, which is the most common evaluation metric in medical image reconstruction.
Revised: [Wang et al., 2003] as a basis of our evaluation, which is the most common evaluation metric in medical image reconstruction.

Original: In all of our experiments, we follow the setup of the fastMRI multi-coil knee track with 8 × acceleration in order to provide comparison with state-of-the-art networks of the Public Leaderboard.
Revised: In all of our experiments, we follow the setup of the fastMRI multi-coil knee track with 8 × acceleration in order to provide comparison with state-of-the-art networks of the Public Leaderboard.

Original: That is, we perform retrospective undersampling of the fully-sampled k-space data by randomly subsampling 12 .
Revised: That is, we perform retrospective undersampling of the fully-sampled k-space data by randomly subsampling 12 .

Original: 5% of whole k-space lines in the
Revised: 5% of whole k-space lines in the

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  For datasets with multiple trainvalidation split runs we show the mean and standard error of the runs.
Revised: various datasets. For datasets with multiple trainvalidation split runs we show the mean and standard error of the runs.

Original: [Table]
Revised: [Table]

Original: phase encoding direction, keeping 4% of lowest frequency adjacent k-space lines.
Revised: phase encoding direction, keeping 4% of lowest frequency adjacent k-space lines.

Original: Experiments on other acceleration ratios can be found in Appendix C. During training, we generate random masks following the above method, whereas for the validation dataset we keep the masks ﬁxed for each k-space volume.
Revised: Experiments on other acceleration ratios can be found in the supplementary. During training, we generate random masks following the above method, whereas for the validation dataset we keep the masks ﬁxed for each k-space volume.

Original: For HUMUS-Net, we center crop and pad inputs to 384 × 384 .
Revised: For HUMUS-Net, we center crop and pad inputs to 384 × 384 .

Original: We compare the reconstruction quality of our proposed model with the current best performing network, E2E-VarNet.
Revised: We compare the reconstruction quality of our proposed model with the current best performing network, E2E-VarNet.

Original: For details on HUMUS-Net hyperparameters and training, we refer the reader to Appendix A. For
Revised: For details on HUMUS-Net hyperparameters and training, we refer the reader to the supplementary.

Original:  E2E-VarNet, we use the hyperparameters speciﬁed in Sriram et al.
Revised: For E2E-VarNet, we use the hyperparameters speciﬁed in Sriram et al.

Original:  The fastMRI dataset [Zbontar et al., 2019] is the largest publicly available MRI dataset with competitive baseline models and a public leaderboard, and thus provides an opportunity to directly compare different algorithms.
Revised: fastMRI – The fastMRI dataset [Zbontar et al., 2019] is the largest publicly available MRI dataset with competitive baseline models and a public leaderboard, and thus provides an opportunity to directly compare different algorithms.

Original: Speciﬁcally, we run experiments on the multi-coil knee dataset, consisting of close to 35 k slices in 973 volumes.
Revised: Speciﬁcally, we run experiments on the multi-coil knee dataset, consisting of close to 35 k slices in 973 volumes.

Original: We use the default HUMUS-Net model deﬁned above with 3 adjacent slices as input.
Revised: We use the default HUMUS-Net model deﬁned above with 3 adjacent slices as input.

Original: Furthermore, we design a large variant of our model, HUMUS-Net-L , which has increased embedding dimension compared to the default model (see details in Appendix A.
Revised: Furthermore, we design a large variant of our model, HUMUSNet-L , which has increased embedding dimension compared to the default model (see details in the supplementary).

Original: We train models both only on the training split, and also on the training and validation splits combined (additional ≈ 20% data) for the leaderboard.
Revised: We train models both only on the training split, and also on the training and validation splits combined (additional ≈ 20% data) for the leaderboard.

Original: Table 1 demonstrates our results compared to the best published models from the fastMRI Leaderboard 3 evaluated on the test dataset.
Revised: Table 1 demonstrates our results compared to the best published models from the fastMRI Leaderboard 3 evaluated on the test dataset.

Original: Our model establishes new state of the art in terms of SSIM on this dataset by a large margin, and achieves comparable or better performance than other methods in terms of PSNR and
Revised: Our model establishes new state of the art in terms of SSIM on this dataset by a large margin, and achieves comparable or better performance than other methods in terms of PSNR and

Original: NMSE.
Revised: NMSE.

Original: Moreover, as seen in the second column of Table 2, we evaluated our model on the fastMRI validation dataset as well and compared our results to E2E-VarNet, the best performing model from the leaderboard.
Revised: Moreover, as seen in the second column of Table 2, we evaluated our model on the fastMRI validation dataset as well and compared our results to E2E-VarNet, the best performing model from the leaderboard.

Original: We observe similar improvements in terms of the reconstruction SSIM metric to the test dataset.
Revised: We observe similar improvements in terms of the reconstruction SSIM metric to the test dataset.

Original: Visual inspection of reconstructions shows that HUMUS-Net recovers very ﬁne details in images that may be missed by other state-of-the-art reconstruction algorithms (see attached ﬁgures in Appendix I).
Revised: Visual inspection of reconstructions shows that HUMUS-Net recovers very ﬁne details in images that may be missed by other state-of-the-art reconstruction algorithms (see attached ﬁgures in the supplementary).

Original: Comparison based on further image quality metrics can be found in Appendix
Revised: Comparison based on further image quality metrics can be found in the supplementary.

Original: H. We point out that even though our model has more parameters than E2E-VarNet, our proposed image-domain denoiser is more efﬁcient than the U-Net deployed in E2E-VarNet, even when their number of parameters are matched as discussed in Section 5.3.
Revised: We point out that even though our model has more parameters than E2E-VarNet, our proposed image-domain denoiser is more efﬁcient than the U-Net deployed in E2E-VarNet, even when their number of parameters are matched as discussed in Section 5.3.

Original: Overall, larger model size does not necessarily correlate with better performance, as seen in other competitive models in Table 1.
Revised: Overall, larger model size does not necessarily correlate with better performance, as seen in other competitive models in Table 1.

Original: Finally, we note that the additional training data (in the form of the validation split) provides a consistent small boost to model performance.
Revised: Finally, we note that the additional training data (in the form of the validation split) provides a consistent small boost to model performance.

Original: We refer the reader to Klug and Heckel [2022] for an overview of scaling properties of reconstruction models.
Revised: We refer the reader to Klug and Heckel [2022] for an overview of scaling properties of reconstruction models.

Original: Stanford 2D – Next, we run experiments on the Stanford2D FSE [Cheng] dataset, a publicly available
Revised: Stanford 2D – Next, we run experiments on the Stanford2D FSE [Cheng] dataset, a publicly available

Original: MRI dataset consisting of scans from various anatomies (pelvis, lower extremity and more) infully-sampled volumes.
Revised: MRI dataset consisting of scans from various anatomies (pelvis, lower extremity and more) infully-sampled volumes.

Original: We randomly sample 80% of volumes as train data and use the rest for validation.
Revised: We randomly sample 80% of volumes as train data and use the rest for validation.

Original: We randomly generate 3 different train-validation splits this way to reduce variations in the presented metrics.
Revised: We randomly generate 3 different train-validation splits this way to reduce variations in the presented metrics.

Original: As slices in this dataset have widely varying shapes across volumes, we center crop the target images to keep spatial resolution within 384 × 384 .
Revised: As slices in this dataset have widely varying shapes across volumes, we center crop the target images to keep spatial resolution within 384 × 384 .

Original: We use the default HUMUS-Net deﬁned above with single slices as input.
Revised: We use the default HUMUS-Net deﬁned above with single slices as input.

Original: Our results comparing the best performing MRI reconstruction model with HUMUS-Net is shown in the third column of Table 2.
Revised: Our results comparing the best performing MRI reconstruction model with HUMUS-Net is shown in the third column of Table 2.

Original: We present the mean SSIM of all runs along with the standard error.
Revised: We present the mean SSIM of all runs along with the standard error.

Original: We achieve improvements of similar magnitude as on the fastMRI dataset.
Revised: We achieve improvements of similar magnitude as on the fastMRI dataset.

Original: These results demonstrate the effectiveness of HUMUS-Net on a more diverse dataset featuring multiple anatomies.
Revised: These results demonstrate the effectiveness of HUMUS-Net on a more diverse dataset featuring multiple anatomies.

Original: Stanford 3D – Finally, we evaluate our model on the Stanford Fullysampled 3D FSE Knees dataset [Sawyer et al., 2013], a public MRI dataset including 20 volumes of knee MRI scans.
Revised: Stanford 3D – Finally, we evaluate our model on the Stanford Fullysampled 3D FSE Knees dataset [Sawyer et al., 2013], a public MRI dataset including 20 volumes of knee MRI scans.

Original: We generate train-validation splits using the method described for Stanford 2D and perform 3 runs.
Revised: We generate train-validation splits using the method described for Stanford 2D and perform 3 runs.

Original: We use the default HUMUS-Net network with single slices as input.
Revised: We use the default HUMUS-Net network with single slices as input.

Original: The last column of Table 2 compares our results to E2E-VarNet, showing improvements of similar scales as on other datasets we have investigated in this work.
Revised: The last column of Table 2 compares our results to E2E-VarNet, showing improvements of similar scales as on other datasets we have investigated in this work.

Original: This experiment demonstrates that HUMUS-Net performs well not only on large-scale MRI datasets, but also on smaller problems.
Revised: This experiment demonstrates that HUMUS-Net performs well not only on large-scale MRI datasets, but also on smaller problems.

Original: 5.2 Ablation Studies
Revised: 5.2 Ablation Studies

Original: In this section, we motivate our design choices through a set of ablation studies.
Revised: In this section, we motivate our design choices through a set of ablation studies.

Original: We start from
Revised: We start from

Original: SwinIR, a general image reconstruction network, and highlight its weaknesses for MRI.
Revised: SwinIR, a general image reconstruction network, and highlight its weaknesses for MRI.

Original: Then, we demonstrate step-by-step how we addressed these shortcomings and arrived at the HUMUS-Net architecture.
Revised: Then, we demonstrate step-by-step how we addressed these shortcomings and arrived at the HUMUS-Net architecture.

Original: We train the models on the Stanford 3D dataset.
Revised: We train the models on the Stanford 3D dataset.

Original: More details can be found in Appendix
Revised: More details can be found in the supplementary.

Original: B. The results of our ablation studies are summarized in Table 3.
Revised: The results of our ablation studies are summarized in Table 3.

Original: First, we investigate SwinIR, a state-of-the-art image denoiser and super-resolution model that features a hybrid Transformer-convolutional architecture.
Revised: First, we investigate SwinIR, a state-of-the-art image denoiser and super-resolution model that features a hybrid Transformer-convolutional architecture.

Original: In order to handle the 10 × larger input sizes ( 320 × 320 ) in our MRI dataset compared to the input images this network has been designed for ( 128 × 128 ), we reduce the embedding dimension of SwinIR to ﬁt into GPU memory (16 GB).
Revised: In order to handle the 10 × larger input sizes ( 320 × 320 ) in our MRI dataset compared to the input images this network has been designed for ( 128 × 128 ), we reduce the embedding dimension of SwinIR to ﬁt into GPU memory (16 GB).

Original: We ﬁnd that compared to models designed for MRI reconstruction, such as E2E-VarNet (last row in
Revised: We ﬁnd that compared to models designed for MRI reconstruction, such as E2E-VarNet (last row in

Original: Table 3) SwinIR performs poorly.
Revised: Table 3) SwinIR performs poorly.

Original: This is not only due to the reduced network size, but also due to the fact that SwinIR is not speciﬁcally designed to take the MRI forward model into consideration.
Revised: This is not only due to the reduced network size, but also due to the fact that SwinIR is not speciﬁcally designed to take the MRI forward model into consideration.

Original: Next, we unroll SwinIR and add a sensitivity map estimator.
Revised: Next, we unroll SwinIR and add a sensitivity map estimator.

Original: We refer to this model as Un-SS .
Revised: We refer to this model as Un-SS .

Original: Due to unrolling, we have to further reduce the embedding dimension of the denoiser and also decrease the depth of the network in order to ﬁt into GPU memory.
Revised: Due to unrolling, we have to further reduce the embedding dimension of the denoiser and also decrease the depth of the network in order to ﬁt into GPU memory.

Original: Un-SS , due to its small size, performs slightly worse than vanilla SwinIR and signiﬁcantly lags behind the E2E-VarNet architectures.
Revised: Un-SS , due to its small size, performs slightly worse than vanilla SwinIR and signiﬁcantly lags behind the E2E-VarNet architectures.

Original: We note that
Revised: We note that

Original: SwinIR operates over a single, full-resolution scale, whereas state-of-the-art MRI reconstruction models typically incorporate multi-scale processing in the form of U-Net-like architectures.
Revised: SwinIR operates over a single, full-resolution scale, whereas state-of-the-art MRI reconstruction models typically incorporate multi-scale processing in the form of U-Net-like architectures.

Original: Thus, we replace SwinIR by MUST, our proposed multi-scale hybrid processing unit, but keep the embedding dimension in the largest-resolution scale ﬁxed.
Revised: Thus, we replace SwinIR by MUST, our proposed multi-scale hybrid processing unit, but keep the embedding dimension in the largest-resolution scale ﬁxed.

Original: The obtained network, which we call
Revised: The obtained network, which we call

Original: Un-MS , has overall lower computational cost when compared with Un-SS , however as Table 3 shows
Revised: Un-MS , has overall lower computational cost when compared with Un-SS , however as Table 3 shows

Original: MRI reconstruction performance has signiﬁcantly improved compared to both Un-SS and vanilla
Revised: MRI reconstruction performance has signiﬁcantly improved compared to both Un-SS and vanilla

Original: SwinIR, which highlights the efﬁciency of our proposed multi-scale feature extractor.
Revised: SwinIR, which highlights the efﬁciency of our proposed multi-scale feature extractor.

Original: Reconstruction performance is limited by the low dimension of patch embeddings, which we are unable to increase further due to our compute and memory constraints originating in the high-resolution inputs.
Revised: Reconstruction performance is limited by the low dimension of patch embeddings, which we are unable to increase further due to our compute and memory constraints originating in the high-resolution inputs.

Original: The most straightforward approach to tackle the challenge of high input resolution is to increase the patch size.
Revised: The most straightforward approach to tackle the challenge of high input resolution is to increase the patch size.

Original: To test this idea, we take Un-MS and embed 2 × 2 patches of the inputs, thus reducing the number of tokens processed by the network by a factor of 4 .
Revised: To test this idea, we take Un-MS and embed 2 × 2 patches of the inputs, thus reducing the number of tokens processed by the network by a factor of 4 .

Original: We refer to this model as Un-MS-Patch2 .
Revised: We refer to this model as Un-MS-Patch2 .

Original: This reduction in compute and memory load allows us to increase network capacity by increasing the embedding dimension 3 -folds (to ﬁll GPU memory again).
Revised: This reduction in compute and memory load allows us to increase network capacity by increasing the embedding dimension 3 -folds (to ﬁll GPU memory again).

Original: However, Un-MS-Patch2 performs much worse than previous models using patch size of 1 .
Revised: However, Un-MS-Patch2 performs much worse than previous models using patch size of 1 .

Original: For classiﬁcation problems, where the general image context is more important than small details, patches of 16 × 16 or 8 × 8 are considered typical [Dosovitskiy et al., 2020].
Revised: For classiﬁcation problems, where the general image context is more important than small details, patches of 16 × 16 or 8 × 8 are considered typical [Dosovitskiy et al., 2020].

Original: Even for more dense prediction tasks such as medical image segmentation, patch size of 4 × 4 has been used successfully [Cao et al., 2021a].
Revised: Even for more dense prediction tasks such as medical image segmentation, patch size of 4 × 4 has been used successfully [Cao et al., 2021a].

Original: However, our experiments suggest that in low-level vision tasks such as MRI reconstruction using patches larger than 1 × 1 may be detrimental due to loss of crucial high-frequency detail information.
Revised: However, our experiments suggest that in low-level vision tasks such as MRI reconstruction using patches larger than 1 × 1 may be detrimental due to loss of crucial high-frequency detail information.

Original: Our approach to address the heavy computational load of Transformers for large input resolutions where increasing the patch size is not an option is to process lower resolution features extracted via convolutions.
Revised: Our approach to address the heavy computational load of Transformers for large input resolutions where increasing the patch size is not an option is to process lower resolution features extracted via convolutions.

Original: That is we replace MUST in Un-MS by a HUMUS-Block, resulting in our proposed
Revised: That is we replace MUST in Un-MS by a HUMUS-Block, resulting in our proposed

Original: HUMUS-Net architecture.
Revised: HUMUS-Net architecture.

Original: We train a smaller version of HUMUS-Net with the same embedding dimension as Un-MS .
Revised: We train a smaller version of HUMUS-Net with the same embedding dimension as Un-MS .

Original: As seen in Table 3, our model achieves the best performance across all other proposed solutions, even surpassing E2E-VarNet.
Revised: As seen in Table 3, our model achieves the best performance across all other proposed solutions, even surpassing E2E-VarNet.

Original: This series of incremental studies highlights the importance of each architectural design choice leading to our proposed HUMUS-Net architecture.
Revised: This series of incremental studies highlights the importance of each architectural design choice leading to our proposed HUMUS-Net architecture.

Original: 5.3 Direct comparison of image-domain denoisers
Revised: 5.3 Direct comparison of image-domain denoisers

Original: In order to further demonstrate the advantage of HUMUS-Net over E2E-VarNet, we provide direct comparison between the image-domain denoisers used in the above methods.
Revised: In order to further demonstrate the advantage of HUMUS-Net over E2E-VarNet, we provide direct comparison between the image-domain denoisers used in the above methods.

Original: E2E-VarNet unrolls a fully convolutional U-Net, whereas we deploy our hybrid HUMUS-Block architecture as a denoiser (Fig. 1).
Revised: E2E-VarNet unrolls a fully convolutional U-Net, whereas we deploy our hybrid HUMUS-Block architecture as a denoiser (Fig. 1).

Original: We scale down the HUMUS-Block used in HUMUS-Net to match the size of the U-Net in E2E-Varnet in terms of number of model parameters.
Revised: We scale down the HUMUS-Block used in HUMUS-Net to match the size of the U-Net in E2E-Varnet in terms of number of model parameters.

Original: To this end, we reduce the embedding dimension from 66 to 30 .
Revised: To this end, we reduce the embedding dimension from 66 to 30 .

Original: We train both networks on magnitude images from the Stanford3D dataset.
Revised: We train both networks on magnitude images from the Stanford3D dataset.

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: We observe that given a ﬁxed parameter budget, our proposed denoiser outperforms the widely used convolutional U-Net architecture in MRI reconstruction, further demonstarting the efﬁciency of HUMUS-Net.
Revised: We observe that given a ﬁxed parameter budget, our proposed denoiser outperforms the widely used convolutional U-Net architecture in MRI reconstruction, further demonstarting the efﬁciency of HUMUS-Net.

Original: This experiment suggests that our HUMUS-Block could serve as an excellent denoiser, replacing convolutional U-Nets, in a broad range of image restoration applications outside of MRI, which we leave for future work.
Revised: This experiment suggests that our HUMUS-Block could serve as an excellent denoiser, replacing convolutional U-Nets, in a broad range of image restoration applications outside of MRI, which we leave for future work.

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce HUMUS-Net, an unrolled, Transformer-convolutional hybrid network for accelerated MRI reconstruction.
Revised: In this paper, we introduce HUMUS-Net, an unrolled, Transformer-convolutional hybrid network for accelerated MRI reconstruction.

Original: HUMUS-Net achieves state-of-the-art performance on the fastMRI dataset and greatly outperforms all previous published and reproducible methods.
Revised: HUMUS-Net achieves state-of-the-art performance on the fastMRI dataset and greatly outperforms all previous published and reproducible methods.

Original: We demonstrate the performance of our proposed method on two other MRI datasets and perform ﬁnegrained ablation studies to motivate our design choices and emphasize the compute and memory challenges of Transformer-based architectures on low-level and dense computer vision tasks such as MRI reconstruction.
Revised: We demonstrate the performance of our proposed method on two other MRI datasets and perform ﬁnegrained ablation studies to motivate our design choices and emphasize the compute and memory challenges of Transformer-based architectures on low-level and dense computer vision tasks such as MRI reconstruction.

Original: A limitation of our current architecture is that it requires ﬁxed-size inputs, which we intend to address with a more ﬂexible design in the future.
Revised: A limitation of our current architecture is that it requires ﬁxed-size inputs, which we intend to address with a more ﬂexible design in the future.

Original: This work opens the door for the adoption of a multitude of promising techniques introduced recently in the literature for Transformers, which we leave for future work.
Revised: This work opens the door for the adoption of a multitude of promising techniques introduced recently in the literature for Transformers, which we leave for future work.

Original: Acknowledgments
Revised: Acknowledgments

Original: M. Soltanolkotabi is supported by the Packard Fellowship in Science and Engineering, a Sloan
Revised: M. Soltanolkotabi is supported by the Packard Fellowship in Science and Engineering, a Sloan

Original: Research Fellowship in Mathematics, an NSF-CAREER under award #1846369, DARPA Learning with Less Labels (LwLL) and FastNICS programs, and NSF-CIF awards #1813877 and #2008443.Appendix A HUMUS-Net baseline details
Revised: Research Fellowship in Mathematics, an NSF-CAREER under award #1846369, DARPA Learning with Less Labels (LwLL) and FastNICS programs, and NSF-CIF awards #1813877 and #2008443.

Original: Our default model has 3 RSTB-D downsampling blocks, 2 RSTB-B bottleneck blocks and 3 RSTB-U upsampling blocks with 3 − 6 − 12 attention heads in the D/U blocks and 24 attention heads in the bottleneck block.

Original: For Swin Transformers layers, the window size is 8 for all methods and MLP ratio ( hidden _ dim/input _ dim ) of 2 is used.

Original: Each RSTB block consists of 2 STLs with embedding dimension of 66 .

Original: For HUMUS-Net-L, we increase the embedding dimension to 96 .

Original: We use 8 cascades of unrolling with a U-Net as sensitivity map estimator (same as in E2E-VarNet) with 16 channels.

Original: We center crop and reﬂection pad the input images to 384 × 384 resolution for HUMUS-Net and use the complete images for VarNet.

Original: In all experiments, we minimize the SSIM loss between the target image x ∗ and the reconstruction ˆ x deﬁned as

Original: [Equation]

Original: fastMRI experiments– We train HUMUS-Net using Adam for 50 epochs with a learning rate of 0 . 0001 , dropped by a factor of 10 at epoch 40 and apply adjacent slice reconstruction with 3 slices.

Original: We run the experiments on 8 × Titan RTX 5000 24GB GPUs with a per-GPU batch size of 1 .

Original: We train

Original: HUMUS-Net-L using a learning rate of 0 .

Original: 00007 (using inverse square root scaling with embedding dimension) for 45 epochs, dropped by a factor of 10 for further 5 epochs.

Original: We trained HUMUS-Net-L on 4 × A100 40GB GPUs on Amazon SageMaker with per-GPU batch size of 1.

Original: Stanford 3D experiments– We train for 25 epochs with a learning rate of 0 . 0001 , dropped by a factor of 10 at epoch 20 and reconstruct single slices.

Original: We run the experiments on 4 × Quadro RTX 5000 16GB GPUs with a per-GPU batch size of 1 .

Original: Stanford 2D experiments– We train for 50 epochs with a learning rate of 0 . 0001 , dropped by a factor of 10 at epoch 40 and reconstruct single slices.

Original: Moreover, we crop reconstruction targets to ﬁt into a 384 × 384 box.

Original: We run the experiments on 8 × Quadro RTX 5000 16GB GPUs with a per-GPU batch size of 1 .

Original: B Ablation study experimental details

Original: Here we discuss the experimental setting and hyperparameters used in our ablation study in Section 5.2.

Original: In all experiments, we train the models on the Stanford 3D dataset for 3 different train-validation splits and report the mean and standard error of the results.

Original: We use Adam optimizer with learning rate 0 . 0001 and train for 25 epochs, decaying the learning rate by a factor of 10 at 20 epochs.

Original: For all unrolled methods, we unroll 12 iterations in order to provide direct comparison with the best performing E2E-VarNet, which uses the same number of cascades.

Original: We reconstruct single slices and do not use the method of adjacent slice reconstruction discussed in 4.4.

Original: In models with sensitivity map estimator,we used the default U-Net with 8 channels (same as default E2E-VarNet).

Original: For Swin

Original: Transformers layers, the window size is 8 for all methods and MLP ratio ( hidden _ dim/input _ dim ) of 2 is used.

Original: Further hyperparameters are summarized in Table

Original: [Table]

Original: networks

Original: downsampling

Original: B

Original: upsampling

Original: separately

Original: We run the experiments on 4 × Quadro RTX 5000 16GB GPUs with a per-GPU batch size of 1 .

Original: C Results on additional accelerations

Original: We observe that HUMUS-Net achieves state-of-the-art performance across a wide range of acceleration ratios.

Original: We perform experiments on the Stanford 3D dataset using a small HUMUS-Net model with only 6 cascades, embedding dimension of 66 , adjacent slice reconstruction with 3 slices and for the sake of simplicity we removed the residual path from the HUMUS-Block.

Original: We call this model HUMUS-Net-S .

Original: We set the learning rate to 0 . 0002 and train the model with Adam optimizer.

Original: We generate 3 random training-validation splits on the Stanford 3D dataset (same splits as in other

Original: Stanford 3D experiments in this paper), and show the mean and standard error of the results in Table 6.

Original: In our experiments, HUMUS-Net achieved higher quality reconstructions in every metric over

Original: E2E-VarNet.

Original: [Table]

Original: D Effect of the number of unrolled iterations

Original: The number of cascades in unrolled networks has a fundamental impact on their performance.

Original: Deeper networks typically perform better, but also incur heavy computational and memory cost.

Original: In this experiment we investigate the scaling of HUMUS-Net with respect to the number of iterative unrollings.

Original: We perform an ablation study on the Stanford 3D dataset with × 8 acceleration for a ﬁxed trainingvalidation split, where 20% of the training set has been set aside for validation.

Original: We reconstruct single slices, without applying adjacent slice reconstruction.

Original: We plot the highest SSIM throughout training on the validation dataset for each network on Figure 6.

Original: We observe improvements in reconstruction performance with increasing number of cascades, and this improvement has not saturated yet in the range of model sizes we have investigated.

Original: In our main experiments, we select 8 cascades due to memory and compute limitations for deeper models.

Original: However, our experiment suggests that HUMUS-Net can potentially obtain even better reconstruction results given enough computational resources.

Original: E Effect of Adjacent Slice Reconstruction

Original: In this section, we demonstrate that even though adjacent slice reconstruction is in general helpful, it is not the main reason why HUMUS-Net performs better than E2E-VarNet.

Original: To this end, we add adjacent slice reconstruction to E2E-VarNet and investigate its effect on model performance.

Original: We run experiments on the Stanford 3D dataset for 3 different random train-validation splits.

Original: We use adjacent slice reconstruction with 3 slices.

Original: We have found that the default learning rate for E2E-VarNet is not optimal with the increased input size, therefore we tune the learning rate using grid search and set it to 0 . 0005 .

Original: We compare the results with HUMUS-Net, where we match the number of cascades in E2E-VarNet, and use the default embedding dimension of 66 .

Original: We do not use adjacent slice reconstruction when training HUMUS-Net in order to ablate its effect.

Original: The results are summarized in Table 7.

Original: We observe that ASR boosts the reconstruction quality of

Original: E2E-VarNet.

Original: However, ASR alone cannot close the gap between the two models, as HUMUS-Net without ASR still outperforms the best E2E-VarNet model with ASR.

Original: [Figure]

Original: [Table]

Original: F Iterative denoising visualization

Original: Here

Original: [Table]

Original: we provide more discussion on the iterative denoising interpretation of unrolled networks, such as HUMUS-Net.

Original: Consider the regularized inverse problem formulation of MRI reconstruction in (2.1) and design the architecture based on unrolling the gradient descent steps on the above problem.

Original: Applying data consistency corresponds to the gradient of the ﬁrst term in (2.2), whereas we learn the gradient of the regularizer parameterized by the HUMUS-Block architecture.

Original: This can be conceptualized as alternating between enforcing consistency with the measurements and applying a denoiser based on some prior, represented by the HUMUS-Block in our architecture.

Original: We show an example of a sequence of intermediate reconstructions in order to support the denoising intuition in Figure 7.

Original: We plot the magnitude image of reconstructions at the outputs of consecutive

Original: HUMUS-Blocks, along with the model input zero-padded reconstruction.

Original: We point out that in general the intermediate reconstruction quality (for instance in terms of SSIM) is not necessarily an increasing function of cascades.

Original: This is due to the highly non-linear nature of the mapping represented by the neural network, which might not always be intuitive and interpretable.

Original: G Vision Transformer terminology overview

Original: Here we provide a brief overview of the terms related to the Transformer architecture, some of which has been carried over from the Natural Language Processing (NLP) literature.

Original: We introduce the key concepts through the Vision Transformer architecture, which has inspired most Transformer-based architectures in computer vision.

Original: For more details, we refer the reader to [Dosovitskiy et al., 2020].

Original: Traditional Transformers for NLP receive a sequence of 1D token embeddings.

Original: Each token may represent for instance a word in a sentence.

Original: To extend this idea to 2D images, the input x ∈ R H × W × C image ( H and W stand for spatial dimensions, C for channel dimension) is split into N patches of size P × P , and each patch is ﬂattened into a 1D vector to produce an input of ﬂattened patches x f with shape N × P 2 C :

Original: [Equation]

Original: In order to set the latent dimension of the input entering the Vision Transformer, a trainable linear mapping E ∈ R P 2 C × D is applied to the ﬂattened patches, resulting in the so called patch embeddings z of shape N × D :

Original: [Equation]

Original: In order to encode information with respect to the position of image patches, a learnable positional embedding E pos ∈ R N × D is added to the patch embeddings:

Original: [Equation]

Original: The input to the Transformer encoder is this N × D representation, which we also refer to in the paper as token representation, as each row in the representation corresponds to a token (in our case an image patch) in the original input.

Original: H Detailed validation results

Original: [Table]

Original: I Additional ﬁgures

Original: [Figure]

Original: [Equation]